8 Key Lessons From My AI_managers Webinar That Attracted 3,000+ Leaders

Recently i had the pleasure of joining a webinar for 3,000 managers curious about AI. Here are the key lessons I've shared with them.

Recently I had the privilege of sharing hard-won AI implementation insights from my years at Allegro, Booking.com, and now as Head of Product at GOG. During the AI Managers webinar that brought together over 3,000 business leaders and technology decision-makers. The engagement was electric – clearly touching on challenges organizations are actively confronting right now.

Before diving into the transformative lessons, I'm excited to announce that I'll be joining the AI_managers course as a special guest instructor where I'll take these concepts even deeper. If you're serious about implementing AI in your organization without the expensive missteps most companies make, this is your opportunity to gain structured, actionable frameworks.

Using the link below you are eligible to a 300zł discount on the course and I get a kickback (full transparency). Looking at the key speakers, the curriculum and the price I think AI_managers is a good value for money, especially if you have a training budget at your company.

Click here to get the 300zł discount.

Now let’s go to the lessons from the webinar. :)

The Fundamental Misconception Crippling Your AI Investments

Traditional IT project management isn't just inadequate for AI initiatives – it's actively harmful.

The reason? You're trying to build a house when you should be growing a garden.

Traditional IT projects are deterministic – clear specifications, predictable timelines, and defined deliverables. You create blueprints, gather materials, and execute. Linear, controllable, measurable.

AI projects operate in a fundamentally different reality. They're probabilistic experiments where success depends on creating optimal conditions rather than following rigid plans.

Let me walk you through the seven critical lessons that will transform how you approach AI project management – lessons learned through both painful failures and breakthrough successes at Allegro, Booking.com, and now at GoK.

Lesson 1: Gardens Outperform Buildings

My earliest and most expensive AI lesson came at Allegro. We faced a critical problem: millions of product listings (like "iPhone 14 32GB Silver Kraków CHEAP!") were treated as unique items despite being identical products. This devastated our recommendation engine performance – instead of suggesting different products, we kept showing users the same product listed differently.

Some of you know this story from one of my previous article. If so, you can skip this lesson :)

5 Unexpected Lessons from Teaching 20,000 People About AI

Our approach? Pure technological ambition.

One of our data scientists discovered an academic paper from eBay using Siamese neural networks for similar matching. It sounded sophisticated and cutting-edge. We committed to a two-sprint implementation.

Fast forward two months and multiple extensions later: our neural network approach achieved only 40% accuracy. Meanwhile, an engineer outside our team built a simple full-text search solution in two weeks that reached 60% accuracy.

The Garden Principle: In traditional IT (building), sophisticated architecture is impressive. In AI (gardening), simplicity often yields better results with a fraction of the resources. Start with the minimal viable solution before pursuing complexity.

Lesson 2: Zero-One Problems Win, Complex Problems Fail

The more isolated and binary your problem, the higher your probability of AI success. Complex, multi-variable challenges often lead to expensive failures.

At Allegro, we transformed our "Hot Deals" carousel (which previously showed everyone the same manually-selected products, including mattresses I had no interest in buying) by asking a simple question: "Can we personalize these recommendations based on user history?"

This crystal-clear, binary problem led to a 700% ROI improvement within just two weeks.

The Zero-One Framework: Evaluate potential AI projects based on how cleanly they can be reduced to binary outcomes:

• Customer service automation: Will customer accept solution? (Yes/No)

• Fraud detection: Is transaction suspicious? (Yes/No)

• Recommendation systems: Will user engage with this item? (Yes/No)

When evaluating AI opportunities, prioritize problems with clear, measurable outcomes over ambiguous, multi-dimensional challenges.

Lesson 3: Domain Experts Outvalue Algorithms

AI models are only as effective as their data foundations – and the most valuable insights often come from human domain experts, not databases.

At Booking.com, we didn't just analyze customer service data – we physically sat with agents, wore headsets, and handled actual customer cases to understand their decision-making process.

When building models to predict customer satisfaction issues, we could have spent months analyzing thousands of data points. Instead, we simply asked experienced agents: "What signals indicate a customer will be dissatisfied?"

Their answer? "Check their reservation history. If they've filed multiple complaints in the past six months, they're likely to be unhappy again."

This single insight, impossible to derive from pure data analysis, fundamentally shaped our modeling approach.

The Expert Integration Protocol: Place your data scientists physically alongside domain experts. Have them experience the actual work processes they're trying to optimize. The most powerful signals often come from human experience, not data tables.

Lesson 4: Simple Heuristics Beat Fancy Algorithms

Before diving into complex AI solutions, establish baselines with simpler approaches:

• Rule-based heuristics

• Standard search algorithms

• Basic filtering techniques

This approach validates problem solvability, establishes performance benchmarks, and creates comparison points – all while delivering faster time-to-value.

The Simplicity Principle: Sophisticated approaches carry three hidden costs most organizations ignore:

• Extended development time

• Increased opportunity cost

• Greater maintenance complexity

Always begin with the question: "What's the simplest possible solution that could work?" When faced with competing approaches, start with the least complex one.

Lesson 5: Safety Nets Prevent Catastrophic Failures

AI models will make unexpected decisions. The question isn't if but when – and the consequences can be severe without proper guardrails.

At Booking.com, our rebooking algorithm recommended a €5,000 luxury penthouse to a customer whose €500 reservation fell through during a major sporting event. The customer happily accepted this free upgrade, creating a significant financial loss.

The Safety Matrix: Create appropriate safety mechanisms based on risk assessment:

• Human oversight for consequential decisions

• Maximum spending/impact limits

• Error detection mechanisms

• Fallback processes

The higher the potential impact of a wrong decision, the stronger your safety mechanisms should be.

Lesson 6: Time-Boxing Breaks the Sunk Cost Trap

AI projects have a natural tendency to expand indefinitely as teams chase incremental improvements. Without strict boundaries, you'll find yourself pursuing diminishing returns while opportunity costs multiply.

This exact trap caught us at Allegro – what started as a two-sprint project stretched into multiple months because we kept believing "just a little more time" would deliver breakthrough results.

The Time-Box Protocol: Treat AI projects as series of time-constrained experiments rather than open-ended initiatives:

1. Define clear hypotheses

2. Set explicit evaluation criteria

3. Establish non-negotiable time limits (usually 2-4 weeks)

4. Evaluate ruthlessly at the deadline

5. Either proceed with promising results or pivot to a different approach

Remember: An AI project that delivers 60% accuracy in two weeks creates more business value than one promising 90% accuracy "eventually."

Lesson 7: Dual Metrics Ensure Meaningful Success

Effective AI implementation requires continuous monitoring across two dimensions:

Technical metrics: Precision, recall, accuracy, F1 score

Business KPIs: Revenue impact, time saved, customer satisfaction

A successful AI solution must perform well on both fronts – technical excellence without business impact is academic, while business impact without technical robustness is unsustainable.

At Allegro, we transformed how we evaluated recommendation engines by moving beyond technical metrics to measure actual business impact. This shift fundamentally changed which models we deployed, prioritizing those with demonstrable commercial value over those with impressive technical benchmarks.

The Dual Measurement System: For every AI initiative, establish paired technical and business metrics. Create dashboards that show both dimensions simultaneously, and never celebrate technical success without corresponding business outcomes.

Lesson 8: The Vendor Selection Protocol That Prevents Expensive Mistakes

Working with external AI vendors? Apply these critical evaluation criteria:

1. Reject "black box" solutions If vendors can't clearly explain their methodology, run. Always verify claims with examples and establish clear evaluation criteria before committing.

2. Avoid academic partnerships unless commercially oriented University collaborations offer theoretical expertise, but their definition of success (papers, research) often diverges from yours (deployment, ROI). Ensure alignment of objectives before committing.

3. Establish model maintenance responsibility Who will retrain the model when performance degrades? Who handles data drift issues? Address these questions before implementation, not after problems emerge.

4. Prioritize business-aligned partners The best AI partners understand your business context, not just technical implementation. They should spend time with your domain experts, not just your data team.

The Bottom Line: Are You Building or Gardening?

AI isn't magic – it's a methodical discipline requiring structured experimentation. Organizations that approach AI with garden-cultivating patience rather than building-construction certainty will unlock transformative potential while competitors waste resources on impressive but unproductive technical showcases.

The most successful AI implementations I've witnessed shared three characteristics:

1. Crystal-clear problem definition

2. Simplicity-first approach

3. Rigorous, binary success criteria

Are you cultivating a garden of possibilities or focusing on technical sophistication? Your approach will determine whether your AI initiatives flourish or wither.

The question isn't whether AI will transform your industry – it's how effectively you'll implement these transformations compared to competitors who understand the garden strategy.