The AI era requires a different kind of experimentation.

I bet your 2022 experimentation playbook is obsolete.

Experimentation is my favorite. I just love it. But this dear old friend of mine is quite different from even a few years ago.

And maybe that’s not such a bad thing. The old way of experimenting had its limits:

• Focus on minor tweaks instead of big bets

• Optimizing superficial surfaces instead of foundational complexity

• Avoiding monetization (‘It takes 6 months of decision approvals & build out. If you are lucky.’)

• Product mostly testing through previous/post, instead of actual A/B tests or feature flags

• Focusing on fast wins - results measured in 2 weeks

Some of this made sense, back when product development was slow. There was so much bloat and so many surfaces that just creating awareness of features and explaining how to use them was a win.

But even then, many of those experiments were just pulling up revenue (if anything). They weren’t actually having incremental progress. But by improving the experience a bit or creating some urgency, we made this thing (which would have happened anyway) …