On day two at Zynga, maybe day three, I got yelled at by my boss. He had just been yelled at by one of the most senior directors in the building, the one in charge of analytics and A/B testing.
What I'd said was simple. If you run enough tests, you will find significant results that are noise. More tests, more spurious findings. Statistically speaking, that's not an opinion. It did not land well in a company that treated the test volume as the product.
My argument then is my argument now. Five to seven people in a room, watching them play, tells you whether the boat is pointed at the right destination. A/B testing tells you how to row faster toward wherever you're already pointed. Both are great. They answer different questions, and the second one can optimize you straight into a worse place.
I'm all for analytics. I spent years there. But those tools optimize against a curve. They don't set it.
One more Zynga story, since we're here. I told all my product teams not to run an April Fools' event. Not because jokes are bad. Because the way we treated players, an April Fools' joke wouldn't be funny to them. Know your relationship with the audience before you try to be cute with it.