Any particular change (like this redesign) might not move the needle, but there exists SUBSTANTIAL evidence that A/B testing can help the bottom line of the company. Read e.g. any of the multiple times the 37 Signals blog covered how they got like 40% lifts to paid account signups by redoing a pricing or landing page.
Do you find me credible on this topic? I'm legally, morally, and practically constrained regarding how much detail I can go into here, but if you trust me: I have recent experience which makes me disagree in the strongest possible way with the generalization of your hypothesis. A recent client CCs me on their weekly Optimizely status report. I've been evangelizing A/B testing for a few years. Every time I read one of those emails my mind gets blown again.
Are you aware of anyone experimenting with the automatic evolution of web pages?
For example, use genetic programming to periodically modify the page served up. Both pages are served up, A/B style, and the page which has the highest metric (eg. $ earned) survives. Repeat.
It would be interesting to see what the results are after an extended period of time.
Having played with genetic algorithms before, that is a hugely interesting idea to me. You would need to take special care on the types of mutations used though. I don't think it is possible to do it well at scale, but it would be very impressive nonetheless.
That's like telling the director of the FBI he knows nothing about law enforcement because he wont reveal facts that he has a professional obligation to not reveal.
But wouldn't there be big gains initially and then, as the easy fruit was plucked, the changes would give less and less and the successful tests would become fewer and fewer?
Likely yes. But what matters is return on time in contrast to any of the other things you could be doing. "always work on the most important thing". Even small percentage point improvements in conversion go directly to the bottom line, which is far from obviously the case when fixing bugs or adding features. And if your bottom line is already pretty big, the marginal value of someone working on optimization is likely to continue to be high, regardless of how plucked the fruit seems to be.
Do you find me credible on this topic? I'm legally, morally, and practically constrained regarding how much detail I can go into here, but if you trust me: I have recent experience which makes me disagree in the strongest possible way with the generalization of your hypothesis. A recent client CCs me on their weekly Optimizely status report. I've been evangelizing A/B testing for a few years. Every time I read one of those emails my mind gets blown again.