Inaugural Testing Thursday Success!

Q: What happens when you combine a bunch of smart testing practitioners, 5 (yes, 5!) different testing vendors, Avinash Kaushik, and free drinks & appetizers?

A: A really good time, with ample helpings of witty banter and intelligent conversation!

Thanks to everybody who came out for the first Testing Thursday, it was great to see such an enthusiastic crowd.  As you can see from the pictures, we got lucky with the weather and people and had to be given the last call warning by the restaurant.

We had a turnout of about 35 people, and Avinash Kaushik gave a short talk about the state of testing and optimization today. 

Here are some of the highlights from that conversation:

• Stop testing button colors! Or, rather, at least test things other than button color!
• Testing vendors have overpromised the value of testing a million things at once. A/B is an extremely powerful place to start, and Avinash never even mentions “multivariate” in his talks about testing.
• A really interesting test was run recently on the value of bidding on your brand keywords. It was tested by only running the branded bidding on specific days of the week, and then comparing the organic traffic %’s & conversion rates on those days against the other days of the week. By doing so, they were able to deduce that organic traffic was slightly cannibalized (if memory serves me right, it was around 11%), but the astonishing finding was that conversion rate went up by 47% on the days of branded keyword bidding. Now that’s a great test.
• Another test was designed for an extremely large advertiser that involved 4 dimensions and some phDs. With some serious math and brainpower, they were able to find phenomenal lift and save millions of dollars. Beyond that, my memory fails me.

Thanks to Omniture for sponsoring the first Testing Thursday, Avinash for taking time out of his busy schedule to attend, and Google for bringing t-shirts, stickers, and picking up the last couple rounds of drinks. (Who knew web optimizers were such a bunch of boozers?!) We’re working on planning the next event now, and will have details out shortly.

Looking forward to the next Testing Thursday!