Keynote: The Jungle of In-house Experimentation

22 Nov 2025

9:55 - 10:30

Keynote Room

Melanie Mueller – Senior Product Manager Experimentation @ Booking

Melanie Mueller at #CH2025

Full keynote video access available for members (tickets here).

Xeï – CRO Specialist, feedback through our #CH2025 attendee survey:

Really interesting to have a different perspective on test quality and what has been done to improve this.

Questions asked by attendees through our #CH2025 app:

What is the ratio of the number of people in the Center of Excellence to the ambassadors in the organization?
You talk about execution, decision-making, and design, but it’s still all about people. How do you ensure the right people stay and the bad apples leave?
How do you know the effect of your experiment will continue over time? or do you run experiments again?
Do you check whether the pre-test data matched to the actual experiment data?
How exactly do you measure something like ‘good decision’ in your plots? Does that not depend on the unknown truth?
How does the Booking.com police have 3 arms? ;-)
Do you correlate ExQual to actual € ROI on a macro scale
What happened in 2022 when execution quality dropped?
Are the ambassadors you have embedded in the individual teams experimenters (specialist) or a person from the respective teams?
Are there elements of quality that you are not capturing that you think should be there?
Rumour is most booking experiments now are non inferiority. are you still doing for-improvement experiments or not so much?
Are different components of the scores weighted depending on importance, or ever modified over time (do you experiment on experiment quality scoring
Did you retest the bad quality experiments and were the results very different?
How do you deal with interference between all your experiments?
What tools do you use?
For the 1/4 experiments that are implemented, how do you decide that
Which teams arent running experiments?