Keynote: The Jungle of Choosing Metrics

Keynote: The Jungle of Choosing Metrics

Kees Mulder – Data Scientist / Statistician in Team Experimentation @ bol

Small video preview

Full keynote video access available for members (tickets here).

Eline – Marketing Automation & UX specialist, feedback through our #CH2025 attendee survey:

Did a really good job, spoke with confidence and a good topic to dive in deeper.

Slides

(direct download link)

Notes

This is the link to the full summary of Kees his talk

Questions asked by attendees through our #CH2025 app:

• What’s your take on the almost religious question “Baysian or Frequentist”?
• Can you comment on using 95% confidence vs 80% confidence? I think the latter is a horrible practice!
• How many experiments do you need on the same KPI to he able to plot local/global metric graphs. And do these experiments need to be on the same step of the customer journey ?
• What’s your opinion on using more global metrics as „guardrail“ metrics (expecting they won’t hit significance) and use a more local primary metric for low traffic tests?
• Completely agree with the distraction element of CTR based pages, but Amazon makes lots of cash, sooo…it probably works sometimes right?
• In the orders chart, many visitors had negative order values, what’s up with that?
• Have you seen pushy optimisation methods (scarcity, urgency) lead to increasing return rates?
• Besides order splitting, what else should we be looking out far?
• Do you take user and experiment variables into account when checking local v global metric correlation? so it could be correlation under certain conditions but not in others?
• How should news & media companies measure engagement? A composite engagement score?
• Do you only have a lot of fun when people are starting to get it right, not just when they are messing up?