Keynote: The Jungle of Science vs CRO

Keynote: The Jungle of Science vs CRO

Marcella Sullivan – CRO Expert @ Sullivan Consulting Ltd

Small video preview

Full keynote video access available for members (tickets here).

Ben – Senior Optimization Consultant, feedback through our #CH2025 attendee survey:

This session was very enjoyable. Marcella was able to clearly and confidently articulate the crossover between scientific rigor and online experimentation without making the subject matter too dry.

Slides

(direct download link)

Notes

This is the link to the full summary of Marcella her talk

Questions asked by attendees through our #CH2025 app:

• Like that you mention there should be a guardrail against harm. How would you advice to assess this in practice.
• How do you cope with a need of a higher significance versus not enough traffic to actually finding a result?
• Bayesian or frequentist?
• Is it possible for the “line” (rigor line) be different in different businesses or in different circumstances? like in different verticals? or iterations? or for high value audiences?
• If we allow people to say “the risk of this one is low, we can ship with less rigour” how do we prevent that burgeoning and everyone using it to push their stuff through?
• How do yoh measure rigour? is it purely confidence levels?
• Having variable levels of rigour makes perfect sense in theory, but in practice it seems like a lot of overhead to establish that without very well defined protocols can cause friction within an org. Could a standardised approach be preferable in scaling quality and quantity?
• What year will the replication crisis tsunami hit the “less rigor” CRO world?
• How do you prevent many discussions before each test when rigour varies?
• What are some other things we can translate over from scientific methods to our cro jungle?
• Can we quantify the required rigour, and the cost of reducing it?
• Science is as much a human endeavor as a/b testing, so all the caveats of CRO should also apply to science?
• If a tiny change—like adjusting font size—seemed to generate x amount of money, and management wanted to claim it, but I didn’t trust the effect without understanding the mechanism, would you still accept it—or would you run a follow-up experiment to see if the results replicate?
• What’s the bigger risk, the false positive? or the false negative?
• We might all agree on the basic hygiene’s of CRO, why is it that we rely on the outcome of CDP’s?