Online Controlled Experiments (A/B Testing) is a game-changer when it comes to decision-making, creating compelling customer experiences, and ensure maximum return of investment (ROI).

A/B testing gives small and large digital companies alike the power to give their users a voice and let them, by their actions, steer the direction of all digital development.

As a concept, Online Controlled Experiments (A/B Testing) is simple. Two or more variants of a webpage or app are shown to users at random to determine which one performs better. However, the execution is often filled with frustration, and companies make serious mistakes.

For experienced A/B test practitioners, this is not surprising. A/B testing is reliant upon proper statistical practices and there are many other potential pitfalls besides pure statistics one encounter.

Also, building a data-driven culture around digital experiment is far from easy. It requires that you have management on board and that they are okay both by being proven wrong by data and to delegate power to run experiments to the people working for them.

I have personally found it next to impossible to find one affordable single source of truth on how to properly conduct A/B tests, learn to identify (and mitigate) the risks one often encounter when doing so while also providing the information on how to communicate and persuade management to give A/B testing the conditions to thrive.

This book change that.

Within the small community of people dedicated to A/B testing, the authors are among the most well-known and cited authorities.

The book is divided into different sections. At the beginning of the book, it is explained what audience each section is suitable for.

The result is a book valuable for anyone interested in the field of A/B Testing. It is easy to read, but it is also "meaty", with a lot of good advice for both new and experienced A/B testing practitioners.

For management and business readers without any prior knowledge, a lot of high-level examples are provided. Since the authors, all have wast experience from digital giants such as Google, LinkedIn, and Microsoft, these examples give a rare glimpse into how digital experiments are conducted in real life within these companies and also the amazing results achieved by doing so.

Something I especially like is that this book is research-based with a clear reference to academic papers (which one can dig deeper into if there is an interest in doing so).

To sum it up, this book is the best one I've read so far on the topic of A/B testing and digital experimentation and I highly recommend it to anyone interested in this field.

Note: during the authors' writing process, I have had the opportunity to give them feedback. For this, I have been mentioned in the acknowledgment section of the book and I have also been given a physical copy free of charge. Still, as one can tell by my review, I choose to purchase a digital copy just to ensure I can constantly access the book through my Kindle and computer when I don't have the physical copy at hand. This by itself should give you an indication of how valuable I consider the information in it to be.

A quick summary:

A/B testing can be a powerful tool for IT companies to test out new ideas and provide data-driven evidence for innovation. However, without a correct mindset or appropriate design, A/B testing can go wrong. The corresponding results will not be trustworthy, and conclusions can be misleading and even damaging. The authors encourage readers to evaluate the trustworthiness of experiment results and share important practical lessons. This book will help a wide range of readers (e.g., executives, product managers, engineers, data scientists/analysts) to avoid the pitfalls and to make the most of A/B testing.

More Detailed reading notes

Part I is designed to be read by everyone, regardless of background. The basics about A/B testing (e.g., the concept of controlled experiments, the benefits) and the examples (Chapter 1 and Chapter 2) are relatively straightforward. The key message of Part I is that trustworthiness of (interpreting) experiment results is a critical issue. To increase the trust in experiments and experiment results, the authors laid out questions/suggestions from different levels:

1. The company/executives level: three questions to answer

Does the company want to build a culture of making data-driven decisions?

Is the company willing/ready to invest in the infrastructure and test to run A/B testing?

Is the current way of assessing the value of ideas poor or not good enough?

If yes, read the tenets (Page 11 to Page 14). More details in Chapters 4, 6, 7, and 8.

2. The middle/design level (for senior manager, product managers)

How to help build a robust and trustworthy experiment platform?

How to design metrics and align them with missions/long-term goals (many goals contradict each other)?

How to ensure that the results are trustworthy in general (internal validity and external validity)?

Is A/B testing the only technique to provide evidence for data-driven decisions?

Detailed answers are in Chapter 4 in Part I, Part II (Chapter 5 to Chapter 9) and Part III (Chapter 10 and Chapter 11).

3. The operation level (for data analysts, engineers)

The list of questions will be much longer and at a more detailed level. A few examples:

How to process logs from multiple sources (e.g., logs from servers, logs from different client types)?

How to choose a randomization unit (user level or session level)?

How to conduct a power analysis?

How to interpret the experiment results (e.g., p-value, confidence interval)?

Details answers are in Part IV (Chapter 12 to Chapter 16) and Part V (Chapter 17 to Chapter 23).

In sum, Part I introduces the basics about A/B testing and lays out the framework for the rest of the book.

As a data scientist on the experimentation platform team in Microsoft, I have worked on many products within Microsoft like Microsoft News, Edge Browser, Office and Windows to run online controlled experiments at scale. As a professional in the area of Online Controlled Experiments, I know the authors of this book and I have worked under Ron Kohavi during his time at Microsoft. I have helped review some of the early drafts of the book and I received a complementary copy of the book from Ron Kohavi who also encouraged me to write an unbiased review.

This book is contains a wealth of technical, methodological, and practical information from years of experience in running online controlled experiments in three large technology companies. The book is very well researched and references a large number of papers in the area of online controlled experiments in software and randomized controlled trials in general.

This book is well suited for readers with different levels of experience in online controlled experiments. The book is very well structured with sections for everyone - from product team leaders and CEOs, to data engineers to data scientists. This book can act a good introduction to Online Controlled Experiments and also serve a reference for advanced topics.

The area of Online Controlled Experiments is still relatively new. There are still a lot of open problems and challenges in this area. This book serves as a good reference to get you up to speed on the current state of art. I would recommend the readers to extend the learnings in this area by building on the foundations discussed in the book.

I like the full spectrum of topics about controlled experiments covered in this book, nicely organized in 23 chapters. Experimentation is a core and fundamental piece of Data Science work in almost all companies that is equipped with DL and DS capabilities. Technical challenges & solutions, best practices, and related stats, however, exist in mostly scattered style, from my personal experiences. This book provides a comprehensive overview of the topic.

I wish that there could be case study walk through about the experiment design and experiment data analysis.

First, I very much still believe what I wrote as a blurb for this book:

"Internet companies have taken experimentation to an unprecedented scale, pace, and sophistication. These authors have played key roles in these developments and readers are fortunate to be able to learn from their combined experiences."

These are exactly the right authors for a book with this topic and intended audience. I'm a co-organizer of the Conference on Digital Experimentation (CODE) at MIT, and we've had Ronny Kohavi and Ya Xu as invited speakers — in fact I think we've had Ronny as an invited speaker every year! This expertise and ability to communicate comes through in this book.

I've also now had a chance to teach with this book at MIT, where it has been immensely helpful in teaching Marketing Analytics for MBA, Masters of Business Analytics, and other students. (I think this may have been the first use of the final text in the classroom, as the authors and publisher helpfully provided us with early access copies.)

"Trustworthy Online Controlled Experiments" is rigorous and detailed enough to make it appropriate for a course, but also deeply connected with practical considerations, which is especially important for professional degree programs. The examples are tilted towards Internet companies, but much of the material is useful more broadly, especially as more and more experiments will be deployed through similar software stacks.

For other instructors, here's what we've used from the book, in the order I used them:

Session titles | Book chapters

Metrics, customer lifetime value & ROI | Ch 6

Randomized experiments Part I & II | Ch 1-2, Ch 17

Advertising and marketing experiments | Ch 18

Product experimentation | Ch 3-4, 7 & 23

Targeting marketing interventions | Ch 20

Network data and multiple units | Ch 22

This book largely replaces my use of "Field Experiments" by Gerber & Green, though I still point more technical students to that as a resource.

This is a great book. The first section explains general concepts about A/B testing. The next section goes into practical examples of A/B tests the authors ran. Lastly there is a more technical section.

A DS would benefit from this book by the quotes alone which should help convince more executives on the need for measurement on critical business objectives.

Love how detailed this book is about all the subtleties and pitfalls in experimentaion.

This arrived in the mail today and I just looked through it. My take is that this is a must-read book for anyone involved in online experimentation. And it is extremely readable: business and product managers as well as data scientists will find it useful. It provides technical and practical information needed to carry out trustworthy experiments and contains a wealth of case studies from Microsoft, Google and LinkedIn.

The HiPPO stress-toy mascot pictured on the cover reminds everyone that the Highest Paid Person's Opinion should not be the default decider of designs. If people apply the learnings from this book, they'll avoid a lot of potentially misleading results that can arise in in the execution and interpretation of online experiments.

This book is a masterpiece on Experimentation!!

Amazing book for someone who is interested in analytics space. Must read for folks who conduct experimentation or are looking for similar roles in big tech.

buen libro

The book has quite a lot of examples on the best practices and infrastructure of online experiments. It focus a lot on particular domains (like browsers), and l found that, that knowledge is not applicable to othe domains. I would have liked more notions on the statistical part, but I also understand that is a practical guide, not a statistics book.

Fantastic book that is packed with practical insights about experimentation and easy to grasp for non-mathematical folks as well.