Distinguished Lecture
Women in Computing

A/B Testing at Google: Scaling to 1000s of Simultaneous Tests

Diane Tang

Google Fellow
Tuesday, February 24, 2015
4:00pm - 5:00pm
EECS 1303

Add to Google Calendar

About the Event

Over the past 20 years, software engineering has shifted from shrink-wrapped software to SaaS. While many people focus on the increasing modularization and ability to have different services proliferate as creating new products becomes easier, a more subtle shift is in the new-found ability to rapidly optimize these services for users via A/B testing.

Google has pushed the bounds of this optimization, and runs live traffic A/B tests to understand the efficacy of various product changes, ranging from UI changes big and small to algorithmic changes. In this talk, we’ll explain how we have pushed the limits on how to optimize software and improve user experience via A/B testing. We’ll start with the tools and infrastructure we developed to scale to running hundreds to thousands of live traffic experiments simultaneously. This infrastructure, while it leads to an ability to scale, also has impact on experimental design, such as what factors can be tested together and what types of experiments are easy or hard to run. Given the scale of the experimentation that we run, we’ll also discuss several best practices in designing robust live traffic experiments and metrics for hundreds to thousands of experiments. This ranges from proper data capture (including the importance of counterfactuals) to metric design to understanding the statistical variance of what we measure (including why, even with billions of queries, we see such large variances in metrics).

However, it’s not just a matter of infrastructure and best practices: establishing a data-driven approach, process, and culture for product development is key to supporting the pace of innovation at Google. We’ll conclude with an overview of the principles that guide our experiments, ranging from user- and product-focus to privacy and security issues to ethics. We’ll discuss how we consider multiple perspectives to ensure robust practices that both protect the user and advance the state of the art for all users.

Time permitting, we'll conclude with a brief discussion of potential implications of our work towards future research in medicine and learning health systems.


Diane Tang is a Google Fellow currently working in Google Life Sciences on building data infrastructure and analytics for biological & medical applications. Prior to 2014, she was a leader on the AdsQuality team at Google. She joined Google in 2003 and has focused on logging, large-scale data analysis & infrastructure, experiment methodology and ads systems. She earned a bachelor's degree in Computer Science from Harvard in 1995 and a master's degree and PhD in Computer Science from Stanford in 2001. She holds many patents and is the author of numerous publications in mobile networking, information visualization, experiment methodology, data infrastructure, and data mining / large data.

Additional Information

Sponsor(s): CSE

Open to: Public