CODE@MIT – OCTOBER 16 2015
Lessons learned in display advertising
Natural experiments at scale
Robert Moakler – (rmoakler@...
CODE@MIT – OCTOBER 16 2015
The $100+ billion question!
Does online advertising really work?
$104.57
$120.05
$140.15
$160...
CODE@MIT – OCTOBER 16 2015
The $100+ billion question!
Does online advertising really work?
Do online ads cause you to ...
CODE@MIT – OCTOBER 16 2015
The usual approach!
Randomized experiments and A/B tests are great!
Campaign AdPSA
CODE@MIT – OCTOBER 16 2015
The usual approach!
But sometimes …
RIGHT
WRONG
Randomized experiments and A/B tests are great!...
CODE@MIT – OCTOBER 16 2015
Natural experiments!
•  Consider the typical setup for the ad serving process
Confounding!
W
Us...
CODE@MIT – OCTOBER 16 2015
Natural experiments!
•  Consider the typical setup for the ad serving process
•  Introduce a me...
CODE@MIT – OCTOBER 16 2015
Natural experiments!
•  Consider the typical setup for the ad serving process
•  Introduce a me...
CODE@MIT – OCTOBER 16 2015
Ad viewability!
Horizontal location (px) Proportion of ads
Verticallocation(px)
Addensity
CODE@MIT – OCTOBER 16 2015
Running in the wild!
•  Natural experiments aren’t always clean or easy
•  We will discuss five ...
CODE@MIT – OCTOBER 16 2015
An online advertising campaign!
•  Our data structure
Analysis window
Viewable ad
Unviewable ad...
CODE@MIT – OCTOBER 16 2015
Longitudinal data!
•  Monitoring
–  Most online advertising campaigns run continually
–  We are...
CODE@MIT – OCTOBER 16 2015
User fragmentation and study period!
•  In reality, our users are defined by cookies.
–  Howeve...
CODE@MIT – OCTOBER 16 2015
User fragmentation and study period!
•  In reality, our users are defined by cookies.
–  Howeve...
CODE@MIT – OCTOBER 16 2015
Validation!
•  How do we know our causal models give reasonable estimates?
•  Use an array of n...
CODE@MIT – OCTOBER 16 2015
Running at scale!
•  Converting our data into something analyzable is a challenge
…
Raw daily l...
CODE@MIT – OCTOBER 16 2015
Summary!
•  Mediators and natural experiments may already exist in your data
•  Running a natur...
CODE@MIT – OCTOBER 16 2015
Thanks!
Robert Moakler – (rmoakler@stern.nyu.edu)
Ekaterina Eliseeva – (keliseeva@integralads.c...
CODE@MIT – OCTOBER 16 2015
Acknowledgments!
Integral Ad Science
Ekaterina Eliseeva
Kiril Tsemekhman
Ana Calabrese
Gijs Joo...
CODE@MIT – OCTOBER 16 2015
References!
Chan, D., Ge, R., Gershony, O., Hesterberg, T., & Lambert, D. (2010, July). Evaluat...
of 20

Natural Experiments at Scale

Rob Moakler presents "Natural Experiments at Scale" at The Conference of Digital Experimentation @ MIT
Published on: Mar 3, 2016
Published in: Science      
Source: www.slideshare.net


Transcripts - Natural Experiments at Scale

  • 1. CODE@MIT – OCTOBER 16 2015 Lessons learned in display advertising Natural experiments at scale Robert Moakler – (rmoakler@stern.nyu.edu) Ekaterina Eliseeva – (keliseeva@integralads.com) Kiril Tsemekhman – (kiril@integralads.com) CODE@MIT 2015
  • 2. CODE@MIT – OCTOBER 16 2015 The $100+ billion question! Does online advertising really work? $104.57 $120.05 $140.15 $160.18 $178.45 $196.05 $213.89Digital ad spending! % change! 2012 2013 2014 2015 2016 2017 2018! Source: www.emarketer.com, “Global Ad Spending Growth to Double This Year” 20.4% 14.8% 16.7% 14.3% 11.4% 9.9% 9.1%
  • 3. CODE@MIT – OCTOBER 16 2015 The $100+ billion question! Does online advertising really work? Do online ads cause you to take some action?
  • 4. CODE@MIT – OCTOBER 16 2015 The usual approach! Randomized experiments and A/B tests are great! Campaign AdPSA
  • 5. CODE@MIT – OCTOBER 16 2015 The usual approach! But sometimes … RIGHT WRONG Randomized experiments and A/B tests are great! Campaign AdPSA
  • 6. CODE@MIT – OCTOBER 16 2015 Natural experiments! •  Consider the typical setup for the ad serving process Confounding! W User features A Served ads Y Convert
  • 7. CODE@MIT – OCTOBER 16 2015 Natural experiments! •  Consider the typical setup for the ad serving process •  Introduce a mediating variable W User features A Served ads Y Convert M Mediator W’ Residual Confounders
  • 8. CODE@MIT – OCTOBER 16 2015 Natural experiments! •  Consider the typical setup for the ad serving process •  Introduce a mediating variable –  Viewability W User features A Served ads Y Convert V Viewable ad W’ Residual Confounders
  • 9. CODE@MIT – OCTOBER 16 2015 Ad viewability! Horizontal location (px) Proportion of ads Verticallocation(px) Addensity
  • 10. CODE@MIT – OCTOBER 16 2015 Running in the wild! •  Natural experiments aren’t always clean or easy •  We will discuss five problems that we have run into and some solutions for dealing with them
  • 11. CODE@MIT – OCTOBER 16 2015 An online advertising campaign! •  Our data structure Analysis window Viewable ad Unviewable ad Conversion Web activity Our users
  • 12. CODE@MIT – OCTOBER 16 2015 Longitudinal data! •  Monitoring –  Most online advertising campaigns run continually –  We are constantly monitoring many campaigns at the event level •  Running an intermediary analysis –  Data is subject to left truncation and right censoring –  We need to account for our residual confounders, W’ •  Use survival analysis –  Cox Proportional Hazards (CPH) model
  • 13. CODE@MIT – OCTOBER 16 2015 User fragmentation and study period! •  In reality, our users are defined by cookies. –  However, people do not just have one cookie! Viewable ad Unviewable ad Conversion Web activity Sarah Cookie 1 Cookie 2 Cookie 3 Bob Cookie 1 Cookie 2 Analysis window Cookie 3
  • 14. CODE@MIT – OCTOBER 16 2015 User fragmentation and study period! •  In reality, our users are defined by cookies. –  However, people do not just have one cookie! •  Some methods we use to account for this –  We define an effect period of 1 week •  Seasonality has a major impact •  Users are selected through iterative simulation and research •  Incremental causal estimates level off after a single week
  • 15. CODE@MIT – OCTOBER 16 2015 Validation! •  How do we know our causal models give reasonable estimates? •  Use an array of negative control tests –  Use the impressions of one campaign to predict an unrelated conversion W User features A Served ads Y Convert W’ Residual Confounders Y Unrelated Event - V Viewable ad
  • 16. CODE@MIT – OCTOBER 16 2015 Running at scale! •  Converting our data into something analyzable is a challenge … Raw daily logs Billions of events HDFS scalable cluster storage Hadoop People browse the web. Advertising events turn into billions of daily events. Raw data is moved to scalable storage optimized for our experimental setup. Users are subsampled and negative controls are chosen in parallel. Reports are run in parallel using stripped down R libraries. Iterative process of simulation and research.
  • 17. CODE@MIT – OCTOBER 16 2015 Summary! •  Mediators and natural experiments may already exist in your data •  Running a natural experiment at scale is not straight forward, because 1.  The longitudinal nature of the data 2.  Users can become highly fragmented 3.  No predetermined start and end dates 4.  Validation of causal models 5.  Billions of events and terabytes of raw data •  Equal parts engineering and modeling •  We explored online advertising, but this setup can apply to a wide variety of industries
  • 18. CODE@MIT – OCTOBER 16 2015 Thanks! Robert Moakler – (rmoakler@stern.nyu.edu) Ekaterina Eliseeva – (keliseeva@integralads.com) Kiril Tsemekhman – (kiril@integralads.com) Grab this deck @ bit.ly/natural-experiments-at-scale
  • 19. CODE@MIT – OCTOBER 16 2015 Acknowledgments! Integral Ad Science Ekaterina Eliseeva Kiril Tsemekhman Ana Calabrese Gijs Joost Brouwer Sergei Izrailev NYU Stern Foster Provost Amazon, Inc. Daniel Hill
  • 20. CODE@MIT – OCTOBER 16 2015 References! Chan, D., Ge, R., Gershony, O., Hesterberg, T., & Lambert, D. (2010, July). Evaluating online ad campaigns in a pipeline: causal models at scale. In Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining (pp. 7-16). ACM. Dalessandro, B., Perlich, C., Stitelman, O., & Provost, F. (2012, August). Causally motivated attribution for online advertising. In Proceedings of the Sixth International Workshop on Data Mining for Online Advertising and Internet Economy (p. 7). ACM. Hill, D. N., Moakler, R., Hubbard, A. E., Tsemekhman, V., Provost, F., & Tsemekhman, K. (2015, August). Measuring Causal Impact of Online Actions via Natural Experiments: Application to Display Advertising. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 1839-1847). ACM. Johnson, G. A., Lewis, R. A., Nubbemeyer, E. I. (2015, October). Ghost Ads: Improving the Economics of Measuring Ad Effectiveness. Available on SSRN: ssrn.com/abstract=2620078 Klein, J. P., & Moeschberger, M. L. (2003). Survival analysis: techniques for censored and truncated data. Springer Science & Business Media. Pearl, J. (2009). Causality. Cambridge university press.

Related Documents