ABSTRACT
The majority of online display ads are served through real-time bidding (RTB) --- each ad display impression is auctioned off in real-time when it is just being generated from a user visit. To place an ad automatically and optimally, it is critical for advertisers to devise a learning algorithm to cleverly bid an ad impression in real-time. Most previous works consider the bid decision as a static optimization problem of either treating the value of each impression independently or setting a bid price to each segment of ad volume. However, the bidding for a given ad campaign would repeatedly happen during its life span before the budget runs out. As such, each bid is strategically correlated by the constrained budget and the overall effectiveness of the campaign (e.g., the rewards from generated clicks), which is only observed after the campaign has completed. Thus, it is of great interest to devise an optimal bidding strategy sequentially so that the campaign budget can be dynamically allocated across all the available impressions on the basis of both the immediate and future rewards. In this paper, we formulate the bid decision process as a reinforcement learning problem, where the state space is represented by the auction information and the campaign's real-time parameters, while an action is the bid price to set. By modeling the state transition via auction competition, we build a Markov Decision Process framework for learning the optimal bidding policy to optimize the advertising performance in the dynamic real-time bidding environment. Furthermore, the scalability problem from the large real-world auction volume and campaign budget is well handled by state value approximation using neural networks. The empirical study on two large-scale real-world datasets and the live A/B testing on a commercial platform have demonstrated the superior performance and high efficiency compared to state-of-the-art methods.
- K. Amin, M. Kearns, P. Key, and A. Schwaighofer. Budget optimization for sponsored search: Censored learning in mdps. UAI, 2012.Google Scholar
- J. Boyan and A. W. Moore. Generalization in reinforcement learning: Safely approximating the value function. NIPS, pages 369--376, 1995.Google Scholar
- O. Chapelle. Modeling delayed feedback in display advertising. In KDD, 2014. Google ScholarDigital Library
- Y. Chen, P. Berkhin, B. Anderson, and N. R. Devanur. Real-time bidding algorithms for performance-based display ad allocation. In KDD, 2011. Google ScholarDigital Library
- Y. Cui, R. Zhang, W. Li, and J. Mao. Bid landscape forecasting in online ad exchange marketplace. In KDD, 2011. Google ScholarDigital Library
- B. Edelman, M. Ostrovsky, and M. Schwarz. Internet advertising and the generalized second price auction: Selling billions of dollars worth of keywords. Technical report, National Bureau of Economic Research, 2005. Google ScholarCross Ref
- Google. The arrival of real-time bidding, 2011.Google Scholar
- G. J. Gordon. Stable function approximation in dynamic programming. In ICML, pages 261--268, 1995. Google ScholarCross Ref
- T. Graepel, J. Q. Candela, T. Borchert, and R. Herbrich. Web-scale bayesian click-through rate prediction for sponsored search advertising in microsoft's bing search engine. In ICML, 2010.Google ScholarDigital Library
- X. He, J. Pan, O. Jin, T. Xu, B. Liu, T. Xu, Y. Shi, A. Atallah, R. Herbrich, S. Bowers, et al. Practical lessons from predicting clicks on ads at facebook. In ADKDD, 2014. Google ScholarDigital Library
- K. Hosanagar and V. Cherepanov. Optimal bidding in stochastic budget constrained slot auctions. In EC, 2008. Google ScholarDigital Library
- V. Krishna. Auction theory. Academic press, 2009.Google Scholar
- K.-C. Lee, A. Jalali, and A. Dasdan. Real time bid optimization with smooth budget delivery in online advertising. In ADKDD, 2013. Google ScholarDigital Library
- K.-c. Lee, B. Orten, A. Dasdan, and W. Li. Estimating conversion rate in display advertising from past erformance data. In KDD, 2012. Google ScholarDigital Library
- X. Li and D. Guan. Programmatic buying bidding strategies with win rate and winning price estimation in real time mobile advertising. In PAKDD. 2014. Google ScholarCross Ref
- H. B. McMahan, G. Holt, D. Sculley, M. Young, D. Ebner, J. Grady, L. Nie, T. Phillips, E. Davydov, D. Golovin, et al. Ad click prediction: a view from the trenches. In KDD, 2013. Google ScholarDigital Library
- R. J. Oentaryo, E.-P. Lim, J.-W. Low, D. Lo, and M. Finegold. Predicting response in mobile advertising with hierarchical importance-aware factorization machine. In WSDM, 2014. Google ScholarDigital Library
- C. Perlich, B. Dalessandro, R. Hook, O. Stitelman, T. Raeder, and F. Provost. Bid optimizing and inventory scoring in targeted online advertising. In KDD, 2012. Google ScholarDigital Library
- D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. van den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, et al. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587), 2016. Google ScholarCross Ref
- A. L. Strehl, L. Li, E. Wiewiora, J. Langford, and M. L. Littman. Pac model-free reinforcement learning. In ICML, pages 881--888. ACM, 2006. Google ScholarDigital Library
- R. S. Sutton and A. G. Barto. Introduction to reinforcement learning, volume 135. MIT Press Cambridge, 1998.Google ScholarDigital Library
- G. Taylor and R. Parr. Kernelized value function approximation for reinforcement learning. In ICML, pages 1017--1024. ACM, 2009. Google ScholarDigital Library
- J. Wang and S. Yuan. Real-time bidding: A new frontier of computational advertising research. In WSDM, 2015. Google ScholarDigital Library
- J. Wang, W. Zhang, and S. Yuan. Display advertising with real-time bidding (RTB) and behavioural targeting. arXiv preprint arXiv:1610.03013, 2016.Google Scholar
- Y. Wang, K. Ren, W. Zhang, J. Wang, and Y. Yu. Functional bid landscape forecasting for display advertising. In ECML-PKDD, pages 115--131, 2016. Google ScholarDigital Library
- C. J. Watkins and P. Dayan. Q-learning. Machine learning, 8(3--4):279--292, 1992.Google Scholar
- W. C.-H. Wu, M.-Y. Yeh, and M.-S. Chen. Predicting winning price in real time bidding with censored data. In KDD, 2015. Google ScholarDigital Library
- J. Xu, K.-c. Lee, W. Li, H. Qi, and Q. Lu. Smart pacing for effective online ad campaign optimization. In KDD, 2015. Google ScholarDigital Library
- S. Yuan and J. Wang. Sequential selection of correlated ads by pomdps. In CIKM, 2012. Google ScholarDigital Library
- S. Yuan, J. Wang, and X. Zhao. Real-time bidding for online advertising: measurement and analysis. In ADKDD, 2013. Google ScholarDigital Library
- W. Zhang and J. Wang. Statistical arbitrage mining for display advertising. In KDD, 2015. Google ScholarDigital Library
- W. Zhang, S. Yuan, and J. Wang. Optimal real-time bidding for display advertising. In KDD, 2014. Google ScholarDigital Library
- W. Zhang, T. Zhou, J. Wang, and J. Xu. Bid-aware gradient descent for unbiased learning with censored data in display advertising. In KDD. Google ScholarDigital Library
Index Terms
- Real-Time Bidding by Reinforcement Learning in Display Advertising
Recommendations
A Unified Solution to Constrained Bidding in Online Display Advertising
KDD '21: Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data MiningIn online display advertising, advertisers usually participate in real-time bidding to acquire ad impression opportunities. In most advertising platforms, a typical impression acquiring demand of advertisers is to maximize the sum value of winning ...
Deep Reinforcement Learning for Sponsored Search Real-time Bidding
KDD '18: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data MiningBidding optimization is one of the most critical problems in online advertising. Sponsored search (SS) auction, due to the randomness of user query behavior and platform nature, usually adopts keyword-level bidding strategies. In contrast, the display ...
Budget Constrained Bidding by Model-free Reinforcement Learning in Display Advertising
CIKM '18: Proceedings of the 27th ACM International Conference on Information and Knowledge ManagementReal-time bidding (RTB) is an important mechanism in online display advertising, where a proper bid for each page view plays an essential role for good marketing results. Budget constrained bidding is a typical scenario in RTB where the advertisers hope ...
Comments