TL;DR

# Abstract

In online display advertising, advertisers usually participate in real- time bidding to acquire ad impression opportunities. In most advertising platforms, a typical impression acquiring demand of advertisers is to maximize the sum value of winning impressions under budget and some key performance indicators constraints, (e.g. maximizing clicks with the constraints of budget and cost per click upper bound). The demand can be various in value type (e.g. ad exposure/click), constraint type (e.g. cost per unit value) and con- straint number. Existing works usually focus on a specific demand or hardly achieve the optimum. In this paper, we formulate the demand as a constrained bidding problem, and deduce a unified optimal bidding function on behalf of an advertiser. The optimal bidding function facilitates an advertiser calculating bids for all impressions with only 𝑚 parameters, where 𝑚 is the constraint number. However, in real application, it is non-trivial to determine the parameters due to the non-stationary auction environment. We further propose a reinforcement learning (RL) method to dynam- ically adjust parameters to achieve the optimum, whose converging efficiency is significantly boosted by the recursive optimiza- tion property in our formulation. We name the formulation and the RL method, together, as Unified Solution to Constrained Bid- ding (USCB). USCB is verified to be effective on industrial datasets and is deployed in Alibaba display advertising platform.

# 解决方案

• 状态$\cal{S}$，广告计划相关特征，包含时间、预算消耗、KPI约束状态等等
• 动作$\cal{A}$，每个时刻$t$，每个agent会采取M+1维的动作向量$\overrightarrow{a_t}=(a_{0t},\cdots,a_{Mt})$，对应参数向量$\overrightarrow{w_t}=(w_{0t},\cdots,w_{Mt})$，下一时刻参数为$\overrightarrow{w_{t+1}}=\overrightarrow{w_{t}}\cdot(\overrightarrow{a_{t}}+1)$
• 奖励$r_t$，定义$r_t$为当前时刻的目标函数值，$r_t=\sum_{i\in O_t} v_i\cdot x_i$，其中$O_t$表示$t$时刻流量集合
• 折损系数$\gamma$，设置为$1$

• 对于时刻$t$，agent会尝试将参数$\overrightarrow{w_t}$修正到$\overrightarrow{w^*_t}$并保持，而不是逐步调整，这加快了agent的学习速度
• 评估网络$Q$可以简化为$\cal{G}$$Q(s_t,\overrightarrow{a_t})$的差异，同时$\cal{G}$可以直接看出哪个action可以取得更好的结果，这样会使得整个算法收敛得更快

# 实验分析

• CB{click}，最大化点击，约束是预算，此时最优出价策略是$b_i=w_0 \cdot \text{pCTR}_i$
• CB{click-CPC}，最大化点击，约束是预算和CPC成本，最优出价是$b_i=w_0\cdot \text{pCTR} +w_1\Bbb{k}_\text{CPC} \cdot \text{pCTR}$
• CB{conversion-CPC}，最大化转化，约束是预算和CPC成本，最优出价是$b_i=w_0\cdot \text{pCVR}_i +w_1\Bbb{k}_\text{CPC} \cdot \text{pCTR}_i$

M-PID方法主要缺点在于需要先验知识设定目标，同时需要调节超参数。USCB相比DRLB主要是收敛效率所带来的稳定性优势。