腾讯广告高可用的深度学习技术架构 唐溪柳

QCon大会

2019/06/25 发布于 技术 分类

QCon  QCon2019 

文字内容
1. 腾讯广告高可用的深度学习技术架构 唐溪柳 腾讯广告总架构师
2. 自我介绍
3. 自我介绍 l 目前主要负责腾讯广告系统技术架构,关注算法和架构设计; l 在深度学习系统技术、检索系统、分布式网络服务、语言模型、大数据处理 等方向有比较深入的了解; l 2012 年加入腾讯广告,主导了腾讯广告检索系统、海量数据分析系统、深度 学习系统的架构设计与开发工作
4. 目录 l 项目背景 l 架构设计 l 特征工程 l 训练平台 l Serving集群 l 系统演进
5. 腾讯广告系统架构图 外部DSP 流量接入平台 ADX 数据回流平台 移动联盟流量 微信流量 腾讯非微信流量 请求日志 回流网关 点击日志 回流网关 曝光日志 回流网关 移动联盟 转化日志 回流网关 反作弊网关 移动联盟SDK 联盟开发者接入 广告播放与精排集群 Mixer and Reranking 数据管理平台 DMP数据 接入 DMP数据 Online Serving DMP数据 透视分析 深度学习平台 Serving集群 用户画像 LBS服务 。。。 训练平台 特征工程 广告投放平台 广点通投 入端 微信广告 投放端 代理商投 放端 广告定向与粗排集群 Retrieval and Scoring Marketing API 数据库代理 数据库集群… 日志与报表处理平台 AD Indexer 实时报表 离线报表 广告系统数据流 实时日志关联 离线日志关联 日志回流消息队列 计费网关
6. 2016年底,广告系统应用机器学习技术面临的挑战 l 海量训练数据, 特征复杂 Ø 比如: “男性 x 年龄28-35 x 关注科技”点击电动汽车广告的可能性较大, 在LR 模型中需要通过四维cross特征表达 l 原始高维稀疏category特征 + cross特征维度可达千亿规模 Ø 从高维稀疏category特征→稠密特征(图像特征)→序列特征(用户行为序列), 越来越需要模型对复杂特征的处理能力,自动挖掘学习特征,传统的线性模型如 LR、FM等越来越难以满足要求 l 线上计算资源有限,latency要求高,在线Infer必须在10ms内完成 l 要求模型具备复杂特征挖掘学习能力, 同时系统架构上要求能提供充分的算力支持
7. 2016年底,广告系统应用机器学习技术面临的挑战 l 离线指标与在线效果存在gap, 需通过在线实验确定效果 Ø 以CTR预估项目为例,不仅要求序准确,也要求值精准:CTR预估值会影响CPM 等收入相关指标,仅用离线AUC来衡量是不够的 l 在线与离线的数据分布存在巨大差异 Ø 比如离线训练的AUC指标衡量的是“不同用户对不同广告的点击率的序”,但是 在线预测的时候关注的是“同一用户对不同广告的点击率的序” l 线上效果受复杂排序算法的影响 Ø 通常排序公式还包含bid、转化率、广告质量因子等,还有规则性的策略 l 因为在线离线gap的存在,需要能快速上线实验迭代,因此要求模型的开发和发布的 成本要低
8. 生产环境持续运维的在线深度学习系统原生的技术债 l 训练和Serving阶段的数据不一致,导致在线、离线差异较大 Ø 监测和确保两者一致是一个贯穿数据生产、模型训练和Serving的系统工程 Ø 各个模块之间良好的接口与解耦合是确保架构稳定的基础,但往往缺乏重视 l 隐式的数据依赖 Ø 对数据的定义和理解在数据流的上下游可能并不一致 Ø 数据的分布随着时间推移或者系统规模的增长发生变更,分布和最初设计特征时 有较大变化,导致效果与预期不一致 l 多种配置在系统的各个阶段,导致无法看到全局效果,配置之间的耦合与依赖没有系 统性的检查和约束
9. 生产环境持续运维的在线深度学习系统原生的技术债 l 缺乏业务领域的抽象 Ø 基于python的API接口在应用于生产过程时,过于灵活,会对效果复现带来一定 的困难,也不利于成功的模型迅速推广复制 l 研究员与工程师的文化冲突 Ø 算法研究员专注于发现更有用的特征,构建更复杂的模型,提升效果 Ø 工程师专注于如何让系统更稳定,更高效,更可扩展 Ø 两种角色看问题的视角不一样,导致关注点不一样,一个好的系统抽象能推动两 者的合力,反之会降低团队效率
10. 腾讯广告利用多种创新技术组合,构建特征工程、训 练平台、Serving集群三段式技术架构、解决广告系 统应用深度学习技术难点 特征工程 训练平台 • 提升特征调研效率 • 研发领域建模API,屏 • 监测在线、离线特征 一致性 • 优化Infer性能,规避 蔽调参、部署上线等技 • 促进特征共享 • 优化特征存储 Serving集群 训练数据流 TFExample 术细节 • 研发模型模版 • 优化单机多卡与多机多 卡性能 算力瓶颈 自描述模型 文件 • 优化模型分发通道, 加快模型更新速度
11. 深度学习与广告系统结合,利用数据与算法提升收入 数据子系统 实时日志 关联 请求日志 曝光日志 特征生产 • 前文特征 • 广告特征 • 用户特征 特征补录 • 将新特征和 已有特征关 联起来 特征存储 • 离线存储 • 在线存储 接入与数据回流 集群 样本抽取 特征服务 点击日志 转化日志 模型训练 模型计算 模型文件 预测结果 模型服务 广告精排集群 广告主第一方数据 用户行为数据 用户画像数据 数据流 模型文件 广告粗排集群
12. 特征工程通过系统化建设,减少存储空间,提升计算 效率,加快特征调研,促进特征共享 l 特征工程系统架构 l 特征工程数据流
13. 特征工程平台系统架构图 Lookalike 自动定向 粗选点击 率预估 动态广告 HBase存储 前文统计特征 生产框架 CF2 HDFS Hive 商品点击 率预估 特征在线服务 DataHub 特征补录 任务调度 Hotwheels 粗选转化 率预估 Marvel存储 Dragon 列式存储 广告特征 生产框架 TerraStore HBase 潜客优选 模型应用 监控告警 Eagle Eye 特征工程平台 训练样本数据流 RocksFlow 数据校验 特征注册 Spark 负反馈 Marvel 数据发布 元数据管理 Dragon Pivot 基础设施
14. 特征工程数据流 特征生产 特征存储 特征计算 (调研环境) 特征计算 (生产环境) 前文统计特征生产 训练样本集生成 广告特征生产 数据 特征存储 用户语义特征生产 … 特征补录 预测样本生成
15. 特征工程通过统一的特征仓库和特征抽取,提高特征 复用率,减少存储空间,提升计算效率 提高数据与特征复用率 • 建设统一的特征仓库 • 使用自研的列式存储引擎Dragon,提升存储效率 • 建设特征抽取框架,基于有向无环图减少重复计算 曝光日志 特征抽取 曝光样本 减少存储空间,提升计算效率 • 存储空间每日从1526GB下降到69GB,减少95% • 全部特征补录时间从12分钟减少到5分钟,提升效 率60% • 特征调研整体流程时间从天级别减少到小时级 训练数据生成 CTR训练数据集 训练数据生成 点击日志 关联用户特征 点击日志 特征抽取 点击样本 训练数据生成 CVR训练数据集 转化日志 关联用户特征 第一方数据 转化日志 特征抽取 第一方数据 转化样本 训练数据生成 第一方数据 训练数据生成 DPA正例 DPA采样负例 关联用户特征 DPA采样负例 关联商品库 数据流与特征复用于生成多个训练集合 DPA采样负例 训练数据生成
16. 特征工程通过抽象的方式,减少特征调研过程中的重 复工作,提升特征调研效率,促进特征共享 l 模型研究员和特征工程解耦合,特征工程建设优化系统,提供简单灵活接口供模型研 究员扩展特征生产部分,使用方只需定制聚合函数 Ø 不需要关注特征存储细节,通过UI访问特征存储的元数据和进行数据质量校验 Ø 特征补录只需要提交补录配置文件 Ø 样本集生产只需要提交Pipeline配置文件,自定义处理逻辑扩展Node节点 l 特征共享 Ø 面向所有模型方提供特征统一存储、分析和离线在线计算能力 Ø 特征在不同模型的进行调研、生产使用的效果数据对各个模型应用透明
17. 特征工程自动监测离线/在线特征一致性 在线/离线不一致的原因 系统化建设避免在线/离线特征不一致 • 在线/离线数据不同源 • 在线与离线数据同源 • 在线/离线数据流处理逻辑不同 • 避免出现数据泄漏与数据穿越 • 在线/离线时间偏差,导致数据发送变化 • • 数据泄露:本来不应该出现在X里的、和目标Y有关的数 据,出现在了X中 数据穿越: 训练模型的中使用了一些特征,而这些特征 在实际用模型去做预测时所获取不到的“未来信息” • 实时监控特征一致性,偏差较大时人工介入处理 在线特征存储 特征注册 在线特征服务 在线预测样 本 特征列式 离线/在线特征一致 存储 性监测 离线特征存 储 离线特征服务 离线训练样本
18. 建设体系化的特征数据流系统,提升各模型的特征数 据线上与线下一致率 Process node Input node l 提升线上和线下一致性的方案 Output node 5 Ø 数据一致性:特征仓库统一存储 Ø 代码一致性:统一采用特征算子 1 2 2 3 特征抽取1 特征离 线存储 6 模型1离线训练 4 特征抽取2 特征 一致率(原) 一致率(新) FeatureName1 88.59% 98.34% FeatureName2 88.69% 98.35% FeatureName3 91.35% 99.66% FeatureName4 91.41% 99.86% FeatureName5 91.41% 99.86% FeatureName6 97.08% 99.86% FeatureName7 97.30% 99.90% 1 22 6 模型2离线训练 特征 仓库 特征抽取1 2 模型1 2 模型2 在线特 征存储 特征抽取2
19. 特征工程输出周期性训练数据流,训练平台加载数据 流与模型配置文件,调度集群进行端到端训练模型 特征工程 HDFS训练样本文件 AllInOne训练平台 训练平台调度训练任 基于Tensorflow底 配置文件 务 层API训练模型
20. 训练平台领域建模API将模型研究与特征工程,模型 训练,模型服务解耦合,提升模型研究的效率 如何提升众多推荐业务模型研究的效率 模型研究与特征工程、模型训练、模型Serving解耦合,各模块独立优化 模型调研配置化 特征工程基于特征算子进行样本抽取,提供统一的样本格式 模型训练基于模型配置进行训练,产生统一的模型格式
21. 训练平台系统架构 l 数据源 Model Train Ø 特征工程提供的样本数据 l 训练数据流 Learner Ø 支持本地和Hdfs两种读取模式 Ø BatchPipeline全量训练(Online/Local) Reader Ops Gradients Ø StreamingPipeline增量训练(Online) Ø 支持数据和模型自恢复 l Learner Ø Reader对数据实现ETL操作 Model Saver Batch/Streaming pipeline Ø Ops Ø 支持大多数op和optimizer Ø 支持tf原生op直接调用 Ø ModelSaver Ø Load Graph&Restore Variable Model Variable Ø Save&Export Model Model Profile l 模型推送 Ø Model包含signature供Serving使用 Ø SourceProfile记录训练详细信息 Signature Example Data
22. 训练平台对模型研究员屏蔽部署运维技术细节,提升 模型调研效率 提交/更新配置 研究人员 客户端 存储模型应用的配置 模型应用 ModelAppId 模型管理 订阅配置,部署任务 训练任务 模型训练 模型评估 模型服务 训练配置 训练状态 评估配置 评估状态 服务配置 服务状态 1.机器资源描述 2.训练样本路径 3.模型结构定义 4.训练方式 5.集群 1.模型路径 2.训练速度 1.机器资源描 述 2.模型路径 3.计算配置 4. 预测结果分 发配置 5.模型提交配 置 1.模型指标 2.预测结果存 储路径 1.加载版本数 2.模型上线阈 值 1.模型路由表 评估任务 模型服务
23. 训练平台领域建模API集成成熟的模型结构,促进模 型复用 Sigmoid cross entropy with logits OP l Ops:常见的建模结构、操作、 Dot OP 运算封装为更高层次的API l Ops以其它Ops/Feature Input作为输入,整个网络通过 Full Connection OP Full Connection OP Full Connection OP Full Connection OP Full Connection OP Full Connection OP Concat OP Concat OP Ops搭积木的方式拼装实现 l 模型文件自描述,离线训练与 在线Serving解耦合,从而实 现统一的在线Serving集群 Reshape OP Reshape OP Reduce Mean OP Reshape OP Reshape OP MaxPooling OP Text CNN OP Embedding OP Embedding OP Conv OP Feature Input(Category) Feature Input(category) Feature Input(Continous) App安装列表 … 年龄 广告图像 Embeddi ng OP Embeddi ng OP Feature Input(Text) … 广告标题
24. 训练平台领域建模API内置多阶Hash技术支持原始 特征Embedding 离散ID特征在Embedding前 多阶Hash技术 映射有两种方式 式限制了能增加的特征数 • 使用Hash函数映射,优点是扩展性强, 缺点是特征冲突会影响效果,但是 Hash冲突的概率远大于人的直觉(生 日问题),且Hash冲突对在线效果影 响大于离线效果影响。 • Embedding过程无状态化,无需引入 合 缺点是扩展性不足,词表需动态管理 征例如用户购买行为,构造词表的方 明显收益 • 使用多个种子进行Hash,将 Embedding之后的多个向量非线性组 • 构造词表,一一映射,优点是没有冲突, 与回收,词表变更时模型需要重新训 练,广告系统存在大量开放的稀疏特 多阶Hash带来工程与效果的 外部字典,训练平台与特征工程平台 • 实现GPU的Hash算法,整个过程作为 创新 神经网络的一部分,参与反向求导, 降低了冲突率和信息损失,降低空间 使用,充分利用显存和GPU的并行算 力 解耦合 提升 • 大幅度降低Hash冲突,提升效果,提 升稳定性 • Hash冲突对在线的效果影响大于对 离线的效果影响
25. 训练平台领域建模API基于多阶Hash技术研发 Embedding Op 端到端处理离散特征Embedding 深度学习模型的第一层是将特征映射到一个Embedding矩阵,广告或者推荐类业务中有大 量稀疏离散特征需要作Embedding处理 V0版 特征数据 训练样本 0.122 0.729 0.222 由特征工程的同学在生成训练数据时将特征Hash为int,在模型中使用 tf.embedding_lookup来做映射,这种方式Hash的桶发生变化时需要重新生成训练数据, app id 广告id 0.123 … title term 0.436 0.924 0.566 … 0.144 0.211 0.516 0.083 … 0.753 N维向量空间 让特征工程与Training有比较强的耦合,而Hash的桶数是Training的超参数之一 使用 tf.string_to_hash_bucket + tf.embedding_lookup 也可以完成Embedding,但是 tf.string_to_hash_bucket只有CPU版本,性能不够好,且占用宝贵的CPU资源 Hash映射 Embedding Lookup 创新 … 原始特征 自研Gather Op实现了一个Tensorflow的OP,在一个OP内完成原始特征到N维向量的映 射, 同时解决一致性和性能问题,且实现CPU和GPU版本,充分利用GPU算力,减少对CPU 的占用,GPU版本让我们可以将整个训练图放进GPU,Tensorflow原生的 embedding_lookup无GPU版本,整个训练图放进GPU在单机多卡的模式下,可以减少 CPU上的锁冲突,提升训练速度 Hash映射 Embedding Lookup
26. 训练平台领域建模Embedding Op优化数据访问 l 优化前:对主机内存锁页,设备在计算时通过总线访问主机内存数据 l 优化后:计算前把数据统一从主机内存拷贝至设备内存,计算时直接从设备内存访问 Host Device offset 1 offset 2 offset 3 … offset N len 1 len 2 len 3 … len N str 1 str 2 str 3 … str N offset buffer pointer cudaGetDevicePtr len buffer pointer str buffer pointer Host Device offset 1 offset 2 offset 3 … offset N len 1 len 2 len 3 … str 1 str 2 str 3 … cudaMemCopy offset 1 offset 2 offset 3 … offset N len N len 1 len 2 len 3 … len N str N str 1 str 2 str 3 … str N
27. 训练平台领域建模Embedding Op优化跨设备数据 传输 l 在特征工程的预处理中,对于字符串类型特征,会在特征前拼接上seed串再进行 哈希 l 对于自研的Embedding Op ,在批量计算时,同一类特征的seed串相同,因此 可以把seed串拷贝至设备内存后再拼接计算,减少跨设备的数据传输量 seed seed + example 1: age example 2: age seed + example 3: age seed + example N: age … … 以batch size=N批量 计算 + 0.642 0.32 0.14 … 0.34 0.78 0.27 0.56 … 0.69 0.13 0.07 0.43 … 0.58 0.23 0.16 0.39 … 0.06
28. 训练平台领域建模Embedding Op减少重复计算 l 在批量计算变长特征时,会为特征长度较短的样本填充默认值补齐 l 默认值到N维向量空间的映射关系是固定的,只需计算1次,因此可以减少重 复计算 Dense Matrix Sparse Matrix feature 11 feature 12 feature 32 feature 41 feature 42 feature 33 feature 12 feature 13 default value feature 21 default value default value default value feature 31 feature 32 feature 33 feature 34 feature 41 feature 42 default value default value SparseToDense feature 13 feature 21 feature 31 feature 11 feature 34
29. 训练平台领域建模Embedding Op 性能提升6倍 l 对自研Embedding单个算子的性能进行测试,GPU版本性能在四种优化都 完成后,处理性能提升了6倍
30. 训练平台领域建模Embedding Op(CPU & GPU实现)
31. 领域建模API Collector OP l 背景: 在线流式训练, 需要dump出模型中已经出现的id信息,对未出现id信息,预测取缺省值 l 特征hash取模查embedding有一定碰撞概率,部分id类特征如广告id空间很大,更新快,固定size embedding随时间累积冲突概率高,Hash取模embedding碰撞可能使个别广告ctr高估 or 低估,整体 影响不大但会造成bad case l 多桶hash取模方案可以降低冲突概率,但hash之后失去了原有的id信息 l Collector OP方案:内置DenseHashTable,最多存储N个 key – value,支持快速判断某个key是否在 表内内置LRU Cache,记录最近见过的N个key当数据表满了后,如果需要插入新的key,会根据LRU Cache删除最老的key回收空间 DenseHashTable Doubly-linked List Key bucket tensor key 3 key 1 key 2 value bucket tensor value3 value 1 value 2 node array tensor link array tensor node 1 node2 node3 prev1 next1 prev2 next2 prev3 next3 dummy tail head
32. TEXTCNN OP l TextCNN 是利用卷积神经网络对文本 进行处理的算法 l 在商业内容推荐业务中,TextCNN相 对其他方式AUC提升0.3% TextCNN 示意图 op { name: "title_term_textcnn" text_cnn { operand: "title_term_embedding" conv_max_pool { conv2d { filters: 64 # 多个少卷积核 kernel_size:'>size: 3 # 3*28的卷积核 kernel_size:'>size: 28 strides:'>strides: 1 # 步长 strides:'>strides: 1 padding: “valid“ # padding方式 activation: “tf.nn.relu” # 激活函数 use_bias: true } } } ……. } TextCNN 配置方式
33. 领域建模训练平台提供多维度模型评估基础能力 l 分维度模型评估 Ø 支持auc在内的多种评估指标和统 计指标 Ø 支持配置化过滤干扰结果 评估指标 统计指标 AUC ACCURACY PRECISION RECALL F1 LOGLOSS BIAS GAUC AVERAGE_PRECISION SAMPLE_COUNT POSITIVE_SAMPLE_COUNT NEGATIVE_SAMPLE_COUNT LABEL_MEAN SCORE_MEAN 评估样本 模型评估 模型 HDFS
34. 训练平台管理多个模型应用,调度CPU/GPU算力资 源完成模型训练,周期性输出模型文件 训练样本 系统的输入 模型应用的配置 训练样本 机器预算 系统功能 部署训练任务,生成模型 评估模型指标 离线批量计算模型预测数据 预测请求 模型应用配置 样本配置 模型训练 模型配置 服务配置 模型评估 模型服务 模型管理 计算配置 批量模型 预测结果 模型 资源配置 模型 模型数据流 实时预测模型服务 模型维度的监控 自动部署 系统能力 集群 接口封装 模型指标 自研算子 模型监控 优化训练速度 提高服务性能 无缝对接上下游 沙箱集群 benchmark集群 正式集群 系统能力 自研算子 接口封装 性能优化 模型托管
35. 训练平台统一管理CPU/GPU/VGPU,优化算力调度 • 机器预算:按照业务申请机器预算, 预算到位后, 由模型系统统一初始化,加入模型训练集群 • 资源配额: 业务的资源配额和实际加入集群资源数保持一致。按照配额优先来保证业务能优先使用资源 • 利用率: 模型系统跟进集群利用率问题,包含GPU利用率和CPU利用率 • 集群划分:集群机器分为 formal、sand、benchmark 环境 沙箱集群 业务 业务配额 机型:VGPU (Nabulet) 1.最大可以使用的资源数。 业务名 2.GPU按卡为单位 称 3.配额优先原则 机器预算 模型 管理 资源 ip 资源 明细 机型 CPU/ GPU/ 磁盘 docker /G7/B7 实际使 0/CG1 用情况 0 资源占 用 上报 资源 使用 状况 机型:GPU (Nabulet) 机型:CPU (Nabulet) 机型:GPU (Nabulet) 机型:CPU (Nabulet) 机型:GPU (Nabulet) 机型:CPU (Nabulet) benchmark集群 机型:VGPU (Nabulet) 正式集群 机型:VGPU (Nabulet)
36. 训练平台通过AllInOne配置文件,将各阶段配置集 中在一起,形成整体的视图 l 模型结构配置,例如双塔模型 l Dataset的hdfs文件路径 l 训练所需的机器资源,CPU/GPU l 优化器类型 l Batch size等超参数
37. 训练平台领域建模API模型定义文件 input { name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name: "title_term" type: BYTES is_var_len: true } op { name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name: "title_term_embedding" embedding { operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand: "title_term" vocabulary_size:'>size:'>size:'>size:'>size:'>size:'>size:'>size:'>size:'>size:'>size:'>size:'>size:'>size:'>size:'>size: 50000 embedding_size:'>size:'>size:'>size:'>size:'>size:'>size:'>size:'>size:'>size:'>size:'>size:'>size:'>size:'>size:'>size: 28 seed: "0" } } op { name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name: "title_term_textcnn" text_cnn { operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand: "title_term_embedding" conv_max_pool { conv2d { filters: 64 kernel_size:'>size:'>size:'>size:'>size:'>size:'>size:'>size:'>size:'>size:'>size:'>size:'>size:'>size:'>size:'>size: "3 x 28" strides: "1, 1" padding: "valid" activation:'>activation:'>activation:'>activation: "relu" use_bias: true … op { name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name: "ad_input_layer" concat { operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand: "title_term_textcnn” operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand: "ad_id_reshape" … axis: 1 } } op { name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name: "ad_input_layer_layer1" full_connection { operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand: "ad_input_layer" hidden_unit_count:'>count: 512 activation:'>activation:'>activation:'>activation: "relu" } } op { name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name: "ad_input_layer2" full_connection { operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand: "ad_input_layer1" hidden_unit_count:'>count: 256 activation:'>activation:'>activation:'>activation: "relu" } } … op { name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name: "dot_product" multiply { operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand: "user_input_layer2 " operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand:'>operand: "ad_input_layer2" } } op { name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name: "sigmoid_cross_entroy" sigmoid_cross_entropy_with_logits { labels: "ad_click" logits: "dot_product" } } dataset_option { num_epochs: 1 batch_size:'>size:'>size:'>size:'>size:'>size:'>size:'>size:'>size:'>size:'>size:'>size:'>size:'>size:'>size:'>size: 128 shuffle_buffer_size:'>size:'>size:'>size:'>size:'>size:'>size:'>size:'>size:'>size:'>size:'>size:'>size:'>size:'>size:'>size: 1000 … } hyper_parameter { optimizer { name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name: "adam" args { name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name:'>name: "learning_rate" float_val: 0.0001 …
38. 切换到领域建模训练平台的优势 l 降低了模型的开发成本 Ø 基于Tensorflow实现,可以基于Tensorflow快速实现模型方案,leverage工业界, 学术界最新的研究成果 Ø 模型开发工作进一步被简化为配置,大部分模型可以通过配置化方法快速生成 l 加快了模型的上线速度 Ø 模型offline->online 完全透传,不需要对线上代码二次开发 Ø 新模型的上线周期从一个月缩短到一周 l 工程和算法工作解耦 Ø 算法工程师focus模型的优化及业务效果分析,后台工程师focus架构,性能优化等
39. 训练平台周期性输出模型文件,模型数据流系统将模 型推送到在线广告系统,模型Serving集群加载模型 文件,提供模型服务 领域建模 训练平台 训练生成 自描述模型文 件 模型Serving集群 模型数据流 广告检索集群
40. 通用特征服务和模型服务专注高性能,可扩展与可运 维性建设 l 实时性 :用户行为特征和上下文特征具有实时性,模型更新频率高 l 模型多:每个推荐业务有自己独立的模型,同一业务在不同流量下有不 同模型 l 可运维性:搭建模型服务工作类似,同一请求需要计算多个模型,模型 之间特征部分重合
41. 模型服务基于Tensorflow Serving改造并持续优化, 提升Serving性能 l 根据广告业务需求进行改造 Ø 替换grpc为部门自研rpc框架 Ø 支持自研的tensorflow算子 l 性能优化 Ø 持续升级tensorflow版本 Ø 优化CPU优化编译选项 Ø 调整网络线程数量
42. 双塔模型充分利用离线算力,减少在线Infer计算量, 优化Infer延迟 Output l全连接模型需要实时计算全部网络,latency比较高,无法上线,对模型 进行简化和删减之后虽然能上线,但是限制了模型的深度与特征的规 Full Connect 模,制约架构的长远发展。 Full Connect l双塔模型可以突破规避在线算力不足的问题 Ø 离线批量计算广告子网络 Ad feature User feature 全连接模型 Ø 在线实时计算用户子网络,并计算用户子网络与广告子网络的点积 Ø 利用近邻检索可以进一步加速点积计算 Context feature Output ad embedding l双塔模型效果和全连接模型效果持平 隐层数 全连接模型 双塔模型 1 0.724 0.723 2 0.725 0.725 3 0.728 0.726 user embedding Full Connect Full Connect Full Connect Full Connect Ad feature 双塔模型 User/Context feature
43. 建设高效P2P文件分发网络,分钟级将双塔模型中广 告子网络Embedding数据分发到广告检索集群 线上预估 模型预测 技术挑战 技术方案 • 十分钟级分发大量数据至数 • 同IDC之内采用自研 千台机器 在线特征 P2P传输,节省CPU与 检索模块 带宽资源,同时提升速 • 直接影响收入,要求具备高 广告列表 度与高可用性 可用性 广告请求 • 跨IDC采用双通道热备, ad1 ad2 … user embed adn 增加容错能力 点击/曝光回流 模型数据 litectr litectr_feeds_exp litectr_mobile_qq litectr_mobile_qq_exp litectr_tencent_news litectr_tencent_video litectr_tencent_video_cpp litectr_wechat_moments moments_litecvr moments_litecvr_cpp 数据大小(MB) 111 112 65 65 30 30 20 90 185 400 更新 20min 20min 20min 20min 20min 20min 15min 20min 10min 30min 离线训练 ad embeds combine 统一数据分发通道 实时logjoin 特征工程 离线模型训练模型 批量广告子网络预测数据 广告精排
44. 研发近邻检索算法,降低在线计算量 l 最近邻域检索问题 Ø 在模长为1的数据集,用非O(n)算法检索最近邻域 Ø 根据论文实现HNSW算法,检索效果与业界性能持平 Ø 与原来检索算法集成,支持求解逻辑表达式检索与最近邻域的交集问题 最大遍历的向量个数 6000 10000 6000 10000 启发式队列长度 600 600 800 800 平均时延(ms) 0.68672 0.68682 0.89138 0.89388 30万100维数据规模的最近邻域检索性能,线性遍历查找需要29秒 l 最大内积检索问题 Ø 模长不为1 Ø 转化为最近邻域检索问题 最大时延(ms) 1.389 1.667 1.573 2.003 召回率 0.847548 0.8476 0.879066 0.879565
45. 通过特征工程、训练平台、Serving集群三段式技术 架构、各模块之间充分解耦合,较好的解决了深度学 习系统原生的技术债 数据(特征 工程) 模型(算法 工程师) 深度学习驱动广 告收入增长 用户 (Serving 集群) 算力(训练 平台)
46. 各个推荐模型效果汇总 l 首次建立服务于多个模型应用的机器学习平台 Ø 广告推荐:CTR预估、CVR预估、动态广 告、自动定向、LookAlike Ø 广告主工具:潜客优选 Ø 用户体验:负反馈模型 l 提升模型调研的效率,模型上线周期从月缩短 到周 l 助力广告业务发展,提升业务收入 业务名称 当前效果(大盘日消费提升) 自动定向 约1% LookAlike 约1% CTR预估 约5% CVR预估 约2% 动态广告 约1% 累计 约10%
47. 广告粗选点击率预估 l 粗选CTR预估背景 l 接入收益 Ø 广告粗选点击率预估模块 Ø Feeds日消耗增量:0.8% Ø 影响CPC计费广告 Ø 腾讯新闻日消耗增量:1% Ø CPC广告的ecpm = 出价(元/千 Ø 腾讯视频日消耗增量:1% 次点击) * CTR Ø 朋友圈日消耗增量:2.2%
48. 广告粗选转化率预估 l 粗选转化率预估背景 Ø 广告粗选转化率预估模块 Ø 影响oCPA和oCPM计费广告 Ø oCPA广告的ecpm = 出价(元/转化) * CVR * CTR l 接入收益 Ø 使用的双塔模型,持续引入特征提升效果 Ø 提升收入1%以上 模型结构 特征配置 AUC Logloss LR(Online) 10特征+组合 0.7669 0.1271 双塔NN 22特征 0.8004 0.1219
49. DPA动态商品广告 l DPA动态商品广告背景 Ø 千人千面, 精细化用户兴趣,创意随兴趣变化而 实时变化 Ø 创意自动生成, 无需广告主制作创意 l 接入收益 Ø 商品推荐模型,日均消耗提升相对DPA baseline提升10% Ø 突破了DPA投放的UV瓶颈,在广告主活动大促 期间带动更多预算,之前存在预算消耗不尽的问 题 Ø 商品pCTR精排,feeds 20%实验上DPA ctr提升 6%, 消耗提升38%;移动内部站点5%实验上 DPA ctr提升1.9%,消耗提升2.3%
50. LookAlike l LookAlike 背景 Ø 基于广告主的一方数据(广告主已有的高质量用户 集),Lookalike 帮助广告主拓展出相似用户,考虑 MORE 到潜在用户和现有用户是相似的,这些相似的潜在用 户被用于广告定向。 提供或抓取现有顾客集 l 接入收益 Ø 离线模型计算转为实时模型计算 一方数据:已消费过的会员 投放数据:已转化过的用户 Ø 提升研发效率 优化项 迁移前 迁移后 分站点单独建模,优化流量变现效率 • 每个模型需要额外申请机器,成本高 • 线上模块需项目组单独运维,人力成本高 • 新增站点模型无需额外部署,修改模型即可上线 引入更多特征(罗卡特征、线上实时特征等), • 引入新特征需要修改大量代码 提升拓展效果 • 特征模块需项目组单独运维,人力成本高 提升模型快速迭代上线能力,节省人力和时间 成本 • 上线新模型需要一条龙修改线下、线上代码 • 模型实验需要额外申请机器,机器利用率低 • 开发新模型训练系统成本高、难度大 • 通过修改配置即可引入新特征 • 简化离线模型搭建流程,线上无需修改代码 • 统一模型训练系统,模型方仅需关注逻辑 找到相似人群
51. 自动定向 l 项目背景: Ø 自动地、精准地为广告主定向目标人群。平台将突破广告主 原有定向,为广告主拿到更多更好的曝光 l 特点: Ø 不需要广告主提供种子包或选择定向,平台直接使用广告历 史数据,为广告优化目标建模。 Ø 基于广告维度建模,可以同时使用用户、流量、广告等多个 维度的数据特征。 l 目前效果 Ø 全流量邀请广告主内测中,大盘新增消耗超5% l 规划 Ø 推广引入更多的广告;特征和模型优化,同时尝试不同优化 目标,优化扩量效果
52. 项目成功的经验总结 l 特征工程、训练平台、Serving集群三段式设计,各个模块之间协议自描述 l 多阶Hash技术优化稀疏特征Embedding,充分利用GPU的并行算力,同时帮助特征 工程与训练平台、Serving集群解耦合 l 通过双塔模型,利用离线训练集群承担模型中计算密集部分,规避在线Infer的算力 与Latency瓶颈 l 通过多种技术组合,深度模型相比LR、FM在拟合能力上有质的飞跃,使所有接入系 统的产品都收获了指标提升
53. 系统2018年全量,是腾讯广告利用数据与模型提升 广告收入的核心能力 日请求数几百亿,峰值百万 l 日提交文件数:1000+ QPS,99%时延约 2ms l 日分发文件数:700w+ l 在线模型数:40+ l 分发至下游检索系统数千台机 l 日上线版本数:400+ l 器,深度学习系统几百台机器
54. 项目的长期意义 l 腾讯广告系统与深度学习强强结合 Ø 为应用深度学习技术提升广告收入、广告主体验与用户体验留出了长远的技术架构 发展空间 l 释放数据潜力,加快机器学习模型迭代速度,提升系统稳定性 l 广告算法团队融入全球Tensorflow生态,Leverage最新的学术与工程进展
55. 项目的下一步规划 l 持续优化性能,单机多卡训练性能提升10倍,分布式训练再提升10倍 l 探索更强的算力的优化实现方式 l 探索新的模型结构在广告业务的应用 l 支持AutoML,提升平台各个模型的调参效率