Google高级软件工程师刘靖-大数据驱动下的谷歌云平台运维

马华楚

2017/11/14 发布于 技术 分类

回顾devops过去十年在软件行业的发展,目前的状况,以及新的技术驱动下,尤其是大数据和云平台的不断变化,给devops工作带来的新的变化和挑战。大数据处理和机器学习技术的日益成熟使得越来越多的企业用户开始实施大数据战略,云平台的服务产品不断升级和更新,给devops的工作带来很多新的变化。

文字内容
1. Devops Practices on google cloud Jeff Liu Google
2. Jeff Liu ● SWE / Devops Google ● SRE - Twitter ● Devops - Splunk ● SRE - Dell
3. Devops Evolution
4. Devops Evolution
5. Devops Evolution ● Cloud/Platform ● Microservices ● Big data ● Machine Learning
6. Cloud and Platforms
7. Microservices
8. Big Data and Machine Learning
9. Devops DevOps (a clipped compound of "development" and "operations") is a software development process that emphasizes communication and collaboration between product management, software development, and operations professionals. --- Wiki “DevOps is the combination of cultural philosophies, practices, and tools that increases an organization’s ability to deliver applications and services at high velocity.” --- AWS “DevOps is where any application starts.” --- Google
11. Dataflow evolution MapReduce Colossus Spanner BigTable PubSub Megastore Millwheel Dremel Google Cloud Dataflow Flume Apache Beam
13. Fully-managed Deploy Tear Down
14. Dynamic Work Rebalancing 100 mins. 65 mins. vs.
15. Unified programming model
16. Dataflow - Integrated Capture Process Google App Engine Cloud Logs Google Analytics Premium Cloud Pub/Sub Cloud Dataproc Batch Stream Cloud Dataflow Real time analytics and Alerts Cloud Monitoring Cloud DataStore Cloud Bigtable Store BigQuery Storage (tables) Cloud Bigtable (NoSQL) Cloud Storage (files) Analyze BigQuery Analytics (SQL) Cloud Dataflow Cloud Dataproc 16
17. Dataflow public static void main(String[] args) throws IOException { ... Pipeline pipeline = Pipeline.create(options); pipeline .apply(PubsubIO.Read.topic(options.getPubsubTopic())) .apply(ParDo.of(new ExtractWords())) .apply(ParDo.of(new Uppercase())) .apply(ParDo.of(new StringToRowConverter())) .apply(BigQueryIO.Write.to(tableSpec) .withSchema(StringToRowConverter.getSchema())); PipelineResult result = pipeline.run();
18. Dataflow java -cp ./ -Dexec.mainClass=com.example.WordCount \ -Dexec.args="--project= \ --stagingLocation=gs://'>gs:///staging/ \ --output=gs://'>gs:///output \ --runner=BlockingDataflowPipelineRunner"
19. Dataflow static class ExtractWords extends DoFn { @Override public void processElement(ProcessContext c) { String[] words = c.element().split("[^a-zA-Z']+"); for (String word : words) { if (!word.isEmpty()) { c.output(word); } } } }
20. Stackdriver import com.stackdriver.api.custommetrics.CustomMetricsPoster Sending a custom metric value, not tied to an instance, collected now CustomMetricsPoster(YOUR_API_KEY).sendMetricDataPoint(YOUR_METRIC_NAME, POINT_VALUE); Sending multiple metrics, all collected now CustomMetricsPoster(YOUR_API_KEY).sendMetricDataPoint(YOUR_METRIC_NAME, POINT_VALUE).sendMetricDataPoint(SECOND_METRIC_NAME, SECOND POINT VALUE);
21. Dataflow static class ExtractWords extends DoFn { @Override public void processElement(ProcessContext c) { String[] words = c.element().split("[^a-zA-Z']+"); for (String word : words) { if (!word.isEmpty()) { c.output(word); CustomMetricsPoster(YOUR_API_KEY).sendMetricDataPoint(“Counting Word”, 1); } } } }
22. Dataflow static class ExtractWords extends DoFn { @Override public void processElement(ProcessContext c) { String[] words = c.element().split("[^a-zA-Z']+"); for (String word : words) { if (!word.isEmpty()) { c.output(word); ….. } } } } Monitoring, Capacity Management, Rate-limiting, security enhancement and etc.
23. Operations as-a-Service Data Pipeline Deployment SDK MicroService Capacity Analytics MicroService SDK RateLimit MicroService SDK SDK Monitoring MicroService Traffic Management MicroService Alerting MicroService
24. Operations as-a-Service
26. “We choose because they are hard” “We choose to go to the moon in this decade and do the other things, not because they are easy, but because they are hard, because that goal will serve to organize and measure the best of our energies and skills, because that challenge is one that we are willing to accept, one we are unwilling to postpone, and one which we intend to win, and the others, too.” ----- J.F.K, September 12, 1962
27. 会议 培训 咨询 • 8月18日 DevOpsDays 上海 • 全年 DevOps China 巡回沙龙 • 11月17日 DevOps金融上海 DevOpsDays 2017·上海站 • EXIN DevOps Master 认证培训 • DevOps 企业内训 • DevOps 公开课 • 互联网运维培训 • 企业DevOps 实践咨询 • 企业运维咨询 商务经理:刘静女士 电话 / 微信:13021082989 邮箱:liujing@greatops.com