KubeCon + CloudNativeCon North America 2018

Automating Research Workflows at BlackRock Matthew Magaldi & Vaibhav Page, BlackRock

1. Automating Research Workflows Matt Magaldi & Vaibhav Page December 12, 2018 ©2017 BlackRock, Inc. All Rights Reserved. BLACKROCK, BLACKROCK SOLUTIONS and ALADDIN are registered trademark of BlackRock, Inc. All other trademarks are the property of their respective owners.
2. Background A brief history of data science at BlackRock
3. Why? • Analytical research and development environments available to researchers and scientists across the firm • Best in class free and open source software tools • On demand resourcing for users • User configurability of research tools and environment • User session resource isolation and security RESTRICTED: NOT FOR FURTHER DISTRIBUTION 3
4. How? RESTRICTED: NOT FOR FURTHER DISTRIBUTION 4
5. What? • Environmental, Social and Governance alert mechanism for portfolio managers based on positioning of funds • Compliance tool to evaluate orders against equity thresholds • Outlier check to improve quality check of daily portfolio analytics • Pricing models for variety of complex financial instruments • Recommender tool based on ticket similarity model • Dashboard to track retail fund activities RESTRICTED: NOT FOR FURTHER DISTRIBUTION 5
6. The New Problem Automate scheduling, manage exceptions, and provide transparency for complex Kubernetes daily batch processes. RESTRICTED: NOT FOR FURTHER DISTRIBUTION 6
7. Workflows Argo • Intuit + KubeFlow • Workflow as CRD Brigade • Microsoft • Workflows as JS • Scheduled on Kubernetes Pods Airflow • Apache (Incubator) • Workflows as Python • Pluggable Operators RESTRICTED: NOT FOR FURTHER DISTRIBUTION 7
8. Simple Idealized Task 1 Task 2 RESTRICTED: NOT FOR FURTHER DISTRIBUTION Task 3 8
9. Complex Reality RESTRICTED: NOT FOR FURTHER DISTRIBUTION 9
10. Complex Reality: Breaking it apart RESTRICTED: NOT FOR FURTHER DISTRIBUTION 10
11. Complex Reality: Simplified RESTRICTED: NOT FOR FURTHER DISTRIBUTION 11
12. Automation • Gateways - produce events internally or process events that originate external to the gateways. • Sensors - define a set of event dependencies and trigger actions. RESTRICTED: NOT FOR FURTHER DISTRIBUTION 12
13. Demo – Image Processing RESTRICTED: NOT FOR FURTHER DISTRIBUTION 13
14. Features • Kubernetes native • Runtime agnostic • Lightweight • Easily extensible • Configurable at runtime RESTRICTED: NOT FOR FURTHER DISTRIBUTION 14
15. Data Sourcing • Schedule workflows to acquire, process, validate and deliver vendor data feeds to users in a consistent & systematic approach. RESTRICTED: NOT FOR FURTHER DISTRIBUTION 15
16. Data Science Fixed Income Notebooks • Help BlackRock investment teams deliver market leading performance (alpha) to their clients • Used for risk scaling, time-series transformation, running optimizations for financial models • Includes equity models, commodity models which run on certain frequency like daily, monthly etc. • Schedule varies from a normal cron job to a complex calendar logic. RESTRICTED: NOT FOR FURTHER DISTRIBUTION 16
17. Takeaways • Automation is a growing area of interest • Staying relevant in the open source community • Argo Events project scope is expanding • Standardizing eventing platform using CloudEvents specifications. RESTRICTED: NOT FOR FURTHER DISTRIBUTION 17
18. More Resources • Argo Project Website - https://argoproj.github.io/ • Argo Events Github - https://github.com/argoproj/argo-events • BlackRock Engineering Blog - https://medium.com/blackrock-engineering/ • BlackRock Homepage - https://www.blackrock.com/ RESTRICTED: NOT FOR FURTHER DISTRIBUTION 18
19. This material is not to be construed as an offer or solicitation to buy or sell securities, or to engage in any trading or investment strategy. The views contained in this material are those of BlackRock as of the date of this report, may change as subsequent conditions vary, and are based on information obtained by BlackRock from sources that are believed to be generally reliable. BlackRock has no obligation to update this material or to notify you of changes. The information in this report is not all-inclusive and is not guaranteed as to accuracy. Model-based methodologies and calculations are based on assumptions and historical correlations between data and other factors, and are not assured to predict future results. BlackRock is not responsible for typographical or clerical errors in this material or in the dissemination of its contents. Reliance upon this material is at the sole discretion of the reader. The analysis provided herein is for informational purposes only and estimated in nature. This material is not intended for distribution to third parties without the prior written permission of BlackRock. The recipient of this report undertakes sole responsibility and liability for investment or other decisions. For additional information on any of the descriptions contained in this document, please contact your Aladdin Relationship Management representative. BlackRock may modify or discontinue any functionality or service component described herein at any time without prior advance notice to you.