RedHat 朱聿明 Building & Releasing Pipeline


2019/07/08 发布于 编程 分类


1. Building & Releasing Pipeline How we are shipping releases in Red Hat 朱聿明 Red Hat Senior Software Engineer
3. We are an open-source Company ● Upstream Community ● Source code from Internet ● Build internally ● Well-tested by QEs ● Distribution as multiple formats
4. We are DevOps ● ● Goal: ○ build and maintain the internal tools that keep Red Hat Engineering running ○ *aid shipping customer-facing products Org: ○ ○ ○ ○ ○ Release Engineering (RCM) System Operations (SysOps) Labs and Capital Management (Labs) Development Automation/Quality Engineering (AQE)
5. We are DevOps ● ● We maintain a bundle of tools for Engineering ○ Most of them focus on product building and release ○ + some tracking tools like Bugzilla, JIRA, etc ○ Many product team setup their tools on the infrastructure we provide ○ Our tools are important parts of product CI/CD workflow We heavily use Red Hat products ○ As Customer ZERO ○ Give feedbacks, report bugs/RFE, fix issues to product line ○ Share best practices with customers
6. The Product Pipeline ● ● The point of Red Hat's Product Pipeline is to reliably and efficiently produce Red Hat Enterprise Products It includes ○ A set of tools maintained by DevOps team ○ Externally-facing services that presents software content ○ ○ Extra tools maintained by various team in Engineering
7. Product Features Defined We know what changes in the software and why Reproducible We can rebuild and content from any time, reliably Auditable We know who built and shipped each update and release Deliverable We know we can present and provide the content consistently
8. Infrastructure Features ● ● Safe Security Policies Opportunities for automation all over the place
9. The Product Pipeline
10. Origin ● SCMs of upstream community ○ GitHub ○ Fedora Pagure ○ ○ SourceForge ○ ...
11. Source Control ● The core of the Source Control Management for build/release is dist-git ○ It is open-source and also used by Fedora and CentOS ○ It is designed to hold content of source rpms specfile + source code archive ● ● The CLI tool is rhpkg ○ rpkg/fedpkg/centpkg for Fedora(CentOS) ○ It provides a set of commands for git management ○ It also provides various build commands against several different build systems Fedora dist-git ○
12. Source Control
13. Build System ● Koji/Brew ○ ○ ● OSBS ○ ○ ● A collection of tools, workflows and integration points that build and release layered container images(currently, Docker images for OpenShift) osbs-client, koji-containerbuild plugin, atomic-reactor, ODCS, OpenShift Origin v3 runtime… MBS ○ ● An RPM-based build system A build metadata management system A build system for Modularity PNC ○ ○ A system for managing, executing, and tracking cross-platform builds A next generation build system for JBoss
14. Koji ● ● ● Fedora Koji - Project - Key Features ○ New buildroot for each build ○ Robust XML-RPC APIs for easy integration with other tools ○ Web interface with SSL and Kerberos authentication ○ Thin, portable command line client ○ Users can create local buildroots ○ Buildroot contents are tracked in the database ○ Versioned data
15. Koji ● Components ○ koji-hub A XML-RPC API provider for various clients including other components ○ Kojid The build daemon that runs on each of the build machines ○ Koji-web A web interface to expose information of koji and provide a set of operations ○ Koji CLI A CLI tool which allows the user to query most of koji data as well as perform actions ○ Kojira A daemon that keeps repo data of buildroot updated
16. Terminology ● ● ● Buildroot ○ A build toolset to build packages from source code ○ An isolated build environment ○ A set of dependencies required by build Package ○ (Legacy) The name of a source rpm ○ The “N”ame of builds Build ○ (Legacy) A particular build of a package - N-V-R - name-version-release ○ This refers to the entire build: all arches and subpackages. For example: kernel-2.6.9-34.EL, glibc-2.3.4-2.19
17. Package Organization ● Tag ○ ○ ○ ○ ● Target ○ ● Target(Build Target) is a pair of a source tag and a destination tag Source tag (build tag) ○ ○ ● The catalogue of builds(inheritable) Tag and tag’s inheritance is configurable Each tag has its own whitelist of valid packages (inheritable) Package ownership can be set per-tag (inheritable) It’s used to generate the repository in the buildroot It contains a list of (latest) builds as build requirements Destination tag ○ The tag where build result is tagged into as A build
18. Building ● Internal ○ ○ ○ ○ ● RPM ■ RPM is built by mock on kojid Images ■ (Legacy) LiveCD, LiveMedia, Appliance images are supported ■ koji uses ImageFactory and Oz to start a VM guest and perform an automated Anaconda installation to build various kinds of disk/container images(qcow2, vmdk, ova, Hyper-V, raw, docker, etc) Maven Win Content Generator (External) ○ ○ ○ ○ A Koji Content Generator is an external service that generates content (jars, zips, tarballs, .npm, .wheel, .gem, etc) which is then passed to Koji for management and delivery to other processes in the release workflow. In koji, CG works as an interface. The 3rd party build system can import a build with its metadata via this API. CG metadata should contain source, buildroot, archives, authoring information OSBS, MBS, etc act as CGs
19. Plugin-able ● Task ○ ○ ● XML-RPC API ○ ● New API could be written as a hub plugin Callbacks ○ ○ ● Task aims to do a certain job(mostly it’s a build job) on kojid TaskHandler could be extended as a kojid plugin easily There are a sort of events predefined Developers are allowed to write hub/kojid plugins to observe those events CLI command ○ End user can define his/her own command as a plugin
20. Misc ● Volume ○ ○ ● Policy ○ ○ ● Koji stores build content on multiple volumes which could be specified per-build by policies NFS is used to share files with other systems Defining a policy on the hub allows you fine control over certain activities in the system At present, policy allows you to control: ■ tag/untag/move operations ■ allowing builds from srpm ■ allowing builds from expired repos ■ managing the package list for a tag ■ managing which channel a task goes to ■ … Channel ○ ○ Builders are categorized into channels Build tasks are assigned to a certain channel then one builder in that channel picks up this task and execute
21. Build System
22. What’s next? ● ● ● ● ● Common build interfaces ○ RPM and new Builder management ○ Builders are managed by Ansible ○ Metal machines, VMs, and Containers? XML-RPC to REST More build type invoked ...
23. OSBS(OpenShift Build Service) ● ● ● OSBS is a collection of tools, workflows and integration points that build and release layered container images(Docker images for OpenShift). Tools: ○ koji-containerbuild plugin ○ osbs-client ○ Atomic-reactor ○ OpenShift Origin ○ Pulp registry ○ ... Integration ○ Dist-git ○ Koji ○ Pub ○ ...
24. OSBS
25. MBS (Module Build Service) ● ● ● ● ● ● ● MBS is the Module Build Service for Modularity Provides a Restful interface for module building and data querying Verifies modulemd and spec file is available and correct Prepares the build environment in the supported build systems ○ Koji ○ Mock Schedules and builds the module components and tracks the build state Emits bus messages about all state changes so that other infrastructure services can pick up the work Fedora’s service:
26. MBS ● But, what’s Modularity? ○ ○ modulemd It is a file that defines which packages get built for which releases It includes a summary and a description, a list of source RPM packages, build information i.e. build order and macros, and usage information i.e. installation profiles and licenses. ○ ○ NSVC: name-stream-version-context A module build contains a set of RPMs as its components
27. MBS ● ● ● ● ● MBS checks modulemd yaml files and enriches it MBS will prepare the build tag configuration on koji for it MBS doesn’t do RPM build by itself. It will ask Koji to build component RPMs Once all RPM builds done, MBS will push modulemd and metadata into Koji via Content Generator API, as A Module Build The module build points to a tag created by MBS which contains all of component builds
28. MBS
29. PNC(Project NewCastle) ● ● ● ● PNC is A system for managing, executing, and tracking cross-platform builds PNC is the primary build system of Middleware products(Red Hat JBoss) PNC aims at a minimum to help ○ improve tooling efficiency problems within the productisation of MW projects ○ reduce build times ○ scale with demand ○ increase capability of supporting more streams PNC is organized by: ○ jBPM: a workflow engine to provide infrastructure to automate processes and tasks ○ OpenShift: horizontal resource scaling of hardware and software resource ○ Several components which are able to be easy extended and scaled
30. PNC ● Orchestrator ○ ○ ○ The main entrypoint into the build system. It exposes a REST API for all the main tasks available in the system It is also responsible for managing the execution of tasks associated with other components Orchestrator consists of three logical parts ■ Metadata Service exposing the DB over the REST API ■ Coordinator managing build jobs ■ Executor preparing the build environment, running the builds and collecting the results ● User Interface ○ Web UI(within Orchestrator) ○ CLI
31. PNC ● Indy ○ ○ ● DA (Dependency Analyzer) ○ ○ ○ ● Analyzing a maven projects pom file helping users to align dependencies to the versions that are used by other products To create new build configurations for the projects dependencies in the PNC Repour ○ ○ ● Indy is a repository manager service used to store build dependencies and produced binaries. The Initial purpose it to make it simpler to work with Maven artifacts and repositories A service used to clone "upstream" source repositories to internal git server and to apply version alignment to use internally build dependencies Version alignment is done by modifying maven pom files using PME (pom manipulator extension). PME ○ ○ PME is a tool to manage pom files Its prime use case in PNC is to automatically increment version for each build
32. PNC ● PNC Builder ○ ○ ● Causeway ○ ● A gateway between PNC and Koji Cartographer ○ ● A Docker image with pre-installed tools to run the builds (java, maven, git) A PNC Build Agent which is used for a communication with the Executor. It is used to get the graph of dependencies for a given POM, which helps the DA service in PNC to construct build configuration sets for a bunch of related projects interactively, avoiding the need for repetition KeyCloak ○ KeyCloak is used to provide Single-Sign-On (SSO) service.
33. PNC
34. COMPOSE ● Terminology ○ Release a collection of software with identity and a life cycle (RHEL-7.4) ○ Compose a snapshot of release content (RHEL-7.4-${date}.n.0) ○ Variant a part of a compose targeted at particular use case (Server, Workstation, …) ● Components: ○ dist-git ○ rhpkg ○ pungi
35. PUNGI ● ● Goal: collect build artifacts from build system and prepare them for shipping Composes are release snapshots that contain release deliverables such as: ○ installation trees ■ ■ ■ ○ ○ (bootable) ISOs kickstart trees ■ ■ ● RPMs repodata comps anaconda images images for PXE boot Inputs: ○ comps.xml defines groups of packages ○ variants.xml a definition of variants for the compose each variant has a list of package groups that should be provided in that variant
37. Release Checklist Tool a.k.a Errata Tool Release workflow management ○ ○ ○ ○ ○ Preparation(build/compose) Testing Document Approvals Auditing Stakeholders ○ ○ ○ ○ ○ Developers Quality Engineering Releasing Engineering Product Security Technical Writers
38. Advisory ● ● Customer updates are delivered in advisories Three advisory types: ○ Security Advisory ○ Bugfix Advisory ○ Enhancement Advisory
39. Integration Points
40. Content Delivery & Management Tools ● ● Those Tools are used for shipping content to customers via distribution platforms Those Tools should supports ○ Automation of Content Delivery with growth of portfolio ○ Management of already shipped content ○ Content testing and verification against staged content ○ Various distribution platforms
41. Content Delivery & Management
42. Tools / Services in FACTORY ● Estuary Restful APIs and A Web UI to provide an overall view based on an item in product pipeline ● MetaXOR Collects and organizes container metadata from multiple sources ● ResultsDB A database to store test results and a REST API for importing and exposing ● WaiverDB A service used to ignore (waive) certain test results for certain test suite ● Greenwave A service on top of ResultsDB and WaiverDB to make a decision for shipping software ● ODCS (On Demand Compose Service) As the name, ODCS is to allow generation of temporary composes(RPM repos) using the REST API calls ● Freshmaker A service scheduling rebuilds of container artifacts as new content becomes available ● BOTAS (Bot-Operated Container Advisory Filing Service) A micro-service for filing Release Checklist advisories for container images rebuilt by Freshmaker
43. Infrastructure ● Bare Metal ● Virtual Machine ● Openstack ● Docker + K8S ● Next...
44. Infrastructure ● KVM -> Openstack -> OpenShift ● Ansible -> Ansible Tower ● Cfengine -> Satellite ● Zabbix -> Prometheus ?
45. Infrastructure
46. Q&A
47. 欢迎关注msup微信公众账号 关注大会微信公共账号,及时了解大会动态、 日程及每日更新的案例! 关注公众号获得 更多案例实践