DevOps pipelines as code – enabling quality with speed
June 12, 2019
Many organizations have recently started their DevOps Journey. Everyone is busy automating everything, with constant talks about continuous integration, automated deployments, Infrastructure as Code, and continuous delivery. In this DevOps wave, there is one piece which is not gaining enough attention: “Pipeline Design”. In this blog and the next, I focus on efficient pipeline design and why it’s so important to maintain just enough pipelines that are repeatable and reusable.
At the heart of DevOps are orchestration tools such as Jenkins, GO CD, and Bamboo which allow us to bring all pieces of automation together in the form of a “Pipeline”. Usually, there is no benchmarking available for these pipelines like which stages they should have, which languages can be used, tools they can use, or how many pipelines should be built. Typically, pipelines are grouped as build pipelines, test pipelines, deployment pipelines, CI pipelines, CD pipelines and nowadays cloud pipelines, too.
While designing pipelines you should consider these architectural principles:
-
DRY (Don’t Repeat Yourself): I have seen setups where duplicate pipelines are created every now and then. Release based pipelines, or a set of pipelines created for every release, add additional maintenance efforts for clean-up post release. During parallel releases, ad hoc admin efforts are required to ensure smooth run of pipelines. Another example is environment-based pipelines—for every environment there is dedicated pipeline. If you have 20 environments, then you have 20 pipelines with the same code but different environment names. We are simply duplicating pipelines for one variable which can be avoided.
-
KISS (Keep it Simple): Recently, I encountered a CI/CD platform where they had 300+ pipelines for applications and a dedicated document explaining which pipeline is doing what and is dependent on which. Always try to keep your pipeline code as simple as possible. Usually a parent pipeline script and its child scripts are written in such a way that they are easy to read and are predictable. Avoid code complexity and unnecessary looping; and instead, make use of functions, common code snippets, and a microservices model. Develop every capability with microservices code quality, build, and deployment and then plug this microservice in the desired pipeline to make it easy to manage.
-
ISOLATE: Categorize your pipelines based on application or activity and isolate them accordingly. If you have 2 applications, it’s better to create 2 separate Jenkins instances and maintain your pipelines in isolation. This also helps to tune SCM, global variables, and configuration accordingly for the required application without impacting others.
-
Group Related Functions: For ease of maintenance, you can group related jobs together such as build jobs and deployment jobs. As a result, you can get a well-orchestrated view of your CI setup.
These Architecture principles help us tackle most of the concerns such as Scalability, Performance, Maintainability, Security, and Operability.
