Robust CD with Spinnaker & CircleCI

Spinnaker bills itself as a continuous delivery platform for the enterprise. Continuous delivery scopes in many technical practices and tools and begins with continuous integration, which in turn also brings its own practices and tools. Products like CircleCI also advertise continuous integration and delivery.

It may seem that teams need to choose one platform over the other. After all they both perform continuous delivery, right? This is not the case. Spinnaker and CircleCI do overlap ever so slightly but are really intended for use together in continuous delivery pipelines. CircleCI is a continuous integration system. Spinnaker is a deployment-pipeline manager triggered from CI, and is not a build server. This post explains their individual uses cases and how to build powerful deployment pipelines by leveraging use of them both.

Separating CD, CI, & Deployment Automation

Continuous delivery requires continuous integration to build a complete pipeline from commit to production. The pipeline commonly flows like this:

  1. Commit code to source control.

  2. Run tests and build artifacts with a CI system (such as CircleCI).

  3. Deploy artifacts through a series of stages (such as dev, staging, or UAT) before hitting production.

CI systems tend to handle the first two points and are increasingly moving towards the software-as-a-service model. They provide on demand scaling—teams can simply pay more as their throughput increases—and support a variety of languages like Node.js, Ruby, or Go to minimize any configuration required by the end user. They also integrate with code hosting platforms like Github or Bitbucket, so new commits are automatically tested.

CI systems differentiate themselves by integrating with other systems for artifact publishing or deployment. They may offer native integrations with Docker registries that publish Docker images at the end of each build or trigger deployments to Google App Engine, AWS Elastic Beanstalk, serverless platforms, and even Kubernetes.

This is how CI systems drift to encompass continuous delivery workflows by adding deployment automation, and products like Heroku Flow and GitLab AutoDevOps combine CI and deployment automation in this way. These tools trend towards light integrations with the deploy targets by simply triggering deploys via APIs and relying on the platform to do the rest. Generally speaking, teams need a well-defined deployment target to leverage these integrations, and custom or hand-rolled deployment infrastructures require custom or hand-rolled integrations.

Deployment automation comes in after CI when artifacts are ready to deploy. These tools do not concern themselves with CI—just on deploying software—and vary in complexity from language- or framework-specific tools to general purpose automation and multi-cloud/multi-artifact support. Tools such as Fabric and Capistrano fit well-known application patterns. Automation tools like Puppet or Ansible allow engineers to automate deployments and infrastructures. And deployment managers like Octopus support deployments to the cloud, on-prem, or a hybrid infrastructure. Each of these tools differentiates itself by a supported application pattern, infrastructure topology, and artifact type. They may also differentiate themselves by automatically creating deployment infrastructures and deployment strategies for increased reliability.

CircleCI is primarily a CI system with deploy automation for common applications. Spinnaker is a purpose-built enterprise-grade deployment manager that supports multiple cloud providers and application artifacts. Enterprises can, and should, leverage both to build quick and reliable deployment pipelines.

Start Your Pipeline with CircleCI

CircleCI is one of the better hosted CI systems, and enterprises may opt for an on-premise version if a cloud service is out of the question. CircleCI provides engineering teams what they need in a CI system:

  • Support for common languages like C++, Node.js, Python, or Docker

  • Automatic test database access for Postgres, MongoDB, and more

  • Source control backed configuration

  • Integration with popular code hosting

Best of all, everything is just shell scripts, so additional teams can write more complex builds and integrations. CircleCI is a fantastic tool to test and build code. However, the deploy integrations are nowhere near as robust as what Spinnaker provides. Luckily, it’s no problem to leverage Spinnaker from CircleCI. All build steps are just bash scripts, so teams can write a deploy step to trigger a deploy from Spinnaker at the end of the run. CircleCI also supports secret environment variables, so builds can authenticate to external systems.

Enterprise-Grade Deploys with Spinnaker

Spinnaker is a mature and battle-tested solution from Netflix that supports multiple artifact types, cloud providers, and deployment strategies. Deployment strategies are the killer features that make Spinnaker stand out from the rest.

The deployment pipeline requires an artifact (e.g., a machine image, JAR file, or Docker image). Spinnaker uses the "bake stage" to produce an immutable image, which is deployed to the big three cloud providers. Spinnaker also relies on standard infrastructure-as-a-service features like computing, networking, and load balancing to build reliable environments. This is simple, but also powerful and scalable. Spinnaker supports Kubernetes too, so teams can migrate (or jump) to newer platforms depending on their technical requirements.

Spinnaker leans heavily on an immutable infrastructure build with Packer. Each deployment produces a new machine image (e.g., an AMI) that is used for the deployment’s lifetime, and old infrastructure is thrown away at the end of the process after each deployment.

Spinnaker does not manage 100% of the infrastructure itself. Teams using AWS will need to create VPCs, subnets, etc. and configure pipeline steps accordingly. This is where tools like Ansible, Chef, or Puppet will round out the deployment pipeline’s infrastructure automation. Spinnaker takes over once the prerequisites are met. It also does not handle things like service discovery, so configuration such as location of other services and monitoring information must be handled in the bake stage.

Additionally, Spinnaker supports the canary, red/black, and custom deployment strategies, and promotion between stages is automatic or manual. A "single pane of glass" GUI presents the state of all pipelines, so any team member can identify what is running where and what the current status is.

Unlike CircleCI, Spinnaker is not a hosted system, meaning teams need to provision and manage it. Teams can also opt for an enterprise-grade managed solution such as Armory with automated canaries, SLA driven rollouts/rollbacks, and automated load testing.

Conclusion

Enterprises need flexible and robust solutions. Building a continuous delivery (or deployment) pipeline with CircleCI and Spinnaker is a great choice. CircleCI meets pretty much any requirement for the build and test phase, and Spinnaker offers enterprises flexibility across cloud providers for migration or failover scenarios, deployment strategies for increased durability, multiple artifact support, and a single UI to view the status of all environments and applications. This choice doesn’t put teams in a box either. The build and deploy systems are loosely coupled, so either can be changed without much effort. Simply put, CircleCI and Spinnaker are a powerful combo for successful CD.