Continuous Delivery & Microservices, Part 2

Here at Armory, we believe that obtaining the value of microservices is a critical component of continuous delivery. A lot of companies are moving from monolith to microservices architecture, but they often end up duplicating effort in their systems which is an inefficient use of resources and leaves the promise of microservices, unfulfilled.

The value of microservices is evident in its ability to add a feature to one component without affecting another. The assumption is that the software development lifecycle (SDLC) cost-per-feature is reduced.

Every organization moving toward this type of architecture has a different understanding of microservices and best practices regarding them. Any instance of an individual microservice is constantly changing making it hard to navigate this transition. The architecture transition, make no mistake, is vast.

This two-part article shares some of the lessons we’ve learned while helping our customers through this journey. In Part One, we shared tips on breaking into using microservices, tooling, and determining service boundaries. Part Two will discuss security, testing, teams, and the most significant challenges we’ve faced.

New Security Requirements

The use of microservices significantly increases the need to secure data. Data is accessed from many different services and locations, and through a number of different APIs, therefore teams have to give access to data on more discrete datasets.

A lot has been written about segregating data and responsibility. For example, one service may be responsible for password data while another is responsible for sensitive user data.

Depending on the size of your organization this can look very different, because the services requesting access to your data may be in different parts of the country. There are three ways to solve this problem:

  • Token Based Authentication: this is a great place to start because it’s simple to implement

  • Mutual TLS: Two-way authentication is when two parties authenticate each other simultaneously. The Mutual TLS method requires that the service which is accessing the data be granted certificates which makes it a secure option. The downside of this methocanard is managing certificates for both parties. Netflix created an open source service called "Lemur" to help with certificate management.

  • Security Groups: these are effective, but are more difficult to implement. Although it becomes harder to apply across VPC and account resources in a security group, the extra security and work are considered to be worth it.

How Testing Changes with Microservices

Integration testing is very error prone and consumes a lot of person-hours, which makes it very costly. Each integration test brings limited coverage of use cases. If you consider all the code path combinations to your application coupled with another application’s code paths, the number of tests written quickly explodes to a number that is unachievable. As a result, managers feel the ROI isn’t there.

As an alternative, we instruct our customers to focus on unit test coverage. A handful of well-selected integration tests can demonstrate the critical points: total failure to essential areas of the application, additional metrics such as SLA/SLO, additional alerting, canary deployments and one-click rollbacks.

Inter-service integration tests defeat the purpose of microservices. If you’re testing how the system works as a whole, then conceptually you’ve done something wrong since these services should be loosely coupled. Essentially, you've just created a monolith that's more complicated to debug.

Instead, consider building a “contingency plan” into the code for when a sub-service fails. For example, Netflix uses a service to predict what you will see in your feed but when that service fails it falls back on a cached version of the feed where no failures exist.

Managing the Biggest Challenges

The most frequent challenges we have come across are dependency management, deployments, monitoring, alerting, SLA/SLO, security, inter-team communication, logging, and debugging practices.

The way our customers solve these problems is to turn operations into an actual "DevOps shop." Creating a "DevOps shop" is critical because microservices bring overhead in the form of duplication. Building great tooling is the only way to make microservices work.

It is important to start treating the application developers (the users of the tool) like customers. The result of doing so will not only improve the tooling for the internal developers, but it will also motivate your team towards a success metric.

For one of our customers, we measured tooling/service adoption as a way to measure success. Measuring success doesn’t have to be complicated.  Simply asking “are they using the internal tool?” was a good enough metric to drive better internal tooling.

Auditing data sources have simplified since the footprint of code that needs access to a data source gets smaller.

Changing How Teams Work

Instead of communication driving software design, software architecture drives communication. Traditionally, businesses have followed Conway’s Law, which is the inverse of what microservices need. The law states "organizations which design systems are constrained to produce designs which are copies of the communication structures of these organizations."

Not only do teams operate and organize differently, but it requires a new set of tooling and process to support this type of architecture, i.e., DevOps. As monoliths get broken into smaller components, the need for smaller teams and their roles become more apparent.  Engineers best skilled for (or who have a strong desire to work on) a distinct type of problem can now be assigned to those microservices.

Building and deploying pipelines is very different from managing a monolith, and your team will need to learn how to work in this new environment.

We’ve just touched the surface of these topics, but hopefully, this has given you an excellent place to start.