Microservices as Code
There are a lot of moving parts to a service-oriented architecture. If your engineering organization is all-in on microservices, deploying a new service is a regular occurrence. Which means getting a service production-ready is a regular task. This can be largely automated using infrastructure-as-code tools like Terraform. But is that really enough?
What’s in a service?
At FullContact, every service needs several components to get to production. Each of which required an engineer to interact with an interface to configure. This quickly became time consuming, error-prone, and difficult to audit. For example, prior to setting up automation, engineers needed to interact with:
- AWS: To provision databases, caches, load balancers, etc.
- Jenkins: To create jobs for testing and building
- DataDog: To configure monitors, dashboards, and alerts
- Spinnaker: To setup deployment pipelines
Issues arise from having all of these manual touch points. How are changes tracked, reviewed, and verified? Do we have backups? Can configurations be copied? How easily can we switch vendors?
Infrastructure as Code
These problems, as it pertains to cloud resources, are solved by infrastructure as code. Tools like CloudFormation and Terraform have made managing cloud resources much easier by putting configurations into version control. With every cloud resource defined in code, every change to that resource is a pull request. So instead of logging into the AWS interface to create a MySQL database, an engineer can open a pull request with a few lines of code. The pull request can then be reviewed by other engineers. Reviews are a great place to discuss resource constraints (this cache needs more memory), change management (when is downtime acceptable for this database upgrade?), and other topics that may have previously required out-of-band communication.
Microservices as Code
So if we can automate cloud resources, why not other microservice components? Instead of just infrastructure as code, why not have the entire microservice defined as code? There are some fantastic open source tools for this. Some specific tools to solve for the components above:
Jenkins Job DSL: Define Jenknis jobs with Groovy. An example job:
DogPush: Define DataDog alerts with YAML. An example alert:
Foremast: Define Spinnaker pipelines with JSON. An example pipeline:
The repositories that hold these configurations should be treated just like any other software project. They should have hooks for validating pull requests. They should have automated builds that trigger on merge. Remember that the goal is to replace the user interface with code, and to do that, you need automation. The open source tools are often built with these concepts in mind, allowing you to setup a simple CI job to sync the configurations with the tools.
High Level Changes
One of the unforeseen benefits of these collected configurations is that high level changes become extremely simple. For example, let’ s say we decide to no longer use a specific Amazon EC2 instance type, which is defined in our Spinnaker pipelines. To solve this previously would have meant auditing every pipeline for that instance type and updating those fields in the UI. With microservices as code, a simple find and replace, and we have a pull request ready to be reviewed by our fellow engineers. On merge, the changes are applied automatically, and the old instance type is no longer in use. We can then take this a step further by having our pull request builder watch for, and fail, if an engineer attempts to use that instance type again.
Putting it all Together
With these systems in place, microservices at FullContact can be stood up entirely with code. A series of pull requests is all that is required to have a fully functioning service. This results in a much faster, safer, and easier engineering process.
March 13, 2020 COVID-19: Our Commitment to Health, Wellbeing, and You FullContact News
February 19, 2020 Creating the Whole Person Picture with the Help of Mobile Advertising IDs API Release, Resource, Marketing & Sales
January 27, 2020 Digital Journal Interview with FullContact President Chris Harrison Interviews