My top-10 things to do when starting out on a project to improve code quality and security through DevSecOps
About 7 years ago I was frustrated with the industry practices where it took us 6 months to provision environments, projects were delayed due to environment issues, manual deployments (UI based code uploads), long deployment times due to deploy-from-build vs deploy-from-release etc.
A lot has changed in the industry since then but sometimes these practices are not consistent. My gold standard is to atleast see the following when engineering and pushing new software, especially distributed software across enterprise and web scale solutions
Here are my top 10 things to engineering a good software from a DevOps
1. Start with automated build pipelines: Automate key steps in your software engineering from build, release, deploy to rollback of code with build engines and build tools. Build software like Jenkins, Bamboo, GoCD etc do this and should have pipelines for each step
2. Test your build: Add unit test and test coverage reports to your builds, this is a great quality step and helps you find pesky issues at a component level quick
3. Test for security early: Do you depend on 3rd party libraries? How do you know if your codebase is not part of the latest vulnerability automatically and regularly? Tools like SonarQube to improve code quality and security
4. Do not hand-craft environments: If it takes weeks to setup environments then it will take months to release anything of value to your end customers. Codify your infrastructure provisioning using infrastructure automation tools. Practice automated provisioning and tear-down
5. Deploy from Release Artefacts: Always building code to deploy can introduce bugs. Deploy from a release artefact that has been through rigorous testing. Promote your release artefact through environment to production instead of building a new version every environment
6. Use Platform APIs and CLIs to automate: Platform and Build tools provide APIs. Write command line interfaces (CLIs) over your build process and your platform to create “headless” (without UI) interface for developers. This is an easy way for them to launch into a terminal, authenticate and deploy and has shown to boost productivity
7. Make your CI tests count: Is your continuous integration (CI) test actually testing integration with downstream interfaces? Build a CI environment, deploy end system service mocks and run tests via tools like Newman (headless Postman), Gatling for functional and load testing of your newly crafted components against a semi-real environment. This will improve your code quality further especially as you head into a real S.I.T (systems integration testing) environment!
8. Automate environment configuration sharing and consistency check: As we head towards complex integration where we use cloud based integration tools to integrate other cloud based services how do we share and manage cross-cloud configurations? At one client this was done via Confluence where all the vendors provided endpoints by environments and others had to copy and configure their environment details, deploy and wait for everything to …fail! We also built our own environment configuration consistency checking tool to reduce errors in environment setup from manual configuration. This was handy at clients where we were integrating across multiple vendor services (their own PaaS’s) and environment configurations were shared via Confluence causing early shakedown issues and time to setup a new environment. We reduced the time to provision new environments from 2-weeks to 3-days
9. Automate API Policy deployment: We find that most engineering teams do well with code and infrastructure but still when it comes to the last leg of managing and securing their APIs they use manual processes. This can cause issues downstream as other programs of work/engineers look to change the same service and without context can accidentally rollback credentials and policy. Automate API policy deployment for security, performance , authentication etc using version control and build tools
10. Run synthetic tests: You have a secure and well-tested pipeline for deploying code but Production is where the rubber hits the road! If your customers are finding issues before you are then there is a gap in our engineering process as we look to not only build a good product but also provide value through it and increase revenue. A good set of functional and security tests against your APIs can provide insights into your services and identify risks faster
What did I miss?
We did not talk about Logging, Tracing, Error Handling, Reliable messages etc that is a topic I believe comes under component engineering and will tackle in a future post
Summary
Engineering distributed software in rich ecosystems within an enterprise or web contexts can be done via drag-drop tools as sold by platform vendors. Be sure to always ask about APIs to support these actions because repeated manual processes can be inefficient and error prone. Look to engineer quality in early and shift testing, security left and into your build pipelines. Manage your environments through automated tools and have the ability to setup/tear-down entire environments with a single command. Finally, know your product service levels functionally via API tests as monitoring tools tell a technical story and you need to know the end-to-end functionality is working on not. Oh also, continuously testing security via APIs helps you discover accidental rollbacks of policies and other issues before attackers do
Alok Mishra 