A Start-Up's Journey to a Fully Integrated Approach to DevOps
Prior to 2008, development and operations were two independent functions - there was a slight overlap in their responsibilities around deployment and monitoring of the code that developers wrote, but largely they worked on very different sets of problems. While developers focused on design, development and testing of features, the operations teams were heads down on infrastructure needs, working on availability, setting up backups and many more. This also meant the two teams did not work closely together and were at logger heads when things went wrong in production – blaming their respective code and servers In the last 8 to 9 years, we have seen a paradigm shift in this model. Today, a much more common setup now is to have a DevOps team that works very closely with developers and together they deliver high quality and timely releases. Interestingly, this trend is observed not just in start-ups, but also MNCs. The three catalysts that have orchestrated this paradigm shift are:
1. Agile Development Methodology: With typical sprints being 2 weeks to 1 month long, it becomes imperative for development and operations teams to collaborate to meet the expectation that an MVP / iteration of the feature will be delivered by the end of the sprint.
2. Proliferation of IAAS Services Such as AWS: These services have significantly simplified traditional Ops activities ranging from server setup to taking backups to keeping infra secure. To developers, what was a complete black box earlier now seems just a configuration or a script away.
3. Micro-services: Micro-services further break down core Ops tasks like provisioning, configuration and deploys into much smaller, more manageable, repetitive and most importantly reusable chunks.
Start-ups are always looking to achieve more with less and this is equally applicable to the DevOps function. In this article we will outline strategies that go a long way towards achieving the same.
From a DevOps standpoint, the key to unlocking the productivity of your entire engineering organization is empowerment through automation. It is critical to make every DevOps task accessible to all developers and QA team members through automation.
We can bring the mindset of empowerment to every aspect of the SDLC. For example, rather than the traditional model of QA providing sign-off on a build and DevOps owning the deployment, a better approach is to fully automate the deployment process so it’s essentially just a button click and then provide your QA team the permission and responsibility of production deploys. It is also critical that you provide QA team training on how to roll back a deploy, how to revert changes, etc. This builds their confidence, makes them self-reliant and prepares them for every eventuality.
A step by step approach towards automation (and empowerment) is:
1. Standardize system requirements, then fully automate them
2. Continuously evolve all DevOps tasks until they are fully automated and can be completely moved into self-serve mode
Standardizing System Requirements:
Without standardizing system requirements, there is no hope of automating tasks around them, from provisioning, to deploy to monitoring. A pre-requisite to standardization is adopting a micro-services approach. Each micro-service will then be made up of a pre-defined set of mandatory and optional components such as - load balancer, reverse proxy, cache, web server, data-store, logging, monitoring etc.
Various factors can drive the choice of these components, such as – team members being familiar with a technology from having worked on them previously, ease of setup and deployment, actual performance and functionality. As the team matures, you are better placed to do a thorough evaluation and make the right choices. However, it is important to acknowledge that you may not always make the right choice and sometimes the right choice may emerge over time (for example – advent of Docker); as long as all systems use the same components, it becomes easier to plan migrating to newer, better solutions.
Continuously Evolving DevOps Solutions:
Here we will outline a simple example of handling poorly written queries in code. With an ROR stack, it is not uncommon to inadvertently write code that translates into very poorly performing queries on the DB and our team faced this challenge as well.
• As a first step, we generated the slow query report manually and shared it with the entire development team. Pockets of virtual teams were formed to address missing indices, re-factor poorly written code etc. and get the applications back into a reasonable state.
• Fresh from this success, as a next step, we decided to run the job daily and mail out the report to all developers to ensure we don’t slip again. However, given our one time effort and the daily frequency of this mail, developers promptly started ignoring it.
• As a third step, we started parsing the results in the slow query log and only sent out reports if there was an increase in count. This clearly indicated that there was a recent regression and hence, made the report actionable.
• Finally, we created a web based dashboard that charts this data as well as includes download links to all previous reports. This helps developers and QA immediately spot regressions in terms of when and how many and helps narrow down culprit commits without any intervention whatsoever from DevOps. In conclusion, a fully integrated DevOps perspective has enabled us to empower our entire engineering organization through relentless automation of all DevOps activities.