Best Practices

The Continuous Saga: how to keep a steady pace without breaking things

How to transition from the “move fast and break things” paradigm to a “keep a steady pace and don’t break things” — all while making sure you don’t jeopardize customer feedback or time to market.

One of the most exciting and important moments for a product and engineering team is when you grow beyond 2-3 contributors to as many as 10-15. In my experience everyone is full of energy, is positive and is engaged — but with more hands on and ideas on the development lifecycle, there is more room for risks and roadblocks.

Imagine a scenario like this…

Your new, bigger team starts to implement multiple changes to the product(s), autonomously, in parallel and without the level of coordination that you used to have when there were just a few people. Sound familiar?

At first, all is okay (exciting even) but in time, product starts to bleed from each and every angle. As a result, the team spends more time fighting fires and eventually, burnout sets in. One by one, each person becomes less engaged and as a team lead, you are left to pick up and resolve resulting personnel and product issues.

As a functional team lead in product or in engineering, it is critical that you prevent a scenario like this in order to ensure the long term success of the business and of course, the happiness of your customers.

Avoid the avoidable

It’s been almost a decade since Jez Humble and David Farley released Continuous Delivery, in which they formalized state-of-the-art principles and practices to craft quality software at scale. The first iPad was just being released; Atari was still a company and Bitcoin was worth $0.01.

Today continuous delivery is not just a nice-to-have — it is a must-have if your goal is to provide high value to your clients. However in order to talk about continuous delivery, we first need to have a broader conversation about what I call the continuous saga: continuous integration, continuous delivery and continuous deployment.

Continuous Integration

The youngest of the three was first mentioned in 1991 in the context of extreme programming. The goal of CI is simply to integrate and test every single change you add to your code base (aka product).

So in other words, every time an engineer adds code to the version that is currently running in production, principles of CI mean that you are integrating and testing the full application before releasing it.

So what?

Continuous integration helps the overall product development process because it:

  • Reduces the number of unknowns that you can find when features are merged (assuming you have a reliable test suite)
  • Resolves, or at least mitigates, the “works on my machine” paradigm
  • Assuming you run the test suite for each commit, CI reduces the probability that your change is not compatible with the codebase version

There are plenty of resources and books about CI and there isn’t any reasonable motivation as to why you shouldn’t implement it.

Continuous Delivery

CI is necessary — but not sufficient alone. With continuous delivery you can take the maturity of your product development process a step further.

Done right continuous delivery can add another layer of automation and security to your team’s operations. After a product version is validated, you package it and deploy it to a staging (or testing) environment. Automatically, without any human intervention.

At that point if the team is happy with one of the versions they can then manually deploy to the production environment.

In this case, the only required human intervention is to decide if a release is going to production or not. However, nothing needs to be prepared for that, everything is packaged, tested, and deployed in a test environment.

As you can imagine continuous delivery is more complex to implement and requires a higher level of automation. The benefits, however, also are greater:

  • Mitigates deployment risks. Smaller changes reduce the risk of deployment failures and make it easier to detect the root cause of identified issues.
  • Decreases time to market. You no longer need to wait months or quarters to roll out a change incrementally. Instead, you can release on demand and involve more your customers in the discovery phase.
  • Reduces costs. Continuous delivery eliminates fixed costs and frees up time for your team to focus on more important and less tedious tasks (don’t let people do what a machine can do).
  • Helps to build better products. Faster, iterative deployment means more feedback from your customers faster.

Continuous Deployment

The final step of the continuous saga is another CD acronym, but this time the ‘D’ stand for deploy.

The goal of this step and approach is to completely remove any human intervention. Any change that passes all quality and testing gates goes straight to production.

I know — from the outside — it doesn’t look much different than continuous delivery, but the implications are enormous. Every time a single line of code is added to your code base, it’s also going directly to your customer hands without exception.

This approach requires a high level of awareness and diligence in the team and in process. You don’t just need to be able to deploy fast, but you also need to recover fast in case of failures.

High risk, high reward, right?

When you remove transition time and the time/energy required by human coordination, your team can test more variations in shorter time and receive product feedback faster.

The level of effectiveness that you can provide to your customers is directly proportional to your level of knowledge of their needs. The more and faster you know about customer needs, the better you can help them.

What to do then?

Which of these processes you’re able to employ depends on the level of maturity of your company and what your goals are. Not all companies can afford to implement continuous deployment, but it’s important to understand each area of process improvement and how they are connected to one another.

You should focus your effort in implementing a solid and reliable first step before moving on to the next one. However, before rewriting guidelines and process, don’t forget to encourage the most important change — the one that takes place with your and the team’s mindset. Performance goes beyond process compliance. Failure to adjust your thinking will result in poor adoption and execution.

In fact, I suspect that this is the reason why many companies still don’t do continuous delivery or fail to launch improved processes. Perhaps they see it as technical and execution activity, when in reality, the fundamentals are deeper and are tied to how you approach product development and who you are as a company.

Marco Ziccardi

best practices development engineering integration processes