Developing Engineering Processes

I’ve had to deal with many different engineering processes over the years. I remember the first time I worked in a code base with unit tests that we were expected to run before checking in code. I was very diligent about doing this, and very upset every time someone broke the build because she hadn’t bothered to make sure her change didn’t break the tests. I also remember the first time I had an engineering process forced upon me that I hated. After years of no required code reviews, no ticketing, and no tracking, a central bureaucracy decided that everyone had to adopt all of these measures at once in a push for standard software development lifecycle management. It felt unnecessary, slow, and burdensome, and no one bothered to explain to us why these changes were happening.

Engineering processes are the place where the rubber meets the road when it comes to structure. Career ladders, values, team structures — all of those are easy compared to the general angst and frustration that you can cause by adopting the wrong engineering processes for your teams. Without any process, your teams will struggle to scale. With the wrong process, they will be slowed down. Balancing the current size and risk tolerance of your team with the processes at hand is the essence of guiding good software development and operational guidelines.

Ask the CTO: Engineering Process

I’m the head of engineering at a small but quickly growing startup. We have very little process right now: there are no code reviews, we use Trello to manage tasks but rarely put everything into that system, and our architecture decisions tend to be made by whomever is working on the project at the time, with my sign-off.

Recently, some engineers have come to me to complain that new people are checking bad code into the systems. They want us to introduce code reviews for all changes. I also just discovered that someone has been writing a new system in Scala, despite the fact that all of the rest of our code is in Ruby. He’s the only person who knows Scala on the team, and I’m afraid of the support burden, but the project is pretty far along so I can’t just kill it.

What should I do? I’m nervous about going from zero process to a bunch of process all at once, but something needs to change!

Think of process as risk management.

As your teams and systems grow, it’s almost impossible for any one person to keep the systems in her head. Because we have a bunch of people coordinating work, we evolve processes around that work coordination in order to make risks obvious.

One way to think about engineering processes is that they serve as a proxy for how hard or rare it needs to be for something to happen. A complicated process should exist only for activities that you expect to be rare, or activities where the risks are not obvious to people. “Complicated” in this case does not only mean a long process. Sometimes, the complexity is in getting sign-off from a group of people who are very busy, or in meeting a very high standard.

This has two important implications. The first is that you should not put a complicated process on any activity where you want people to move quickly and where you believe the risk for change in that activity is low or that the risks themselves are obvious to the whole team. If you want to do code review for all changes, make sure that the process for code review is not so onerous that the team slows down significantly on minor changes, because that will impact your whole group’s productivity.

The second implication is that you need to be on the lookout for places where there is hidden risk, and draw those hidden risks out into the open. There’s a saying in politics that “a good political idea is one that works well in half-baked form,” and the same goes for engineering processes. The processes should have value even when they are not followed perfectly, and that value should largely lie in the act of socializing change or risk to the team as a whole.