I remember my very first experience with complex project management quite vividly. I was a first-time tech lead and my team was undertaking a very complex task. We had an existing system that we had scaled to its breaking point. After throwing just about every hack in the book at it, we decided it was time to figure out how to run it across several machines. This was back in the very early days of distributed systems, when most software developers really didn’t know all that much about the best practices of creating them. But we had a great team of smart people, and we felt confident that we could figure this out.
We did figure it out, slowly but surely. We spent a long time thinking about design and the different ways of breaking up our computations so they made sense when computed across multiple machines. And then, one day, my boss Mike pulled me into his office and told me I needed to make a project plan.
It was one of the worst experiences ever.
I had to take this incredibly complicated set of tasks and try to figure out which ones depended on other ones. I had to think of all kinds of dependencies. How would we make it work in the complex testing framework we depended on? How would we deploy it? When did we need to order hardware to test it? How long would integration testing take? The questions just kept coming. I would go into Mike’s office, sit across from him at this big wooden desk, and we would go over task descriptions, dates, and breakdowns. He would help me do some of it, and then send me off with the parts that needed more work.
This was not something I enjoyed doing. It is burned into my memory as a series of frustrating and tedious steps where I had to overcome uncertainty and the fear of making mistakes, the fear of missing pieces, in order to create a plan that would pass Mike’s judgment. Then we had another round of tedious work to put it into a format that we could present to the leadership team, so that they would accept it. It almost killed me. And it was one of the most important learning experiences of my career.
Doesn’t agile software development get rid of the need for project management? No. Agile software development is a great way to think about work because it forces you to focus on breaking tasks down into smaller chunks, planning those smaller chunks out, and delivering value incrementally instead of all at once. None of this means that you don’t need to understand how to do project management. You’ll have projects that for whatever reason can’t be completed in a single sprint, or even two small sprints. You’ll need to estimate project length for your management team, and give some detail on why you believe things will take that long. There are some projects, usually described by words like infrastructure, platform, or system, that require architecture or significant advanced planning. When faced with this kind of project, which includes many unknowns and relatively hard deadlines, you will find it doesn’t fit so well into the standard agile process.
As you move forward in your career, you need to understand how to break down work that has complexity beyond the scope of what you can do as an individual. Project management for a long-running, team-based project is not what most people consider fun. I find it tedious and sometimes kind of scary. I want to be building and getting value, not trying to think about how to break down a project that still has very fuzzy implementation details. I’m afraid that I will be held accountable and that I could miss something important in the process that will make the project fail. But the alternative is the project failing slower, not faster.
Project management isn’t something that needs to happen in detail for every single effort, and it’s overused in some organizations. I don’t even like hiring project managers because they often act as a crutch for engineers to use instead of learning to think through their future work and ask real questions about what they’re doing and why, and their presence means that you have more waterfall-style projects instead of an agile process. Still, project management has to happen, and as tech lead, you should be doing it when it is needed, especially for deeply technical projects.
Ultimately, the value of planning isn’t that you execute the plan perfectly, that you catch every detail beforehand, or that you predict the future; it’s that you enforce the self-discipline to think about the project in some depth before diving in and seeing what happens. A degree of forethought, in places where you can reasonably make predictions and plans, is the goal. The plan itself, however accurate it turns out, is less important than spending time on the act of planning.
Back to my first project management experience. Did the project run perfectly according to plan? Of course not. There were bumps, bugs, unexpected delays, and things that we missed. However, amazingly, we still delivered the project fairly close to on time, and there was no string of sleepless nights required to get there. We managed to make the changes needed to move this complex system into a distributed deployable artifact, all while working against the master code branch with 40 other developers making concurrent changes of their own. All of this was possible because we had a great team, and we had a plan. We had thought through what success looked like, and we had identified some of the risks that might cause failure.
Since that frustrating series of meetings with Mike, I’ve had my own series of project planning meetings where I was the one sitting in Mike’s place, and across from me was Carlo, or Alicia, or Tim. They each felt the frustration of the plan lacking detail, and they each went away and did the uncomfortable work of thinking about things that aren’t code, that couldn’t be perfectly predicted. And they’ve each led complex projects to successful outcomes thanks to this work, and are better equipped to build bigger systems and lead larger teams now that they understand what breaking down a project really means.