Over the last 5 years I have been working with long running product teams. It has constantly been a desire of mine to deliver high quality software. However, when working on long running Product teams, team health and sustainability is significantly more important than when acting as a consultant/contractor. In this post I discuss some of my history and observations about how understanding risk can help Product Engineering teams be better equipped for long term success.
Project vs Product
When I first started (professional) software development it was the dotcom boom. We built websites, e-commerce and sales force automation services and tools. I enjoy looking back on the scrappy nature of how we FTP’ed changes up, ran single servers on bare metal rented in datacentres. We ran our own relational database servers, typed in license codes and did front-to-back work. I was the graphics designer, the UI designer, App dev, DB dev and ops person. I did the integrations. Not a test in sight. No regulations. RUP and waterfall were considered best practice.
I then took these skills into contract and consultancy work. I left the comfort of my mentor and went to learn new things. I discovered TDD, Build Servers and XP. I saw the value in my role as providing expertise in technical areas like application architecture, concurrency, UI design, relational database design and specific technology skills like WPF and Rx. The contractor life suited me well. I believe that I was able to have high levels of autonomy on both where I worked and what I worked on. I was able to find amazing talent to work with where I could grow, and even sometimes offer growth opportunities back too.
This lifestyle was an exciting double-edged sword; on one hand I was exposed to challenges that required certain expertise not widely available, on the other hand there is a persistently high expectation of being able to deliver on this expertise. My peers and I worked very hard to be top of our game. We offered a quality service to accelerate the success of complex finance software projects. Looking back, I am proud at what we delivered and they way we delivered it.
However, it was always project work. At the end of the project, the customer owned our work. We walked away. All care and no responsibility.
Becoming an owner
More recently, I swapped teams. I am now the customer. I am part of the team that owns this work before, during and after the project. Now I get to see the impact of some of the consulting and contract culture that has defined our industry. I am grateful for progress that concepts like the Agile Manifesto, Scrum, Kanban, and related books and tools have provided our industry. Most of which were written and endorsed by consultancies. It all made so much sense as a consultant. Shipping quality fast was the aim. We would make sure we put ourselves out of job, but do such a superb job that we would be invited back for the next project.
However now I see that while this was great, it only is part of the equation if you actually own the work. I had to make a shift from Project Delivery to Product Delivery.
While the concepts of ownership were familiar to me through SOA, MSA, DevOps, they were familiar like how a spectator of a Marathon race is familiar with running a Marathon. What I didn’t appreciate was the phycological burden of that ownership model. How the confidence at the start of the project started to look like arrogance at the end of the project. The fear of the on-call pager. The anxiety of making an incorrect design decision and how that would be judged by others, how that might affect your team, or even your family (as you stayed late to complete the project or were woken by pager alerts).
I was introduced to the concept of Product, and it wasn’t for much later that I realised that this permanence created a new dimension to how I would consider software delivery. As a contractor I wanted to create “good quality software”, but without skin in the game, this was an almost arbitrary measure. It was roughly measured as “a responsive and stable user experience, delivered quickly”. This didn’t mean just front-ends were delivered. Servers and Databases still have users, and I aimed to ensure they were responsive and stable too. In the long shadow of larger financial institutes, this measure heuristic was a useful proxy for what the customer generally wanted. However, when I joined a Product team, this one-size-fits-all approach wasn’t always as useful to me anymore.
The Anxiety Arrogance See-Saw
The challenges I faced in a Product environment were many and diverse. Sometimes, speed to market was more important that total over all quality. Sometimes quality could not be compromised. The measure of quality kept changing from performance to cost, to compliance, to scale, to availability, to delivery time. No longer did I have the luxury of being surrounded by top quality consultants with only the project delivery to consider. While there were still talented folk, many were managing people and things (like Projects, Platforms, Policies). The volatility, uncertainty, complexity and ambiguity of Product work lead to what I now call the Anxiety Arrogance see-saw.
Anxiety can come from the hard to answer questions like:
- What are we building, and why?
- How good does it have to be?
- When is it due?
- When do I start?
- When should I stop?
This can be exhibited as Imposter Syndrome and Analysis Paralysis.
On the other hand, we can be overly confident:
- Why haven’t we started yet?
- It is only a {dismissive statement here}
- Other people have done this before
- The Product Owner told me it should be easy
- Apparently, I just drop this widget on the web site
This can be exhibited as JFDI and “How hard can it be?” Sometimes it is the same person on the see-saw, sometimes arrogant, then ashamed when the project starts to fail. They overcompensate next time in an anxious attempt to cover every possibility of failure, but land in analysis paralysis. However, other times this is a two person see-saw; one who is quick to move and leaves a mess behind and the other who is increasingly cautious as they must clean up more of the mess that gets left behind. The slower one goes, the fast the other goes to compensate. The more mess one makes, the more cautious the other is to compensate.
Understanding Risk
I empathised with people in each situation because I had been in each situation.
My observation was that while neither behaviour was entirely wrong, they sure weren’t right. Both had valid motivations to achieve business outcomes, but both failed to understand and address risk.
There are many ways to manage risk, but I wanted to find one that my team members of all abilities could practically apply every day. I believe that if I could provide them with a simple model to approach risk, then we would be able to provide a psychologically safer and more rewarding workplace.
The first thing I would want them to do is:
- Be able to identify the risk
- Define the owner of the risk
- Measure and understand the risk
- Manage the risk
Identify, Own, Measure, Manage
Identify Risk
The first step in our approach to risk is to identify it. For my Product Engineers, I want them to be able to first identify risks so they can categorise them. I want the Product Engineers to focus on risks to the Product’s failure. For a Product Engineer we can start by breaking these down to Delivery Risks and Operational Risks, or, risks when “building it” and risks when “running it”. Once we can loosely categorise the risk, this makes it easier to decide who is likely to own the risk. The goal here really is just to understand if it is the Product Engineering team that owns the risk or not.
Risk owners
Part of the Anxiety-Arrogance seesaw can be attributed to inappropriate perception of ownership of risks. I have Product Engineers be unnecessarily worried about risks they don’t own, and possibly have not control over. I have also seen Product Engineers operator with almost reckless abandon as they are unaware of the risks that they own.
To help my teams, we have segregated our risks into multiple layers. First we make it explicit that value delivery is about not just building it, but also making sure it runs and continues to deliver value over time. So we have Build Risks and Run Risks. These are then further broken down into categories of “idea” vs “execution”.
This gives us a simple table of:
| Risk | Phase |
| Build | Idea |
| Build | Execution |
| Run | Idea |
| Run | Execution |
We can then simply apply owners now to our table. Engineers should be responsible for the execution of the project; “Building the thing right”. Product should be responsible for the idea; “Building the right thing”.
| Risk | Phase | Owner |
| Build | Idea | Product |
| Build | Execution | Engineer |
| Run | Idea | Product |
| Run | Execution | Engineer |
Just making it clear who owned what, immediately saw a reduction in stress in the team. It was like a weight was lifted from the shoulders. This also resulted in some better communication within the team. Knowing who was covering what, meant a less double-work and less things falling through the cracks.
Measure Risk
Measuring risk can be challenging. Many of us want to put an exact number on a risk. However, risk tends to be vague. This can be where VUCA rears its head again. To help people process risk, I have found a simple combination of Likelihood and Impact to be a useful 2 axis model. In this chart, things closer to the top right are riskier. Using this model, we don’t need to have exact numbers, we can instead roughly gauge if something is more or less likely than another to occur. We then can independently evaluate the impact of our identified risks. We again can roughly measure these just against each other.
This chart above plots 4 risks. One risk is identified as low-likelihood and low-impact, so it is reported as a low risk. Two are identified as medium risk as they are ranked as high in one dimension but low in the other. Finally, we have identified a fourth risk as both high-likelihood and high-impact and as such is reported as high risk. This model just gives us a better feel for which things we can consider carrying more risk than another. A simple look to the top right tells us what to be concerned about.
Manage Risk
Once we had some simple models to help us understand why we were feeling either overly confident or overly stressed, this allowed to take the next step which was to manage our risks. Defining risk ownership allowed shedding the load of some risks, but with the risk that were left to the Engineering team, we were still left with trying to understand how to manage them. The dominant response to most risks from the Engineers was to try to solve or mitigate the risk. However, we discovered this is only one of our options.
- Accept the risk – The (impact x likelihood) was just too low to worry about
- Transfer the risk – Move the risk entirely or partially to someone or something else
- Reduce the risk – Reduce either the likelihood or impact of the risk
- Avoid the risk – Make the risk disappear either by reducing the impact or likelihood to zero, or changing the game such that this risk was no longer relevant.
Many were surprised to know that accepting or transferring the risk was acceptable. As engineers do, they also then figured out that they could use a combination of these to reduce risk. For example, they could reduce the risk to such a point that it was now either acceptable or transferable.
Conclusion
Moving from a project developer to a Product Engineer has made me aware of the extra joy/burden of the long-term ownership of a Product. This added pressure of really living the “build-it, ship-it, run-it” lifestyle has led me to use a simple risk management framework to find direction in the foggy ambiguity of Product Delivery. By being able to identify and categorise risk allows my team to understand who should own the risk. Then for risks that we own, we can further group and measure risks. These models focus on the build phase and the run phase of a product lifecycle. To provide focus, I exclude other phases such as discovery or end-of-life and decommissioning work. These first steps of understanding our risks allows us to manage them which ultimately helps us with our goal of providing a stable and predictable value delivery pipeline while maintaining a healthy and enjoyable working environment.
Which parts of your SDLC do you face risks with? Are you and your team well prepared to deal with them?
Lee Campbell 