DaveWentzel.com            All Things Data

The Sunk Cost Trap

"We want you to float on Project Automation and get back to us in a week with your thoughts.  We are a little concerned that this project has cost us $14 million and 14 months.  It is scheduled for code freeze in 4 months and we have yet to see a working proof-of-concept."  The worry was audible in the VP's voice.  
 
And this is how it starts.  I both relish and fear these moments.  On one hand I love coming in and saving a failing project, on the other hand, you never win and friends doing it.  You see, there was a customer perception that our enterprise software required too much manual intervention and Project Automation (PA) was meant to relieve some of that burden.  It was a contractual demand from our largest customers.  We either succeeded or we went out of business.  
 
I quickly began spending my days in the PA team room, at first absorbing what I was hearing, then asking questions, then offering suggestions.  When you get to that last stage the reticence is noticeable from your co-workers.  Within two weeks it was clear to me that most of the team viewed this as a death march project.  The team knew key aspects needed to change if it was to be a success but everyone was hesitant to make those changes knowing that there was only 4 months left.  Left alone, inertia guaranteed this project would fail.  
 
Within three weeks I had a working list of features that, as implemented, would NOT meet customer requirements regardless of time or money spent at this point.  The approach was wrong.  I also worked out a better approach that would meet the requirements, albeit scaled back for Version 1, and could be coded-up and unit tested within the remaining 3 months.  Granted, this would require some serious hunkering down and probably some OT, but we would show success.  
 
It took another week until I could convince the entire team that the only way they could show success was to ignore the sunk costs and change direction.  Within that week I had a somewhat-working proof-of-concept that I could demo and the team was won over.  It took another week to convince management to ignore the sunk cost and change direction.  Management was not happy about time and money being flushed away, but it was the only way to show success.  
 
We delivered Project Automation about a month late and with scaled-back requirements.  In the enterprise software development space, that's an unqualified success!  And we worked almost no overtime (I strongly believe in work/life balance).  By the time the release was GA'd we already had a service pack to address the missing requirements and even added a few new features that we realized were lacking after usability testing.  Customers were thrilled.  
"We want you to float on 'Project Automation' and get back to us in a week with your thoughts.  We are a little concerned that this project has cost us $14 million, 140 man-years, and 14 months of time.  It is scheduled for code freeze in 4 months and we have yet to see a working proof-of-concept."  If 'worry' has a sound it was audible in the VP's voice that day.  
 
And this is how it starts.  I both relish and fear these moments.  The moment that I am asked to work on an impossible project.  On one hand I love coming in late and saving a failing project, on the other hand, I never win friends doing it.  And I've failed at one of these opportunities.  That's not braggadocio.  There were times I went days without sleep to get to success.  It's just my nature, and I'm neither the only one nor the exception to the rule.  
 
Project Automation (PA) was critical to the future of our software.  You see, there was a customer perception that our enterprise software required too much manual intervention and PA was meant to relieve some of that burden.  It was a contractual demand from our largest customers.  We either succeeded at PA or we went out-of-business.  The details of what PA did are irrelevant to this story.  Just understand that it was critical.  
 
I wasn't the first person asked to help out with PA.  Like I said, there are lots of people far better than me.  For months management tried to get some control.  They tried the obvious things...shorter sprints, a move from scrum to kanban, enforced pair programming, more unit testing, less unit testing.  Nothing worked.  Those obvious things, while often helpful, will not turn around a failing project.  I know that, management sometimes does not.  
 
I quickly began spending my days in the PA team room, at first only listening.  Soon I began asking questions, then later offering suggestions.  When you get to that last stage the reticence is noticeable from your co-workers.  Within two weeks it was clear to me that most of the team viewed this as a death march project.  The team knew key components needed to change if it was to be a success but everyone was hesitant to make those changes knowing that there were only 4 months left.  Left alone, inertia guaranteed this project would fail.  
 
Within three weeks I had a working list of features that, as implemented, would NOT meet customer requirements regardless of time or money.  The approach was wrong.  I worked out a better approach that would meet the requirements, albeit scaled back, and could be coded-up and unit tested within the remaining 3 months.  Granted, this would require some serious hunkering down, but we would show success.  
 
It took another week until I could convince the entire team that the only way they could show success was to ignore the sunk costs and change direction.  Within that week I had a somewhat-working proof-of-concept that I demo'd and the team was won over.  It took another week to convince management to ignore the sunk cost and change direction.  Management was not happy about time and money being flushed away, but it was the only way to show success.  More out of desperation for their failing business than faith in my ideas, they acquiesced.  The PA team would change direction and do it my way.  
 
We delivered Project Automation about a month late and with scaled-back requirements.  In the enterprise software development space, that's an unqualified success!  And we worked almost no overtime (I strongly believe in work/life balance...and it was NHL playoff time too).  By the time the release was GA'd we already had a service pack to address the missing requirements and even added a few new features that we realized were lacking after usability testing.  Customers were thrilled.  We remain(ed) in business.  

Why did this project succeed?  It wasn't because I'm some kind of architect-superstar.  We know that's not true.  It was a success solely because I could convince others to ignore the sunk costs, especially so late in the development cycle, and switch focus to something else that would work instead.   

The Sunk Cost Trap

I feel I'm constantly explaining the sunk cost trap to project and product managers.  Recognizing a sunk cost trap when you see it in your software projects can save your company.  

A sunk cost in accounting parlance is a cost that has already been incurred and can't be recovered.  A sunk cost is not necessarily bad if it is a cost that can yield a positive return in the future.  The sunk cost trap arises when people cannot see that an investment will not ever show a return, yet they continue to waste resources pursuing it.  

There are lots of pithy aphorisms that deal with the sunk cost trap that you might've heard:  

  • "When you first find yourself in a hole, stop digging."
  • "Your first loss is your best loss."  
  • "Don't throw good money after bad."  
  • "Let bygones be bygones."  

Each of these aphorisms is conveying the message, more or less, not to fall for the sunk cost trap.  Yet people fall for this fallacy all the time.  We have the mistaken belief...it is really very irrational...that we've already committed a lot of resources to a given endeavor and we are past the point of no return.  We must continue with the chosen path and see it through.  The sunk cost trap, like all fallacies, can cost you and your company a lot of money when not identified.  

It is important as a software architect to not be a victim of fallacies.  This may even be the most important part of a software architect's job description.  

In your everyday life you probably fall for the sunk cost trap more than you realize.  Have you ever finished watching a two-hour movie that you were not enjoying after the first half hour?  You already had thirty minutes invested in it right?  It might get better so you keep watching.  Or you've finished a terrible dinner at a posh restaurant because you aren't the type to complain and walk out without paying.  If you're going to pay anyway, why not finish the meal?  Or you hold onto a bad stock in your portfolio because it is down too much to sell now, even though you'd never buy more of it today.  

Try to recognize fallacious activity. If you suddenly realize that a bad decision was made and it is preventing you from attaining your goals, and a better alternative exists, then you need to ensure that you change.  The truly good architects will do this...the mediocre architects will focus on the sunk costs and will try, come hell or high water, to make a subpar solution work at any cost.  

Prevention and Cure

Here are some tricks that I use to prevent sunk cost traps in my projects and overcome common objections when they do arise:   

  • "Fail early instead of fail late".  I preach this.  Never embark on a 6 month (or longer) project.  Instead, pare down the scope until you can have a proof-of-concept (POC) deliverable in a few sprints.  In your sprint retrospectives you should be evaluating whether or not the POC is working and if there are any alternatives that may be better.  POCs are really just "science projects" but many people are not willing to deviate after a few sprints due to the sunk cost trap.  I just finished a very lengthy blog series on various POCs I did using NoSQL solutions.  As soon as we found an issue that caused a given product to be sub-optimal for our needs we dropped that POC immediately.  We did not continue to throw good money at it just because we already wrote a few hundred lines of code.  I dropped a couple NoSQL POCs after just a few hours because of fundamental flaws I saw in the products given our requirements.  
  • Understand lost opportunity costs.  By continuing to invest in something that is sure to fail you lose TWICE.  You are flushing money down the toilet on the existing solution and you can't properly invest in an alternative that is more promising (that is the "lost opportunity cost").  Management fails to understand opportunity costs constantly.  For instance, senior, strategy-minded architects are forced to perform support tasks because there is a shortage of support personnel.  The opportunity cost is the senior architect cannot think about how to relieve the technical debt that causes inflated support costs in the first place.  
  • "This was a learning cost, albeit an expensive one".  At a minimum, make your mistakes educational so they won't happen again.  

For years after Project Automation I still had managers griping about the $14 million "learning cost" we spent before we changed direction.  This is entirely the wrong thing to focus on.  It isn't that we wasted $14 million, it is that we are still in business and are profitable.  I can assure you that we never had another fiasco like PA again.  Of course we made sure we had better project governance, but most importantly we recognized when the sunk cost trap was causing us to not do the right things.  

Add new comment