Why NASA Projects Miss Deadlines and Blow Budgets

(spectrum.ieee.org)

92 points | by rbanffy 2134 days ago

11 comments

  • whatshisface 2134 days ago
    >NASA managers have come to believe―usually correctly, Martin said―that “projects that fail to meet initial cost and schedule goals will receive additional funding and subsequent scientific and technological success will overshadow budgetary and schedule problems.”

    There's something else hidden behind this statement: NASA managers have also come to realize that proposals that under-estimate budgets are likely to look more attractive than proposals that accurately or over-estimate them. As a result, the equilibrium shifts and a certain percentage gets taken off the estimation for everything.

    • maxxxxx 2134 days ago
      This pretty much how you do it in a company too. You will never get the budget for 3 years but if the estimate is for 1 year then after that year you usually get more.

      I have also seen people getting penalized for delivering early because their estimate wasn't correct.

    • AngryData 2133 days ago
      Yup, if you need 10 million for a project, you quote 3 million and ask for more funding later on, otherwise you will never get any funding at all as they balk at the true costs of research and development.
      • sgt101 2133 days ago
        Where I work this gets you fired.

        We have a benefits tracking requirement, you put in a 3 gets you 9 case (borderline fundable - probably too small and not sufficient payback, but you might get it) and they go back and ask for another 7 over 2 years and you will leave on the ask. If you put 3 gets you 30 and then ask for another 7 you may well leave on the ask, you may get to stay for the realisation of the 30, but then you go. Probably you go on the ask because someone else will be parachuted into the project (if it's believed that the 30 really is on the table).

        • buro9 2133 days ago
          I worked in a place (a bank) where an estimate that was more than 10% out in either direction would get you fired. Their view was that they valued accurate estimations that highly.

          But when I was there it was already being worked around... managers would create lots of smaller projects that delivered cross-functional behaviour to multiple projects and allowed them to externalise and hide the overages in what might appear to only be dependency lines to existing internal projects. The managers of multiple projects would collude to sponsor each others externalities so that they all had a layer of protection from being fired, obscuring what the costs were actually contributing to.

          The reason I was there was precisely to build a revenue recognition type system for their project management system to try and identify which projects had asked for some other project to start.

          But after that I worked at a UK govt department and they'd become so good at tracking who had been first to demand the creation of another project that their problem became one of legacy... no-one could sponsor even upgrading their long obsolete software or hardware, because even though it was a thing that benefited everyone, the project that demanded it was going to be the one that paid for it (literally paying for the certification, hardware, software, training, etc) which meant if you were the team that said "We can't support this on IE6" then you had to pay to replace laptops, OS, to get onto a later version.

          My conclusion... if we don't embrace a pragmatic reality of what things cost, if we try and manage the costs tightly... the side effects usually have unintended consequences that are worse than facing or embracing the original costs.

          • Someone 2133 days ago
            ”I worked in a place (a bank) where an estimate that was more than 10% out in either direction would get you fired.”

            If so, I would see only two rational approaches: find another job, or hugely overestimate, and, unless things go very wrong, twiddle your thumbs for most of the time (or think of/work on a side project or study)

            It’s very hard to estimate even the simplest ‘projects’ within 10%. For example, try estimating how long your trip home or your next bathroom break or coffee machine visit will take, and see how often you get within 10% (if you say “5 minutes” that gets you 30 seconds either way)

            • AtlasBarfed 2132 days ago
              I'd overestimate and then invent work to get to 5% under budget. You can always test more.
          • amelius 2133 days ago
            > My conclusion... if we don't embrace a pragmatic reality of what things cost, if we try and manage the costs tightly... the side effects usually have unintended consequences that are worse than facing or embracing the original costs.

            Yes, but sometimes there is competition between projects. In that case it may pay to lie about the actual cost.

            • longerthoughts 2133 days ago
              It might pay for the people on that project, but I think the point was that this ends up being worse for the organization. Because of the incentives at the individual level, as you pointed out, I agree that it's not realistic to expect project leads to be more pragmatic. I do, however, think organizations should be making more of an effort to realign incentives so that what's best for the project is also best for the organization. Unfortunately, solutions like the "within 10% or you're fired" model create enough pressure that people evidently just find ways to cheat the system because their greatest concern is now keeping their jobs.
          • sgt101 2133 days ago
            10% either way is very tight. I think everyone's keen to allocate capital well, but if you see a way to pull 30% out of a project and are incentivized to decline then that's not good.
        • marsokod 2133 days ago
          One of the problem with NASA (and other companies/agencies of the sort) is that projects take a very long time, at least 10 years and usually 20 years. You cannot fire the guy who officially did the estimate at the beginning: he is not on the project anymore, most probably not in the same company, maybe retired, and usually has grown to a position much more powerful.

          Underestimating on the contrary leads you to not getting funding for the projects you are responsible for, and that is a nice way to not get a promotion.

          As in Python, it is easier to ask for forgiveness than permission.

    • mysterydip 2134 days ago
      This seems to be common in the defense industry: overpromise and underdeliver, because they know good money will be thrown after bad in an attempt to finish a bad project once they get the contract.
      • kevin_thibedeau 2133 days ago
        Until Rumsfeld started waving the cancel wand and ruined some plans.
        • forapurpose 2133 days ago
          Is there any evidence that Rumsfeld was more effective at managing budgets than other Secretaries of Defense? He liked to talk big, but he talked big about Iraq and was a complete failure and liar (including on the budget).

          Gates did a lot, canceling Cold War era projects such as the F-22 rather than throw good money after bad, especially with the U.S. fighting two insurgencies (for which advanced fighter planes were a waste). His successor Ash Carter refocused the Defense Department on planning and developing for future warfare, especially in the 'cyber' domain.

    • skywhopper 2133 days ago
      Like other commenters, I see nothing unique here to NASA or any engineering project of any scale, public or private. There are always unanticipated challenges, proposers are incentivized to underestimate time and costs up front, and these two things combined will always result in cost overruns and lateness.
      • Scoundreller 2133 days ago
        Right. Shipping beta software/hardware to meet timelines and fixing later with a firmware upgrade (or, a for-pay upgrade...) isn't an option for them.
    • triggercut 2133 days ago
      There's another slightly related aspect that a lot of people tend to overlook, that being the ability to capitalize your costs.

      Expense always gets pushed out as far as possible. R&D costing too much? Push it to CAPEX. CAPEX costing too much? Push it to OPEX.

      Unfortunately humans are fantastic at being able to create systems where by we can easily defer anything that doesn't suit us right now.

      This being said, I have no idea how NASA capitalize their assets. Accountants being accountants leads me to believe it's either squeezed into the same way as everything else, or there's some esoteric standard invented just for them.

    • kaybe 2133 days ago
      I know this in reverse: You want 2 years, so you ask for 4 and you will get 2. They always cut something out, even if you give an accurate number, so better put something in for them to cut.
  • tomohawk 2133 days ago
    The kind of engineering you have to do to make a project successful at a place like NASA is very different than many other places. If it's a larger program, you need to make sure you have participation scattered across congressional districts, or at least certain important ones. This reduces defunding risk, but introduces lots of program management risks.

    Once you've successfully become known as a jobs creator, then your job is to spend as much as possible. You can't be looking at the efficiency of getting the program completed - you have to be looking at the efficiency of spending taxpayer money as efficiently as possible in congressional districts. The less work performed per dollar, the more efficient you are in getting the money into the economy. Hiring an engineer at $300/hour instead of $100/hour - why risk getting the cheap guy when there is no benefit to you?

    Efficiency in government programs has a very different meaning than in other programs.

  • acidburnNSA 2134 days ago
    Important to note. A lot of complex systems people turn to the NASA systems engineering handbook [1] thinking it will solve all their cost and schedule issues. It may help and is insightful. But there's no silver bullet.

    [1] https://www.nasa.gov/feature/release-of-revision-to-the-nasa...

    • iancmceachern 2134 days ago
      Thank you for posting this, ordering my copy now!
  • forapurpose 2134 days ago
    My projects are very hard to budget and schedule accurately, especially to the degree they involve skilled labor, and my projects tend to be much smaller than NASAs, with far less cutting edge tech and development and with much less demanding requirements.

    Can any large organization reliably and accurately budget, schedule, and then implement large, complex projects? Which ones? How do they do it?

    • iancmceachern 2134 days ago
      This was my thought.

      Could it be that NASA takes on the toughest, most daring and most technically complex projects around?

      On the medical device projects I work on delays, overruns and issues come into play often. I can only imagine if the medical device I was working on not only had to not injure the patient, but do it in a vacuum, survive the radiation and survive launch!

      • forapurpose 2133 days ago
        And don't forget the 'serviceability' of your product in the Kuiper Belt.
    • nradov 2134 days ago
      The US Navy created Program Analysts and Review Technique (PERT) for projects like that. It seems to have worked pretty well. https://en.m.wikipedia.org/wiki/Program_evaluation_and_revie...
      • forapurpose 2133 days ago
        > The US Navy created Program Analysts and Review Technique (PERT) for projects like that. It seems to have worked pretty well.

        The U.S. military, including the Navy, makes plenty of headlines for over-budget, over-schedule projects. For the Navy look at their highest profile recent projects: Ford-class aircraft carriers, Zumwalt destroyers, and, IIRC, the LCS.

        That said, I know nothing about PERT.

    • abecedarius 2134 days ago
      https://dominiccummings.files.wordpress.com/2017/02/201702-e... claims to outline why Apollo-era NASA was effective in this way and offer lessons for people today. I don't know anything about it myself.
    • user5994461 2133 days ago
      >>> Can any large organization reliably and accurately budget, schedule, and then implement large, complex projects? Which ones? How do they do it?

      Yes, it's fairly doable to set the project budget and schedule. Most organizations can do that fairly accurately.

      The thing is that most people don't understand what a budget is. For instance, when a project is given 2M dollars and a year. It doesn't mean that the project will costs 2M dollars and be completed in a year, not at all. It means that there will be a few people allocated on it, with maybe some hardware, and we will see what we get at the end of the year.

  • greeneggs 2133 days ago
    "Too big to fail" makes it sound like a fixed-cost fallacy; we've spent $X billion on this project, so we had better not cancel it now!

    But in fact it could also be the opposite of the fixed-cost fallacy. Project X is worth $1 billion, and is funded for $1 billion. A year later, the $1 billion is spent, but another $1 billion is needed to finish. At this point, your choice to spend $0 and get nothing or spend $1 billion and get the project done. The choice is obvious; we've already decided the project is worth $1 billion.

    The problem is that a year later you face the same choice again. And then again. Provided the marginal increment is always $1 billion, you'll always agree to it, because the project is worth that.

    • Eridrus 2133 days ago
      Running projects where you only make decisions when you run out of money seems like the sign of incredibly poor management.

      Once you start noticing that budgets/timelines are slipping you can start revising your estimate. If revised estimates don't start converging, you basically don't actually know how much the project will cost, and saying it will cost another billion is just straight lying.

      • CamTin 2133 days ago
        "Running projects where you only make decisions when you run out of money seems like the sign of incredibly poor management."

        You're right, but this is the kind of management that the U.S. Constitution requires. The Executive branch "runs" the program, but the people allocating money (Congress) are only asked for input when the cash box is empty.

        In a smaller (say, pre-Civil War-sized) country with a much much higher (constituent/representative) ratio, and with far fewer things for the Federal government to keep track of, Congress could reasonably exercise oversight of all or most of the things that they had appropriated money for. When 535 people are trying to oversee the largest, most expensive, most geographically distributed organization ever, in the history of the world, it all breaks down.

        The best a legislator can reasonably do is, when somebody comes to Congress for a top-up on their bank account, to farm out a staffer to read some reports about a project and make a suggestion about whether it should get more money or get cut off. Most of the millions of line items in the Federal budget don't get even this much attention.

        • Eridrus 2133 days ago
          Sure, there are a whole host of reasons Congress is bad at this, but there is nothing rational about repeatedly authorizing $1bn in funding for something with a projected benefit of $1bn.

          Congress can't manage it, but in an ideal world they could demand good management practices and refuse to fund things that go over budget without evidence of good management.

          • CamTin 2133 days ago
            Yeah, not in this specific example, but under certain (somewhat artificial) circumstances, it could be.

            Imagine that your car doesn't start and you have it towed to a 100% trustworthy mechanic. He says that he knows what is wrong and it will cost you $500. You need the car, and $500 is far less than what it would cost you to replace the car. You authorize the mechanic to fix what he thinks is wrong for $500.

            He calls you the next day to say he did the $500 worth of work but, just as he was finishing, he found something else wrong. The car won't work at all without this new work. It also costs $500. Because it is still worth more $500 to you to have a working car, you authorize this work as well.

            This can continue indefinitely, even far past the point where you could have just bought a new car with the money you spent fixing this one. At every point along the way, the most prudent-, rational-seeming thing to do was to spend the $500, but this ultimately leads you to ruin: no working car, and your savings totally depleted.

            This outcome is similar to that in the (also artificial) circumstances of a dollar auction (https://en.wikipedia.org/wiki/Dollar_auction).

            Still, many things is real life approximate this dynamic, if not exactly, then in part. It's valuable to recognize the pattern so that you can recognize and short-circuit it in situations where you would otherwise "rational" yourself into a corner. In a dollar auction, you can pick the optimal move, which is not to play at all. In certain real-life circumstances, that isn't really an option.

            • Eridrus 2133 days ago
              The rational thing to do is to not be limited by the go/no go choice and seek more complete information.

              This game illustrates the limits of blind cost-benefit analysis, but that is not the entirety of rational thought.

      • greeneggs 2133 days ago
        I was specifically thinking of the Webb space telescope. Per Wikipedia, the budget estimates indeed laddered up in this way. [1] I would be mad at Congress if they decided to cancel it because of the next $1 billion cost increment.

        [1] https://en.wikipedia.org/wiki/James_Webb_Space_Telescope#Cos...

        • Eridrus 2133 days ago
          Why would you be mad? In 2006 they had budgeted 4.5 billion and already spent 1 billion, cancelling it then would have let us spend 7bn+ on projects that weren't mismanaged.
    • DennisP 2133 days ago
      But it's worth saying no to the extra billion anyway, because otherwise you can be exploited.

      My brother's a Navy officer, and described to me an Air Force base he visited. They'd gotten a certain amount of money, and spent a lot of it on luxurious accommodations including a large swimming pool. They ran out of money without having gotten around to building a runway. Of course they got another round of funding for that.

  • mcguire 2133 days ago
    "Another reason NASA management plays down a project’s technical complexity and risk is to get the project sold, Martin noted. Once sold, a project almost immediately becomes too big to fail."

    One example: Ares (https://en.wikipedia.org/wiki/Ares_I?wprov=sfla1). Originally sold as reusing the shuttle's SSRBs, it was too heavy. So they said, "we'll just add another segment to the SSRB" and didn't mention that that would mean a complete redesign and effectively a new booster.

    I work as a NASA contractor on enterprise software and see the same issues: things are sold as new, exciting, cutting edge, fast, and easy. Then they turn into a fiasco.

  • woodandsteel 2134 days ago
    On the other hand, deep space probes like the Mars rovers seem to be generally on time, and at least reasonably within budget (and fabulously successful scientifically). I wonder what is making the difference.
    • wolf550e 2134 days ago
      NASA Science Directorate, which operates Mars rovers and sends planetary probes, is mission driven. NASA human spaceflight is a jobs program (it invents make work in order to excuse spending money) that is fatally afraid of taking on a project they might fail at, because that might endanger the spigot of endless government money for all the pet technologies (for the people inside NASA) and jobs in districts (for the funders in congress).

      See this talk by Dan Rasky (inventor of PICA heatshield) for a perspective from the inside: https://www.youtube.com/watch?v=g3gzwMJWa5w

      See almost any talk by Robert Zubrin for a perspective from outside, for example: https://www.youtube.com/watch?v=EaNKtIY6JpQ

      • jonhendry18 2133 days ago
        "is fatally afraid of taking on a project they might fail at, because that might endanger the spigot of endless government money for all the pet technologies "

        And people could die

        • rbanffy 2133 days ago
          A lot of projects fail without taking human lives with it. Ares was dropped after one single test flight of a simpler Ares I.

          Arguably it shouldn't even have been attempted - the sheer stupidity of the whole idea of having a single large solid rocket propelling you up without the possibility of being throttled down and with the chance of having burning solid fuel raining on your parachute in case you needed to bail out was probably swept under the rug too many times for comfort.

          And that clusterfuck was stopped before it'd be able to kill anyone. In the meantime, it produced some knowledge on large solid fuel 1st stages that may someday compete with SpaceX's tech.

          • wolf550e 2133 days ago
            This is only re: your last sentence.

            It won't, solid rockets are inherently limited in their safety (for the reasons you pointed out), fuel efficiency and ability to be rapidly reused.

            The fuel efficiency will never be that of kerolox, methalox or hydrolox engines [1].

            Landing a solid the way Falcon 9 lands is impossible (there is no engine on the way back, and if you can shut it down safely and then relight it then you can't throttle it), and when you recover the parts of the rocket that are not fuel, "refueling" solid rocket is basically re-manufacturing (the STS boosters were re-manufactured, without any economy).

            In general, the use of solid boosters for crewed NASA missions is a folly that was mandated by congress to funnel money into ATK (manufacturer of many military solid rockets and explosives from hand held to ICBM/SLBM, now Northrop Grumman). Ares I, Liberty and OmegA, and the SLS boosters, are just more of the same.

            1 - https://en.wikipedia.org/wiki/Solid-propellant_rocket#Perfor...

            • rbanffy 2133 days ago
              Indeed. The SRBs are a stupid idea. Someone must build a decent rocket factory in the same congressional district so that NASA can make safe rockets.

              It makes sense if refueling is just filling them with fuel in an automated way (think a giant extrusion device printing layers of fuel inside a rotating empty rocket). That way, the cost of flying a rocket becomes downtime+fuel.

          • greglindahl 2133 days ago
            OrbitalATK (now part of Northrop Grumman) is bidding Omega, which has a big solid first stage, for EELV2.
        • wolf550e 2133 days ago
          Wasting billions of dollars trying to prevent a few astronauts from dying, while that money spent on saving lives more efficiently could have saved thousands of lives, is called "statistical murder".

          https://reason.com/archives/2012/01/26/how-much-is-an-astron...

          • jonhendry18 2133 days ago
            Sure, but that's cold comfort if you work on something that blows up some astronauts and a teacher.
      • woodandsteel 2133 days ago
        Thank you, that is a very insightful comment and it answers exactly what I was asking about.

        Maybe if SpaceX starts sending people to Mars at 1/100 the cost NASA plans to, congress will finally end the NASA manned space program.

    • jimrandomh 2134 days ago
      If you miss your launch window for going to Mars, the delay is two years; since you can't be 1 month late and have things still be okay, the initial schedule is more realistic and delays are much less tolerated.
    • mturmon 2134 days ago
      Delays in the Mars program do happen. MSL and its Curiosity rover had a delay of one launch window (18 months): http://www.planetary.org/blogs/guest-blogs/20081204-msl-laun...

      This is small compared to some other delays, of course.

      (Perhaps this forgetting illustrates the point of the article.)

  • kk58 2133 days ago
    On studying the joint cost and schedule analysis document highlighted in the article. It is evident why NASA has problems. The estimation of cost and schedule delay distributions is subjective and not Bayesian. This leads to outcomes severely influenced by optimism bias.i work on similar problems for an energy company and it's clear that Bayesian methods with broad based machine learning models can rectify a lot of these issues
  • sytelus 2133 days ago
    TLDR; The GAO identified the culprits as “risky management decisions, unforeseen technical challenges—some avoidable and some not—and workmanship errors.”

    Looks like nothing we didn't knew. NASA wants very high success rates on missions which leads to spending enormous sums to drive down probability of failures. On the other hand, Musk can blow up rockets dozens of times and keep moving at a fraction of the cost. The second biggest factor is simply the unknowns. It's not that we have launched giant telescopes in space dozen times and have became very good at predicting costs.

  • hessenwolf 2134 days ago
    NASA: Swingers.

    I don't mind if you upvote or downvote me. Tell me your experience.

  • rurban 2134 days ago
    One should add that this didn't happen with the old NASA in Alabama under the germans. This only started with LBJ ditching the Apollo team, the liquid rockets, and splitting up NASA into Houston and California under US management. It went downhill since then.

    The NASA never really was about space travel, because then they would have kept the Apollo tech and liquid rockets. But military needed only solid fuel rockets so they switched and developed the shuttle. And now they have no proper tech anymore at all.

    • bryananderson 2134 days ago
      This is a wholly inaccurate picture of NASA’s organization then and now. The centers in Alabama, California, and Houston were all subsidiaries of NASA headquarters in Washington, and remain so today. Control was never held by the Alabama center, nor was it transferred to any of the other field centers. Von Braun was the director of Marshall (the Alabama center) throughout the Space Race until well after LBJ.
      • Herodotus38 2133 days ago
        Thanks, I thought the OP's post was odd, as I'd never heard of Alabama being the center, and I've been more interested in spaceflight for the last 4 years. I don't have enough historical knowledge in this area to definitively rule out what was posted, and it seemed plausible that there was tension between solid and liquid development. I am currently readin Von Braun's 'Space Frontiers' and enjoy it. Any recommendations for a next book about the history of US rocketry?
        • bryananderson 2133 days ago
          Andrew Chaikin’s ‘A Man on the Moon’ is a wonderful history of the Space Race.
      • mcguire 2133 days ago
        Yes and no.

        Yes, NASA HQ is in Washington and always has been. On the other hand, the centers have always had a lot of autonomy; moreso in the past, and the fight to coalesce overlapping facilities is messy. (Source: Work for the NASA Enterprise Applications Competency Center (yes, that's a thing) and get to see the sausage being made.)

        JPL is one fun example. It's really only loosly associated with the rest of NASA. And it does most of the unmanned, science flight programs, using DOD launch vehicles.

        That goes back to the early days of NASA following Sputnik and the Explorer I vs von Braun/MSFC/Alabama teams. MSFC has been the center of launch system development since, for the manned side of things. And the shuttle, in retrospect, seems not to have been a great idea.

        Tl;dr: To paraphrase Ghandi, NASA organization might be a really good idea.

      • rurban 2133 days ago
        Laughable. Washington gave all control to Von Braun after the Sputnik desaster, and Von Braun delivered dramatically. They only cut his budget, because Von Braun wanted to fly humans to Mars, and Washington wanted intercontinental rockets with nuclear payloads only. LBJ did the first dramatic cuts by favoring the Californian solid rocket development and moved control away from Alabama to the new Houston site. And then Nixon killed it completely and went with the Shuttle program, out of Von Braun hands.
    • Herodotus38 2134 days ago
      I'm not sure why you are being downvoted. I'll admit I've never heard this opinion before but I'm interested in learning more. Do you have any links or books that go into more detail?