page updated on August 24, 2017

Why Do Government IT and Software Projects Fail?

If you pay taxes, you have plenty of opportunities to be outraged at how badly public agencies are mishandling technology. Consider, for example, the rollout of the US Affordable Care Act's website, healthcare.gov. When the site launched in October 2013, it couldn't handle the initial rush of traffic. Of course, that wasn't so bad—because, as it turns out, the rest of the software running behind the site didn't actually work either.

While healthcare.gov worked better by the end of 2013, the state of Oregon's health care exchange, coveroregon.com (now moved to a new domain), has been in worse shape. Workers have had to help people fill out paperwork on paper to sign up for expanded health care coverage.

It's not always easy to predict how popular a website will be when it launches, but it's also not as if no one in Oregon has ever built a website before. (In fact, it's fairly easy to get a good estimate of the number of people who might visit coveroregon.com, based on the number of people who are eligible to buy insurance from the exchange.)

Have these projects failed? That depends on your definition of software project failure. If it's acceptable that these systems will eventually perform the needed functions and will eventually allow users to do what they want to, then success or failure must be judged on the long term. If, however, you define success as "delivering the required features at the desired level of quality at or before the promised delivery date and not exceeding the expected cost", then these projects are failures. (Several studies over the past 30 years suggest that up to 70% of software projects in general do not meet these success criteria.)

You're probably not outraged by these two failures, even if you pay taxes in the US and especially Oregon. In fact, you probably don't find them surprising. Even if you're generally favorable to the idea of state and federal governments, you probably believe that government IT and software projects often end up expensive failures.

They often do. This brings to mind Tolstoy's Anna Karenina principle: successful software projects are all alike. Failed software development projects all fail in their own ways. Yet there are some similar threads.

Many Customers with Divergent Goals

Many government technology projects have to serve multiple masters. The accounting department has the final responsibility to write checks to pay for contractors and software and consultants. Perhaps a state or federal IT department has to direct all software deployments, or at least review all requirements documents. Any software which affects a big political program (healthcare.gov in general and the Cover Oregon failure are two timely examples) has political considerations which may put it under the purview of the chief executive's office or a commission appointed by a governor or president. There's likely a procurement office—whether attached to the accounting department or not—that puts together the contract and oversees it, after running it through the legal department, of course.

Oh, and there are eventually some rank-and-file users of the government ERP software or the municipal accounting software or the building permit software or the specially-made government antivirus software. Somehow their interests get represented too.

Of all of those groups with an interest in figuring out what to build, who will build it, how much it should cost, when it needs to be done, and what it should do, who's in charge? When one or more of those groups disagree on some facet of the software or its schedule or its cost or its quality, which group wins? Who decides? Who talks to the people who are actually building or deploying the software, and who do they talk to when they have questions? What happens when the assumptions of one group don't match the assumptions of another group and no one has ever said anything about it, because, of course, they're assumptions that everyone takes to be true? (Why, of course in government accounting software you need a fiscal year that starts in September—or June or April—and not January. Everyone knows that.)

Procurement Processes Assume Construction Schedules

Governments have been building things for a long time: aqueducts, granaries, roads, bridges, monuments, even palaces and cathedrals. Millennia of building experience means that every educated adult knows how to build something. First you gather the requirements. Then you design it. Then you build it.

Sure, buildings and bridges fall down sometimes and sewer and bypass projects go way over budget and schedule and, yes, sports arenas never earn back what they cost, but everyone knows that gather, design, and build is the only true way to build things, even if it doesn't always work.

That's how you build software, right?

Writing software is just like a bridge, except you don't have to excavate millions of tons of dirt or pour millions of tons of concrete and you're not crossing hundreds of feet of rapidly flowing water and working around billions of dollars of shipping traffic. Also you're not blocking one lane of a busy freeway for several hours every day for several weeks while you pour pavement and concrete and stripe and restripe lanes. Other than that—other than the fact that software is ephemeral and not a physical object—writing software is exactly like a construction project.

Unlike a bridge, software can change until the last moment. In fact, it's altogether common for a programmer to realize that the approach he or she took could be more efficient or more effective or simpler with this one little change. It's all too common that, near the end of a project, a user will make an offhanded remark about what the software really should do and only then will people realize that there's a feature missing.

Programmers can implement these missing features. Sometimes they can do it quickly. This is an advantage of software. Malleability in the face of changing requirements (or at least the changing understanding of requirements) is essential to the nature of software. That's why software exists. (For a full treatment of this idea, see the work of Charles Babbage and, later, Alan Turing to build universal computing machines.)

Government procurement offices generally seem oblivious to this reality of software. That's not so rare; even software companies seem oblivious to it. Software is malleable, sure, but software has an even more ineffable quality. Because the purpose of software, especially in a government, is to automate manual processes and to create new things which did not exist before, it's not At all obvious what those new things should do before real users attempt to use it. In other words, everyone knows what a bridge should do: it should get people and goods from one side of a river to the other. Not everyone knows what public sector software should do when it comes to electronic health care records, for example. Not everyone immediately knows what software in a government call center for health insurance exchange signups should do because that software has never existed, and it's certainly not been something studied for thousands of years.

Software is very, very difficult to plan in detail from the start unless you've written something very much like it before—and if you've already written something very much like it, why not just reuse what you have as much as possible?

Also unlike a bridge: you can (and should) test software from the very start of the project. If you plan correctly, you can verify that the software is behaving as you intend—as it must behave—from the first few lines of code. Unlike a bridge, you can test the system from extant end to extant end as you build it. Saving the testing process until the end means that when you blow past the schedule, you'll skimp on testing and you still won't have time to fix things that you did wrong.

Yet government procurement policies tend to insist on treating software development like construction projects, even knowing that the latter tend to blow past deadlines and budget guidelines. That leads to another problem.

No Consequences For Failure

Because public sector IT projects seem like big public works projects moving millions of tons of dirt and concrete and steel, obviously they have to be done by big companies with thousands of employees and big contracts. Right? Right?

No Consequences for Contractor Failures

In a dispute between a couple of dozen people at the Oregon Department of Health and, let's say, Oracle, Accenture, IBM, or SAP, who's going to win? (Answer: anyone but the Oregon Department of Health.) Given the consulting arm of a big company with a fat government contract and a small army of lawyers writing the consultant's side of the contract (that army of lawyers has a full time devotion to meeting government procurement standards to the letter, even if those standards make no sense in the context of software), who is the contract going to favor?

Oracle or Accenture or IBM Global Services or SAP will negotiate a good contract with the government and will not lose money. The software will be done when it's done and it will cost what it will cost and if that deadline or final bill doesn't look like what the procurement department thought it negotiated, there are no consequences for that failure. The state of Oregon may sue Oracle for breach of contract, but it will have a difficult time proving its case. Oracle isn't going to go out if its way to go above and beyond its contractual duties because there's no penalty for failure. Besides, it's easy enough to pin the failure on the flawed procurement process or the lack of leadership from the customer's side. (Oracle will argue that as well.)

No Consequences for Employee Failures

It's popular to pick on government employees, but that's an unfortunate and lazy trend. What's more disturbing is the lack of incentives for employees to perform good work. The incentives are all wrong.

Unlike private sector developers and IT employees, public employees often have a public employee union. These unions have the laudable goals of protecting workers from exploitation, but they often also have the effect of making it difficult to fire or replace workers who are incompetent. They also tend to produce management structures where promotions are based on time spent in the department (or government at any level) and not on any particular skill or interest. Even worse, there are few incentives to succeed; pay raises and bonuses are often not based on performance but the ability to continue in a job for many years.

This is not to say that the world of enterprise software development is always better; it's lazy to imagine that the world outside of public sector IT is full of genius youths playing foosball in the Googles and Facebooks of the world. Large companies (and even some of the consulting organizations helping government software projects fail) are often full of negative producing workers. Yet even there, these workers eventually have the possibility of losing their jobs due to layoffs or staff reductions or sheer incompetence.

Government Software Projects are Tied to Political Deadlines

While "It'll be done when it's done" is often an excuse to avoid committing to a delivery date, sometimes it's also an acknowledgement that doing things right takes time. It makes sense that you can't build a skyscraper in a week, but how hard can it be to change the font on a web page? That depends.

Software is ephemeral, but software has its own rules. Programming is creating a vocabulary to describe a problem and its solution, and if you get that vocabulary wrong or if you discover that it's incomplete, you may have to rethink quite a few assumptions. Things that may be easy to describe in one vocabulary may be quite difficult in another.

It's difficult to predict when software will be done in the sense that it does everything you've ever wanted, but it is possible to deliver working software that does a few things completely and improves every week or month or quarter. Yet it's still treated like a bridge, and that bridge has to cross the river and go under an underpass and not disturb an endangered snail's breeding area before the governor or mayor or president leaves office or launches his or her re-election bid or gets recalled.

Sometimes it doesn't even matter what the software is or what it should do; you have six months to create and debug and test and deploy something that no one has ever done before and you have twelve individual bosses to answer to and no one can agree on a single set of requirements, but the contracts were already signed. Figure it out as you go. Good luck getting it right.

The reality is that these deadlines are for political announcements. The consequences of missing the deadline are less than the consequences of not making the announcement date. Besides, you know that that municipal traffic planning software project will go on for another six to nine months anyway. The deadline was always a polite fiction even before it was announced.

In another sad twist, it's possible to get yourself assigned to a project that will last the length of one or mayor's term. When that person leaves office, expect his or her successor to cancel the project. Don't worry; you'll have several months to attach yourself to the next project that'll get you four years closer to your pension.

Can IT Projects Be Saved?

Why do IT projects fail? Because they're never given the chance to succeed. The history of failed software projects examples shows this to be common.

Is this the way of the world? Is it impossible for a state or a federal department to get good software or deploy IT infrastructure on time and under budget? In a word, no. If a project manager, administrator, or other executive in charge of hiring, firing, and signing checks really wanted to handle technology well, and if that person had the power to make things happen, it's possible that government software projects could succeed.

You'd still have to deal with the people problem, especially for internal software projects. Could you hire or recruit the right people? Could you overcome the lack of incentives for success? Could you prevent political interference and help a project survive past the point at which its sponsors leave public service?

Government software projects will never succeed as long as they're treated like construction projects, run by multiple competing departments, tied to political timelines, and performed by contractors who have no incentives to succeed and who face no consequences for failure. Fix those problems, and it's possible to get better software. Yet that is the subject for another article.