Fixing government IT procurement

Note: this post discusses the NZ government's IT procurement, but these concepts apply equally well to IT procurement in governments of other countries, and in organisations and businesses worldwide...

Today in New Zealand, local and central government procurement of IT services and software uses a model which, if you ask many of us in the local IT industry, produces unsatisfactory results. Despite many years of tweaks and refinements, the current RFP model and various vendor panels have not made measurable improvements. Many of us in the domestic industry believe a more drastic change is required.

The purpose of this post is to explain the problems we see and to sketch out a different model I believe would be a substantial improvement over what we currently have.

Some issues with the status quo

The current system has some obvious issues, for example the tendency to award procurement contracts, particularly for large bespoke software development projects, to the lowest bidder. This has seldom, if ever, ended well.

Other issues with the current system seem a bit more subtle. The one which strikes me as most problematic is this:

Most government tenders specify more than just a set of business requirements to be filled by a software implementation. By exceeding this brief, they're presupposing the solution. They almost always stipulate functional requirements, including compatibility requirements with existing systems employed, for integration purposes.

What seems to go largely unnoticed is that these functional requirements revolve around proprietary protocols, formats, or schema, dictated or unilaterally controlled by existing suppliers. These are local monopolies granted to existing suppliers which, in turn, substantially constrain government agencies' ability to seek competitive solutions from the broadest range of suppliers, and are particularly prone to excluding smaller domestic service providers and suppliers while favouring incumbents.

False attribution

These limited monopolies, which include the substantial majority of software solutions and services used in governments today because they're almost all proprietary, have the effect of increasing the perceived cost of alternative solutions which can compete on capability but are not fully compatible with these proprietary incumbents, which, in turn, acts as a barrier to adoption.

Government procurement officers mistakenly attribute the cost penalty of being "incompatible" to the incoming supplier rather than attributing the cost of migration to an incompatible solution to the business practices of the incumbent which was previously granted a monopoly on that particular service. This is a form of "lock-in" which seems almost entirely unacknowledged.

It is also extremely prone to abuse by powerful suppliers (the largest of whom individually have market capitalisation in excess of New Zealand's GDP, effectively rendering them "above the law" here) who actively exploit their advantage through monopoly rents, and who quite deliberately use the perceived high cost of switching to other suppliers to maintain their lucrative positions.

Software development is an immature industry. Many types of software that most of us use every day didn't exist a decade ago, some not even a couple years ago. As a result, many software suppliers assert that they need to create proprietary (supplier-defined and unilaterally changeable) formats, protocols, and schemas to allow themselves to pioneer these new and (sometimes) useful software applications. Examples of this would include the emerging "Internet of Things", some areas of biotechnology and gene manipulation, Augmented Reality, and a bunch of other specialist niches to which mainstream software users are yet to be exposed.

The freedom to add to, or change, formats, schema, and protocols is a reasonable need when pioneering brand new software domains, where developers are making up whole new classes of software as they go. But where a software domain has matured to the point where there are multiple competing solutions, and the problems of that domain are well understood, those proprietary formats cease to be justifiable, and instead become a barrier to entry for would-be competition.

In order to compete against the market leader, competitors typically have to offer "compatibility" with the market leader's solution. This is either done via

1. licensing agreements with the leader (which are expensive and risky - the market leader has a monopoly of their proprietary standards, and can therefore charge what they like, and they also have an incentive to be less than cooperative), or by
2. reverse engineering - a somewhat less expensive route, but very risky, because software can be extremely complex, and the market leader is often known to change things capriciously if a fast-follower catches up too quickly. 

These proprietary de facto standards are a stark contrast to open standards, which should be adopted in any established software domains. To be useful, an open standard, must be developed by multi-supplier groups, with a well-defined change process, be unencumbered by patents, royalty free to anyone wanting to implement them, and ideally have objective compliance vetting systems and multiple independent software implementations. 

Proprietary standards create huge risks for competition. The market leader retains dominance by ensuring that would-be competitors cannot offer real interoperability with their solution. They do this by constantly shifting the goal posts. They have full control over their proprietary standards and can arbitrarily alter them at their whim, or use threats of software patent litigation, or in some jurisdictions (with rampant corporate influence like the US), push through government policy to make reverse engineering legally nebulous. These risks result in lower investment in competing technologies, and long periods of extreme dominance. This pattern has resulted in a handful of dominant software companies attaining a scale seen previously in human history only during times of colonial excess which has ruined entire cultures.

Mutually incompatible repetitive inefficiency at scale

New Zealand has 67 regional authorities. Each of these, though being geographically distributed, has roughly the same remit and the same set of problems for which it requires automated systems. There are likely to be in excess of a hundred different areas in which the typical territorial authority deploys a software solution.

Given the similarities between authority obligations, one might assume that the councils would work collectively, collaboratively, and methodically to catalogue their software requirements, perhaps with the aid of private sector software requirements analysts. They could publish a set of well defined standards for private sector suppliers to meet, using their substantial combined buying power to influence the market's offerings, and achieve a consistent set of applications across all of those authorities, with many flow-on benefits like lower costs of training, support, and procurement thanks to consistency and economies of scale.

Sadly, if one assumed this, one would be painfully disappointed when shown what actually happens. There is almost no collaboration. Procurement is ad hoc, uncoordinated, and authorities actually pay IT suppliers to lock them into proprietary software solutions that are often incompatible with those used by most other neighbouring councils.

This is a classic "divide and conquer" model where each territorial authority pays license fees for each application, when these amounts in aggregate across all 67 authorities could probably have funded the development of these applications from scratch each year. Multiply this incredibly wasteful process by the hundred or more software implementations at each authority to get an idea of the scale of the problem.

To provide a concrete example: the amalgamation of IT systems procured by seven former city councils - with scores of mutually incompatible systems - into a single set of coherent systems for the new Auckland City "super council", has already cost the taxpayer well in excess of $1 billion dollars, with precious little to show for it. That's a cost of about $250 per person in New Zealand so far, and there's no end in sight. That's just to merge systems that all do the same things... but in mutually incompatible ways. That, my friends, is running to stand still and it's an inexcusable waste. It's an illustration of the bureaucratic inefficiency we are currently suffering all around the country (and we're by no means alone here in New Zealand).

Similar inefficiencies pervade the procurement of software by our 20 District Health Boards and many state owned entities like our Universities and Crown Research Institutions.

Unfortunately central government, with significant scale and overlap between agencies and departments, and their software requirements, is little better. So let's just say there is plenty of room for improvement.

A modest proposal

Having described a problem so pervasive in the status quo of government software procurement - which has led to a situation I have characterised, without hyperbole, as a hostage situation - I would like to offer a proposal for fixing this very precarious situation: a better procurement model for New Zealand government.

A quick win: mandate open standards compliance

We could stop digging ourselves an ever deeper hole of technical debt very quickly, with the stroke of a pen, in fact. 

Our government could mandate that all software procured by government in established software domains (as determined by a panel of private sector experts with diverse pecuniary interests, chosen, not by government, but by the private sector) must comply with relevant open standards (which, in turn, would need to be compliant with the definition I provided, to ensure they are not "fauxpen" standards - proprietary standards masquerading as open).

This would have the immediate effect of putting dominant suppliers pushing proprietary standards on notice, and provide certainty to all suppliers that if they invested in building software that is compliant with the relevant open standards, they would face far fewer non-technical barriers to entry into the government procurement market. It would also break the proprietary lock-in hostage situation we currently have.

Government procurement costs with specific suppliers are not generally available (due to the hugely over-used "commercial sensitivity" excuse), so I can only guess here, but those savings could approach hundreds of millions of dollars - perhaps even a billion - per year. And all of this without any loss of capability. If incumbent suppliers are able to meet these criteria, and compete successfully on the level playing field this creates, there would not even be much drastic change in the short term.

The UK took this approach in 2014. Unfortunately, their commitment has flagged somewhat due to withering condemnation from incumbent (and very politically influential) proprietary suppliers, and massive pressure from their legions of lobbyists. I trust that the NZ government, with its reputation for ethical practice, would be far more robust in maintaining its commitment in the face of commercial pressures from almost exclusively overseas multinational corporations. This could be one of those opportunities for New Zealand government to display real global leadership and vision. 

Next steps

I recommend that representatives of a critical mass of regional authorities (from a diverse range of authorities of varying sizes, small to large) work with a panel of software experts (representing a diverse range of pecuniary interests, formally engaged by government) to catalogue the functional requirements for each system currently in use.

These requirements would be converted by a combined government and industry task force into a set of open standards, ideally building on international precedent, using - and contributing to - existing standards where-ever possible to achieve the greatest possible scale and to distribute the cost of maintaining these standards.

As each application area has applicable open standards defined, a Request for Proposals would be issued. This would stipulate that any solution which could meet an objective assessment of compliance with the open standards would be considered.

Fringe benefits

Of course, this standardisation creates other potential opportunities as well. Collectively purchasing solutions delivers economies of scale. In fact, beyond collective bargaining opportunities with single private sector suppliers, our government could do something even more interesting: seed a market place in which vendors could compete around a single open source software reference implementation of each open standard. This might well be necessary in some niche areas where the private sector does not see enough profit in the traditional proprietary software model. 

In this case, the government would commission an open source software supplier or even a community of developers, to create a compliant reference implementation and make it available under a suitable open source license, as stipulated by NZGOAL's Software Extension. This would then provide a working software solution which suppliers could be engaged to implement, support, maintain, and improve on a competitive basis, because any supplier would have equal access to the code to do so. In some cases, this might even be done by teams within government. 

An additional benefit of this approach, which I think few in government fully appreciate, but is codified in the D7 Charter in which New Zealand is a participant, is that it would make good on New Zealand's promise to open source its IT systems, not only for its own benefit, but also for that of the other six D7 participating nations.

In this way, New Zealand would gain access to high-value taxpayer funded software that could be rolled out across the country. But also, with interntional collaboration and standardisation, the support and maintenance costs would shared among multiple countries. It would also allow domestic IT suppliers, both small and large, to parley their successes supporting New Zealand government users into exporting services to other government markets adopting that best-of-breed open source software pioneered right here.

Coda

The real value in software

To those concerned that a shift to open source software use in government would cost us, as a nation: we should always remember that the value of a software solution is not reflected by cost of developing that solution or the licenses fees the developer charges. These are pure costs. Software is a tool. The value of a software application should be measured in the value it brings to those who use it, not in the profit of those who wrote it.

Open Standards are necessary, but not sufficient

Also, as Peter Harrison points out in the comments, open standards shouldn't be seen as a panacea for incompatible software systems, where full practical interoperability is the ultimate goal. Instead open standards are (practically) necessary, but not sufficient, to that goal: full interoperability between applications created by different software and suppliers. What open standards do is crucial, though: they create a baseline of rules available to anyone for permissionless implementation. Businesses can use the standards specification to direct their investment in software development. They can do this with greater confidence than the otherwise fully speculative flight of fancy that is necessary to undertake reverse engineering of a different software, created by another party, who sees your efforts as a direct (possibly existential) threat, leading, inevitably to retaliation and evasive action.

Open standards compliance doesn't take you all the way to "full interoperability", but it does have the beneficial effect of providing an objective metric - namely compliance with the open standard - to circumvent the "false attribution" problem I describe above. Those suppliers who fail to implement software complying with the open standard properly can be shown to be the ones not playing fair (i.e. proper attribution of blame), rather than those whose software might be standards compliant, but still appear (from a users' perspective) to be incompatible. Also, the open standard helps to focus the problem purely on elements of incompatibility rather than succumbing to irreconcilable compatibility problems due to gross structural differences likely to exist between any two completely independent solutions in the same domain.

Update 2018-10-17: thanks to a bunch of folks who have either republished this (ITP's TechBlog Pt 1, Pt 2), recommended it to their communities (NZRise), or published commentary on it (NBR) - note: some of those links are privileged and not available to anonymous visitors.

Update 2019-12-06: some people within the European Union are concerned with what they're calling "Software Colonialism" - this is a German (English subtitles) documentary on it...