The integration tax: why commercial data never scaled, and what a marketplace changes

1. The first feed

The first commercial feed we bought was a good decision, and it cost what a good decision costs. There was a contract to negotiate: licence terms, use restrictions, how long we could keep the data, who inside the organisation was allowed to see it, what happened when the subscription lapsed. Then there was the integration, which nobody budgeted for because it looked small. The vendor delivered over an SFTP drop. We wrote a job to pull it, a parser for the vendor's file layout, a mapping from their schema into ours, a credential to rotate, a cadence to watch, and an alert for the morning the drop was empty.

One negotiation. One build. One place to check. It worked, and it still works. That is what one vendor costs, and it is not unreasonable. The problem is the word one.

2. The second feed

The second vendor was a different company with a different lawyer and a different engineering team, which meant a second contract with its own terms and a second delivery mechanism with its own habits. This one was a REST API: pagination, rate limits, a token that expired, a response shape that changed between versions. Nothing from the first build carried over. Two feeds from two places over two service mechanisms, and that was the honest description of our commercial data programme.

The second feed did not double the work. It did something worse: it introduced a second kind of work. Two schemas to reconcile into one picture. Two sets of credentials with two rotation schedules. Two failure modes that looked nothing alike. Two people who understood how the plumbing worked, neither of whom understood the other's. The compounding had begun, and it was already showing up in the places that matter: the time between deciding we needed data and the moment an analyst could use it.

3. The tenth feed

By the tenth vendor we were maintaining ten of everything. Ten agreements with ten renewal dates. Ten schemas, none of which agreed on what a timestamp or an identifier was. Ten ways to pull data: the SFTP drop and the REST API, then an S3 bucket, an emailed CSV, a portal somebody logged into on a Friday, a message queue, a vendor SDK that only ran on one operating system. The integration backlog became the real procurement calendar. We could sign for new data faster than we could land it, so we stopped signing.

That is the ceiling most Defence and commercial data teams hit, and it is not a budget ceiling. It is an integration ceiling, and the literature on integration has been describing it for twenty years. The standing argument for a hub, from the enterprise integration patterns, is arithmetic: point-to-point connections between n participants grow with the square of n, and a canonical data model is what turns the format problem linear again.¹ For a single buyer wiring N feeds the raw count is N, but every one of those N is a full one-off cost, and the moment a second programme, office or system needs the same feeds, the count heads toward M times N and the translation pairs head toward the square.

The survey data says the same thing from the other direction. In a 2025 benchmark of more than a thousand enterprise IT leaders, 95 percent said they struggle to integrate data across systems, and on average only 29 percent of an organisation's applications were connected at all.² The people who are supposed to be using the data are the ones paying the tax: data practitioners report spending about 45 percent of their time loading and cleaning data before they can do anything with it.³

The public-sector buyer has already named the problem in its own words. Soliciting a commercial data consortium in 2025, the Office of the Director of National Intelligence wrote that the intelligence community "has a commercial data acquisition duplication problem and a costly commercial data replication and storage problem."⁴ The Government Accountability Office found Defence and intelligence agencies not taking advantage of available commercial imagery because procurement was "slow and cumbersome,"⁵ and its 2026 review of commercial data in national security space found officials hesitant to buy over licensing costs, perceived use restrictions and concerns about long-term access.⁶ Read those findings together and the pattern is plain. The data exists. The hesitancy is about the work of bringing it in and keeping it.

4. What a marketplace changes

A marketplace earns its keep by taking the compounding out. It does that with three ones.

One agreement. We post what we need once, in plain language, and every order that follows runs through the same marketplace agreement. There is no tenth negotiation because there is no second one. Terms, payment and recourse are the marketplace's job, settled once, not re-litigated per vendor.

One schema. Every vendor delivers in the same normalised, rights-tagged form, so the picture an analyst sees is one picture, not ten reconciled ones. The format problem goes linear because the canonical model is the marketplace's, not ours to build.

One service we already run. Delivery is not a download, a bucket or a credential. It lands in a system we already operate, through an interface we already have. Nothing new to wire. Nothing to fetch.

The rest of the mechanics follow from the structure. The request is anonymous: vendors see what we need, never who we are, so asking does not reveal intent or budget to the whole supply side. Vendors are vetted before they can see an order, so vetting is a gate at the front, not a filter we run ourselves at the back. And the price is set by competition, because more than one vendor is answering the same posted need, rather than by whatever the tenth negotiation happened to produce.

5. Inside your own instance

The third one is where /MARKET differs from a catalogue or a broker. A catalogue stops at a file in a bucket. A broker stops at the introduction. /MARKET finishes the order: delivery lands in the buyer's own BLUESTAQ /ARQ^™ instance, the governed record store the buyer already runs, through /ARQ Interop on open standards: REST, Kafka, OpenAPI, JDBC/ODBC, SDKs and MCP. The data arrives normalised and rights-tagged, and the licence terms agreed on the marketplace are enforced after delivery by /ARQ access control, record by record, so the use-restriction anxiety the auditors describe has an answer that is architectural rather than contractual.

Everyone keeps what is theirs. The vendor keeps its data and its rights. The buyer keeps its instance, its keys and its tools, and can move the data on to an enclave or the edge under the same controls. Interop is the component of /ARQ that connects what you already run, so the marketplace is not asking anyone to adopt a new system in order to buy data; it is delivering into the one that is already there. /MARKET is one of four equal products, with /TRUTH, /ARQ and /LAUNCH, on one secure, open platform.

6. The 300th vendor

This is not a design sketch. /MARKET runs today with 300+ vetted vendors across space, air, maritime, land and cyber, and more than 10K orders have gone through it. On the live market, order to data takes under 12 minutes, as published. Every one of those orders was one agreement, one schema and one delivery into a service the buyer already ran.

Which brings us back to the arithmetic we started with. The first feed cost a contract and a build. The second cost two of everything. The tenth cost us the programme. Through the marketplace, the three-hundredth vendor costs what the first did, and the team spends its time on the data instead of the plumbing.