What an integration layer really costs: a TMS backend counted line by line
Most cost-of-software posts are written about someone else's system, from the outside, by someone who never had to keep it running. This one is written from the inside.
We're taking apart a transport management system — a TMS. Orders arrive from SAP. A driver and a vehicle get matched against requirements for both the vehicle and the stops. A trip runs through waypoints with time windows, checklists at each stop, and handover reports with damage records when a vehicle goes out and comes back. On top of that: a GPS stream into partitioned tables, tracking for trips, places and counterparties, WiFi-based presence detection, a mobile app for drivers, and a public zone where the customer can see where their delivery is.
Fifteen modules. Three nodes behind a load balancer, RabbitMQ and Kafka three nodes each, Redis, PostgreSQL, a separate identity server.
Why it was built at all. The TMS used to be bought as a service from outside vendors, and the risk grew. When a core business process runs on an external service, risk converts directly into money: downtime you can't influence, fix timelines set by someone else's priorities, dependency on a single supplier when terms change. While the risk is small, buying beats building. When it grows, the arithmetic flips — and building your own becomes a way to get control back, not a way to save money.
So it was brought in-house.
And here's the thing worth saying immediately, because it breaks the usual frame of every tech-debt discussion. This system is sixteen months old. First commit April 2025, 931 commits so far.
This is not inherited 2000s legacy whose author left years ago. It's a young greenfield project. 175,000 lines of backend in sixteen months — and it has already accumulated an infrastructure layer that now has to be dismantled.
Disclosure. I'm the architect and developer of the backend under analysis, and also the author of the ecosystem it relies on. So what follows isn't a critique of someone else's decisions — it's my own bill. And that's exactly why measurements are kept separate from conclusions and the numbers are broken out line by line: argue with the lines, not with my good faith.
Scope: backend only. The web frontend is built by a separate team, the mobile app by a separate developer. Neither appears in these numbers by a single line. Everything below accumulated inside one server-side codebase.
Part of the redb / redb.Route series — recent posts first:
About the team, because it changes how you read the numbers
The backend wasn't written solo, and it wasn't written in a vacuum. The team had:
- a systems analyst — requirements arrived analysed, not paraphrased in chat;
- a QA engineer — what got built was verified by someone other than its author;
- a project manager — the full classic role: budget, deadlines, priorities, work order, risk assessment.
This matters in two ways.
First: the overhead counted below is not the result of a broken process. The easiest explanation for 105,000 lines of infrastructure is "they had no analysis, no testing, nobody minding the shop." That explanation doesn't work here. All of it was in place. The infrastructure layer didn't grow instead of process — it grew alongside it.
Second: every hour in the tables below is development only. Analyst, QA and management time isn't in there by a single hour, though the project wouldn't have shipped without them. Total cost of ownership is higher than my numbers, not lower.
What's actually in there
Counted from the repository, vendor libraries excluded.
| Files | Lines | |
|---|---|---|
| C# — application layer, own engine, container | 851 | 121,466 |
| SQL — schema, migrations, reporting, reference data | 28 | 17,341 |
| synapse-XML — WSO2 MI artefacts | 206 | 14,564 |
| OpenAPI contracts | 20 | 5,235 |
| Razor — operator dashboard and host boilerplate | 54 | 5,224 |
| Java — bus connector and class mediators | 18 | 4,099 |
| Own JS and CSS | 26 | 3,999 |
| Maven — build descriptors for bus modules | 7 | 3,235 |
| Total written by humans | 1,210 | ≈ 175,200 |
| On top: generated migration artefacts | 4 | 17,407 |
Those last two rows need explaining, because they usually get shovelled into one pile and the difference between them is the whole point.
Generated migration artefacts — 17,407 lines across four files. No human wrote them and no human reads them: they're model snapshots the ORM tooling produces by itself. But they live in version control, they show up in pull requests, and they conflict on merge. They're excluded from the "written" count deliberately — a separate row is more honest. We'll come back to them.
Build descriptors for the bus — 3,235 lines across seven modules, averaging 460 lines each. That's not routing logic, it's Maven configuration: how to package a module into an installable artefact. For comparison, the .NET build descriptor in the same system is 30 lines. A fifteen-fold difference in a file that does nothing except explain how to build everything else.
Six languages describing one system, nearly all with their own build pipeline, their own debugging story, their own release cycle.
Layer one: WSO2 MI and synapse-XML
Let's start with the layer that usually escapes cost accounting entirely, because people habitually file it under "configuration."
That's wrong. Synapse-XML is code. Declarative, narrow, limited in expressive power — but code: it has branching, error handling, transformations, external calls, and message flow control. The difference from C# isn't whether it's code. It's the length of the feedback loop: artefacts reference each other by name, expressions over the message evaluate at runtime, and a meaningful share of mistakes surface not at build time but in production.
One thing I should correct up front, because it's a common misconception — WSO2 MI does have a debugger. You can set breakpoints on mediators in Integration Studio and inspect the message context. "You debug it with logs" is false, and I wrote exactly that in the first edition of this post until someone corrected me. The tooling exists. What's expensive here is something else.
The deployment unit is not a file but a whole module: projects are packaged by Maven into a Carbon Application, a .car archive, and that's what ships to the bus. You can see it in the repo as <packaging>car</packaging> across six build descriptors.
195 artefacts, and what they cost
A model with explicit assumptions. Rates are per artefact, covering writing plus debugging — not typing speed. Adjust for your reality; the table is broken out so you can edit it row by row.
| Artefact type | Count | Hours each | Hours |
|---|---|---|---|
| Sequences | 96 | 4–8 | 384–768 |
| Endpoints | 42 | 2–4 | 84–168 |
| Proxy services | 10 | 6–12 | 60–120 |
| APIs (routing + definition) | 6 | 8–16 | 48–96 |
| Scheduled tasks | 4 | 3–6 | 12–24 |
| Local entries, templates | 12 | 1–2 | 12–24 |
| Forking the Kafka connector: reading foreign code, fixing, extending | 1 | 60–120 | 60–120 |
| Java class mediators (1,670 lines) | 5 | 25–50 | 125–250 |
| Deploying and configuring the bus, clustering, training the team | — | 300–500 | 300–500 |
| One-off total | 904–1,708 |
The Java sitting underneath
Artefacts aren't the end of it. Next to them sit 4,099 lines of Java across 18 files, and they split into two stories.
A fork of the official Kafka connector — 2,429 lines across 12 files. This one deserves telling in full, because it's more typical than it looks.
The vendor has a Kafka connector. It's in their repository, it installs normally, it's part of the box. The problem is that it didn't work properly: it had bugs.
Then comes the fork in the road familiar to anyone living on someone else's platform — wait for the vendor, or fix it yourself. Waiting wasn't an option, so the project forked the connector and fixed it in-house.
The price looks low in the moment: reading foreign code and patching it is cheaper than writing a transport from scratch. But the price arrives later and never leaves.
A fork takes you off the upgrade path. The version is pinned to a snapshot — an unpublished state of the vendor's branch. Every platform upgrade now means: take the new connector, re-apply your patches, verify nothing drifted. Or don't upgrade, and accumulate divergence.
And you now maintain Java you didn't design. Nearly two and a half thousand lines of someone else's architecture to re-learn every time something breaks.
This isn't about a bad vendor. It's a property of the model: when the platform is someone else's, any defect in it becomes your permanent line item — either you wait, or you fork and pay for the fork forever.
Class mediators — 1,670 lines. These are Java classes wired directly into XML sequences wherever the XML ran out of expressive power. Their list describes the boundaries of the language better than any argument:
| Mediator | Lines | What it does |
|---|---|---|
| Image scaling | 433 | Resizing and thumbnails — binary data handling |
| OAuth token cache | 402 | State that outlives a single request |
| Unique key pool | 401 | Thread-safe block allocation of identifiers with auto-refill |
| Thread pool diagnostics | 250 | Runtime introspection |
| Token validation | 183 | Custom security logic |
Every line is a place where a developer hit "XML can't do that" and switched languages. Binary data, cross-request state, concurrency, runtime introspection, non-trivial security — all outside what a declarative routing format was designed for.
The third row is worth a pause. A block-allocating identifier pool with auto-refill is the classic answer to "hitting the database for every key is too expensive." Here it had to be implemented as a Java mediator living inside the integration bus. The same mechanism shows up elsewhere in this story — but that's the next post.
Two real failure modes
Not hypotheticals. These are the ones that actually cost nights.
Sender and consumer on one connection — mutual deadlock.
The mechanics: request-reply means the sender, having posted a message, waits for the answer. If a consumer is sitting on the same RabbitMQ connection, that wait occupies the shared channel and everything stops. The exchange doesn't proceed; the connection hangs.
What it looks like from outside is worse: consumers silently stop reading the queue. They don't crash, don't log an error, don't raise an alert. You find out from the consequences — messages piling up.
The cure is separate connections: distinct factories for sending and receiving, identical parameters, declared separately. You can see it directly in the system's configuration — every factory has separate sender and listener blocks with identical settings. Duplication that exists purely to work around platform behaviour.
Temporary RPC queues don't survive a cluster rebuild. When the container nodes get rebuilt, the temporary reply queues on the bus-side producers don't come back. The usual fix is restarting the bus node — by hand.
Layer two: EF Core
Here's a picture familiar to anyone who has run a project on Entity Framework. With one caveat worth holding onto for this whole section: everything below accumulated in sixteen months, not over years.
| Count | |
|---|---|
DbSet declarations |
51 |
| Lines of model configuration in one method | ~1,500 |
| Files with eager-loading chains | 70 (40 + 30 nested) |
Files calling SaveChanges |
40 |
| ORM models with navigation properties | 54 |
| SQL views assembling object graphs into JSON | 5, of which one is used, four are dead code |
| Generic repository + specifications | yes |
| Migration files | 3 |
And now the question all that counting was for: what of this actually requires an ORM?
The answer from the code: two partitioned tables, five views, and three raw SQL calls. Everything else is servicing the ORM itself.
Fifteen hundred lines of model description and a migration system exist to serve two partitioned entities and three queries. Partitions and raw SQL require no ORM at all — they work through a direct connection.
The five views that were a workaround
The five SQL views deserve their own note. They assemble object graphs into JSON — that is, they do on the database side the work the ORM couldn't do acceptably. Four of the five are used by nobody: dead code that nobody dared delete, because it's unclear who might be looking at it.
| Work | Hours |
|---|---|
| Modelling and configuration: 54 models, ~1,500 lines of mapping | 120–220 |
| Eager-loading chains and manual graph assembly across 70 files | 180–340 |
| Generic repository + specifications | 40–80 |
| Five SQL views as a workaround for graph loading | 40–80 |
| Migrations, snapshots, resolving their merge conflicts | 60–100 |
| One-off total | 440–820 |
Layer three: what we wrote ourselves
Now the most interesting thing in this system, and the reason it's worth taking apart at all.
Running alongside WSO2 MI was a home-grown integration engine. Not a wrapper, not helpers — a full routing framework in the spirit of Apache Camel, written for .NET.
| Component | Size |
|---|---|
| Routing engine | 173 files, 27,529 lines |
| Module container with dashboard and REST API | 54 files, 11,117 lines |
| Total | ≈ 38,600 lines of infrastructure code |
Inside the engine:
- eight transports — Kafka, RabbitMQ, Redis, Cron, Timer, SEDA, Direct, Validator, each with its own consumer, producer and endpoint;
- 43 abstraction files —
IComponent,IConsumer,IProducer,IEndpoint,IProducerTemplate: Camel's vocabulary, verbatim; - 41 files of expression language — expressions and predicates, compiled rather than interpreted;
- 19 route definition files — i.e. its own DSL;
- 16 processors and 4 files of transaction handling.
And the container isn't "a host that starts modules." By inventory it has module lifecycle management, hot-reload, leader election, per-route metrics, a REST API and a web dashboard.
| Work | Hours |
|---|---|
| Routing engine: transports, DSL, expressions, processors | 1,200–2,100 |
| Module container: lifecycle, hot-reload, clustering, dashboard, API | 740–1,300 |
| One-off total | 1,940–3,400 |
That's 1.2–2 person-years, and the single most expensive line in the entire estimate.
The obvious question is why a system that already had an enterprise bus needed a second integration layer. The answer is in the previous section: because the bus covered the declarative part and stopped there. Everything past a branch went into Java, every missing transport went into a fork, every stateful thing went into a mediator.
Add up the workarounds and you find you've written half an engine anyway — just smeared across someone else's extension points.
Layer four: the identity server
Authentication runs on WSO2 Identity Server. It brings four databases that have to be created, seeded, versioned, backed up and upgraded along with the product.
227 service tables. For scale: the business schema of this TMS — trips, orders, drivers, vehicles, places, counterparties, tracking, everything — is 53 tables.
Four to one in favour of authentication infrastructure.
The bus talks to it through nine confirmed integration points: token issue, refresh, revoke, JWT decode and validation, password recovery and reset, user binding, group sync, adding a user to a group.
And one detail worth stopping on. Among the confirmed flows is service-account token retrieval for two channels — mobile and the control panel.
A service account is machine-to-machine. No user, no browser, no redirect. One part of the system proving to another that it is what it says it is.
The path of that request, per the architecture documentation, goes through several layers: client → bus → identity server → backend.
There's also a token cache implemented as a Java mediator inside the bus — 400 lines that wouldn't exist if the traffic were lower. The first validation goes to the identity server, which issued the token; after that, validation happens bus-side without touching it.
| Work | Hours |
|---|---|
| Deployment, four databases, backup and upgrade procedures | 150–300 |
| Nine integration points on the bus side | 120–250 |
| Token cache as a Java mediator | 60–120 |
| User and group synchronisation | 120–180 |
| One-off total | 450–850 |
What it costs every year
One-off is half the story. The other half arrives annually.
Two stacks instead of one
The application layer is .NET, the integration layer is Java with Maven. That's two build pipelines, two dependency sets, two upgrade cycles and two areas of expertise inside one team.
In practice it means "change how this integration behaves" needs a person who knows both — or a handoff between two people, with losses at the seam.
| Recurring work | Hours/year |
|---|---|
| Maintaining synapse-XML artefacts | 200–350 |
| Keeping the connector fork in sync with upstream | 60–120 |
| Java mediators: separate pipeline, separate competence | 80–150 |
| ORM: migrations, snapshot conflicts, query tuning | 150–300 |
| Identity server: upgrades, four databases, certificates | 120–250 |
| Diagnosing silent failures (see the two failure modes above) | 140–230 |
| Annual total | 750–1,400 |
The summary
| Layer | One-off hours |
|---|---|
| WSO2 MI: 195 artefacts, Java, fork, deployment | 904–1,708 |
| EF Core: models, mapping, chains, migrations | 440–820 |
| Own engine and container | 1,940–3,400 |
| Identity server: databases, integration, caching | 450–850 |
| Total | 3,734–6,778 |
Round it: 3,700–6,800 hours one-off.
That's 2.2–4 person-years — before a single line of business logic.
Annually: 750–1,400 hours, i.e. 0.45–0.85 of a permanent headcount, purely to keep the layer running.
The rate to apply is the fully loaded hourly cost to the company: salary, taxes, workspace, management, holidays and idle time. Not the offer number — that's the usual mistake, and it understates the result two- or threefold.
Infrastructure against application code
One more cut worth taking. Out of the 175,000 lines written by humans:
| What it is | Lines |
|---|---|
| Own routing engine and module container | 38,646 |
| SQL: schema, migrations, reporting, reference data | 17,341 |
| WSO2 MI artefacts plus build descriptors | 17,799 |
| Contracts, dashboard, own JS and CSS | 14,458 |
| ORM configuration, models, repositories, workaround views | 12,672 |
| Java: connector fork and class mediators | 4,099 |
| Infrastructure total | ≈ 105,000 |
| Application code: rules, calculations, processing, module routes | ≈ 70,000 |
Rules, calculations, validation, processing — the part anyone actually pays for — is about forty percent of what was written. The rest is data transport, configuration, plumbing and manoeuvres around tool limitations.
Roughly three to two against the application code: for every two lines doing the job, three lines exist to make doing the job possible. In my experience that's not an anomaly — it's the norm for enterprise systems on a ready-made integration platform. And those three lines are what this post is about.
Worth repeating: this ratio is inside a single backend. Web and mobile are other teams and aren't in the numbers.
What these numbers don't cover
The section without which the estimate would be dishonest.
This is a model, not a measurement. Nobody kept parallel time tracking per artefact. Hours-per-item is expert judgement and every row can be disputed separately. That's exactly why the tables are broken out rather than given as a total.
Some of this work would have been necessary anyway. Business logic — rules, calculations, validation — doesn't depend on the stack it's written in. It isn't in the numbers above and shouldn't be: only the infrastructure layer is counted.
Development hours only. Analysis, testing and project management aren't in the tables, though the team had all three. Total cost of ownership is higher than my figures — by how much depends on how your roles are split, so I'm not inventing a multiplier.
ESBs have real strengths. A ready-made bus gives you out-of-the-box adapters you'd otherwise write by hand, one uniform administration point, and a model a team already fluent in it understands. If you have that competence in-house, some of my estimates are too high for you.
The organisation could have bought some of this. Then the saving converts into licences and into not developing your own, rather than into person-hours.
And most importantly: the system works. It holds production, serves the SAP integration and the mobile clients — and, again, it's sixteen months old. In sixteen months the team stood up from scratch what used to be bought as a service. This isn't a post-mortem of a failure. It's an accounting of the price paid for a specific set of architectural decisions. The price may well be justified; the question is whether you know it.
What's next
Everything above is the "before" state. The system is leaving it: the integration bus, the identity server and the ORM are all being removed.
Three of the four layers get dismantled. The one that stays is the one that cost more than the other three combined.
The next post is about that transition — a module-for-module comparison, the same zone, the same job, bus artefacts and a Java mediator on the left, a rewritten module on the right. And the headline result of that comparison was the opposite of what you'd expect from a migration post: there is now more code, not less. Why that's the correct outcome is the next one.
If you have a similar stack and different numbers, post yours in the comments. I'm happy to revise mine — converging on a realistic model is more interesting than defending the first draft.
Sources and releases: github.com/redbase-app. About the redb store: redbase.app.