redb 4.0: XML routes, lazy references, unique keys and WS-Trust in one major release

redbredb.Routeredb.Tsakredb.Identity

A lot of this release was built burning the midnight oil. Feedback from people using the stack kept feeding both the thinking and the fixes: comments on articles, GitHub discussions, reports from live deployments. After a marathon stretch of work, 4.0.0 is out.

This is a major release for the whole ecosystem at once. The redb.Core store, the redb.Route integration engine, the redb.Tsak runtime and the redb.Identity OpenID server all ship under one version number: 76 packages on NuGet (up from 66 in 3.7), seven container images on GHCR, and archives for Windows and Linux. The Pro edition stays free and needs no license key across the entire 4.x line.

There is enough here that each product deserves its own deep dive, and those are coming. This post is the short version, close to a list: what's new, what was fixed on security, and what to know before upgrading. Full changelogs are at redbase.app/releases.

redb.Route: XML routes, a test kit and data formats

Ten new packages and a noticeable rework of the engine itself.

A shared vocabulary: the EIP book

Wire Tap, Content-Based Router, Splitter, Aggregator, Content Enricher, Claim Check, Idempotent Receiver: none of these names were invented for redb.Route. They all come from Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions by Gregor Hohpe and Bobby Woolf (Addison-Wesley, 2003), a catalog of 65 messaging patterns. Apache Camel, WSO2 Micro Integrator and redb.Route are all built on it, which makes it the common vocabulary of integration: once you recognize a pattern by name, you recognize it in every one of these tools. The redb.Route DSL verbs follow Camel's spelling: WireTap, Choice, Split, Aggregate, Enrich, ClaimCheck, IdempotentConsumer.

The whole catalog is free on the authors' site, with a diagram and a short explanation for every pattern: enterpriseintegrationpatterns.com. The chapters on Message Routing and Message Transformation are the place to start: per-partner formats, translators between them and a single point of entry are covered there pattern by pattern.

New packages

  • redb.Route.Xml. A route is described declaratively in .route.xml and loads into the same fluent DSL you write in C#. The whole document is parsed at load time, and every problem comes back in one list: an unknown element with a suggestion, a malformed expression, an endpoint scheme that isn't registered. The XSD is generated from the element registry, context.xml declares the context's components and beans, and a generator prints the XML back out as C#. The redb store is reachable straight from the markup through <redbGet> and <redbQuery>.
  • redb.Route.TestKit. Test a route without its brokers and without touching the route: AdviceRoute with ReplaceFrom, MockEndpoints("kafka://*"), a MockEndpoint with expectations and scripted replies, NotifyBuilder. No dependency on any test framework.
  • redb.Route.Templates. Message bodies from Scriban or Liquid templates. The output type drives escaping of substituted values, templates compile at Start(), and the sandbox blocks method calls.
  • redb.Route.DataFormats.Csv, .Protobuf, .Avro, .Yaml, plus Base64, GZip and Zip in redb.Route itself. Protobuf and Avro support Confluent framing.
  • redb.Route.JsonTransform. JSON-to-JSON transformation with JSONata on a native .NET engine.
  • redb.Route.Cache. Cache as an EIP: a Cache(key, ttl)…EndCache() scope and a cache: component, in process or over any IDistributedCache.
  • redb.Route.XPath2. XPath 2.0 for what 1.0 can't do: regular expressions, sequences, if, for, some and every.

The engine

  • One expression language. Conditions, values and ${...} templates go through one grammar and one compiler instead of three engines. A broken condition now fails Start() instead of quietly turning into a constant on live traffic. New in the language: stats(target, metric) for endpoint statistics, format(value, pattern, culture), uuid() and the % operator.
  • The bean: component. Your own code as an ordinary endpoint: the object comes from the context registry or is created with constructor DI, and URI parameters bind to its properties.
  • Interception and completion in the Apache Camel shape: Intercept, InterceptFrom, InterceptSendToEndpoint, OnCompletion.
  • Aggregation without lambdas. A library of ready-made AggregationStrategies (collect bodies into a list, concatenate, sum, min and max by expression, merge into a header or property), Enrich without a strategy, and a throttle whose limit comes from the message. Aggregate, RecipientList, DynamicRouter, IdempotentConsumer, Enrich and PollEnrich accept an expression, not just a delegate.
  • REST DSL in redb.Route.Http. Rest("/api/orders").Get("/{id}") with JSON binding, Consumes/Produces, and OpenAPI 3.0.3 served at {basePath}/openapi.json.
  • Metrics from inside a route. IExchange.Context and UseMetricsSnapshot() make the OpenTelemetry layer readable in process.
  • No more Newtonsoft.Json. jpath moved to JsonPath.Net (RFC 9535) on System.Text.Json, which leaves redb.Route with a single JSON model.

Load and hosting

  • Per-endpoint admission limits. HTTP, SOAP, AS2 and gRPC gained maxConcurrentRequests and a wait queue: an excess request gets 429 with Retry-After before the pipeline runs, and a neighboring route on the same port keeps its own budget. Rejections land in a separate Rejected counter, so shedding load never looks like failing.
  • concurrentConsumers=auto on RabbitMQ, AMQP 1.0, IBM MQ, SQS and MQTT: one consumer per processor, with a minimum of two. The default is still one, which preserves message order.
  • WebSocket and SignalR on the shared Kestrel host, next to HTTP, gRPC, SOAP and AS2: one port, handshake authentication supplied by the host, and a backplane seam for scale-out.
  • Trusted proxies on the shared host. X-Forwarded-For is walked right to left past every listed proxy, X-Forwarded-Proto restores the client's scheme, and every transport on the host sees the real address. A reader's comment on the 3.7 write-up prompted this one.

Connectors

  • Kafka: a commit on partition revocation no longer loses a record that was read but not yet processed, and seekTo applies per partition.
  • S3: the consumer no longer loses objects, and connection settings actually reach the AWS SDK.
  • Firebase: FCM multicast and topic management, copy and signed URLs in Storage, streaming downloads, and databaseId for Firestore projects with several databases.
  • Llm: system prompt caching on Anthropic with cache token counters, a fixed preamble ahead of the conversation history, HTTP/2 keep-alive so long answers survive VPNs and NAT, speech-to-text through stt://, and file downloads from Telegram.
  • SOAP: the consumer serves TLS, a route knows who called it, and the route chooses the fault code.
  • TLS without a certificate no longer opens a plaintext port anywhere: WebSocket, SignalR and TCP refuse to start in that configuration.

redb.Core(.Pro): lazy references and unique keys

Everything below behaves the same on PostgreSQL, MSSQL and SQLite, in Free and in Pro.

  • Lazy references. A reference at the depth boundary arrives as a stub carrying id, scheme and hash, and the first access to Props loads exactly that object. A reference that should stay lazy at any depth is marked virtual, with EnableLazyReferences switched on. A collection of stubs loads in one query through LoadReferencesAsync, with no N+1. Saving the parent, computing hashes and serializing never wake a stub.
  • The ValueUnique object key. A readable string (order number, SKU, external id), unique within the scheme, with upsert by that key through SaveByUniqueAsync.
  • [RedbUnique] on Props fields. Uniqueness enforced by a database index for scalars of any type, including decimal, dates and byte[]; for fields of nested classes; for a whole nested class, array or dictionary, where the key is the content of the subtree; and for collection elements within one object or across the scheme. GetByUniqueAsync is a single index probe, and a violation surfaces as one RedbUniqueViolationException on all three databases. Duplicates already sitting in existing data go into a report instead of failing startup.
  • A contract for schema upgrades. The schema module is applied at application startup. If the application's role can't change the schema, startup stops with a typed RedbSchemaOutdatedException, and the script for your DBA comes from GetUpgradeScript() or redb schema --upgrade.
  • CancellationToken across the entire async API. A cancelled SaveAsync rolls back completely, and the token is never read after the commit: "cancelled" can never mean "but actually saved".
  • Write interceptors (IRedbSaveInterceptor) in the EF style: SavingAsync can adjust the object or veto the save, SavedAsync sees the outcome, and the same pair exists for deletes. Under ChangeTracking the interceptor receives the applied diff with property paths, ready to feed your own audit journal.
  • Transaction isolation level on demand through BeginTransactionAsync(IsolationLevel), with DbErrorClassifier.IsSerializationFailure for retrying serializable transactions.
  • Maintenance: AnalyzeAsync refreshes planner statistics in one call on any database, and GetIndexStatsAsync reads index health.
  • LoadJsonAsync(id, depth) returns an object as raw JSON with no CLR type, and RedbObject.ToString() returns the same canonical JSON.
  • byte[] is stored as one BLOB instead of a row per byte, and the old layout converts itself. A scheme remembers the namespace of its type, so two Order classes from different projects can no longer adopt each other's scheme.
  • Performance: indexes on the _values FK columns, faster ChangeTracking, SQLite hashes stored as BLOB(16), and an index on _value_string on MSSQL too.
  • Fixes that affected 3.x: saving a parent whose reference was set by id only overwrote the referenced object; DateOnly on existing databases read back as 0001-01-01; MSSQL cut nested references in LINQ results at depth 1 where PostgreSQL and SQLite used 10.

redb.Tsak: a cluster on database keys and modules from XML

  • The cluster rests on database unique keys. Locks, nodes, groups and module assignments are backed by redb.Core partial unique indexes. Two nodes starting on an empty database no longer create two cluster roots: the node that loses the race adopts the winner. Assignment writes carry the leader's epoch, so a stale leader can't place one module on two nodes. An existing database backfills its keys on the first start.
  • Modules from XML. A .tpkg can carry nothing but .route.xml and context.xml, with no assembly at all, and such a package hot-reloads just like a C# module. An XML route in the worker sees every markup element from the shared layer: <cache>, <rest>, <redbGet>.
  • The dashboard tells shedding from silence. A Shedding Routes panel shows routes that hit their admission limit, and endpoints now show Rejected and Bytes Out. Page auto-refresh survives a transient error, and monitoring charts no longer freeze under the cursor.
  • About page: dashboard and node versions, uptime, and for admins the list of redb assemblies actually loaded.
  • Trusted proxies: Tsak:Http:TrustedProxies takes addresses and CIDR networks, after which the API key throttle and module routes see the real client behind a chain of proxies.
  • The Audit and Dead-letter pages work on PostgreSQL. The daily tsak_audit_log retention sweep started working on PostgreSQL at the same time, so on a node that has lived on Postgres for a while, check the table's size before the first sweep runs.
  • Cluster protocol: a heartbeat no longer reverts a concurrent cordon, a cordoned node gets no new modules, and a clustered route stopped through the API starts again.
  • Cluster-ready defaults: on a shared database, Quartz ships in clustered mode (on SQLite it switches itself off, with a warning in the log), and a node's ApiEndpoint is detected automatically instead of every node registering localhost:9090.

redb.Identity: WS-Trust, token audiences and a console built from pages

  • WS-Trust facade (redb.Identity.Soap). A third transport after HTTP and gRPC: Issue, Validate, Cancel and Renew over SOAP, for systems that generate their clients from WSDL. Issuer, client registry and token store are shared, so a client registered over HTTP gets a token over SOAP. The facade refuses to start without TLS, and mTLS with certificate thumbprint pinning is available.
  • RFC 9068 audiences in access tokens. The resources of granted scopes and the application's new AccessTokenAudiences go into aud, and the management API accepts only tokens that carry its own audience.
  • Contracts in a carrier package of their own. Hot-reloading one module no longer splits the shared DTOs into two copies, which used to make typed bodies vanish on their way through direct-vm://.
  • Uniqueness on redb.Core primitives. On MSSQL, ClientId, ScopeName and ReferenceId are protected by an index for the first time, and an external identity can no longer be linked to two local users by a race.
  • An admin console built from pages instead of dialogs. Federation providers get a full page with claim mappings, a claim mapper's owner is now selectable, users are created through a wizard, and there's a new API resources catalog plus a dashboard with KPIs and audit by category.
  • DPoP behind a TLS-terminating proxy, and the correct client_id in the audit log for clients that send credentials in the form body.

Security

Some of these came in as external reports; others turned up in our own audit of the facades and the dashboard. The mechanics of each will be in the per-product deep dives. If you run 3.x in production, this list alone is a reason to upgrade.

  • redb.Identity: an HTTP header could impersonate any user. Internal-only headers are now stripped at the edge of all three facades: HTTP, gRPC and SOAP.
  • redb.Identity: the consent page took the application name and scope list from its own URL, and the consent form could be submitted on behalf of another user without their session. The page now renders from a server-signed ticket, the user comes only from the session, and the forms check Origin and refuse to be framed.
  • redb.Identity: introspection answered any authenticated client about any token. It now answers only members of the token's audience.
  • redb.Tsak: dashboard pages opened without a login. Every page is now protected, and access tiers match the worker API: audit and users for Admin, dead-letter and logs for Operator.
  • redb.Tsak: module signatures are verified on every load path, process startup included, and the bytes loaded are exactly the bytes verified. Passwords no longer reach the audit log, sessions are revalidated every five minutes, and the login throttle keys on username and client address together.
  • redb.Core: the default password hasher is now bcrypt. Legacy SHA256 hashes still verify and move to bcrypt on the next password change.

Breaking changes

The list is short, but go through it before upgrading.

  • redb.Tsak and redb.Identity clusters upgrade with a stop. Nodes of one group must not run mixed versions: stop the whole group, deploy 4.0, start it again. The first start backfills the keys.
  • A large database needs a maintenance window. The upgrade adds columns and builds partial indexes under exclusive locks. It's easier to apply GetUpgradeScript() ahead of time from one node and start the application with AutoApplyDatabaseUpgrades = false.
  • redb.Core: Props-level lazy loading is gone entirely (EnableLazyLoadingForProps, WithLazyLoading(), the lazyLoadProps parameter). _objects._value_string is capped at 450 characters, and a database holding longer values refuses the upgrade with a list of objects to move to _note or Props. Accessing Props on a reference beyond the depth now goes to the database; for Blazor WebAssembly there's LazyReferenceAccess = Throw. Assemblies compiled against 3.x need a rebuild.
  • redb.Route: the string forms of LoopExpression, DelayExpression and ThrottleExpression became Loop, Delay and Throttle, and SetBodyExpression("…") and its siblings became SetBody(Expr("…")). Newtonsoft.Json no longer arrives transitively. ${...} renders numbers and dates culture-invariant, ${...} in a consumer URI fails Start(), and zero, "0", "no" and "off" are now false in conditions.
  • redb.Identity: a backend that introspects another client's tokens (the classic SPA and BFF pair) must be added to that client's AccessTokenAudiences. The SQLite audit table changed its timestamp column type, so delete SQLite audit files before upgrading.

Getting it

dotnet add package redb.Core
dotnet add package redb.Postgres        # or redb.MSSql / redb.SQLite
dotnet add package redb.Postgres.Pro    # Pro: free, no license key

dotnet add package redb.Route
dotnet add package redb.Route.TestKit

Libraries target net8.0, net9.0 and net10.0; host applications and images are built on .NET 10. Tsak and Identity ship as archives for Windows and Linux (redb-tsak, redb-identity) and as images on GHCR. The easiest way to try Tsak is the redb-tsak-stack image, which runs the worker and the dashboard in one container.

Packages: nuget.org/profiles/relikt. Source: github.com/redbase-app.

What's next

Each product gets its own deep dive: redb.Core with code for lazy references and keys, redb.Route with the test kit and XML routes, and Tsak and Identity with their own stories. A VS Code extension for XML routes, with a text mode and a graph editor, is in the works.

Thanks to everyone who left comments, opened discussions and sent reports from production: a good part of this release grew out of them.

If this was useful — a ⭐ on GitHub helps others find it.

More of my writing: redbase.app/articles, and on dev.to.