x-db: &db ConnectionStrings__Postgres: "Host=$;Database=app;Username=..."

x-cluster: &cluster Tsak__Cluster__Enabled: "true" Tsak__Cluster__ClusterName: app Tsak__Cluster__GroupName: app

x-rmq: &rmq Tsak__Contexts__default__RabbitMq__Host: $ Tsak__Contexts__default__RabbitMq__Exchange: app.events

services: app-core: image: ghcr.io/redbase-app/redb-tsak-worker:3.7.2-net10 environment: <<: [*db, *cluster, *rmq] Tsak__Modules__AssemblyPaths__0: /app/modules Tsak__Cluster__NodeId: app-core-1 Tsak__Cluster__ApiEndpoint: http://app-core:9090 volumes: [ "./output/core:/app/modules" ]

app-api: image: ghcr.io/redbase-app/redb-tsak-worker:3.7.2-net10 environment: <<: [*db, *cluster, *rmq] Tsak__Modules__AssemblyPaths__0: /app/modules Tsak__Cluster__NodeId: app-api-1 Tsak__Cluster__ApiEndpoint: http://app-api:9090 volumes: [ "./output/api:/app/modules" ]

app-integration: image: ghcr.io/redbase-app/redb-tsak-worker:3.7.2-net10 environment: <<: [*db, *cluster, *rmq] Tsak__Modules__AssemblyPaths__0: /app/modules Tsak__Cluster__NodeId: app-integration-1 Tsak__Cluster__ApiEndpoint: http://app-integration:9090 volumes: [ "./output/integration:/app/modules" ]

One web console for all three workers.

tsak-web: image: ghcr.io/redbase-app/redb-tsak-web:3.7.2-net10 environment: <<: [*db, *cluster] Tsak__Web__Mode: cluster Tsak__Web__ServiceApiKey: $

Two things there are worth noticing.

First, **the three workers run the same image**. The official `redb-tsak-worker`, not something built for this project. The only differences are the mounted modules folder and the `NodeId`. A custom image is what you build when you want the modules baked in rather than mounted, and that is a choice rather than a requirement.

Second, **the web console does not list the nodes**. In `cluster` mode it reads them from the cluster topology, which lives in the same database. Add a fourth worker and it shows up in the dashboard on its own.

The cluster forms around that shared database: leader election, node registration, heartbeats and module distribution are data in redb rather than separate infrastructure. No ZooKeeper, no etcd, no Consul, no Redis to install. A single worker with clustering on is already a working one-node cluster: it elects itself leader and starts handing out routes.

## Why microservices are the better default

The monolith layout works honestly, and for a small system it is simpler. Past a certain size, though, separate processes give you what in-process isolation cannot.

**Failure stops at the container boundary.** `AssemblyLoadContext` isolation protects you from version conflicts and from one module reaching into another's statics. It does not protect you from an `OutOfMemoryException`, from a native library that took the process down, or from a leak that ate the working set. A separate container does.

**Limits go on the thing that needs them.** An integration module pulling hundred-megabyte XML and an API module returning JSON need different memory. In one process they share a single ceiling, and that ceiling is set by the worst case.

**You scale what is loaded.** Three API replicas and one integration replica is an ordinary picture. In a monolith you scale everything together, including the parts that did not ask for it.

**Release cadences come apart.** Updating one `.tpkg` does not make you think about the other eight.

**Ownership boundaries match team boundaries.** Each team gets its worker, its modules, its row in the dashboard and its set of endpoints.

**The honest price.** Splitting turns free in-process `direct-vm://` calls into network calls. That is the cost, and it should be paid deliberately. The consolation is that the route code changes its address, not its logic: a producer publishes to a logical endpoint, and what that resolves to (`seda:` inside the process, `rabbitmq:` between processes) is configuration. The sensible tactic is to keep tightly coupled modules in one worker and cut along the boundary where the conversation is already asynchronous through a broker.

## Microservices do not take hot reload away

Splitting into microservices usually costs you hot reload. The reasoning goes: a service is a container, an update is a new image, a new image is a pod restart. Want it faster, build your own plugin system.

In Tsak hot swap does not depend on topology. It lives in the worker, and there can be any number of workers.

```bash
# Updating a module on a running worker: replace one file.
cp ./output/Orders.tpkg /app/modules/

HotReloadService notices the changed timestamp and performs a graceful swap: it brings up a new AssemblyLoadContext, lets it settle, waits for the old context to finish its in-flight messages, stops it and releases it. Nothing is dropped, the process does not restart, and the other modules never notice.

Deleting a file is a first-class deployment operation rather than an error: rm modules/Orders.tpkg stops every module of that package atomically, closes its transports and connections and disposes its isolated context. Neighbouring packages keep running.

In a cluster RollingUpdate kicks in and nodes update in sequence. There is never a moment when zero nodes run the new version, and never a moment when in-flight messages are lost.

The knobs you actually turn in production:

Key	Default	What it does
HotReload:ScanIntervalSeconds	10	How often the module directories are scanned.
HotReload:RollingUpdate	true	In a cluster, nodes update one after another rather than together.
HotReload:StartupTimeoutSeconds	60	How long to let the new version settle before retiring the old one.
HotReload:KeepVersions	2	How many previous versions stay available for a one-command rollback.
HotReload:RemovalDebounceScans	2	How many consecutive scans a file must stay missing to count as deleted. Protects against atomic replacement, where the file vanishes for an instant.
HotReload:AdditionStabilityScans	2	How many consecutive scans a new file must hold the same size and timestamp before it is opened. Protects against reading a half-copied archive.
HotReload:Collectible	false	Full unload of the assembly context. Off deliberately, explained near the end.

The last two arrived in 3.7.0, after working out why a large archive an operator dropped in by hand was sometimes picked up half-written.

Configuration reloads hot as well: editing context.json or {Module}.config.json re-merges the layers and restarts the affected context. The worker stays up.

Groups, nodes, assignments: slice the isolation however you like

The topology is a three-level tree, and it lives in redb as ordinary objects:

cluster:default                      scheme _tsak_clusters
 └── group:default:default           scheme _tsak_groups
      ├── node:default:worker-1      scheme _tsak_nodes
      ├── node:default:worker-2
      └── node:default:worker-3

Every level is a boundary you can put to work.

A cluster separates environments and products. Different ClusterName values in one database do not see each other.

A group isolates leader election, assignment and rebalancing. Inside one cluster you can keep an edge group for the workers facing outward and a batch group for the nightly processing, and a re-election in one leaves the other alone.

A node is a worker. It registers itself, sends a heartbeat every 15 seconds, and is evicted from the registry after 60 seconds of silence. The leader lock is taken for 30 seconds and renewed; every state change is stamped with the leader's epoch, so a leader that lost the election cannot corrupt state after the fact.

A context is one more level, this time inside a worker. A named context joins several modules under one property bag and one lifecycle; an anonymous one is created for every module that was not assigned anywhere.

{ "Tsak": { "Contexts": {
    "api": { "Modules": ["Api.Orders", "Api.Catalog"], "AutoStart": true }
}}}

Out of those pieces almost any layout assembles: from one worker holding every module to nine workers holding one each, grouped by area of responsibility.

What the cluster adds on top:

• Module distribution across nodes. The leader spreads contexts over the live nodes and reassigns them when a node joins or leaves. The strategy today is round-robin, weighted ones are planned.
• Active-passive for a consumer route, out of the box. A route reading Kafka or firing on a schedule runs on exactly one node. The node dies, the lock expires, a neighbour takes over. No duplicates, no downtime.
• Cordon and uncordon. A node steps out of assignment without stopping: tsak cluster cordon node-2, its modules move to the others, and the node can be serviced.
• Scheduled work as cluster singletons. The built-in daily jobs (audit and dead-letter retention sweeps) are marked .Cluster(true) and run on one node rather than on every one.
• Swappable coordination. ILeaderElection, IDistributedLock, INodeRegistry, IClusterCoordinator, IClusterBootstrap and IAssignmentManager are interfaces. If coordinating through the database is not what you want, register your own implementation, say one over Kubernetes Lease objects, with a single DI line before AddTsakCluster(). Nothing else changes.

One dashboard over every worker

The web console is a separate Blazor Server process that finds the nodes itself in cluster mode. Its sidebar splits into three groups: the whole cluster, the sections of the selected node, and settings. This is an operator's workplace: the console is where you act, not only where you look.

Cluster overview

Section	What is there
Dashboard	Node statuses, a status donut, metric sparklines, a sortable and filterable node table.
Cluster	The three-level topology tree, module assignments, per-node health, click-through into a node.

Inside a node, eleven sections, switched in the same sidebar without losing the selected node:

Section	What is there
Overview	Process cards: CPU, working set, managed memory, threads, thread-pool queue, collections per GC generation.
Contexts	The node's route contexts with status and endpoint count, plus start, stop and restart.
Endpoints	Consumer and producer endpoints per route.
Routes	Every route of every context at once: status, message count, error rate, click-through.
Route detail	One route in full: definition, current state, the exchanges in flight right now, recent diagnostics.
Watchdog	Suspected and hung routes with stop and restart buttons.
Modules	Loaded modules: name, version, status, dependencies, description.
Scheduler	Quartz jobs: group, cron, state, next fire time, pause, resume, fire now.
Monitoring	Four live Chart.js graphs: CPU, memory, threads, garbage collection. Refreshed every ten seconds, twelve hours of history.
Logs	The ring-buffer log viewer with search, level filter and tail mode.
Audit	The trail of admin actions, who pressed what.
Dead-letter	Failed exchanges: list, replay, discard.

Settings

Section	What is there
Auth & Users	API keys and accounts: create, revoke with confirmation.

Login is its own page: in cluster mode the accounts come from redb, in standalone mode from configuration.

Four items on that list deserve their own paragraph.

Per-node monitoring. The graphs are drawn from the worker's own metric history rather than from an external system. Prometheus and Grafana plug in alongside and are not going anywhere, but seeing what one node is doing right now does not require standing them up.

Exchanges in flight. The route detail shows which exchanges are sitting in a route at this moment. When a queue is not draining, "is it stuck or just slow" is answered by looking, not by attaching a debugger to production.

Watchdog. The service continuously classifies routes and tells suspected apart from hung. It will restart them on its own if you let it.

Replaying a failed exchange. The replay button in Dead-letter performs exactly one replay even if two operators press it at the same instant: the claim is taken with a conditional UPDATE, and the database decides which of the two won.

The console runs on its own design system, no Bootstrap, no MUI, no Tailwind: CSS variables, system fonts, dark and light themes, inline SVG icons.

Management, not just observation

The dashboard is one of three heads. Under all of them sits the same REST API, 70 endpoints across 16 controllers, all speaking JSON.

Group	Endpoints	About
/api/health	3	Kubernetes probes: startup, live, ready. Auth-exempt.
/api/system	6	Health, metrics, metric history, process info, effective configuration, loaded assemblies.
/api/contexts	8	List, start, stop, restart, reset route states, endpoints, remove.
/api/contexts/{ctx}/routes	8	Routes: start, stop, force-stop, in-flight exchanges, metrics.
/api/modules	6	List, remove, upload, validate signature, roll back.
/api/scheduler	8	Scheduler: status, scheduled jobs, running jobs, pause, resume, fire now.
/api/cluster	6	Status, nodes, rebalance, remove node, cordon, uncordon.
/api/watchdog	6	Status, alerts, enable, test alert.
/api/exchanges	3	Dead-letter queue: list, replay, discard.
/api/logs	3	Incremental tail, file list, download.
/api/diagnostics	2	Cluster-wide and per-route dumps.
/api/auth	3	API keys: create, list, revoke.
/api/users	5	Users.
/api/audit	1	The admin-action trail.
/api/lifecycle	1	The lifecycle event feed.
/api/dashboard	1	An aggregated snapshot for the console in one round-trip.

The API itself is built nicely: it is an ordinary route context called _system, one HTTP listener whose pipeline reads "header bridge, auth, controller dispatch". Tsak manages itself with the same engine it runs your routes on.

The second head is the CLI: tsak, one binary, 57 commands, connection profiles, tabular output for humans and JSON for CI.

tsak login http://prod-1:9090 --key $PROD_KEY --profile prod
tsak profile use prod

tsak context list                    # a table
tsak context list --output json      # for jq in a pipeline

tsak route force-stop orders route-1 # take down a hung route
tsak route inflight orders route-1   # see what is sitting in it
tsak dlq replay 42                   # replay a failed exchange
tsak cluster cordon node-2           # step a node out of assignment
tsak module deploy ./Orders.tpkg     # ship a module
tsak module rollback Orders          # go back one version

The third head is a typed C# client, for automating operations from your own code:

services.AddTsakClient(o => { o.BaseUrl = "http://tsak-prod:9090"; o.ApiKey = key; });

public class Ops(ITsakApiClient tsak)
{
    public async Task RestartFailedAsync(CancellationToken ct)
    {
        var contexts = await tsak.ListContextsAsync(ct);
        foreach (var c in contexts.Where(c => c.Status == "Failed"))
            await tsak.RestartContextAsync(c.Name, ct);
    }
}

On access: keys are stored as an HMAC-SHA256 hash, the raw key is never persisted, comparison is constant-time, a key carries roles, an expiry and a revocation, and a revoked key stops being accepted across every node within thirty seconds. Inter-node calls use the same authentication: there is no implicit trust between nodes.

Since 3.7.0 all of it is also closed by default. The management API binds 127.0.0.1 rather than 0.0.0.0, so exposing it is a deliberate act. A roleless key is denied instead of being treated as admin. The console has a real server-side cookie session, the password is checked against a BCrypt hash, and the login is rate-limited.

Kubernetes, Prometheus, Jaeger

External observability plugs in without a shim, because Tsak was written for containers from the start.

Three probes, not one. Split by pod lifecycle phase, all three auth-exempt.

startupProbe:   { httpGet: { path: /api/health/startup, port: 9090 } }
livenessProbe:  { httpGet: { path: /api/health/live,    port: 9090 } }
readinessProbe: { httpGet: { path: /api/health/ready,   port: 9090 } }

The difference between liveness and readiness is deliberate. Liveness intentionally does not check module health, otherwise a rolling update would turn into a restart loop. Readiness is stricter: any context in a non-running state takes the pod out of the load balancer without restarting it, and the cluster redistributes assignments meanwhile.

Prometheus with no extra port. With Tsak:Metrics:Prometheus:Enabled the metrics are served at /metrics on the same port as the API. The OpenTelemetry listener sits on loopback, and the facade route proxies it out. No second port to open, and on Windows no URL ACL, because Kestrel binds the sockets rather than HttpListener.

metadata:
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port:   "9090"
    prometheus.io/path:   "/metrics"

The exporter is pre-flighted: if the loopback bind ever fails, Tsak logs a warning and runs without metrics. An optional exporter has no business taking the worker down.

Jaeger and any OTLP collector. Traces leave through the standard OTLP exporter, Tsak:Tracing:Otlp:Enabled plus an endpoint. The redb.Route ActivitySource is registered in the OpenTelemetry pipeline, so the spans opened by route processors and transports are collected on their own. The service name in Jaeger comes from Tsak:Tracing:ServiceName, which is exactly right in a microservice layout: each worker carries its own name, and the trace is stitched across the broker.

Ready-made artifacts. The repository ships Kubernetes manifests (a Deployment with the correct probes, a Service, a ServiceMonitor for Prometheus Operator), an importable Grafana dashboard, and a local Prometheus + Grafana + Jaeger stack behind one docker compose.

Pod identity. In a cluster the NodeId has to survive a restart or assignments drift. The downward API binds it to the pod name:

env:
  - name: Tsak__Cluster__NodeId
    valueFrom: { fieldRef: { fieldPath: metadata.name } }
  - name: POD_IP
    valueFrom: { fieldRef: { fieldPath: status.podIP } }
  - name: Tsak__Cluster__ApiEndpoint
    value: http://$(POD_IP):9090

Graceful termination. Set Tsak:Shutdown:TimeoutSeconds five seconds below terminationGracePeriodSeconds so that deregistering the node keeps its buffer. The order is SIGTERM, cluster deregistration, context drain, scheduler shutdown, log flush. SIGKILL never gets its turn.

On top of that the worker counts for itself: process metrics with twelve hours of history sampled every ten seconds (4320 points), per-context and per-route metrics (messages per second, error rate, in-flight count), a two-thousand-entry ring-buffer log queryable over REST and in the console, and diagnostic dumps per route and across the cluster.

Identity arrives as a package in the same worker

The same module format carries a finished product. A full OAuth 2.1 and OpenID Connect server ships as a set of .tpkg files.

Four packages: redb.Identity.Core (the server itself: schemes, stores, MFA, WebAuthn, federation, audit, key rotation) plus three transport facades, Http, Grpc and Soap. The facades are thin bridges with no business logic.

It lays out however you want, on exactly the same logic as your own modules:

• One worker: Core + Http, and you have an OP on its port.
• Two workers: Core + Http faces outward, Core + Grpc serves service-to-service calls. Different cluster groups, different limits, different exposure.
• Inside your own worker: put redb.Identity.Core next to your module and call it straight from your route.

// No HTTP, no serialization, no loopback: the same exchange.
From("http:0.0.0.0:5090/api/login")
    .To("direct-vm://identity-token");

This is the shared-runtime benefit in its purest form: two products that know nothing about each other end up in one process and talk without a network, because both speak in route addresses.

The substance is tested and proven: OIDC Core, OAuth 2.1, introspection, dynamic client registration, Device Code, PAR, JAR, DPoP, backchannel logout, SCIM 2.0, TOTP, OTP over SMS and email, WebAuthn. The official OpenID Foundation conformance suite passes with zero failures on the Config OP and Basic OP profiles.

And it is observed by the same dashboard as everything else: Identity routes appear in the shared list, its metrics in the shared graphs, its logs in the shared buffer.

Your own worker, when you want one

The module stays yours and runs anywhere. The entry point is either a type implementing ITsakModule or a public static class InitRoute with a main(IRouteContext) method. The second is an Apache Camel style convention, and anyone can call it.

A debug host in full:

var services = new ServiceCollection();
services.AddLogging(b => b.AddSimpleConsole());
services.AddRedb(o => o.UseSqlite("Data Source=echo_demo.db"));

var sp = services.BuildServiceProvider();
await sp.GetRequiredService<IRedbService>().InitializeAsync(ensureCreated: true);

var ctx = new RouteContext(sp, contextId: "echo-worker");
ctx.AddService(typeof(ILoggerFactory), sp.GetRequiredService<ILoggerFactory>());

EchoModule.InitRoute.main(ctx);   // the exact method the Tsak worker calls

await ctx.Start();

Fifty lines, and the module runs under a debugger in your IDE with breakpoints and single-stepping. That beats attaching to a hot-loaded assembly context inside a running worker.

The same trick works when you want a host of your own: your DI, your configuration, your metric collection. The redb.Tsak sources are Apache 2.0, the packages are on NuGet, and there is no closed runtime in the middle.

Five configuration layers: why one image serves nine workers

This is the mechanism that lets the three services above start from a single image.

Layer 1: Tsak:Contexts:default                 base for every context
Layer 2: Tsak:Contexts:{name}                  settings of one context
Layer 3: modules/{Module}/context.json         the module's infrastructure defaults
Layer 4: modules/{Module}/{Module}.config.json the module's business settings
Layer 5: Tsak:Contexts:{name}:Override         operations get the final word

The layers deep-merge: nested objects complete each other rather than replacing wholesale. A module brings sensible defaults inside its own archive, operations overrides what has to differ in production, and the module's code anticipates none of it.

The practical consequence for secrets: LDAP and SMTP passwords and JWT signing keys arrive through the Override layer from environment variables and land in named connection factories. They are not in the .tpkg, not in endpoint URIs, not in the logs and not in the dashboard.

On top of that, the worker's shared layer already carries 28 redb.Route connectors: RabbitMQ, Kafka, AMQP, Azure Service Bus, IBM MQ, SQS, Redis, S3, Elasticsearch, SQL, gRPC, SOAP, AS2, SignalR, WebSocket, MQTT, TCP, mail, files, FTP, SFTP, LDAP, Telegram, Firebase, LLM and the rest. A module talking to Kafka does not carry the driver with it: the worker already has it.

What it costs

The boundaries worth knowing before you adopt it.

Assembly-context unloading is off by default. HotReload:Collectible = false, deliberately: Reflection.Emit (used by XmlSerializer, serialization generators and compiled regular expressions) does not survive an unload. The price is that old assembly contexts stay in memory until the process restarts. Their count is exposed as the LeakedAlcCount metric, so it is visible. For a worker updated once a week this is immaterial; for one updated twenty times a day, a nightly restart settles it.

There is one distribution strategy. Round-robin. Weighted strategies are planned, the IAssignmentManager interface for them already exists, and your own implementation can be plugged in today.

Coordination lives in the database. Leader election, locks and the node registry are rows in redb. The upside is that no separate membership infrastructure is needed. The downside is that the database becomes a participant in coordination. If that does not suit you, all six coordination interfaces are replaceable in DI, for instance with an implementation over Kubernetes Lease objects, leaving redb as storage for modules and keys only.

Quartz in a cluster wants a real database. RAMJobStore for development, AdoJobStore for production. The schema is created on first start, with no DBA action required.

Trying it out

The fastest path is a single redb-tsak-stack container: worker and web console in one image, in the spirit of rabbitmq:management.

docker run -p 9090:9090 -p 8085:8085 \
  -v ./modules:/app/worker/modules \
  ghcr.io/redbase-app/redb-tsak-stack:3.7.2

The API comes up on 9090 and the dashboard on 8085. Drop a .tpkg into ./modules and ten seconds later the module is in the list and its routes are on the graphs. Neither a database nor a broker is needed for a first run: the default storage is in-memory.

When it comes to spreading across machines, the separate redb-tsak-worker and redb-tsak-web images take over, as in the compose file above. Ready-made templates for all four cases (worker only, console only, stack, stack with PostgreSQL) live in the repository under publish/docker/.

Without Docker there is a self-contained archive: unpack, run, put the modules next to it. That is the monolith layout in one file, with the same dashboard and the same API. Image and archive signatures are verified with cosign.

The current number on the line is 3.7.2. The libraries target net8.0, net9.0 and net10.0; applications and images are built on .NET 10 and tagged -net10. Pro stays proprietary but free, with no licence key, across the whole 3.x line, clustering included: there is no node limit. The worker's main unit suite passes in full on this line, 647 of 647 on .NET 10.

• Sources: github.com/redbase-app/redb-tsak
• Releases and archives: redb-tsak/releases
• Images: the stack in one container, everything on GHCR
• NuGet packages: nuget.org/profiles/relikt
• The ecosystem as a whole: redbase.app

Starting with the monolith layout is easier, and moving to microservices can wait until there is a reason. The pleasant part is that the move costs an edit to a compose file rather than a rewrite.

If this was useful — a ⭐ on GitHub helps others find it.

More of my writing: redbase.app/articles, and on dev.to.