A directory is an integration: file exchange in .NET without cron, FileSystemWatcher or hand-rolled pollers

2026-08-25redb.Route

Ask anyone who does integrations for a living what the money actually moves through, and files will be on the list. The bank drops a statement into a directory on a share. An ERP exports orders as XML on a schedule. An EDI partner pushes .edi files to an FTP server. Billing lands a couple of gigabytes of CDR files overnight. A retail chain sends stock levels as CSV because that is how it was in 2009 and nobody plans to change it. File exchange is not legacy on its way out. It is a working transport with exactly one property: it looks simpler than it is.

It looks simple because the first version takes an hour. Directory.GetFiles, a loop, File.ReadAllBytes, process, delete. Then real life starts. You picked the file up while the partner was still writing it, and half an order batch went into the database. Processing threw, and the file is already deleted. Two instances of the service grabbed the same file and created duplicate payments. Someone dropped an antivirus .tmp into the directory. FileSystemWatcher missed events once the directory moved to an SMB share. A deploy went out mid-processing and an 800 MB file ended up neither here nor there. Each of these gets its own workaround, and a year later you own a 2000-line half-framework nobody wants to touch.

redb.Route.File closes that list with a transport. A directory becomes an ordinary message source in a route, exactly like a Kafka topic or an HTTP endpoint: polling with filters, waiting until the file is fully written, idempotency, a policy for what happens after processing, atomic writes through a temp file. Same DSL, same EIP patterns after From, same telemetry. Here is what that looks like, and why a native connector inside the ESB beats a poller you wrote yourself.

Files in a minute

services.AddRedbRoute(route =>
{
    route.Services.AddRedbRouteFile();
    route.AddRouteBuilder<MyRoutes>();
});

using redb.Route.File.Fluent;

// Read incoming CSVs, hand the parsed result to a queue
From(FileDsl.Read("/data/incoming")
        .Include("*.csv")
        .MinAge(2000)
        .SortBy("Modified")
        .MoveTo("/data/processed"))
    .Log("Picked up ${header.redbFile.Name}, ${header.redbFile.Length} bytes")
    .To("direct://parse");

// Write the result out atomically
From("direct://export")
    .To(FileDsl.Write("/data/outgoing")
        .FileName("${header.orderId}.json")
        .TempPrefix(".tmp-"));

That is the whole setup. No external dependencies, one registration line, and the file scheme is then available both fluently and as a URI string.

An endpoint is a string, or a builder

Every endpoint reads two equivalent ways: a type-safe builder in code, and a plain URI that lives in appsettings.json and changes without a rebuild. They compile to the same thing, because the builder literally assembles the string.

// Fluent
From(FileDsl.Read("/data/incoming").Include("*.csv,*.xml").Recursive().Delete())

// String (identical)
From("file:///data/incoming?include=*.csv,*.xml&recursive=true&delete=true")

On Windows the path is written file:///C:/data/incoming and the leading slash is dropped for you. A small thing, and reliably the one that bites when a config moves between machines.

The central problem: a file that is still being written

No file system gives you a "writing finished" event. The partner opened a stream, is pushing 300 MB over a slow link, and your poller has already seen the name in the directory. The question is not whether this bites you, only how soon. The connector offers six independent mechanisms, and production setups usually combine two.

Minimum age. The cheapest one: leave the file alone until N milliseconds have passed since the last write.

FileDsl.Read("/data/incoming").MinAge(5000)

Wait for the size to settle. The Changed strategy watches size and modification time at an interval and releases the file only once it has stopped growing for a configured period. This is the right answer for large exports that take minutes to write.

FileDsl.Read("/data/incoming")
    .ReadLock("Changed")
    .ReadLockCheckInterval(1000)   // how often to look
    .ReadLockMinAge(5000)          // how long it must stay unchanged
    .ReadLockTimeout(120000)       // how long to wait at most

A marker file. The MarkerFile strategy creates name.redbLock next to the target using FileMode.CreateNew, which is atomic at the OS level. A second instance that sees the marker skips the file. This is the working answer to "two pods read the same share": exactly one wins the race, and the loser never even opens the file.

FileDsl.Read("/mnt/share/in").ReadLock("MarkerFile")

An exclusive handle. The FileLock strategy opens the file exclusively and holds the handle for the duration of processing. While it is held, nobody else opens that file for reading or writing. One detail worth spelling out: the handle is taken with delete permission, so your own post-processing can delete or move the file without waiting for the lock to drop, and the body is read through the handle that is already open rather than by opening the file a second time.

FileDsl.Read("/data/incoming").ReadLock("FileLock")

A rename. The Rename strategy moves the file to an internal name and works on it there. If the rename fails, someone else holds the file and the route leaves it alone. Pleasant side effect: while the file is being processed it is not in the directory under its original name, so another poller matching on a glob simply does not see it.

FileDsl.Read("/data/incoming").ReadLock("Rename")

A signal file from the sender. EDI classic: the partner writes order.csv, finishes it, and only then drops an empty order.csv.done. Until the signal is there, the file does not exist as far as the route is concerned.

FileDsl.Read("/data/incoming").DoneFileName("${file:name}.done")

${file:name} and ${file:name.noext} are both substituted, so the order.csv plus order.done convention works too. The signal file is removed for you once processing succeeds.

Separately: the consumer ignores internal names starting with .redb_ and anything starting with the configured tempPrefix. So a producer writing into the same directory through a temp file cannot feed itself an endless loop. That is the kind of detail a hand-rolled poller discovers on day three.

What happens to the file afterwards

Exactly one policy per endpoint, and trying to set two fails validation at startup rather than silently at runtime.

Policy	Effect	When to reach for it
`Noop()`	The file stays where it is	Read-only directory, or no write permission
`Delete()`	Removed after success	An inbox where only the fact of receipt matters
`MoveTo(dir)`	Moved to an archive	The production default: there is always something to show
`PreMove(dir)`	Moved BEFORE processing	Several consumers on one directory

PreMove deserves a note, because it solves a problem people usually solve badly. The file is moved into a working subdirectory first and read only afterwards. A move within one file system is atomic, so a second instance simply does not find it and moves on. As a bonus, the directory listing tells you at any moment what is currently in flight.

FileDsl.Read("/data/incoming")
    .PreMove("/data/processing")
    .MoveTo("/data/archive")

Now failure. If processing threw and the exception was not marked handled, post-processing does not run: the file stays exactly where it was and comes back on the next poll. That is deliberate, because "delete the file we could not process" is the worst available option. The flip side is that a genuinely broken file will be retried forever, which is why on a file route OnException is part of the construction rather than a nice-to-have:

OnException(typeof(FormatException))
    .Handled(true)
    .To(FileDsl.Write("/data/quarantine"));

A handled exception completes the exchange successfully, a copy lands in quarantine, and the original goes through the normal policy.

The same applies to a case people rarely plan for: the file was in the directory but could not be read. Someone holds it exclusively, permissions are wrong, the volume dropped. That is a failure for that file, not an empty message. The route will not receive zero bytes dressed up as a successful read, the file stays put, and the next poll picks it up. Obvious, right up until a hand-rolled poller hands you an empty byte[] and the import faithfully records zero orders.

Idempotency

A flag turns it on. The default key is built from the full path, the modification time and the size, so a file overwritten under the same name counts as new, while the same file is never processed twice.

FileDsl.Read("/data/incoming").Noop().Idempotent()

Noop() + Idempotent() gives you "read the directory, change nothing in it, each file exactly once". It is the only way to work with a directory you have no write access to, and it is also the most common shape when integrating with someone else's share.

The registry of processed keys lives in process memory, which is worth keeping in mind: after a restart it is empty. Combined with Delete or MoveTo that does not matter, because the processed file is no longer in the directory. Combined with Noop, a restart means a second pass, and de-duplication for that case belongs further down the route.

Selection: what to take, and in what order

Processing order almost always matters in file exchange. A nightly batch of 4000 files consumed in arbitrary order gives you stock levels that drift away from reality.

FileDsl.Read("/data/incoming")
    .Include("*.csv,*.xml")      // comma-separated globs, * and ?
    .Exclude("*.tmp,~*")
    .SortBy("Modified")          // Name, NameDesc, Modified, ModifiedDesc, Size, SizeDesc
    .MaxMessagesPerPoll(200)     // batch size per pass
    .Recursive()
    .Delay(1000)                 // poll interval, 500 ms by default

MaxMessagesPerPoll together with sorting is your load regulator. A partner comes back after a day of downtime and drops 40 000 files, and you work through them in predictable batches from oldest to newest instead of materialising a 40 000-element list and taking the process down.

Globs are matched case-insensitively, which on Linux saves you from the classic "they sent ORDERS.CSV and the mask says *.csv".

Writing: temp plus rename, not "however it goes"

The producer solves the mirror image of the same problem. If you write straight into a directory a partner is reading, sooner or later they read half a file.

From("direct://export")
    .To(FileDsl.Write("/data/outgoing")
        .FileName("orders-${header.batchId}.json")
        .TempPrefix(".tmp-")
        .AutoCreate());

The body goes to .tmp-orders-42.json and is renamed to the target name only once it is fully written. A rename inside a file system is atomic, so the partner sees either nothing or the finished file. There is no intermediate state. On an exception the temp file cleans itself up.

The target name comes, in priority order, from the FileName option (full expressions over headers and body), from the incoming redbFile.Name header, and failing both it is generated as redb-{guid}. That last fallback is handier than it sounds: a From(kafka).To(file) route works without configuring a name at all.

What to do when the target already exists is configured separately:

`FileExist`	Behaviour
`Override`	Overwrite (default)
`Append`	Append to the end
`Fail`	Throw
`Ignore`	Skip the write silently
`Move`	Set the existing one aside as `.bak`, then write
`TryRename`	Rename the existing one with a timestamp suffix

Ignore and Fail look exotic exactly until the first export where writing the same name twice means shipping the goods twice.

The file name is untrusted input

Look again at the priority order above: with no FileName set, the name arrives in the redbFile.Name header. And somebody put it there: an incoming partner file, an HTTP upload, a field from a queue message. Which means that in a typical From(http).To(file) route, the name of the file you create is chosen by whoever sends the request.

So by default the producer does not let a write escape its own directory. A name like ../../etc/cron.d/backdoor, or an absolute path, gets you an exception rather than a file somewhere surprising. The absolute path case is worth knowing about on its own: plain Path.Combine in .NET silently discards the base directory and hands back what it was given, so "but we concatenate with the base" is not a defence.

// default: writes stay inside /data/outgoing
FileDsl.Write("/data/outgoing")

// deliberately allow writing outside
FileDsl.Write("/data/outgoing").JailStartingDirectory(false)

The option has the same name on the local file system, FTP and SFTP, so it is one rule across all three transports.

An append log

A useful mode of its own: append with a separator. A receipt log, an audit trail, a cumulative daily CSV all become a single route step.

From("direct://audit")
    .To(FileDsl.Write("/var/log/app")
        .FileName("audit-${dateFormat(now(), 'yyyyMMdd')}.log")
        .FileExist("Append")
        .AppendChars("\n"));

Putting the date in the name expression means daily rotation falls out for free, with no separate scheduler.

Headers: the file name travels the whole route

The consumer puts a full set of metadata on the message under the redbFile. prefix, available downstream in any expression, predicate or processor.

Header	Contents
`redbFile.Name`	`order-42.csv`
`redbFile.NameOnly`	`order-42`
`redbFile.Extension`	`.csv`
`redbFile.AbsolutePath`	Full path at read time
`redbFile.RelativePath`	Path relative to the polled directory
`redbFile.Parent`	Parent directory
`redbFile.Length`	Size in bytes
`redbFile.LastModified`	`DateTimeOffset`

After writing, the producer sets redbFile.NameProduced to the path the file actually landed on. On top of that, ContentType is inferred from the extension for .json, .xml, .csv, .txt, .log and .html, so the next step already knows what it was handed.

The practical payoff shows up in routing by name, and file exchange has more of that than anyone would like:

From(FileDsl.Read("/data/in").Recursive().MoveTo("/data/archive"))
    .Choice(c => c
        .When(Header("redbFile.Extension").isEqualTo(".xml").Matches,
              w => w.To("direct://xml"))
        .When(Header("redbFile.Name").startsWith("INV_").Matches,
              w => w.To("direct://invoices"))
        .Otherwise(o => o.To("direct://unknown")));

Large files: do not read the whole thing

The body is a byte[] by default, which is convenient and completely unacceptable for a two-gigabyte CDR file. StreamBody hands over an open stream instead of an array, and the stream is closed along with the exchange.

From(FileDsl.Read("/data/cdr").Include("*.dat").StreamBody().MoveTo("/data/done"))
    .Split(ex => ReadLines((Stream)ex.In.Body!))
    .To("direct://cdr-record");

Streaming body plus splitter is what a splitter in an ESB exists for: the file is consumed line by line and memory holds one record instead of the whole file.

The file system as a cloakroom

There is an adjacent problem, and it shows up specifically in file integrations. The body is large, and the intermediate steps of the route never look at it. Downstream there is a broker, an HTTP call, another service, and each of them dutifully drags two hundred megabytes it has no use for through itself.

The Claim Check pattern (Hohpe and Woolf) handles this like a cloakroom: the body is checked in, and a ticket travels the route in its place. The body is unpacked where it is actually needed.

The store is pluggable, and for the file world the natural choice is the file system itself. FileClaimCheckRepository writes each claim as its own file alongside its metadata and TTL. For large bodies that is cheaper than memory, and on a shared file system it survives both a restart and the work moving to a neighbouring instance.

private readonly IClaimCheckRepository _claims =
    new FileClaimCheckRepository("/data/claims", TimeSpan.FromHours(6));

From(FileDsl.Read("/data/incoming").Include("*.zip").MoveTo("/data/archive"))
    .ClaimCheck(_claims, ClaimCheckOperation.Set, "${header.redbFile.NameOnly}")
    .To("direct://notify")                      // a ticket travels on, not the archive
    .ClaimCheck(_claims, ClaimCheckOperation.GetAndRemove, "${header.redbFile.NameOnly}")
    .Process(UnpackAndImport);                  // body restored, ticket redeemed

The original body type is remembered in headers, so what comes back is what went in rather than a bare byte array. The store can be registered under a name on the context and referenced by string, or left unspecified entirely, in which case steps share the context-wide store.

There is also a keyless stack mode, Push and Pop: check the body in before an enrichment call, take it back afterwards, nesting handled for you. Handy when the middle of a route needs a call that your two hundred megabytes only get in the way of.

Local directory, FTP and SFTP are the same route

This is the connector's main architectural bet. Polling, filters, sorting, idempotency, doneFileName, post-processing, atomic writes and the existing-file strategies are implemented once, in a shared redb.Route.GenericFile base. The local file system, FTP and SFTP are three implementations of file operations on top of it.

The practical consequence is direct. When a partner says "we no longer mount the share, collect from our SFTP", you change the source rather than rewriting the intake logic:

// before
From(FileDsl.Read("/mnt/partner/in").Include("*.edi").MoveTo("archive"))

// after
From(SftpDsl.Directory("/upload/in")
        .Host("sftp.partner.com").Username("edi").Password("{{sftp-pass}}")
        .Include("*.edi").MoveTo("archive"))

Option names, post-processing semantics and failure behaviour all match, because it is literally the same code. Migrating between transports takes a minute rather than a sprint.

Shutting down without losing a file

The consumer is built on the graceful shutdown machinery shared by every redb.Route transport: on Stop the directory polling stops first, then the engine waits for exchanges already in flight, and only then the route goes down. A file that was being processed during a deploy is neither abandoned halfway nor left suspended between PreMove and the archive.

For file exchange this matters more than it does for brokers: an unacknowledged broker message returns to the queue, while a file gets no second chance automatically.

Boundaries

An honest list of what the connector does not do, so you do not find out in production.

Boundary	How it is
Polling, not file system events	No `FileSystemWatcher`. Polling is predictable, survives network shares and loses nothing when a buffer overflows. The price: latency up to `delay`, 500 ms by default
The idempotency registry lives in process memory	It survives a restart only together with `Delete` or `MoveTo`, which take the file out of the directory. It is also not shared between instances: separate them with `preMove` or a `readLock`, not with hope
The idempotency key is built from file metadata	Path, modification time, size, or your own pattern with `${file:name}`. Header and body expressions are not available here: the key is needed before the file has been read, otherwise the check loses its point
Processing within a poll is sequential	One consumer works through a batch in order. Parallelism comes from further down the route, or from several routes on different globs
`readLock` strategies are local file system only	FTP and SFTP have `doneFileName`, `minAge` and `preMove` in their place
The polled directory is not created for you	If it is missing, the poll cycle is skipped quietly. Auto-creation exists on the producer side via `AutoCreate`

Where this plugs in

The value of a file connector is not in reading a directory. That is Directory.GetFiles. The value is that after From the rest of redb.Route is available: splitter and aggregator, content-based routing, de-duplication, retries, circuit breaker, transactions, distributed tracing. A typical nightly intake looks like this:

From(FileDsl.Read("/data/incoming")
        .Include("orders_*.csv")
        .DoneFileName("${file:name}.done")
        .SortBy("Modified")
        .MaxMessagesPerPoll(500)
        .PreMove("/data/processing")
        .MoveTo("/data/archive"))
    .Log("Batch ${header.redbFile.Name}")
    .Split(ex => ParseCsv(ex.In.Body))
        .To("sql:INSERT INTO orders(...) VALUES(...)?dataSource=#pg")
    .End()
    .To("kafka://orders-imported");

Twenty lines instead of a bespoke service, and every line states a decision rather than the mechanics of carrying it out. The producer also opens an OpenTelemetry span on write, so "where did that file go" stops being a question for the logs.

File exchange is never going to be fashionable. It is going to stay, and the difference between "we have a file integration" and "we have a reliable file integration" is measured in exactly the details listed above: file age, markers, atomic rename, order and batch size. It is pleasant when someone has already written them.

Package: redb.Route.File on NuGet; sources and the full option reference in the connector README. Files are one more transport in the redb.Route family, next to Kafka, RabbitMQ, SFTP, AS2 and the rest: same From → … → To, same EIP, same observability. The only difference is that the input is a directory somebody fills whenever it suits them.

If this was useful — a ⭐ on GitHub helps others find it.

More of my writing: redbase.app/articles, and on dev.to.