Settlement file debatch
Each night Acme’s payment provider drops off one big file. Inside it are hundreds of payments the provider collected that day, one line per payment, and each line should mark one of Acme’s invoices as paid. The provider leaves the file on a server for Acme to collect, early the next morning. The goal is to match every payment to its invoice and mark it paid before the finance team starts work. The important part is what happens to a bad line. Sometimes a payment refers to an invoice Acme cannot find, a typo or a payment for something not yet billed. Handling the file as one all-or-nothing batch would let that single bad line fail the entire night’s work. Instead, each payment is handled on its own, so the good ones all go through and only the few that cannot be matched are set aside for a person to look at in the morning.
So the work is: collect the nightly file, break it into one payment at a time, and mark each invoice paid, with the bad lines pulled out individually rather than sinking the whole batch. Breaking a batch into its independent records is a well-known move, the splitter (or debatch), and it is the right one here because one payment’s fate has nothing to do with the next one’s.
Here is how Art2link ESB builds it. Collecting the file is the same scheduled-download shape used elsewhere: a Scheduler receive port ticks at 06:00 (a receive port is an entry point that starts a flow, and this one runs on a timer), then a two-way SFTP Caller send port fetches the night’s dated file, settlement_2026-06-04.csv, from the provider’s drop directory. A send port is an exit that acts on a message, and an adapter is the connector that teaches it how to talk to one kind of system, here a file server over SFTP. Everything moves through the bus, the shared message backbone that flows publish onto and subscribe from.
The difference is what happens on the way back in. The fetched file is labelled with a fallback message type of SettlementFile, then a disassembler pipeline component (a small reusable piece of code that runs inside a port to reshape or split a message) breaks it apart: one published SettlementTransaction per record, the component labelling each emitted message so the bus carries clean JSON, not the raw file. The file was just the provider’s packaging, not the real unit of work. A SQL Caller send port then subscribes to that transaction type (a subscription is a plain rule a port matches against messages on the bus) and calls a reconciliation procedure: match on payment reference, mark the invoice paid, record the fee.
{{Message.MessageType}} == "SettlementTransaction"
This is the example to contrast with the invoice line loop. Debatch when the records are independent, transaction 214's fate has nothing to do with transaction 215's. Keep the batch whole and loop when the elements only make sense inside their parent.
When it fails. Independence is the point. A transaction whose payment reference matches no invoice fails its procedure alone: the shared exception type SettlementFailed re-publishes it payload-intact and the O365 Mail alert reaches finance operations, while the other records complete. The morning email lists three unmatched payments, not one failed night; each is replayed from tracking after the invoice mismatch is resolved. The fetch port carries the same exception type, so a missing file raises the same alarm, the re-published payload tells you whether it was the whole file or a single record.
{{Message.MessageType}} == "SettlementFailed"
Build it, step by step. The steps run in dependency order: every object is created before the object that selects it.
Under the application’s Message types, create the three types this flow routes on. A message type is a name plus a format, no schema required:
| Name | Format | Purpose |
|---|---|---|
| SettlementFile | CSV | the provider’s nightly raw file as it downloads, the fallback type the disassembler component reads; never a map source |
| SettlementTransaction | JSON | one settlement record, the unit of work the splitter publishes and classifies |
| SettlementFailed | JSON | the exception type the failure path publishes under (Step 8) |
Promotions live on the message type, so add them while you are here. The reconcile port’s parameters (Step 7) bind these values, and adapter parameters are plain strings: they bind promotions, not body paths.
| Promotion | Path |
|---|---|
| PaymentRef | $.paymentRef |
| InvoiceNumber | $.invoiceNumber |
| Net | $.net |
| PaidOn | $.paidOn |
Three external parties, three credentials. Under the application’s Authentications, the dialog asks for a Name, the Adapter it pairs with, and a Definition, the credential shape that adapter offers, with the Application preset; the Definition decides the config section that follows.
| Setting | Value |
|---|---|
| Name | ProviderSftp |
| Adapter | SFTP Caller |
| Definition | SFTP Password Authentication |
| Username / Password | the provider’s SFTP login |
| Host Key Fingerprint | the provider server’s fingerprint |
| Setting | Value |
|---|---|
| Name | InvoiceDbSql |
| Adapter | SQL Caller |
| Definition | SQL Server Connection |
| Connection String (Database Config) | the invoice database’s connection string, credentials included |
| Setting | Value |
|---|---|
| Name | O365Ops |
| Adapter | O365 Mail Sender |
| Definition | Microsoft Graph |
| Tenant Id / Client Id / Client Secret (Graph AuthConfig) | an app registration with Mail.Send granted |
Create the table and procedure in the invoice database; the reconcile port (Step 7) calls the procedure. An unknown invoice THROWs, that is the per-record failure the tutorial relies on, raising SettlementFailed for that one transaction while the rest post. Matching on the payment reference keeps it idempotent under replay.
CREATE TABLE dbo.Invoices ( InvoiceNumber VARCHAR(40) NOT NULL PRIMARY KEY, Status VARCHAR(20) NOT NULL CONSTRAINT DF_Inv_St DEFAULT 'Open', PaidAmount DECIMAL(18,2) NULL, PaymentRef VARCHAR(40) NULL, PaidOn DATE NULL ); CREATE OR ALTER PROCEDURE dbo.ReconcilePayment @PaymentRef VARCHAR(40), @InvoiceNumber VARCHAR(40), @Net DECIMAL(18,2), @PaidOn DATE AS BEGIN SET NOCOUNT ON; IF NOT EXISTS (SELECT 1 FROM dbo.Invoices WHERE InvoiceNumber = @InvoiceNumber) THROW 50001, 'Unknown invoice for settlement', 1; -- Idempotent: a replayed payment with the same ref changes nothing UPDATE dbo.Invoices SET Status = 'Paid', PaidAmount = @Net, PaymentRef = @PaymentRef, PaidOn = @PaidOn WHERE InvoiceNumber = @InvoiceNumber AND (PaymentRef IS NULL OR PaymentRef <> @PaymentRef); END;
Under the application’s Pipeline components, create the disassembler SettlementCsvDisassembler: one message per data row, each stamped with a file id derived from the row’s paid_on date, read from the file itself rather than configured (components are data-driven), so a transaction can be traced back to its batch. Classification (SettlementTransaction) happens on each outgoing message.
The provider’s nightly settlement CSV as it downloads:
payment_ref,invoice_no,gross,fee,net,paid_on PSP-771202,INV-20391,250.00,4.85,245.15,2026-06-04 PSP-771203,INV-20402,89.90,1.92,87.98,2026-06-04 PSP-771204,INV-20377,1180.00,21.10,1158.90,2026-06-04
One debatched SettlementTransaction as the splitter publishes it:
{
"paymentRef": "PSP-771202",
"invoiceNumber": "INV-20391",
"gross": 250.00,
"fee": 4.85,
"net": 245.15,
"paidOn": "2026-06-04",
"fileId": "settlement_2026-06-04"
}using System.Text.Json; using CC.Art2link.Pipelines.Domain.Models.PipelineComponents; public sealed class SettlementConfig { // No per-message values here: the file id is derived from the rows. } public sealed class SettlementCsvDisassembler : PipelineComponentBase<SettlementConfig> { public override string Name => "SettlementCsvDisassembler"; protected override Task<PipelineComponentOutput> ExecuteAsync( PipelineComponentInput input, SettlementConfig config, CancellationToken cancellationToken) { try { var lines = input.Body .Split('\n') .Select(l => l.TrimEnd('\r')) .Where(l => l.Length > 0) .ToList(); var messages = new List<PipelineMessage>(); foreach (var line in lines.Skip(1)) // skip header row { var c = line.Split(','); var json = JsonSerializer.Serialize(new { paymentRef = c[0], invoiceNumber = c[1], gross = decimal.Parse(c[2]), fee = decimal.Parse(c[3]), net = decimal.Parse(c[4]), paidOn = c[5], fileId = $"settlement_{c[5]}" // from paid_on }); messages.Add(new PipelineMessage { Body = json, MessageType = "SettlementTransaction" }); } return Task.FromResult(new PipelineComponentOutput { Success = true, Messages = messages }); } catch (Exception ex) { return Task.FromResult(new PipelineComponentOutput { Success = false, ErrorMessage = $"SettlementCsvDisassembler: {ex.Message}", Exception = ex }); } } }
Create the Scheduler receive port SettlementClock; its tick is what wakes the fetch port (Step 6).
| Setting | Value |
|---|---|
| Adapter | Scheduler |
| Repeat Every | 1 |
| Repeat Unit | Days |
| Daily Window | opening 06:00:00 AM |
Create a two-way SFTP Caller send port SettlementFetch with the download command against the drop directory, fetching everything in it: the provider writes exactly one dated file there each night (settlement_2026-06-04.csv), so the fixed directory is the stable part of the convention, not the filename. Subscription: {{Config.PortName}} == "SettlementClock". (The dropdowns can also create types and components inline; building the dependencies first keeps each dialog a selection.)
| Setting | Value |
|---|---|
| Adapter | SFTP Caller |
| Way | Two |
| Host | the provider’s server |
| Command | Download |
| Remote Directory | /outbound/settlements |
Set Authentication to ProviderSftp, from Step 2.
The fetched file rides back as the response and re-enters the bus through the port’s Response stages: set Message Type Fallback to SettlementFile, from Step 1, then select the SettlementCsvDisassembler, from Step 4, to debatch it.
Create a send port ReconcilePayment on the SQL Caller, one-way, calling the procedure from Step 3. Subscription: {{Message.MessageType}} == "SettlementTransaction".
| Setting | Value |
|---|---|
| Adapter | SQL Caller |
| Way | One |
| Command Type | StoredProcedure |
| Command Text | dbo.ReconcilePayment |
Set Authentication to InvoiceDbSql, from Step 2.
The procedure’s four values go under Input Parameters, one key/value row each, binding the promotions defined on SettlementTransaction in Step 1:
| Parameter | Value |
|---|---|
| PaymentRef | {{Promoted.SettlementTransaction.PaymentRef}} |
| InvoiceNumber | {{Promoted.SettlementTransaction.InvoiceNumber}} |
| Net | {{Promoted.SettlementTransaction.Net}} |
| PaidOn | {{Promoted.SettlementTransaction.PaidOn}} |
On both send ports, SettlementFetch and ReconcilePayment, set Exception Message Type to SettlementFailed, from Step 1.
Create the OpsAlert O365 Mail port for finance operations subscribing on {{Message.MessageType}} == "SettlementFailed":
| Setting | Value |
|---|---|
| Authentication | O365Ops, from Step 2 |
| From | noreply@acme.example |
| To | ops@acme.example |
| Subject | Art2link alert: {{Message.MessageType}} |
Everything you configure is live the moment you save it, there is nothing to deploy. Start the send ports first, ReconcilePayment, OpsAlert and SettlementFetch, then the receive port SettlementClock.
Set Tracking to Enabled + Body on every port the flow touches, so you can walk each run step by step, with its message body, in tracking.
Stage a test file with a handful of transactions, one of them referencing a non-existent invoice, and trigger the tick. To avoid waiting for the 06:00 window, temporarily set SettlementClock to Repeat Every 1 / Minutes, then restore the nightly window once the test passes.
In tracking: one fetch, N published transactions, N−1 successful reconciliations, and exactly one SettlementFailed alert for the mismatch. Fix the invoice and replay just that record.