Skip to content
Art2link ESB v2.02 LTS HomeDocumentationBlogContact
Tutorials/Routing & pub-sub/Settlement file debatch

Settlement file debatch

Each night Acme’s payment provider drops off one big file. Inside it are hundreds of payments the provider collected that day, one line per payment, and each line should mark one of Acme’s invoices as paid. The provider leaves the file on a server for Acme to collect, early the next morning. The goal is to match every payment to its invoice and mark it paid before the finance team starts work. The important part is what happens to a bad line. Sometimes a payment refers to an invoice Acme cannot find, a typo or a payment for something not yet billed. Handling the file as one all-or-nothing batch would let that single bad line fail the entire night’s work. Instead, each payment is handled on its own, so the good ones all go through and only the few that cannot be matched are set aside for a person to look at in the morning.

So the work is: collect the nightly file, break it into one payment at a time, and mark each invoice paid, with the bad lines pulled out individually rather than sinking the whole batch. Breaking a batch into its independent records is a well-known move, the splitter (or debatch), and it is the right one here because one payment’s fate has nothing to do with the next one’s.

Each payment marks one invoice paid, on its own nightly file Split the file one payment each Mark invoice paid matched payment Mark invoice paid matched payment No matching invoice set aside for a person, rest still complete

Here is how Art2link ESB builds it. Collecting the file is the same scheduled-download shape used elsewhere: a Scheduler receive port ticks at 06:00 (a receive port is an entry point that starts a flow, and this one runs on a timer), then a two-way SFTP Caller send port fetches the night’s dated file, settlement_2026-06-04.csv, from the provider’s drop directory. A send port is an exit that acts on a message, and an adapter is the connector that teaches it how to talk to one kind of system, here a file server over SFTP. Everything moves through the bus, the shared message backbone that flows publish onto and subscribe from.

The difference is what happens on the way back in. The fetched file is labelled with a fallback message type of SettlementFile, then a disassembler pipeline component (a small reusable piece of code that runs inside a port to reshape or split a message) breaks it apart: one published SettlementTransaction per record, the component labelling each emitted message so the bus carries clean JSON, not the raw file. The file was just the provider’s packaging, not the real unit of work. A SQL Caller send port then subscribes to that transaction type (a subscription is a plain rule a port matches against messages on the bus) and calls a reconciliation procedure: match on payment reference, mark the invoice paid, record the fee.

EXPRESSIONsubscription
{{Message.MessageType}} == "SettlementTransaction"
settlement file (SFTP fetch) Splitter one msg per record BUS SQL Caller mark invoice paid unmatched record dead-letter + ops alert Invoice DB

This is the example to contrast with the invoice line loop. Debatch when the records are independent, transaction 214's fate has nothing to do with transaction 215's. Keep the batch whole and loop when the elements only make sense inside their parent.

When it fails. Independence is the point. A transaction whose payment reference matches no invoice fails its procedure alone: the shared exception type SettlementFailed re-publishes it payload-intact and the O365 Mail alert reaches finance operations, while the other records complete. The morning email lists three unmatched payments, not one failed night; each is replayed from tracking after the invoice mismatch is resolved. The fetch port carries the same exception type, so a missing file raises the same alarm, the re-published payload tells you whether it was the whole file or a single record.

EXPRESSIONfailure subscription
{{Message.MessageType}} == "SettlementFailed"
Stamp every record with its batch. Have the splitter carry the file id onto each transaction (a canonical field, here fileId), so "which file did this payment come from?" is answerable, a whole batch can be traced after the fact, and a re-fetched file can be recognised, reconciliation should be idempotent against the payment reference regardless.
Why a component, not a map. A map’s source format must be XML or JSON, so the settlement CSV can never be a map source, and debatching a flat file is a pipeline component’s job in any case. The component reads the raw CSV body and emits SettlementTransaction JSON, classifying each message so the bus carries JSON. If a normalization step were ever wanted on top, it would run on that JSON, never on the CSV.

Build it, step by step. The steps run in dependency order: every object is created before the object that selects it.

Before you start. Create the Application this flow lives in, a Name plus a code-safe Namespace, under Applications, and select it, so every artifact you create below lands inside it.
1
Step One
Create the message types
The three types

Under the application’s Message types, create the three types this flow routes on. A message type is a name plus a format, no schema required:

NameFormatPurpose
SettlementFileCSVthe provider’s nightly raw file as it downloads, the fallback type the disassembler component reads; never a map source
SettlementTransactionJSONone settlement record, the unit of work the splitter publishes and classifies
SettlementFailedJSONthe exception type the failure path publishes under (Step 8)

Promotions live on the message type, so add them while you are here. The reconcile port’s parameters (Step 7) bind these values, and adapter parameters are plain strings: they bind promotions, not body paths.

Promotions on SettlementTransaction
PromotionPath
PaymentRef$.paymentRef
InvoiceNumber$.invoiceNumber
Net$.net
PaidOn$.paidOn

2
Step Two
Create the Authentications

Three external parties, three credentials. Under the application’s Authentications, the dialog asks for a Name, the Adapter it pairs with, and a Definition, the credential shape that adapter offers, with the Application preset; the Definition decides the config section that follows.

Provider server, for the fetch port
SettingValue
NameProviderSftp
AdapterSFTP Caller
DefinitionSFTP Password Authentication
Username / Passwordthe provider’s SFTP login
Host Key Fingerprintthe provider server’s fingerprint
Invoice database, for the SQL Caller
SettingValue
NameInvoiceDbSql
AdapterSQL Caller
DefinitionSQL Server Connection
Connection String (Database Config)the invoice database’s connection string, credentials included
Alert mailbox, for O365 Mail
SettingValue
NameO365Ops
AdapterO365 Mail Sender
DefinitionMicrosoft Graph
Tenant Id / Client Id / Client Secret (Graph AuthConfig)an app registration with Mail.Send granted

3
Step Three
Create the database objects

Create the table and procedure in the invoice database; the reconcile port (Step 7) calls the procedure. An unknown invoice THROWs, that is the per-record failure the tutorial relies on, raising SettlementFailed for that one transaction while the rest post. Matching on the payment reference keeps it idempotent under replay.

The invoices table and the reconcile procedure
SQLInvoices table + ReconcilePayment
CREATE TABLE dbo.Invoices (
    InvoiceNumber VARCHAR(40)   NOT NULL PRIMARY KEY,
    Status        VARCHAR(20)   NOT NULL CONSTRAINT DF_Inv_St DEFAULT 'Open',
    PaidAmount    DECIMAL(18,2) NULL,
    PaymentRef    VARCHAR(40)   NULL,
    PaidOn        DATE          NULL
);

CREATE OR ALTER PROCEDURE dbo.ReconcilePayment
    @PaymentRef    VARCHAR(40),
    @InvoiceNumber VARCHAR(40),
    @Net           DECIMAL(18,2),
    @PaidOn        DATE
AS
BEGIN
    SET NOCOUNT ON;
    IF NOT EXISTS (SELECT 1 FROM dbo.Invoices WHERE InvoiceNumber = @InvoiceNumber)
        THROW 50001, 'Unknown invoice for settlement', 1;

    -- Idempotent: a replayed payment with the same ref changes nothing
    UPDATE dbo.Invoices
    SET Status = 'Paid', PaidAmount = @Net,
        PaymentRef = @PaymentRef, PaidOn = @PaidOn
    WHERE InvoiceNumber = @InvoiceNumber
      AND (PaymentRef IS NULL OR PaymentRef <> @PaymentRef);
END;

4
Step Four
Create the pipeline component

Under the application’s Pipeline components, create the disassembler SettlementCsvDisassembler: one message per data row, each stamped with a file id derived from the row’s paid_on date, read from the file itself rather than configured (components are data-driven), so a transaction can be traced back to its batch. Classification (SettlementTransaction) happens on each outgoing message.

The nightly file

The provider’s nightly settlement CSV as it downloads:

CSVsettlement_2026-06-04.csv
payment_ref,invoice_no,gross,fee,net,paid_on
PSP-771202,INV-20391,250.00,4.85,245.15,2026-06-04
PSP-771203,INV-20402,89.90,1.92,87.98,2026-06-04
PSP-771204,INV-20377,1180.00,21.10,1158.90,2026-06-04
One debatched transaction

One debatched SettlementTransaction as the splitter publishes it:

JSONSettlementTransaction
{
  "paymentRef": "PSP-771202",
  "invoiceNumber": "INV-20391",
  "gross": 250.00,
  "fee": 4.85,
  "net": 245.15,
  "paidOn": "2026-06-04",
  "fileId": "settlement_2026-06-04"
}
The component code
C#SettlementCsvDisassembler.cs
using System.Text.Json;
using CC.Art2link.Pipelines.Domain.Models.PipelineComponents;

public sealed class SettlementConfig
{
    // No per-message values here: the file id is derived from the rows.
}

public sealed class SettlementCsvDisassembler
    : PipelineComponentBase<SettlementConfig>
{
    public override string Name => "SettlementCsvDisassembler";

    protected override Task<PipelineComponentOutput> ExecuteAsync(
        PipelineComponentInput input,
        SettlementConfig config,
        CancellationToken cancellationToken)
    {
        try
        {
            var lines = input.Body
                .Split('\n')
                .Select(l => l.TrimEnd('\r'))
                .Where(l => l.Length > 0)
                .ToList();

            var messages = new List<PipelineMessage>();
            foreach (var line in lines.Skip(1))   // skip header row
            {
                var c = line.Split(',');
                var json = JsonSerializer.Serialize(new
                {
                    paymentRef    = c[0],
                    invoiceNumber = c[1],
                    gross         = decimal.Parse(c[2]),
                    fee           = decimal.Parse(c[3]),
                    net           = decimal.Parse(c[4]),
                    paidOn        = c[5],
                    fileId        = $"settlement_{c[5]}"   // from paid_on
                });
                messages.Add(new PipelineMessage
                {
                    Body        = json,
                    MessageType = "SettlementTransaction"
                });
            }

            return Task.FromResult(new PipelineComponentOutput
            {
                Success = true, Messages = messages
            });
        }
        catch (Exception ex)
        {
            return Task.FromResult(new PipelineComponentOutput
            {
                Success      = false,
                ErrorMessage = $"SettlementCsvDisassembler: {ex.Message}",
                Exception    = ex
            });
        }
    }
}

5
Step Five
Create the Scheduler receive port
General

Create the Scheduler receive port SettlementClock; its tick is what wakes the fetch port (Step 6).

SettingValue
AdapterScheduler
Repeat Every1
Repeat UnitDays
Daily Windowopening 06:00:00 AM

6
Step Six
Create the fetch port
General

Create a two-way SFTP Caller send port SettlementFetch with the download command against the drop directory, fetching everything in it: the provider writes exactly one dated file there each night (settlement_2026-06-04.csv), so the fixed directory is the stable part of the convention, not the filename. Subscription: {{Config.PortName}} == "SettlementClock". (The dropdowns can also create types and components inline; building the dependencies first keeps each dialog a selection.)

SettingValue
AdapterSFTP Caller
WayTwo
Hostthe provider’s server
CommandDownload
Remote Directory/outbound/settlements
Pair the credential

Set Authentication to ProviderSftp, from Step 2.

Response side

The fetched file rides back as the response and re-enters the bus through the port’s Response stages: set Message Type Fallback to SettlementFile, from Step 1, then select the SettlementCsvDisassembler, from Step 4, to debatch it.


7
Step Seven
Create the reconcile port
General

Create a send port ReconcilePayment on the SQL Caller, one-way, calling the procedure from Step 3. Subscription: {{Message.MessageType}} == "SettlementTransaction".

SettingValue
AdapterSQL Caller
WayOne
Command TypeStoredProcedure
Command Textdbo.ReconcilePayment
Pair the credential

Set Authentication to InvoiceDbSql, from Step 2.

Input Parameters

The procedure’s four values go under Input Parameters, one key/value row each, binding the promotions defined on SettlementTransaction in Step 1:

ParameterValue
PaymentRef{{Promoted.SettlementTransaction.PaymentRef}}
InvoiceNumber{{Promoted.SettlementTransaction.InvoiceNumber}}
Net{{Promoted.SettlementTransaction.Net}}
PaidOn{{Promoted.SettlementTransaction.PaidOn}}

8
Step Eight
Wire the failure path
Set the Exception Message Type

On both send ports, SettlementFetch and ReconcilePayment, set Exception Message Type to SettlementFailed, from Step 1.

Create the OpsAlert send port

Create the OpsAlert O365 Mail port for finance operations subscribing on {{Message.MessageType}} == "SettlementFailed":

SettingValue
AuthenticationO365Ops, from Step 2
Fromnoreply@acme.example
Toops@acme.example
SubjectArt2link alert: {{Message.MessageType}}

9
Step Nine
Start the ports and test
Start in dependency order

Everything you configure is live the moment you save it, there is nothing to deploy. Start the send ports first, ReconcilePayment, OpsAlert and SettlementFetch, then the receive port SettlementClock.

Turn on tracking

Set Tracking to Enabled + Body on every port the flow touches, so you can walk each run step by step, with its message body, in tracking.

Stage a test file and trigger the tick

Stage a test file with a handful of transactions, one of them referencing a non-existent invoice, and trigger the tick. To avoid waiting for the 06:00 window, temporarily set SettlementClock to Repeat Every 1 / Minutes, then restore the nightly window once the test passes.

Check the run in tracking

In tracking: one fetch, N published transactions, N−1 successful reconciliations, and exactly one SettlementFailed alert for the mismatch. Fix the invoice and replay just that record.

Turn bodies off in production. Enabled + Body is the most expensive tracking level, extra processing and database space, and it records every payload including successful runs. Once the flow is proven, set the ports to Only on Error instead: failures still capture the full body for diagnosis, while healthy runs are not recorded.