Middleware that keeps an ERP, a CRM and a storefront agreeing about the same customer, the same order and the same piece of stock. Three systems, three different ideas of what an identifier is, and no shared transaction between any of them.
Last updated 2026-08-19.
Built with Node.js, Express 5, chokidar, ftp-srv, Salesforce Apex REST, OAuth2, SAP file interface.
A fastener manufacturer runs SAP for operations, Salesforce for sales, and a storefront for customers. All three hold a version of the same customer, the same order and the same stock figure. Keeping them in agreement was the job of a commercial integration platform, and this project replaced it.
I built the middleware that took over. It is a Node process that runs an FTP server SAP can drop files onto, watches those folders, transforms whatever lands there into the shape each downstream system wants, sends it, and then handles the part everyone forgets: what happens when one of the three says no.
Replacing a licensed integration platform is a specific kind of brief. The contract already existed and was not mine to change: SAP was already configured to write to particular folders, the Salesforce REST resources were already deployed, and both ends had already agreed on field shapes. The requirement was to reproduce that contract exactly, on something the client owns, that one person can run on one machine.
The unglamorous framing is the accurate one. This is not a product with users. It is a piece of plumbing whose entire job is to be correct about identity and honest about failure, because the failure mode is not a broken page. It is a sales order that exists in Salesforce, does not exist in SAP, and shows as reserved stock on the storefront.
The interesting decision is that failure is modelled as a file. When a handler cannot deliver something, it does not just log and drop it. It writes the original payload plus the error text into an error folder that is itself watched, which triggers an error handler that reports the failure back into Salesforce as a record. The same mechanism the happy path uses carries the sad path. That is why the system has 32 handlers for 14 object types.
This ran in UAT against real Salesforce and real SAP files. It never carried production traffic, so this page contains no volume, uptime or incident figures. The Status tab says exactly what was and was not exercised.
Three systems, each of which is correct about a different part of the business, none of which can see the other two, and a commercial integration platform already sitting between them that this project replaced.
This is not a greenfield story. Before any of this, the three systems were joined by a commercial ETL platform, and the work I was brought in for was to replace it with something the client owned outright. The Salesforce org still carries the evidence: a connected app and a remote site setting named after that platform, both still sitting in the org retrieve alongside the REST resources my middleware calls.
That changes the nature of the problem in a way worth being explicit about. The requirements were not gathered from scratch. They were already encoded in a running system, in the folder names SAP was configured to write to, in the Apex resources that were already deployed and receiving traffic, and in the field shapes both ends had already agreed on years earlier. My job was to reproduce a contract I did not design, exactly, and then be better at the parts that were painful.
It also sets the bar. Replacing a commercial integration platform with four npm packages is only a good idea if the result is genuinely easier to run and easier to reason about than the thing it replaced. That constraint is behind most of the decisions on the Engineering tabs, and it is the reason the dependency count is a design goal rather than an accident.
| System | Authoritative for | Blind to |
|---|---|---|
| SAP | Stock, pricing, invoices, dispatch, the actual sales order once it is real | Anything a salesperson did that has not been committed yet, and everything the storefront knows about a customer |
| Salesforce | The relationship, the pipeline, complaints, what was promised | Whether the warehouse can actually fulfil it |
| Storefront | What a customer sees, orders and pays for | Whether the price and stock it is showing are still true |
Every one of those systems is right within its own boundary. The problem is not that any of them is wrong. It is that a single business fact, one customer placing one order for one product, has to exist in all three at once, and each of them will give that fact a different primary key.
SAP calls a customer by its customer number. Salesforce calls the same customer by its record ID. The storefront calls it by its own ID. None of these three can be derived from the other two. There is no shared key, no shared database, and no distributed transaction that could make all three writes succeed or fail together.
the same customer, three times over SAP customer "7100000453" Salesforce crmid "a0X5j00000ABCDEfg" Storefront ecomid "cus_91be2" nothing in any of those three strings tells you the other two
So the actual job is not moving data. It is maintaining a mapping between three identifier spaces, where the mapping itself only comes into existence as a side effect of a successful create in each system, and where any of those creates can fail independently.
This is the constraint that shapes everything. SAP writes files. It will connect to an FTP endpoint, drop a JSON document into a folder, and consider the matter closed. It will not wait for a response, it will not retry on your behalf, and it will not tell you whether the thing it sent worked out. If you want to acknowledge something, you write a file back and SAP picks it up on its own schedule.
Once you accept that, the shape of the solution is forced. The integration surface has to be a folder structure, the protocol has to be file naming and file placement, and acknowledgement has to be its own separate flow rather than a return value. A large part of the design is just taking that seriously instead of fighting it.
The failure that matters is partial success. A customer creation goes to Salesforce and succeeds, then goes to the storefront and fails. Now Salesforce believes the customer exists everywhere, the storefront has never heard of them, and SAP is waiting for a file that describes a customer with an ID from a system that rejected it.
There is no rollback available. You cannot un-create the Salesforce record because Salesforce already accepted it and something else may already reference it. So the only honest option is to make the partial state visible and routable: record precisely which leg failed, with which error, carrying whichever identifiers were successfully obtained, and put that where a human or a downstream system will actually see it. That requirement is the reason the error path in this system is as large as the happy path.
A single Node process that is simultaneously an FTP server, a file watcher and an HTTP API, wired so that every route in and out of the system converges on one routing table.
SAP Salesforce / Storefront
| |
| FTP PUT *.json | HTTPS POST /
v | source-system, type
+---------------------------------------------------+ |
| ftp-srv, embedded, custom WindowsFileSystem | |
+---------------------------------------------------+ |
| file lands on the share |
v v
+------------------+ +---------------------------+
| chokidar watcher | POST /inbound | Express, 3 endpoints |
| 23 folders | -----------------> | / /inbound /health |
+------------------+ localhost only +---------------------------+
shared secret |
v
routing table lookup
folder path or headers
|
+---------------------+---------------------+
v v
transform helper transform helper
per object type per object type
| |
v v
Salesforce Apex REST Storefront REST
OAuth2 client_credentials OAuth2 + refresh
one Windows host, one process, four npm dependencies
The whole thing runs on express, chokidar, ftp-srv and dotenv. HTTP calls out use the fetch that ships with Node. There is no queue library, no ORM, no logging framework, no ETL framework. That was a deliberate call: this process runs on a machine inside the client's network that somebody else will eventually have to maintain, and every dependency is a thing that can break during an upgrade on a box nobody wants to touch.
It is worth being straight about the cost of that choice, because it shows up later on this page. No queue means no durable retry and no dead letter queue. That is a real limitation, not a hidden virtue, and the Debt tab treats it as such.
SAP drops a file. The watcher notices, tells the server, the server finds the handler that owns that folder, and the handler transforms and fans the payload out to Salesforce and the storefront independently. Success archives the file. Failure writes an enriched copy into the object's error folder.
Salesforce or the storefront POSTs to the root endpoint with source-system and type headers. The handler forwards to the opposite system, merges the identifier that comes back into the payload, and writes the merged result into an outbound folder for SAP to collect.
SAP writes its own verdict back onto the share, into a success or error folder. Those folders are watched too, so SAP's reply is just another inbound file with its own handler, which pushes the SAP identifier into Salesforce and the storefront.
The reason all three look the same is that they are the same. There is one notion of an event here: a JSON file appeared in a folder, or an HTTP request arrived with two headers. Both resolve to a handler through a lookup table built at startup. Adding an object type is writing a transform helper and a handler file, and the routing picks it up on the next restart with nothing to register.
The watcher does not call the handler directly. It sends an HTTP POST to 127.0.0.1 with the file path in a header, and the server routes from there. That looks like an unnecessary hop, and in a sense it is.
It buys two things. The routing table, the auth check and the logging setup all live at the HTTP boundary, so a file arriving and a request arriving go through the same funnel and produce the same shape of log. And the watcher stays a dumb producer that knows nothing about handlers, which means anything else that can make an HTTP request can also inject work without becoming part of the process. The cost is that a local network call can now fail, which is why the notification has its own retry with exponential backoff.
Salesforce sits behind Apex REST resources that accept a single string parameter containing serialised JSON. So every payload heading to Salesforce is wrapped: the real object is stringified and put under a json key, and the whole thing is sent as the body. The transform helpers do this per object type.
what the storefront receives what Salesforce receives
{ {
"customer": "7100000453", "json": "{\"customer\":\"7100000453\",
"name1": "...", \"name1\":\"...\"}"
"ecomid": "" }
}
That single quirk has a long tail. It means a payload cannot be inspected by looking at the top level keys, it means the error handler has to work out whether a file it picked up is already wrapped before it wraps it again, and it means every merge function has to unwrap, merge and rewrap. Half the awkwardness in the transform layer traces back to this one contract detail.
The integration surface is a folder tree. That is the API, and the folder a file sits in is the only thing that says what it means.
FTP_Share/
CustomerCreation/
inbound/ SAP drops new or changed customers here
outbound/ the middleware writes SAP-bound files here
Success/ SAP writes its acknowledgement here
error/ SAP writes a rejection here, and so does the middleware
archive/ processed inbound files end up here
Complaint/ same five
SO/ same five
OpenSO/ same five
Invoice/ Discount/ Payment/ Refund/ PriceBook/
SKUOpening/ SKUUpdate/ DispatchStatus/
StockReservation/ StockUnreservation/
Fourteen object types, each with the same five folders. The uniformity is the point: a new object type is a folder tree that looks exactly like the last one, and anybody debugging can guess where to look without asking.
This is the subtle part of the contract. An error folder is written to by two completely different producers. SAP writes rejections into it, because that is how SAP reports that it refused something. The middleware also writes into it, when its own outbound call to Salesforce or the storefront failed.
Those two producers do not write the same shape. SAP writes a flat object. The middleware writes the Salesforce envelope, an object with a single json key holding a string. Both land in the same folder and both trigger the same handler, so the handler has to detect which it is holding. It checks whether the json property is a string. If it is, the file is passed through untouched, because wrapping it again would produce a doubly encoded payload that Salesforce would deserialise into a string rather than an object.
function transformInboundForSf(inbound) {
if (typeof inbound.json === "string") return inbound; // already wrapped
return { json: JSON.stringify(inbound) }; // flat, from SAP
}
Six lines, and they exist entirely because one folder has two authors with two conventions. It would have been cleaner to give the middleware its own error folder separate from SAP's. It was not done that way because the error folder is also the place a human looks when something has gone wrong, and splitting it means looking in two places to answer one question.
Names are built rather than copied. An archived or errored file is renamed to include an ISO timestamp with colons and dots replaced, because the original SAP file names are not unique across time and a second file with the same name would silently overwrite the first in the archive. Outbound files are named after the business identifier plus that timestamp, so the SAP side can see at a glance which record a file concerns.
inbound file customer_7100000453.json archived as customer_7100000453_2026-03-20T12-05-09-974Z.json outbound to SAP 7100000453_2026-03-17T05-30-08-242Z.json error from storefront complaint_ecommerce_2026-03-17T05-13-26-611Z.json error from Salesforce complaint_salesforce_2026-03-17T05-23-35-061Z.json
The error file name encoding the source system is a small thing that pays for itself immediately. When there are forty files in an error folder, being able to see that seven came from the storefront and two from Salesforce, without opening any of them, is the difference between a two minute triage and a twenty minute one.
A file has no delivery receipt. When the middleware writes an outbound file for SAP, it has no way of knowing whether SAP ever read it. When SAP writes an acknowledgement, it has no way of knowing whether the middleware processed it. The only evidence of progress is that a file appeared somewhere else later.
That is why the acknowledgement folders exist as first class flows rather than as an afterthought, and it is also the honest reason this system cannot claim end to end guarantees. It can guarantee that it acted on every file it saw. It cannot guarantee that it saw every file, and neither can anything else in this arrangement.
Three identifiers per record, none derivable from the others, all of which have to end up written back into all three systems.
The account transform helper opens with the mapping written out as a comment, and the rest of the system follows it consistently: SAP identifies a customer by customer, Salesforce by crmid, the storefront by ecomid. Every object type repeats the same triple with its own field names, and the error payloads normalise them into sap_id, crm_id and ecom_id.
| System | Field on the wire | Where it comes from |
|---|---|---|
| SAP | customer, salesorder, complaint_no depending on object | Present on the inbound file, or returned in the acknowledgement file after SAP creates the record |
| Salesforce | crmid | Read out of the Salesforce response at successRecords[0].crm_id after a successful create |
| Storefront | ecomid | Read out of the storefront response at successRecords[0].ecomid after a successful create |
This is the part that makes the flow shape what it is. You do not know the Salesforce ID until Salesforce has accepted the record. You do not know the storefront ID until the storefront has accepted it. And SAP's ID does not exist until SAP has processed the outbound file and written back an acknowledgement, which happens minutes or hours later on SAP's own schedule.
So the mapping assembles itself over time, in pieces, across two protocols. A record can be in a state where two of its three identifiers are known and the third does not exist yet, and that state is not an error. It is the normal condition of every record for some window of its life.
Each transform helper carries a pair of merge functions whose entire job is to fold a newly learned identifier back into a payload without losing what was already there. They are deliberately defensive, because the response shape from either system can be missing the field entirely if something partially succeeded.
function mergeOutboundAndEcomResponse(outbound, ecomResponse) {
const sfPayload = {
...outbound,
ecomid: ecomResponse.successRecords?.[0]?.ecomid || outbound?.ecomid || "",
};
return { json: JSON.stringify(sfPayload) };
}
The fallback chain matters more than it looks. It reads the new identifier if one came back, otherwise keeps whatever the payload already carried, otherwise writes an empty string rather than undefined. Empty string survives JSON serialisation into a field Salesforce can accept and later fill in. Undefined disappears from the serialised payload entirely, and a missing field and a blank field mean different things to a system that is trying to work out whether it has seen this record before.
When SAP finishes creating a record it writes an acknowledgement file into that object's success folder. That folder is watched, so the acknowledgement is picked up like any other inbound file, and its handler pushes the SAP identifier into Salesforce through the /acknowledgment resource and into the storefront through a link endpoint. That is the moment the third identifier lands, and the moment the mapping is complete in all three systems.
The error acknowledgement is the mirror image. It goes to Salesforce through /acknowledgmentError, carrying whichever identifiers are known plus the error text, so that the CRM ends up holding a record of the failure attached to the account or order it concerns.
The identifier triple is verifiable in the middleware source. In the Salesforce org retrieve I could confirm the receiving side of it: there are Apex REST resources at /account, /acknowledgment and /acknowledgmentError matching exactly what the middleware posts to, and a custom sap_error__c object with error_message__c and object_type__c fields plus lookups to Account, Sales Order and Stock Reservation, which is where a reported failure lands.
What I could not confirm from the org retrieve: a dedicated field holding the storefront identifier. The retrieve is old and partial, so I am recording that as unknown rather than as evidence it does not exist. The middleware certainly sends ecomid, and something on the Salesforce side accepts the payload.
Two lookup tables, both built at startup by reading a directory, and one rule that decides which table applies.
At boot, the inbound middleware reads four handler directories, requires every .js file it finds, and keys each module by the folder path it exports. The outbound middleware does the same across two directories and keys by the source system and type the module exports. Neither has a registration file to keep in sync.
inbound table key: absolute folder path
C:\FTP_Share\CustomerCreation\inbound -> handler
C:\FTP_Share\CustomerCreation\success -> handler
C:\FTP_Share\CustomerCreation\error -> handler
outbound table key: two request headers
salesforce / account -> handler
salesforce / paymentRemarks -> handler
ecommerce / soCreation -> handler
The reason for auto-discovery is that this system grew one object type at a time over eight months, and each new type is the same four files. A registry would have been a fifth file that somebody forgets, and forgetting it produces a handler that exists, looks correct, and is never called. That failure is silent and genuinely unpleasant to diagnose.
Each handler computes its own folder with path.resolve against the share root from the environment, and the router computes path.dirname of the incoming file path and looks it up. Both sides go through path.resolve, so separator style and relative segments normalise to the same string on both sides.
It is worth naming what this depends on. The lookup is an exact string match on a Windows path, and it only works because Windows treats paths case-insensitively at the filesystem layer. On the live share the acknowledgement folders are named Success with a capital S, while the code resolves success in lower case. The two match on Windows and would not match on Linux. That is a portability constraint I would fix by normalising case in the lookup key rather than by renaming folders, since SAP's configuration also points at those folder names.
The inbound discovery validates: it checks that the module exports a folder and that handler is a function, logs and skips anything that fails, and wraps the require in a try so that one broken file cannot stop the other thirty from loading. The outbound discovery does none of that, and carries a TODO saying so.
// inbound.js
if (handlerModule.folder && typeof handlerModule.handler === "function") {
handledDirectories[handlerModule.folder] = handlerModule.handler;
} else {
console.debug(false, `[INBOUND] Skipping non-handler file: ${handlerPath}`);
}
// outbound.js
// TODO: Add validation
sourceSystemsAndTypes[handlerModule.sourceSystem][handlerModule.type] =
handlerModule.handler;
The asymmetry is real and I am not going to dress it up. The inbound directories contain a helper factory that is not itself a handler, which forced the validation there. The outbound directories happen to contain only handlers, so the missing check never bit. It is a latent trap: the first non-handler file dropped into an outbound directory registers undefined against a route, and the failure surfaces later as a type error inside a request rather than as a startup complaint.
The router is narrow about what it will act on. The path has to end in .json, the operation has to be an upload or a rename, and the parent directory has to be in the table. A rename additionally requires that the old parent directory was not in the table, so moving a file between two watched folders does not fire twice.
Anything else returns 200 without logging. That silence is deliberate: the share carries temp files, partial writes and the occasional stray document, and a log entry per irrelevant filesystem event turns the log into noise exactly when it needs to be readable.
Every rejection on the inbound endpoint returns 200. A non-localhost caller gets 200, a bad shared secret gets 200, missing headers gets 200. The intent is that the notifier should not retry something that will never succeed, and the failure is recorded in the log rather than in the status code.
The leak is that the router itself returns 500 when a handler throws, and the watcher retries on any non-ok response up to five times. A handler that fails before sending anything is harmless to retry. A handler that fails after its outbound calls have already succeeded, for example while writing its archive copy, gets replayed, and there is nothing in the system that would recognise the replay. It is a narrow window, but it is the one place where the retry policy and the absence of idempotency meet.
From SAP writing a file to the file being archived, with the two systems downstream deliberately isolated from each other.
The FTP server is embedded in the same process, which means SAP connects to the middleware directly rather than to a separate FTP daemon that would then need its own monitoring. It runs with anonymous access off, a single credential pair from the environment, and passive ports pinned to a fixed range so a firewall rule can be written once.
The interesting part is a custom filesystem subclass. SAP's client lists directories with a wildcard, DIR *.json, and the library's default filesystem treats that path as a file, fails to stat it, and errors before listing anything. The subclass overrides two methods: when a path contains a wildcard, stat returns the parent directory instead so validation passes, and list strips the pattern, lists the parent, and filters by extension. Everything without a wildcard falls straight through to the parent implementation.
That is roughly forty lines that exist entirely because of how one client formats one command. It is also the kind of thing that never appears in a design document and takes a day to find.
A watcher that fires on file creation will happily hand you a file that FTP is still writing. The watcher configuration handles this with awaitWriteFinish, holding the event until the file size has been stable for two seconds with a 100 millisecond poll. Depth is capped at one level so a nested archive folder does not generate events, initial contents are ignored so a restart does not reprocess everything already on the share, and hidden files and non-JSON files are dropped before anything else happens.
The ignoreInitial setting is a real tradeoff rather than a free win. It prevents a restart from replaying the entire share, which would be a genuine mess given there is no deduplication. It also means a file that arrives while the process is down is never seen at all. Given the choice between silently reprocessing hundreds of old files and silently missing the ones that arrived during a restart, the second is the smaller failure, but it is still a failure and the Debt tab lists it.
The watcher posts the file path to the local endpoint with a shared secret header, and retries up to five times with a doubling delay starting at one second. If all five fail it logs a critical error and gives up, leaving the file sitting in the inbound folder untouched.
This is the one retry loop in the system, and it covers the least likely failure, a local HTTP call to the same process. The calls that actually go over a network, to Salesforce and to the storefront, have no retry at all. That inversion is worth stating rather than hiding: the retry was added where a failure was observed during development, not where the risk analysis would have put it.
A handler reads and parses the file, then makes two completely separate attempts, each in its own try block with its own error variable. Salesforce failing does not stop the storefront call, and neither failure throws out of the handler.
let hasError = false, sfError = "", ecomError = "";
try { sfResponse = await sendToSalesforce(...); }
catch { hasError = true; sfError = error.responseBody?.errorRecords?.[0]?.message
|| error.responseBody?.message; }
try { ecomResponse = await sendToEcommerce(...); }
catch { hasError = true; ecomError = error.responseBody?.errorRecords?.[0]?.error
|| error.responseBody?.error; }
Independence is the right default here because the two systems serve different purposes and neither depends on the other having the record. A complaint that reaches Salesforce but not the storefront is still worth having in Salesforce. Blocking the second call on the first would convert one system being briefly unavailable into both systems missing the data.
The detail worth noticing is the error extraction. Both systems return structured errors, but with different field names, one nesting the text under message and the other under error, so the client layer attaches the parsed response body onto the thrown error and each handler digs the human readable line out with its own fallback chain. Without that, every error file would say Unexpected status code: 400 and nothing else, which tells a support person nothing they can act on.
When either leg fails, the handler writes the entire original payload plus both error fields into the object's error folder. Not a summary, and not just the error: the full inbound document, so that whoever or whatever picks it up has everything needed to understand and potentially replay the case without going back to find the original file.
That enriched file then triggers the error acknowledgement handler through the ordinary watcher path, which reports it into Salesforce. So a failure inside the middleware becomes a record in the CRM through exactly the same mechanism that a business document uses. There is no separate error pipeline to maintain.
On success the file is not moved. It is copied to the archive immediately, and a timer deletes the original from the inbound folder three minutes later. This was introduced so the SAP team could see the file still sitting where they put it while they checked whether it had been processed, which was the practical reality of joint testing across two teams and two systems.
It is honest to call this what it is: a testing affordance that is still switched on in the inbound handlers, with the plain rename that would be correct for production sitting commented out directly beneath it. It was removed from the acknowledgement handlers in a later commit but remains in the inbound path. It also relies on an in-memory timer, so a restart within that three minute window leaves the original in place permanently, where ignoreInitial guarantees it will never be looked at again.
Salesforce or the storefront pushes a record in, the middleware forwards it, folds the returned identifier into the payload, and leaves a file for SAP.
Everything outbound arrives on the root endpoint with two headers, source-system and type, and Basic authentication. The middleware chain checks the headers are present, checks they resolve to a registered handler, checks the credentials, and only then routes. A missing header is a 400 with the message naming which header, and an unknown type is a 400 that lists the types that are handled for that source system.
Listing the valid options in the rejection is a small thing that mattered a lot in practice. The consumer on the other end is somebody configuring an Apex callout who cannot see this codebase, and a 400 that says which values would have worked turns a support conversation into a self-service fix.
The ordering inside the account handler is deliberate and commented as such. The storefront call happens first, because that is where the ecommerce identifier comes from. Then the response is merged into the payload. Then the merged payload is written to the outbound folder for SAP. The comment in the source says it plainly: save first so the file exists for SAP even if anything after it fails.
That is the correct instinct for a system with no transaction. Identify the one artefact that downstream work depends on, produce it as early as it can possibly be correct, and treat everything after it as best effort. The alternative ordering, doing all the notifications and writing the file last, means a failure in a non-essential step costs SAP the document it was waiting for.
The handler's own docblock describes a six step flow including sending the merged data back to the storefront to update the CRM identifier, and the architecture document describes the same thing as a three way cross-acknowledgement: create in the storefront, acknowledge back to Salesforce with the storefront identifier, then acknowledge back to the storefront with the Salesforce identifier.
The code does not do that. It calls the storefront once, merges, and writes the file. The cross-synchronisation was removed in a commit that says so directly, Removing cross sync from account flow. The docblock above the function was not updated with it, and neither was the architecture document.
Two documents and one comment describe a flow the code stopped performing. I have left all three descriptions in place on this page rather than picking one, because the gap between them is more informative than either version alone. The Doc vs Code tab lists every instance I found.
When the storefront call fails, the handler writes an enriched error file, prefixes the message with a marker naming the stage that failed, and returns that same payload to the caller with a 200 status. Salesforce receives a body describing the failure rather than an HTTP error.
The reasoning is that a 500 back to an Apex callout produces a callout exception, which in Salesforce is handled where the callout was made and frequently means the record simply does not get its update. Returning 200 with a body the Apex code can read lets the Salesforce side record what happened against the record it already has. Whether that is the right call depends on how disciplined the caller is about inspecting the body, which is a coupling I would flag in a design review, but the failure is captured in the error file regardless of what the caller does with the response.
Every outbound handler wraps its whole body in a catch that writes an error file named UNEXPECTED_ERROR with a timestamp, pulling whatever identifiers it can find off the raw request body with a chain of fallbacks. If even that write fails, it falls back to constructing a minimal payload in memory so the caller still receives a structured response rather than a hanging request.
Two layers of fallback around an error path is usually a smell. Here it earns its place, because this is the boundary where a payload shape nobody anticipated arrives from an external system, and the difference between a logged stack trace and a file on the share is whether anybody finds out.
One factory produces every error handler, and the SAP acknowledgement is treated as an ordinary inbound file rather than as a special case.
The four object specific error handlers are each about fifteen lines. They resolve their folder, name their Salesforce endpoint, name an optional storefront endpoint, and call a factory that returns the actual handler. All the behaviour lives in one place.
const folder = path.resolve(process.env.SFTP_ROOT_DIR, "CustomerCreation", "error").toString();
const handler = createErrorHandler("CustomerCreation", folder,
"/acknowledgmentError", "/customers/link");
module.exports = { folder, handler };
This is the one abstraction in the codebase I would defend without hesitation, because it was extracted after the second copy rather than designed up front, and because the thing it varies is genuinely just configuration. Everything else in this system is a concrete file doing a concrete job.
The factory sends the error to Salesforce, then optionally to the storefront, each in its own try block, each logging on failure and continuing. Nothing throws out of the handler. The file is left in the error folder and explicitly not archived, with the log line saying it remains available for retry.
The honest reading of that line is that it describes an intention rather than a mechanism. Nothing re-reads the error folder. The watcher fires on file creation, and a file that is already there when the process starts is skipped by ignoreInitial. So if Salesforce is unavailable at the moment an error acknowledgement is attempted, the report is lost, and only the file remains as evidence. Retrying it means moving the file out and back in by hand.
Error payloads reach the factory from SAP, from Salesforce outbound handlers and from storefront outbound handlers, and each spells the identifier fields differently. The factory normalises with an explicit fallback chain per identifier before building either downstream payload.
const ecomId = logData.ecom_id || logData.ecomid || logData.Ecom_Id || "";
const crmId = logData.crm_id || logData.crmid || "";
const sapId = logData.sap_id || logData.sapid || logData.complaint_no
|| logData.Complent || "";
That chain is archaeology rather than design. Each alternative is a real spelling that a real producer emitted, including one that is simply misspelt. The correct fix is a shared serialiser that every producer uses, and I would take that over the fallback chain in a rewrite. What the chain buys in the meantime is that a record with a valid identifier under an unexpected key still gets linked correctly instead of arriving with a blank field.
Four objects have success acknowledgement handlers: customer creation, sales order, open sales order and complaint. These are the objects SAP actually creates records for, and therefore the only ones where SAP has an identifier to report back. The rest are one way pushes where SAP is the source and there is nothing to acknowledge.
The handler pushes the SAP identifier to Salesforce through /acknowledgment and to the storefront through a link endpoint, then archives the file. That is the last of the three identifiers landing, and after it every system holds all three.
There is a second, older error mechanism still present in the codebase. A writer utility puts files into a single shared OutboundErrors folder, and a handler registers that folder and forwards to a different Salesforce endpoint, /errorAcknowledgement. The architecture document describes the two paths as being consolidated.
Traced against the current code, that consolidation is already finished, just not tidied up. The watcher's folder list does not include OutboundErrors, so nothing there is ever detected. The live share has no OutboundErrors folder at all. The handler registers itself at startup and can never fire. The writer is not called from the current handlers, which all write into object specific error folders instead.
So the legacy path is not a second active route. It is dead code that still looks alive because it registers at startup and appears in the architecture document as a current mechanism. That is exactly the kind of thing worth writing down before somebody spends an afternoon debugging why their error file was ignored.
Four trust boundaries, four different mechanisms, and two OAuth clients that cache differently for reasons worth explaining.
| Boundary | Mechanism | Note |
|---|---|---|
| SAP to the FTP server | Username and password, anonymous access disabled | One credential pair from the environment, checked in the login event |
| Watcher to /inbound | Shared secret header plus a localhost allowlist | Both checks return 200 on rejection, so a failed check is invisible to the caller and recorded in the log |
| Salesforce or storefront to / | HTTP Basic | One credential pair shared by both source systems |
| Middleware to either system | OAuth2 client credentials | Tokens cached in process memory, never written to disk |
The storefront client does the textbook thing. It stores the expiry the server reported, treats the token as stale five minutes before that expiry, keeps the refresh token, tries a refresh first when the cache is cold, and falls back to a fresh client credentials grant if the refresh fails, clearing the refresh token so the next attempt does not repeat a call that is known to fail.
The Salesforce client caches for a flat thirty minutes regardless of what the token response says. That is a fixed assumption about a value the server is willing to tell you, and it is wrong in both directions: too long and calls fail with an expired token until the cache turns over, too short and tokens are requested more often than necessary.
The reason it is fine in practice is that the failure is self correcting rather than silent. An expired token produces a 401, which throws, which the handler catches and turns into an enriched error file. It is a bad thirty minutes rather than lost data. Reading expires_in the way the storefront client already does is a small change and it is on the list.
An environment variable, SKIP_OUTBOUND_AUTH, short circuits the entire outbound authentication middleware when set to the string true. It exists because the Salesforce side and the middleware were being configured by different people at different times, and being blocked on credentials while trying to verify a payload shape wastes a scheduled joint testing window.
It is a genuine risk and it should not survive into a production deployment. The safer version is the same switch refusing to engage unless NODE_ENV is not production, which is a three line change I would make before any handover. Naming it here rather than leaving it out is the point of the tab.
The outbound auth middleware contains the Basic authentication logic twice, once for the Salesforce branch and once for the storefront branch, decoding the same header against the same two environment variables. The only difference between them is the system name in the log line.
The obvious reading is that it should be one block. The reading I would actually defend is that the branches were separated in anticipation of the two systems eventually holding different credentials, which is what they should hold, since sharing one credential pair between two callers means neither can be rotated independently and a leak from either compromises both. The duplication is the smaller problem. The shared credential is the real one.
Secrets live in a .env file next to the code on the host, excluded from version control by .gitignore along with the FTP host key. There is no secret manager. Transport to SAP is plain FTP rather than FTPS or SFTP, despite the environment variables being named as though it were SFTP.
The mitigating context is that this process is intended to run on a machine inside the client's network, talking to SAP on the same network, and the repository's own guidance for maintainers says to weigh production readiness against that deployment model rather than treating local FTP as an internet facing risk. That is a reasonable position for a UAT deployment on an internal segment. It stops being reasonable the moment the host is reachable from anywhere else, and that is a deployment decision the code cannot make for itself.
No logging framework. A wrapped console, one file per request, and a retention policy that refuses to delete evidence.
Each inbound file and each outbound request initialises its own log file at the start and closes it at the end. So debugging one complaint that failed at half past five means opening one file that contains that entire operation and nothing else, rather than filtering a shared log for the interleaved lines belonging to one request.
For a system where the traffic is naturally discrete, one business document at a time, that is a better fit than a single stream. It would not scale to high concurrency, where the file handle churn would become the bottleneck, but it matches this workload well.
Log files are named so that the newest sorts to the top of a plain directory listing. That is not for me. It is for whoever opens the log folder in Windows Explorer on the host at some point in the future, sorts by name because that is the default, and needs the most recent thing first without knowing anything about the system.
Every log call takes a boolean as its first parameter, which controls whether a level and timestamp header is printed before the message. True starts a new annotated entry, false continues the current one as an indented detail line.
console.log(true, `[INBOUND] Request received`);
console.log(false, ` Operation: ${operation}`);
console.log(false, ` New Path: ${newPath}`);
It is an unusual signature and it would not survive a code review at most places. What it produces is a log where a multi line event reads as one block instead of as five separately timestamped lines, which is the difference between scanning a file and parsing it. I would still prefer a structured logger with a child context per request, and that is what I would reach for now, but the reasoning behind this one is sound rather than accidental.
The logger carries dedicated helpers for outgoing and incoming HTTP calls, recording the request, the response and both bodies. The Salesforce token call passes literal placeholder strings in place of the credentials and the token, so the exchange is visible in the log while the secret is not.
Capturing full payloads is the right tradeoff in an integration system specifically. When a partner system rejects a document, the first question is always what exactly was sent, and reconstructing it from a transform function and an input file is slow and error prone. The cost is that the logs contain customer data, which is another reason the host and its log directory matter.
Cleanup runs every twenty four hours. It walks the log tree, and for any file older than the retention window it reads the contents and checks for an ERROR or WARN tag before deleting. Files containing either are kept regardless of age. Files it cannot read are kept, on the explicit reasoning that failing to read a file is not a reason to destroy it.
So the retention policy is age based for successes and unbounded for failures. Given that the whole purpose of these logs is diagnosing failures, and that a failure is often reported weeks after it happened by somebody noticing a mismatch in a report, keeping the failures indefinitely is exactly right. The obvious catch is that the error logs grow forever, which is a problem worth having compared to the alternative.
One inconsistency worth recording: the cleanup scheduler starts a twenty four hour interval when the module is required, but the immediate startup run is commented out. A process restarted daily therefore never runs a cleanup at all, because it never stays up long enough to reach the first interval.
The architecture document was accurate when it was written. Nine specific things have moved since, and I am recording both versions rather than quietly adopting whichever is convenient.
This tab exists because the useful part of an integration handover is not a description of the system. It is knowing which parts of the existing description you can still trust. Every row below was found by reading the document and then reading the code that implements the thing it describes.
| What the document says | What the code does | Verdict |
|---|---|---|
| The Salesforce account outbound flow performs a three step cross-acknowledgement, sending the storefront identifier back to Salesforce and the Salesforce identifier back to the storefront | One call to the storefront, a merge, and a file written for SAP. The cross-synchronisation was removed in a commit that names it | Historical. The handler's own docblock still describes the removed steps |
| Fourteen SAP inbound handlers, three success acknowledgement handlers, three Salesforce outbound handlers | Fifteen, four and four | Stale count. Refund and payment remarks were added later and never reached the document |
| The object table lists fourteen object types | Refund exists as a folder, a helper, a handler and a live folder on the share, and is absent from the table | Incomplete |
| The folder tree includes SOStatus/inbound and OutboundErrors/ | Neither folder exists on the live share. SOStatus is watched and has a handler; OutboundErrors is registered but not watched | SO status is registered but dormant. OutboundErrors is dead |
| Two error paths exist and are being consolidated | The consolidation has already happened. The legacy path cannot fire, because its folder is not in the watcher list | Historical |
| Delayed archiving is a testing mode with a three minute delay | Still active in the inbound handlers, with the production rename commented out beneath it. Removed from the acknowledgement handlers in a later commit | Partial. True for inbound, no longer true for acknowledgements |
| chokidar 3.5.3 and dotenv 17.2.3 | chokidar 3.6.0 and dotenv 17.4.2 | Drift, harmless |
| No mention of a health endpoint | GET /health returns status, environment and a timestamp, added in a later commit | Document predates the endpoint |
| Success acknowledgement folders are named success | The live share names them Success with a capital S | Both are true. They match only because Windows paths are case-insensitive |
The SO status handler posts to a Salesforce resource named /SoStatus. The Salesforce org retrieve I was given contains an Apex REST resource with the URL mapping /sostatusupdate and nothing mapped to /SoStatus. Those are different strings, not a difference in casing.
I am not calling that a bug, because the retrieve is old and partial and the SO status flow also has no folder on the live share, so it may simply be a flow that was specified and then not finished on either side. It is recorded as unknown, and it is the first thing I would check against a current org before anybody relies on that flow.
The document was written to describe the system and then the system kept moving, which is the normal fate of architecture documents. What makes it worth publishing the diff is that three of these nine items are the kind that cost real time: a described flow that does not run, a described error path that cannot fire, and a described folder that does not exist. Someone joining this project would reasonably trust all three.
None of this was corrected in place. The document still says what it says, and this page records both readings. Silently merging the two would have produced a tidier page and destroyed the only record of what changed.
What was actually built, what actually ran, and what I am not going to claim.
| Status | |
|---|---|
| The middleware | Complete and running. Eighty six commits between December 2025 and August 2026 |
| Deployment model | Designed for a single Windows host inside the client's network, started with node main |
| Production | Never. This ran in UAT against a Salesforce sandbox and real SAP-format files |
| Exercised flows | Customer creation, complaint, sales order, open sales order and their acknowledgements, verified by residual artefacts on the share dated March 2026 |
| Registered but dormant | SO status. Handler and watcher entry exist, folder does not |
| Dead but present | The legacy OutboundErrors path, and a standalone FTP server script at the repository root that duplicates the embedded one and is never required |
Ninety four files remain on the live share from real runs, dated between the ninth and twentieth of March 2026. They include archived customer creations, archived acknowledgements, complaint outbound files named after real SAP document numbers, and error files from both the storefront and Salesforce. Those artefacts are why the exercised flows above are stated as fact rather than as intent.
They also record the failures. There are error files from both directions in the complaint folder within minutes of each other, which is what a joint testing session looks like from the outside.
I verified the receiving end rather than taking it on trust. The org contains Apex REST resources whose URL mappings line up with the endpoints the middleware posts to: /account, /acknowledgment, /acknowledgmentError, /Invoice, /payment, /materialtocrm, /OpenSalesOrder, /stockreservationftp, /stockunreservationftp and /stockdetailsupdate. There is a custom sap_error__c object with error message and object type fields and lookups to Account, Sales Order and Stock Reservation, which is where a reported failure lands as a record.
There are also flag fields across the standard objects with names like Post_to_SAP__c, Sent_to_SAP__c and CRM_TO_SAP_StockReservation__c, which is the mechanism that decides when Salesforce pushes a record outward.
The org also still carries a connected app and a remote site setting named after the commercial ETL product this middleware replaced. Neither was removed when the integration moved over, which is worth knowing before anybody audits that org: they are live configuration entries pointing at a system that is no longer in the path.
The honest list, ordered by what would actually bite first in production.
Nothing in this system records that it has seen a file or a payload before. The same document processed twice produces two Salesforce writes and two storefront writes. There is no key, no hash, no store, and no check.
The narrow path where this becomes live rather than theoretical is the one described in the routing tab: a handler that throws after its outbound calls have succeeded returns 500, and the watcher retries five times. The fix is small and specific. A content hash of the file recorded alongside its outcome, checked at the top of every handler, would close it without adding a dependency. That is the first thing I would build next.
The only retry loop guards a local HTTP call between two parts of the same process. The calls that actually cross a network, to Salesforce and to the storefront, have none. A transient 503 from either becomes an error file immediately, when a short backoff would have resolved most of them without a human ever seeing it.
Error files are written and deliberately never archived, and the log says they remain for retry. Nothing re-reads them. The watcher only fires on creation, and files present at startup are ignored, so a failed error acknowledgement is never attempted again. Making the error folder a real retry queue means scanning it on a timer with a backoff and an attempt count in the file name, which is perhaps forty lines and would change error handling from evidence collection into recovery.
Delayed archiving schedules a deletion three minutes out with setTimeout. A restart inside that window loses the timer, leaves the original in the inbound folder, and ignoreInitial guarantees nothing will ever look at it again. It is not data loss, since the archive copy was written immediately, but it leaves the share looking like there is unprocessed work when there is not.
The test script is the npm placeholder that exits with an error, and there is no test directory. For a codebase that is mostly transform functions taking one object and returning another, that is the cheapest possible thing to fix: the helpers are pure, they need no mocks, and a single file of assertions over one real payload per object type would catch the entire class of bug where a field is renamed on one side and silently becomes undefined on the other. That is the second thing I would build.
A note on how this list should be read. Almost everything here is a consequence of the same decision, which was to ship a system with four dependencies and no infrastructure so that one person could hand it to a client's IT team. I would make that trade again for a UAT deployment. Production would need the first three items closed before I would sign off on it.
I built the middleware. The systems on either side of it belong to other people, and part of the work was accepting that.
The entire Node middleware: the embedded FTP server and its Windows filesystem subclass, the watcher, both routing middlewares and their auto-discovery, all thirty two handlers, the transform helpers for fourteen object types, the two API clients and their token caching, the logger and its retention policy, and the error and acknowledgement architecture.
On the Salesforce side I worked on the integration surface: the Apex REST resources the middleware posts into, the custom fields and objects that hold the SAP and storefront identifiers and the error records, the automation that decides when a record is pushed outward, and the connected app and OAuth configuration that lets the middleware authenticate.
The Salesforce org here is a large one that long predates this work, with hundreds of classes and dozens of automations that have nothing to do with this integration. I have deliberately kept the claims above to the integration surface, and everything on this page about Salesforce is limited to what I could verify in the org retrieve: the REST resources whose URL mappings match the endpoints the middleware calls, the sap_error__c object, and the SAP flag fields. The retrieve is old and partial, so it is a floor rather than a complete picture.
SAP is not mine in any sense. I never saw inside it. My entire view of SAP is the shape of the JSON it writes and the folders it reads, which is exactly the view the middleware has.
Very little of the difficulty was in the code. It was in the negotiation of contracts across three teams who each had a working system and no interest in changing it. The stringified JSON envelope, the folder naming, the five field spellings of the same identifier, the wildcard directory listing: none of these were decisions I got to make, and all of them are the reason particular pieces of this codebase look the way they do.
The design position I held throughout was that the middleware absorbs the awkwardness rather than pushing it outward. Nobody else had to change anything to accommodate it. That is why the transform layer is per object type rather than generic, and why there is a fallback chain instead of a schema. It is more code in one place in exchange for no coordination cost anywhere else, and with three teams on three schedules that was the correct trade.
The things I would defend: modelling failure as a routable file, keeping the dependency count at four for a system somebody else has to maintain, writing the SAP-bound file before any optional work, and extracting the error handler factory after the second copy rather than before the first.
The things I got wrong: shipping without idempotency in a system that has a retry loop, putting the only retry on the least risky call, leaving a testing affordance switched on with the production path commented out beneath it, and letting the architecture document drift far enough that three of its statements now describe flows that cannot run. The last one is the one that bothers me most, because it is the cheapest to have prevented.
Four things this project taught me that I would carry into the next integration.
I went in expecting the hard part to be moving and transforming data. The transforms turned out to be the easy half, mostly field renaming. The hard part was that a single business fact has three identities, each of which only comes into existence when a different system says yes, and that there is a long window during which a record legitimately has two of its three identifiers and is not in an error state.
Everything structurally interesting in this codebase, the merge functions, the acknowledgement flows, the identifier fallback chains, the enriched error payloads, exists to manage that window. If I were briefing someone on their first integration, this is the thing I would tell them to design first and design deliberately.
The decision I am most confident about is that a failure produces a file that flows through the same routing the happy path uses, and ends up as a record in the CRM. An error that is only a log line requires somebody to go looking. An error that is a file in a folder that is watched, and a record attached to the account it concerns, arrives at the person who can act on it.
The corollary I got wrong is that making failure visible is not the same as making it recoverable. The error files are excellent evidence and are not a queue. I stopped one step short, and one step short is a very common place to stop.
There was a version of this project where I pushed for an API on the SAP side, or for a message broker in the middle, and the timeline for that is measured in quarters of somebody else's roadmap. SAP writes files. Building the integration around that constraint instead of against it is why this shipped.
The file interface is genuinely worse than a queue in every technical respect, and it was still the right foundation, because it is the one that did not require another team to change anything. Recognising which constraints are technical and which are organisational, and not spending effort fighting the second kind, is most of what made this project tractable.
Nine statements in the architecture document no longer match the code, and three of them describe flows that cannot run. Every single one drifted because a change was made in the code and the document was a separate artefact somebody had to remember to update.
What I would do differently is not discipline. It is structure: keep the parts of the description that can go stale next to the thing they describe, so that changing the flow and changing the sentence are the same edit, and let a document at this level describe intent and constraints rather than handler counts and folder names. The counts were out of date within weeks. The reasoning is still accurate today.
Four dependencies. Whenever I reread this codebase I expect to regret that, and I do not. The absence of a queue is a real limitation with real consequences that are listed plainly in the Debt tab. But this process runs on one machine inside a manufacturer's network, and the person maintaining it after me is likelier to be an IT generalist than a Node developer. Every dependency I did not add is a thing that cannot break during an upgrade on a host nobody wants to touch.
The version of this system with a broker and a worker pool and a durable store is better engineering and would have been worse for this client. Knowing which of those two you are being paid for is the actual skill.