6 posts tagged with "SAP Integration"

Every integration tool has two speeds: the one in the benchmark, and the one you actually experience on the first day. For erpl-rev those two numbers were embarrassingly far apart. It replicates 10 million BSEG-shaped rows in about a minute, and 2.5× faster than that since August. Getting to the point where you could run that benchmark took an afternoon.

Not because anything was hard. Because it was scattered. A type-T destination here. A function group there. Eight Z_DUCKDB_* function modules. Fourteen ABAP objects. A reginfo line in a file only Basis can touch. Four documents, each correct, none of them the whole path.

This post is the whole path, and it is now short enough to time with a stopwatch.

Most SAP extracts are narrow. You want three columns out of a table with a hundred million rows, on a schedule, and you want it to finish before the batch window closes. That shape is the normal case for an incremental load — and it was the one shape sap_read_table could not read in parallel at all.

partitions fixes that by splitting the scan across rows. On 2.9 million rows it is 4.5× faster at eight workers. Getting there meant taking apart the invariant the scan had been built on, which is the part worth writing about.

In June I wrote up erpl-rev — the little registered RFC server that lets ABAP call out into DuckDB — and it replicated 10 million rows of a 400-column BSEG-shaped table in about a minute. That post did not just claim a number, it explained why the number was what it was.

Since then I put it under a profiler to find out what was still on the table. Two things came back, pulling in opposite directions. A step that genuinely earned its place in June still had about a third of the ingest path left in it. And a constraint I had treated as fixed — SAP's proprietary RFC library — turned out to be removable altogether.

Here is a problem that shows up on every hybrid-cloud project eventually. You have a container in the cloud — an AWS ECS task, a Fargate job, an analytics or AI workload — and it needs data that lives in an on-prem network: a Postgres database, or an SAP system reachable only from inside the corporate LAN.

The data is right there. The container just can't reach it. And the moment you ask, the answer from the network team is no: no inbound firewall port to the on-prem network, no public IP on the database, no exception. That's not obstruction — it's the correct default. An open port into your ERP network is exactly what you don't want.

So the usual workarounds start: a site-to-site VPN, a bastion host with SSH port-forwards, an IP allowlist someone has to maintain, a reverse proxy in a DMZ. Each one is a standing piece of infrastructure, and each one is a thing that can be misconfigured into an open door.

There's a smaller way to do this, and it doesn't open a port at all.

If you have ever worked an SAP EDI or ALE interface, you have seen an IDoc file: a flat text file where every line is exactly 1063 characters, and the payload — the field called SDATA — is a 1000-character run with no delimiters. No commas, no tags, no header. Just a fixed-width blob whose meaning lives in a segment definition inside SAP.

That's the problem. To read what's actually in an IDoc you open SAP: transaction WE60 for the segment layout, WE02 to look at the document. Outside SAP, an IDoc file is opaque. You can't grep it in any useful way, you can't load it into a warehouse, and you certainly can't diff two of them field by field.

erpl_idoc is a DuckDB extension that fixes that. It reads IDoc files as SQL tables, decodes the SDATA blob into typed columns, writes byte-valid IDocs back out from a query, and converts flat ⇄ IDoc-XML — all in plain SQL, and all offline. No SAP connection, no RFC libraries, no middleware.

And it goes both ways. The same extension that lands an inbound file as a table also produces an outbound one: compose the records in SQL, COPY them to disk, and you get a byte-valid IDoc a SAP file-port will accept. Read and write, on one file engine.

Summary​

This article compares ERPL's approach to SAP integration with traditional methods, highlighting performance improvements, ease of use, and cost benefits. We'll examine real-world scenarios and provide concrete examples of how ERPL transforms SAP data integration workflows.