# ERPL — DuckDB for SAP, Microsoft 365, and Dynamics 365

ERPL is the DuckDB extension family for enterprise data integration. SAP (RFC, BICS, ODP, IDoc files, Datasphere, SAC, ABAP/ADT), Microsoft 365 (SharePoint, Excel, Teams, Outlook, Planner, Entra ID), Microsoft Dynamics 365 (Business Central, Dataverse), and open standards (OData, HTTP, Delta Sharing) — all queryable from one SQL engine.





# BICS Protocol Deep Dive

This guide covers the BICS (BI Consumer Services) interface in ERPL: how to discover InfoProviders and BEx queries, build OLAP cross-tabs, fetch result sets, and trace end-to-end lineage from source tables to BEx queries — all from SQL in DuckDB.

**What is BICS?:**

**BICS (BI Consumer Services)** is the SAP interface that tools like SAP Analysis for Office use to consume SAP Business Warehouse (BW) data. ERPL talks to BICS through the **SAP NetWeaver RFC SDK** — it calls the `BICS_PROV_*` RFC function modules to open a query, manipulate its OLAP state, and read result sets. It is **not** a separate HTTP/web protocol: there is no BICS port, no XML payload, and no SAML. Connectivity and authentication are exactly the same as for the [RFC protocol](/docs/erpl/rfc.md) — a DuckDB secret of type `sap_rfc`.

## How it works

ERPL uses **two RFC paths**:

*   **Query execution** drives an OLAP query through the BICS provider modules (`BICS_CONS_CREATE_DATA_AREA`, `BICS_PROV_OPEN`, `BICS_PROV_GET_INITIAL_STATE`, `BICS_PROV_SET_STATE`, `BICS_PROV_GET_RESULT_SET`, `BICS_PROV_CLOSE`). These power `sap_bics_begin` / `rows` / `columns` / `filter` / `result`.
*   **Metadata & lineage** read BW dictionary tables via `RFC_READ_TABLE` (e.g. `RSRREPDIR`, `RSZCOMPDIR`, `RSDCUBE`, `RSTRAN`, `ROOSFIELD`, `RSHIEDIR`, `RSOOBJXREF`). These power the `sap_bics_show*`, `sap_bics_meta_*`, and `sap_bics_lineage_*` functions.

## Connecting

Every BICS function uses a DuckDB secret of type `sap_rfc` — the same secret used by the RFC extension. Create one once per session (the example below uses the ABAP Platform Trial defaults):

```
CREATE SECRET abap_trial (  TYPE    sap_rfc,  ASHOST  'localhost',  SYSNR   '00',  CLIENT  '001',  USER    'DEVELOPER',  PASSWD  'ABAPtr2023#00',  LANG    'EN');
```

If a single unnamed/default secret exists, ERPL uses it automatically. When you keep several secrets, pass the one you want with the `secret` named parameter that **every** BICS function accepts, e.g. `sap_bics_show_cubes(secret => 'abap_trial')`. For encrypted connections, add the SNC parameters (`SNC_QOP`, `SNC_MYNAME`, `SNC_PARTNERNAME`, `SNC_LIB`) to the secret — see the [security guide](/docs/security.md). All examples below assume a usable secret exists and omit the parameter for brevity.

## Discovering objects

```
-- InfoProviders (cubes, ADSOs, CompositeProviders, …)SELECT * FROM sap_bics_show(obj_type => 'INFOPROVIDER');-- Just cubes, or just queries (dedicated helpers with search)SELECT * FROM sap_bics_show_cubes();SELECT * FROM sap_bics_show_queries();-- Search by technical name or textSELECT technical_name, text, typeFROM sap_bics_show_queries(search => '0D_FC_NW_C01_Q0011', search_in_key => true);
```

`sap_bics_show(obj_type => ...)` accepts `INFOPROVIDER`, `CUBE`, `QUERY`, or `INFOAREA`, and returns `technical_name`, `text`, `type`, `cube_name`, `is_folder`, `last_changed`, `last_changed_by`, and `level`.

## Querying a cube (OLAP cross-tabs)

BICS queries are **stateful**. You open a query state, identify it with an `id`, mutate that state with separate calls, and finally read the result set by the same `id`. Unlike ordinary table functions, the BICS query functions are **not nested inside one another** — they are chained by passing the state `id` string from one call to the next.

```
-- 1. Open a query state on the cube and give it an idSELECT state_id, state_versionFROM sap_bics_begin('0D_NW_C01', id => 'q1');-- 2. Place characteristics on the ROWS axis (the cross-tab "drilldown")SELECT state_id, state_versionFROM sap_bics_rows('q1', '0D_NW_PROD', op => 'SET');-- 3. Place a characteristic on the COLUMNS axisSELECT state_id, state_versionFROM sap_bics_columns('q1', '0CALMONTH', op => 'SET');-- 4. Read the result set for this stateSELECT * FROM sap_bics_result('q1');
```

### Axes, not column projection

`sap_bics_rows` and `sap_bics_columns` are **OLAP axes**, not a SQL column projection. They decide which characteristics are drilled into the rows vs. the columns of the cross-tab — the same as dragging a characteristic onto Rows or Columns in Analysis for Office. The key figures of the cube are aggregated for whatever drilldown you choose. The `op` parameter controls how each call changes the axis:

| `op` | Effect |
| --- | --- |
| `SET` | Replace the axis with exactly these characteristics |
| `ADD` | Append characteristics to the existing axis |
| `REMOVE` | Remove the named characteristics from the axis |

Each mutating call returns the updated `state_id` and an incrementing `state_version`, so you can confirm the state advanced. The **columns of `sap_bics_result`** are the cube's own characteristic and key-figure technical names — for `0D_NW_C01` that includes `"0D_NW_PROD"`, `"0D_NW_NETV"` (net value), `"0D_NW_QUANT"` (quantity), and so on. Because these names start with a digit, quote them with double quotes in SQL.

**Fetch results in one call:**

`begin`, `rows`, `columns`, and `filter` accept `return => 'RESULT'` to return the result set directly instead of the default state description (`return => 'DESCRIBE'`). For example, `SELECT * FROM sap_bics_rows('q1', '0D_NW_PROD', op => 'SET', return => 'RESULT')` applies the axis change and hands back data in a single statement.

### Filtering members

`sap_bics_filter` restricts a characteristic to one or more member values (a background filter / "slice"). Pass the characteristic, then one or more member keys as positional arguments:

```
SELECT state_id FROM sap_bics_begin('0D_NW_C01', id => 'f1');SELECT state_id FROM sap_bics_rows('f1', '0D_NW_PROD', op => 'SET');-- Single-member filter: Division = 7 ("High Tech")SELECT state_id FROM sap_bics_filter('f1', '0D_NW_DIV', '7', op => 'SET');SELECT * FROM sap_bics_result('f1');
```

```
-- Multi-member filter: pass several values to one filter() callSELECT state_id FROM sap_bics_filter('f1', '0CALMONTH', '202401', '202402', '202403',                                     op => 'SET');-- op => 'ADD' / 'REMOVE' adjust an existing selection instead of replacing itSELECT state_id FROM sap_bics_filter('f1', '0D_NW_DIV', '15', op => 'ADD');
```

You can also seed a filter at `begin` time with the `filters` named parameter; ad-hoc `sap_bics_filter` calls remain the flexible way to slice an open state.

## Hierarchies

```
-- List hierarchies available for an InfoObjectSELECT * FROM sap_bics_show_hierarchies(info_object => '0D_NW_PROD');-- Read a hierarchy as a flat node listSELECT * FROM sap_bics_hierarchy('0D_NW_PROD_HIER');-- Pin a version / key date and return a recursive treeSELECT * FROM sap_bics_hierarchy('0D_NW_PROD_HIER', date_to => '20151231', as_tree => true);
```

`sap_bics_hierarchy` returns `node_id`, `parent_id`, `child_id`, `next_id`, `info_object`, `node_name`, `node_value`, `date_from`, `date_to`, `level`, and `path` — enough to rebuild the tree yourself or filter by `level`.

## Describing structures

```
-- Describe a cube: characteristics + key figuresSELECT technical_name, text FROM sap_bics_describe('0D_NW_C01');-- Describe a BEx query on a cube (adds its variables)SELECT technical_name, text, variablesFROM sap_bics_describe('0D_NW_C01', '0D_FC_NW_C01_Q0011');-- Describe an open query state by its idSELECT * FROM sap_bics_describe(id => 'q1');
```

`sap_bics_describe` returns `technical_name`, `text`, and the `characteristics` / `keyfigures` structs (plus `variables` for the query overload). For attributes of a single InfoObject:

```
SELECT info_object, data_type, length, decimalsFROM sap_bics_describe_infoobject('0D_NW_PROD');
```

`sap_bics_describe_infoobject` returns `info_object`, `data_type`, `conv_exit`, `output_length`, `length`, and `decimals`.

## Presentation properties (AO-style)

`sap_bics_set_char_prop` toggles per-characteristic display properties that match the radio controls in SAP Analysis for Office's _Properties_ panel. The mutation persists in the BICS state and is honoured by the next `sap_bics_result` call.

```
-- Build a 1D cross-tab on countrySELECT * FROM sap_bics_begin('0D_NW_C01', id => 'q1');SELECT * FROM sap_bics_rows('q1', '0D_NW_CNTRY', op => 'SET');-- Display member texts instead of keys ("Germany" vs "DE")SELECT * FROM sap_bics_set_char_prop('q1', '0D_NW_CNTRY', 'DISPLAY', 'TEXT');-- Show both key and text concatenatedSELECT * FROM sap_bics_set_char_prop('q1', '0D_NW_CNTRY', 'DISPLAY', 'BOTH');-- Sort by member descendingSELECT * FROM sap_bics_set_char_prop('q1', '0D_NW_CNTRY', 'SORT', 'DESC');-- Fetch with the new presentationSELECT * FROM sap_bics_result('q1');
```

| `prop` | Allowed `value` | Maps to BICS state field |
| --- | --- | --- |
| `DISPLAY` | `KEY` \| `TEXT` \| `BOTH` | `RESULT_SET_PRESENTATION` bitflag (KEY=4, TEXT=32) |
| `TOTALS` | `SHOW` \| `HIDE` | `RESULT_VISIBILITY` ('A' / 'N') |
| `SORT` | `ASC` \| `DESC` \| `NONE` | `RESULT_SET_SORTING.DIRECTION` ('A' / 'D' / '') |

The DESCRIBE payload of `state_rows`, `state_columns`, and `state_free` carries `{display, totals, sort}` so clients can read the current values without an extra round-trip.

**Grand-total row visibility:**

BICS does not expose a state field for the SUMME / "Overall Result" grand-total row. `TOTALS='HIDE'` is persisted to per-characteristic `RESULT_VISIBILITY` but the server still returns the grand-total row in `sap_bics_result`. Clients that want AO's "Hide Result" behaviour can filter the row whose row-characteristic value matches `SUMME` / `Overall Result` / localised variants — that's what AO itself does.

### Interactive: `bics-tui`

`bics-tui` (in [`bics/examples/tui/`](https://github.com/DataZooDE/erpl-bics/tree/main/examples/tui)) is a terminal UI that wraps every BICS function above into a Textual app modelled on SAP _Analysis for Office_: log on, browse cubes / queries, build a cross-tab by moving characteristics between **Rows / Columns / Background Filter**, filter members, and toggle **Display / Sort / Totals** for a focused characteristic or **Scaling Factor / Decimal Places** for a focused key figure — all from a context-sensitive **Properties** panel on the right. Every server-side action is mirrored into an always-on SQL recorder, so the resulting script replays cleanly in a vanilla DuckDB shell.

#### Logon

The logon screen pre-fills the standard ABAP Platform Trial credentials — overwrite for your own system. The fields are wired straight into a DuckDB `CREATE SECRET` of type `sap_rfc`.

![bics-tui Logon screen](/assets/images/bics_tui_logon-b37cd93d84a8202f524dbc2898019c6c.png)

#### Analysis screen

Three-column layout: cross-tab grid on the left, design panel in the middle (Data Source / Columns / Rows / Background Filter / Key Figures), context-sensitive Properties panel on the right. Hotkeys are shown next to each section header.

![bics-tui Analysis screen with 0D_NW_PROD on Rows and 0D_NW_CNTRY on Background Filter](/assets/images/bics_tui_analysis-649b4f7be73a8b532535305a1a1600ac.png)

#### Properties — Characteristic on axis

Focus a characteristic in any axis list and the Properties panel switches to three radio groups: Display (Key / Text / Both), Totals (Show / Hide / Conditional), Sort (None / Asc / Desc). Display and Sort round-trip via ([`sap_bics_set_char_prop`](#presentation-properties-ao-style)); Totals Hide is applied client-side (BICS does not expose a state field for the grand-total row).

![bics-tui Properties panel for a focused characteristic](/assets/images/bics_tui_props_char-7a7e31437df602b6dd3ad0de6a12875a.png)

#### Properties — Key figure

Focus a key figure and the panel switches to Scaling Factor and Decimal Places. Both are client-side numeric formatting — BICS state carries no field for them. The KF columns in the cross-tab re-format live as you change the controls.

![bics-tui Properties panel for a focused key figure](/assets/images/bics_tui_props_kf-576ed44f529ef702855f8f8dcb9b3b27.png)

#### Run it

```
GEN=ninja make release          # from erpl monorepo rootcd bics/examples/tuiuv syncLD_LIBRARY_PATH=/path/to/erpl/nwrfcsdk/linux/lib uv run python -m bics_tui
```

The screenshots above are regenerated headlessly via [`scripts/capture_screens.py`](https://github.com/DataZooDE/erpl-bics/blob/main/examples/tui/scripts/capture_screens.py) (Textual's `export_screenshot` → SVG → PNG with a mock session for layout-only capture; no SAP connection needed).

## Lineage Tracking (for SAP BW experts)

**For BI Administrators:**

This section covers lineage tracking from ERP tables through DataSources, InfoProviders, and BEx queries. It reads the BW dictionary tables via `RFC_READ_TABLE`, so it does not need an open query state.

### Complete lineage as edges

`sap_bics_lineage_edges()` returns the BW data flow as a flat edge list spanning ERP source tables → DataSources → transformations → InfoProviders → BEx queries. Each row is one source-to-target edge with these columns:

| Column | Description |
| --- | --- |
| `edge_type` | Edge category (e.g. transformation, query-element) |
| `src_kind`, `src_name`, `src_field` | Source object kind, name, and optional field |
| `tgt_kind`, `tgt_name`, `tgt_field` | Target object kind, name, and optional field |

Use cases:

*   **Impact analysis** — what happens if a source table changes?
*   **Data governance** — track data flow and transformations
*   **Documentation** — automatic lineage documentation
*   **Compliance** — audit data lineage for regulations

```
-- All edges in the systemSELECT * FROM sap_bics_lineage_edges();-- Edges related to a specific BEx query — filter in SQLSELECT *FROM sap_bics_lineage_edges()WHERE tgt_name = '0D_FC_NW_C01_Q0008' OR src_name = '0D_FC_NW_C01_Q0008'ORDER BY src_kind, tgt_kind;-- Scope the underlying RFC reads to one object (faster on large landscapes)SELECT * FROM sap_bics_lineage_edges(scope => '0D_NW_C01');
```

### Forward trace from a source

`sap_bics_lineage_trace()` walks the graph forward from a specific source object/field — perfect for "if I change this object, what downstream BW objects break?". The result includes a `hop` column and a `path` column (the chain of objects walked) so you can see the full downstream blast radius.

```
-- Everything downstream of an InfoProviderSELECT * FROM sap_bics_lineage_trace(source_object => '0D_NW_C01')ORDER BY hop;-- Field-level traceSELECT * FROM sap_bics_lineage_trace(  source_object => '0D_NW_C01',  source_field  => '0D_NW_NETV');
```

### Lineage of a single query

`sap_bics_query_lineage()` resolves the lineage rooted at one BEx query:

```
SELECT * FROM sap_bics_query_lineage('0D_FC_NW_C01_Q0011');
```

### Lineage as a JSON graph

`sap_bics_lineage_graph_json()` returns the full lineage as a JSON document, suitable for visualization libraries (D3, Cytoscape, etc.) or export to external graph tools.

```
SELECT * FROM sap_bics_lineage_graph_json();
```

### Metadata mining

The `sap_bics_meta_*` functions expose the BW dictionary tables directly.

**Several metadata functions require an argument:**

To avoid scanning an entire BW landscape by accident, some functions refuse a bare no-argument call and return a message telling you which parameter to supply. The functions that **require at least one argument** are `sap_bics_meta_providers` (`type`), `sap_bics_meta_datasources` (`appcomp`), `sap_bics_meta_transformations` (`active_only`), `sap_bics_meta_query_stats` (one of `query_name` / `from_date` / `to_date`), `sap_bics_meta_objxref` (`tlogo` / `objnm` / `tlogo_dep`), and `sap_bics_meta_hcpr_mapping` (`composite_provider`). The field-level helpers below take the object name positionally.

```
-- InfoProvider metadata (type is required: CUBE | ADSO | HCPR | ODSO | ODSVIEW)SELECT * FROM sap_bics_meta_providers(type => 'CUBE');-- DataSource metadata (application component is required)SELECT * FROM sap_bics_meta_datasources(appcomp => 'NODE0000');-- Transformation metadataSELECT * FROM sap_bics_meta_transformations(active_only => true);-- Query directory (no argument required)SELECT * FROM sap_bics_meta_queries();-- Query usage / runtime statisticsSELECT * FROM sap_bics_meta_query_usage('0D_FC_NW_C01_Q0011');SELECT * FROM sap_bics_meta_query_stats(query_name => '0D_FC_NW_C01_Q0011');-- Query elements (structures, selections, formulas)SELECT * FROM sap_bics_meta_query_elements('0D_FC_NW_C01_Q0011');
```

### Field-level & cross-reference metadata

```
-- DataSource fields (DataSource name is positional)SELECT * FROM sap_bics_meta_datasource_fields('0FI_GL_4');-- InfoProvider fields (provider name is positional; pass provider_type to disambiguate)SELECT * FROM sap_bics_meta_provider_fields('0D_NW_C01', provider_type => 'CUBE');-- Transformation field mappings (transformation id is positional)SELECT * FROM sap_bics_meta_transform_fields('<transformation-id>');-- CompositeProvider (HCPR) components and their part-provider mappingSELECT * FROM sap_bics_meta_hcpr_components('<composite-provider>');SELECT * FROM sap_bics_meta_hcpr_mapping(composite_provider => '<composite-provider>');-- InfoObject catalog (no argument required; filter with iobjnm/iobjtp)SELECT * FROM sap_bics_meta_infoobjects(iobjnm => '0D_NW_PROD');-- Object cross-reference (which objects reference which)SELECT * FROM sap_bics_meta_objxref(tlogo => 'CUBE', objnm => '0D_NW_C01');
```

The exact rows these return depend on what is configured in your BW system; the demo objects above are populated on the ABAP Platform Trial, while DataSources and transformations are typically richer on a productive landscape.

**HCPR:**

`HCPR` is the SAP TLOGO type for a **HANA CompositeProvider**, not a "hierarchy change pointer". The `*_hcpr_*` helpers describe CompositeProviders and the part-providers they union/join.

## Real-world examples

### Net value by product

Build the state with separate statements, then query the result. Each call returns the `state_id`, so you can ignore those rows and read the data from `sap_bics_result`:

```
SELECT state_id FROM sap_bics_begin('0D_NW_C01', id => 'rep1');SELECT state_id FROM sap_bics_rows('rep1', '0D_NW_PROD', op => 'SET');SELECT state_id FROM sap_bics_filter('rep1', '0D_NW_DIV', '7', op => 'SET');SELECT  "0D_NW_PROD"  AS product,  "0D_NW_NETV"  AS net_value,  "0D_NW_QUANT" AS quantityFROM sap_bics_result('rep1')ORDER BY net_value DESCLIMIT 20;
```

### Combine BW data with ERP master data

Once the `rep1` state above exists, its result set joins to ERP master data read over RFC:

```
-- BW net value per product, joined to ERP material master via RFCWITH bw_sales AS (  SELECT "0D_NW_PROD" AS product, "0D_NW_NETV" AS net_value  FROM sap_bics_result('rep1')          -- the state built above),erp_materials AS (  SELECT MATNR AS material, MTART AS material_type, MEINS AS base_unit  FROM sap_read_table('MARA', MAX_ROWS => 10000))SELECT  b.product, m.material_type, m.base_unit, b.net_value,  CASE    WHEN b.net_value > 1e9 THEN 'High Value'    WHEN b.net_value > 1e8 THEN 'Medium Value'    ELSE 'Low Value'  END AS value_categoryFROM bw_sales bLEFT JOIN erp_materials m ON b.product = m.materialORDER BY b.net_value DESC;
```

## Performance & good practice

*   **Slice early with filters and axes.** Restricting members with `sap_bics_filter` and limiting the drilldown with `sap_bics_rows` / `sap_bics_columns` is what reduces the result set — there is no SQL-style column projection at the BICS layer.
*   **Reuse the query state.** `begin` opens a server-side state; keep mutating the same `id` rather than re-opening for each variation. `state_version` confirms each change landed.
*   **Scope metadata reads.** Pass `scope`/object arguments to the `lineage`/`meta` functions so the underlying `RFC_READ_TABLE` calls stay bounded on large landscapes.
*   **Reuse connections.** ERPL caches RFC connections per secret; running several BICS statements in one DuckDB session avoids repeated logons.

## Troubleshooting

**Query or cube not found**

```
-- Names are case-sensitive; search the catalogSELECT * FROM sap_bics_show_queries(search => 'Q0011');SELECT * FROM sap_bics_show(obj_type => 'CUBE', search => '0D_NW_C01');
```

**Empty result set**

```
-- Re-check filter member keys (use keys, not display texts)SELECT * FROM sap_bics_describe_infoobject('0D_NW_DIV');-- Drop filters to confirm the cube has data at allSELECT * FROM sap_bics_begin('0D_NW_C01', id => 'probe', return => 'RESULT');
```

**Metadata function errors with "requires named parameters"** This is the guardrail described above — supply the named argument it asks for (e.g. `sap_bics_meta_providers(type => 'CUBE')`).

### Enable tracing

ERPL has a built-in trace facility. Enable it to see the exact RFC calls and payloads:

```
SET erpl_trace_enabled = TRUE;SET erpl_trace_level   = 'DEBUG';   -- TRACE | DEBUG | INFO | WARN | ERRORSET erpl_trace_output  = 'console'; -- console | file | both
```

## For SAP BW experts: the BICS interface in detail

BICS in ERPL is an **RFC interface**, not a network protocol of its own: it drives the same `BICS_PROV_*` consumer modules that SAP Analysis for Office uses, over the NetWeaver RFC SDK with your `sap_rfc` secret. Expand the deep dive below for the wire-level mechanics.

**BICS protocol deep dive** — handle lifecycle, state model & versioning, result-set decoding, variables, metadata, security

#### The handle lifecycle

A BICS session juggles three server-side handles, each a 4-character token:

1.  **Application handle** — `BICS_CONS_CREATE_DATA_AREA` creates the consumer data area and returns `E_APPLICATION_HANDLE`. This is the umbrella context for the session.
2.  **Data-provider handle** — `BICS_PROV_OPEN` opens the InfoProvider (or BEx query) and returns `E_DATA_PROVIDER_HANDLE` plus `E_VARIABLE_CONTAINER_HANDLE`. ERPL opens with `I_STATE_VARIABLE_MODE = 'U'` and `I_OPTIMIZE_INIT_VERSION = 5`. A variable-container handle of `0000` is **normal** for queries that declare no BEx variables — it is not an error.
3.  **Initial state** — `BICS_PROV_GET_INITIAL_STATE` (with `I_RETRIEVE_SEL_SPACE_OPT = 'X'`) returns the complete default OLAP state: the characteristic catalog, the rows/columns/free axis assignments, the selection (filter) state, and the per-characteristic presentation state.

At session end `BICS_PROV_CLOSE` is called best-effort for the variable-container, data-provider, and application handles in turn (failures are swallowed).

The full mapping from SQL function to RFC module:

| ERPL step | RFC function module(s) | Key parameters |
| --- | --- | --- |
| `sap_bics_begin` | `BICS_CONS_CREATE_DATA_AREA` → `BICS_PROV_OPEN` → `BICS_PROV_GET_INITIAL_STATE` | `I_DATA_PROVIDER_INFO_PROVIDER`, `I_DATA_PROVIDER_NAME` (query) |
| `sap_bics_rows` / `columns` / `filter` / `set_char_prop` | `BICS_PROV_SET_STATE` | `I_S_ROWS`, `I_S_COLUMNS`, `I_S_FREE`, `I_TH_CHARACTERISTICS`, `I_TSX_SELECTION_STATE`, `I_VALIDATE = 'X'` |
| `sap_bics_result` | `BICS_PROV_GET_RESULT_SET` | `I_MAX_DATA_CELLS = 1000000` |
| `sap_bics_describe` (query) | `BICS_PROV_GET_DESIGN_TIME_INFO`, `BICS_PROV_VAR_GET_VARIABLES` | `I_VARIABLE_CONTAINER_HANDLE` |
| (session end) | `BICS_PROV_CLOSE` | one call per handle |

#### The state model and `state_version`

Although the OLAP state physically lives on the BW server between `SET_STATE` calls, ERPL also **persists each state snapshot client-side** in a DuckDB table keyed by `(id, version)`. Every mutation — placing a characteristic on an axis, adding a filter, or changing a display/sort/ totals property — saves a new snapshot and increments `state_version`. That is why a typical `begin → rows → columns → filter` sequence returns versions `1 → 2 → 3 → 4`. Because snapshots are immutable and addressed by id, several independent query states can coexist in one DuckDB session, and you can branch a state by re-using an earlier id.

`BICS_PROV_SET_STATE` carries the whole OLAP state, not a delta. The notable sub-structures:

*   **Axes** — `I_S_ROWS` / `I_S_COLUMNS` / `I_S_FREE` and their characteristic-reference tables describe which characteristics sit on the rows axis, the columns axis, and the _free_ (background-filter) axis. `op => 'SET'` clears the target axis first (key figures are kept on the column axis), `'ADD'` appends, and `'REMOVE'` moves a characteristic to the free axis.
*   **Selection state** (`I_TSX_SELECTION_STATE`) — one entry per filtered characteristic, each holding a `SELECTION` list of `{SIGN ('I'/'E'), OPERATOR ('EQ'/'BT'/…), LOW, HIGH}` rows. This is the BICS equivalent of an ABAP range table.
*   **Per-characteristic presentation** (`I_TH_CHARACTERISTICS`) — the fields `sap_bics_set_char_prop` manipulates: `RESULT_SET_PRESENTATION` (a bitflag, `KEY=4`, `TEXT=32`, `BOTH=36`), `RESULT_VISIBILITY` (`'A'` show / `'N'` hide / `'C'` conditional for totals), and `RESULT_SET_SORTING.DIRECTION` (`'A'` / `'D'` / `''`).

#### Decoding the result set

`BICS_PROV_GET_RESULT_SET` returns a multidimensional cross-tab, not a flat table, which ERPL flattens:

*   **Members & presentation** — `E_T_MEMBER` lists the axis members; each points into a `E_T_MEMBER_PRESENTATION` table via a `PRESENTATION_INDEX_FROM/TO` range. The display mode decides which presentation entry is used: `KEY` takes the key, `TEXT` the text, `BOTH` concatenates them.
*   **Row / column tuples** — `E_T_ROWS` and `E_T_COLUMNS` give the member references per axis position, including a `LEVEL` field that becomes the synthetic `"<char>_HIER_LEVEL"` integer column you see for hierarchy drilldowns.
*   **Data cells** — `E_T_DATA_CELLS` is a **sparse** `{ROW, COLUMN, VALUE}` list (cells equal to zero are omitted by the server). ERPL initialises every cell to `NULL` and fills only the returned ones, so an absent cell is `NULL`, not `0.0`. Cell `VALUE`s are always `DOUBLE`; axis member columns are always `VARCHAR`.
*   **Grand total** — the "Overall Result" / `SUMME` row is an ordinary row in `E_T_ROWS` whose member key is the literal `SUMME`; it is not a separate structure (see the note under _Presentation properties_ above for hiding it).
*   **Cell budget** — ERPL requests up to `I_MAX_DATA_CELLS = 1,000,000` cells per fetch. A drilldown that would exceed that must be narrowed with filters or fewer axis characteristics.

#### Variables

`BICS_PROV_VAR_GET_VARIABLES` retrieves a query's variable definitions (used by `sap_bics_describe`), but **variable _filling_ is not implemented** — there is no `BICS_PROV_VAR_SET_VARIABLES` call. Queries that declare variables execute with their default/empty values; restrict the result with `sap_bics_filter` instead.

#### Metadata and lineage

Discovery, `sap_bics_meta_*`, and `sap_bics_lineage_*` do **not** open a query. They read BW dictionary tables through `RFC_READ_TABLE` — among them `RSRREPDIR` and `RSZCOMPDIR` (queries), `RSDCUBE` (InfoCubes), `RSTRAN` (transformations), `ROOSFIELD` (DataSource fields), `RSHIEDIR` (hierarchies), and `RSOOBJXREF` (cross-references). Hierarchy nodes come from `RSNDI_SHIE_STRUCTURE_GET3`; single-InfoObject attributes from `BAPI_IOBJ_GETDETAIL`.

#### Security

Because BICS rides on RFC, its security model is the RFC model:

1.  **Authentication** — the `sap_rfc` secret (user/password) or SNC for certificate-based, encrypted logon. There is no Basic Auth or SAML at the BICS layer.
2.  **Authorization** — standard BW analysis authorizations (RSEC) govern which InfoProviders, queries, and characteristic values a user may read.
3.  **Transport security** — use SNC (or an SSH tunnel) to encrypt the RFC connection on untrusted networks.
4.  **Auditing** — SAP-side security audit log (SM19/SM20) records the RFC logons.

## Next steps

### 🚀 Ready for More?

*   [ODP Protocol Guide](/docs/erpl/odp.md) - Delta replication from SAP
*   [RFC Protocol Guide](/docs/erpl/rfc.md) - Read SAP tables and call functions
*   [Function Reference](/docs/reference/erpl-functions.md) - Complete API docs

### 🔧 Advanced Topics

*   [BICS Lineage Tracking](/docs/guides/advanced/bics-lineage-tracking.md) - Track data lineage
*   [Performance Tuning](/docs/guides/advanced/performance-tuning.md) - Optimize BW queries
*   [Real-World Use Cases](/docs/examples/real-world-use-cases.md) - Complete scenarios

### 💡 Examples

*   [ERPL Examples](/docs/examples/erpl-examples.md) - More real-world BICS examples
*   [Integration with Python](/docs/guides/integration/python-pandas.md) - Use BICS data with Pandas

---

**Need help?** Check our [troubleshooting guide](/docs/reference/troubleshooting.md) or browse [more examples](/docs/examples/erpl-examples.md).

# functions

# ODP Protocol Deep Dive

This comprehensive guide covers the ODP (Operational Data Provisioning) protocol in ERPL. Learn how to extract and replicate SAP data with delta/full modes, manage subscriptions, and track changes.

**What is ODP?:**

ODP (Operational Data Provisioning) is SAP's standard for data extraction and replication. It provides automatic subscription management, change tracking (Insert/Update/Delete), and supports multiple contexts including BW DataSources, SAPI, ABAP CDS, and SLT extractors.

## ODP Architecture

Like [BICS](/docs/erpl/bics.md), ODP in ERPL runs entirely over the **SAP NetWeaver RFC SDK** — it calls the `RODPS_REPL_*` RFC function modules and reuses the [RFC](/docs/erpl/rfc.md) extension's `sap_rfc` secret. There is no separate ODP network endpoint to configure.

## Understanding ODP Contexts

ODP supports multiple contexts for different data sources:

*   **BW** - SAP BW DataSources and InfoProviders
*   **SAPI** - Extraction APIs (SAP Application Interface)
*   **ABAP\_CDS** - CDS Views (Core Data Services)
*   **SLT** - SAP Landscape Transformation
*   **HANA** - SAP HANA views and tables

## Basic Extraction

### List Available Contexts

```
-- List all available ODP contextsSELECT * FROM sap_odp_show_contexts();-- Returns columns technical_name, text, release. Example rows:-- technical_name | text                                  | release-- ABAP_CDS       | ABAP Core Data Services               | SBC758-- BW             | SAP NetWeaver Business Warehouse       | SBW758-- HANA           | HANA Information Views                 |-- SAPI           | DataSources/Extractors                | SDE758
```

### List Data Sources

```
-- List data sources in BW contextSELECT * FROM sap_odp_show('BW');-- List data sources in SAPI contextSELECT * FROM sap_odp_show('SAPI');-- List data sources in ABAP_CDS contextSELECT * FROM sap_odp_show('ABAP_CDS');
```

### Describe Data Source Structure

```
-- Describe BW DataSource structureSELECT * FROM sap_odp_describe('BW', 'VBAK$F');-- Describe SAPI extractor structureSELECT * FROM sap_odp_describe('SAPI', '2LIS_11_VAHDR');-- Describe CDS view structureSELECT * FROM sap_odp_describe('ABAP_CDS', 'I_SALESORDER');
```

### Basic Full Load

```
-- Full load from BW DataSourceSELECT * FROM sap_odp_read_full('BW', 'VBAK$F');-- Full load from SAPI extractorSELECT * FROM sap_odp_read_full('SAPI', '2LIS_11_VAHDR');-- Full load from CDS viewSELECT * FROM sap_odp_read_full('ABAP_CDS', 'I_SALESORDER');
```

## Delta Replication

### Full vs. delta in ERPL

ERPL exposes two distinct extraction functions:

*   `sap_odp_read_full(context, data_source)` is **one-shot**: it opens a FULL cursor on the SAP side, streams the snapshot, then auto-closes the cursor when the scan ends. No state survives.
*   `sap_odp_read_delta(context, data_source, subscriber_process)` keeps a **server-side delta pointer** keyed by `subscriber_process`. The first call performs SAP's auto-DELTAINIT — it returns the full current snapshot AND registers the pointer. Subsequent calls with the same `subscriber_process` resume from the pointer and return only the changes since then.

Pick a stable `subscriber_process` per pipeline (e.g. `'NIGHTLY_ETL_BUPA'`). It is the key that ties successive calls together. Close cursors with `PRAGMA sap_odp_close_delta_cursor` (graceful) or `PRAGMA sap_odp_drop` (hard reset). Inspect them with `sap_odp_show_cursors()`.

### Initial Delta Load (DELTAINIT)

```
-- First call: auto-DELTAINIT. Returns the full current snapshot AND registers-- a server-side delta pointer under subscriber_process 'NIGHTLY_ETL'.SELECT * FROM sap_odp_read_delta('BW', 'VBAK$F', 'NIGHTLY_ETL');
```

### Subsequent Delta Loads

```
-- Every subsequent call with the same subscriber_process returns only the-- changes since the previous call.SELECT * FROM sap_odp_read_delta('BW', 'VBAK$F', 'NIGHTLY_ETL');-- Categorize changes by ODQ_CHANGEMODE — ODP exposes the operation kind.SELECT  ODQ_CHANGEMODE,             -- C = Create/Insert, U = Update, D = Delete  COUNT(*) AS record_countFROM sap_odp_read_delta('BW', 'VBAK$F', 'NIGHTLY_ETL')GROUP BY ODQ_CHANGEMODE;
```

**Concurrency:**

Do not run two `sap_odp_read_delta` calls with the same `subscriber_process` in parallel — they will race the server-side pointer.

### Probe before extract

Use `sap_odp_get_last_modified` to skip pipelines when nothing has changed since the last run — it returns a single timestamp without opening a cursor.

```
-- Last-modified probe (returns 0.0 for unknown ODP names)SELECT * FROM sap_odp_get_last_modified('BW', 'VBAK$F');-- Skip extract if untouchedWITH probe AS (  SELECT last_modified FROM sap_odp_get_last_modified('BW', 'VBAK$F'))SELECT *FROM sap_odp_read_delta('BW', 'VBAK$F', 'NIGHTLY_ETL')WHERE (SELECT last_modified FROM probe) > 20260516000000.0;
```

### Recovering an interrupted fetch

```
-- recover => true re-streams the last unconfirmed packet without advancing-- the pointer. Use after a network blip or process crash mid-fetch.SELECT * FROM sap_odp_read_delta('BW', 'VBAK$F', 'NIGHTLY_ETL', recover => true);
```

### Releasing the cursor

```
-- Graceful: keep the subscription registered, just close its current pointer.-- The next sap_odp_read_delta call with the same subscriber_process resumes.PRAGMA sap_odp_close_delta_cursor('BW', 'NIGHTLY_ETL', 'VBAK$F');-- Hard reset: wipe the subscription. The next call performs DELTAINIT again.PRAGMA sap_odp_drop('BW', 'ERPL', 'NIGHTLY_ETL', 'VBAK$F');
```

## Advanced Topics

### Preview Data Without Subscription

```
-- Preview data without creating subscriptionSELECT * FROM sap_odp_preview('BW', 'VBAK$F');-- Preview with limitSELECT * FROM sap_odp_preview('BW', 'VBAK$F', max_rows => 100);
```

### Selection and Filtering

ERPL pushes most DuckDB SQL clauses down to RFC automatically:

```
-- Column projection: the SELECT list determines which columns RFC returns.SELECT VBELN, ERDAT, KUNNR, NETWRFROM sap_odp_read_full('BW', 'VBAK$F');-- Tune parallelism and project columns explicitly.SELECT * FROM sap_odp_read_full(  'BW',  'VBAK$F',  threads => 4,  columns => ['VBELN', 'ERDAT', 'KUNNR', 'NETWR']);
```

For server-side row filtering, ODP uses a structured `filters` parameter built from the `ODP_SELECT_SIGN` / `ODP_SELECT_OP` enums. See the [function reference](/docs/reference/erpl-functions.md#odp-types) for the exact shape.

## Advanced: Subscription Management (For SAP Experts)

**For SAP Administrators:**

This section covers advanced subscription management, cursor handling, and recovery strategies.

### View Active Subscriptions

```
-- ERPL-owned subscriptions (default)SELECT * FROM sap_odp_show_subscriptions();-- All subscribers on the SAP system (cross-team visibility)SELECT * FROM sap_odp_show_subscriptions(ERPL_ONLY => FALSE);-- Cross-subscriber visibility on a single source (RODPS_REPL_ODP_GET_SUBSCR)SELECT * FROM sap_odp_get_subscriptions('BW', 'VBAK$F');
```

`sap_odp_show_subscriptions` returns `queue_name, subscriber_type, subscriber_name, subscriber_proc`. `sap_odp_get_subscriptions` additionally exposes `model_name` and the numeric `subscription_id`.

### Check Extraction Cursors

```
-- View extraction cursorsSELECT * FROM sap_odp_show_cursors();-- Narrow to delta cursors created by a specific subscriberSELECT * FROM sap_odp_show_cursors(  subscriber_name => 'ERPL',  replication_mode => 'DELTA');
```

Cursors created by `sap_odp_read_delta` appear here with `is_delta_extension=true`; the `subscriber_proc` column matches the `subscriber_process` argument you passed.

### Closing a delta cursor

`PRAGMA sap_odp_close_delta_cursor` releases the cursor on the SAP side while keeping the subscription registered. The next `sap_odp_read_delta` call with the same `subscriber_process` resumes from the same pointer. Idempotent: returns `'CLOSED'` or `'NOT_FOUND'`.

```
PRAGMA sap_odp_close_delta_cursor('BW', 'NIGHTLY_ETL', 'VBAK$F');
```

### Resetting a subscription

`PRAGMA sap_odp_drop` invokes `RODPS_REPL_ODP_RESET` to wipe the subscription entirely. The next `sap_odp_read_delta` call with the same subscriber tuple performs DELTAINIT again.

```
-- PRAGMA sap_odp_drop(odp_context, subscriber_name, subscriber_process, odp_name)PRAGMA sap_odp_drop('BW', 'ERPL', 'NIGHTLY_ETL', 'VBAK$F');
```

Prefer `sap_odp_close_delta_cursor` for normal pipeline shutdown; reach for `sap_odp_drop` when a cursor is stuck or you intentionally want to restart from a fresh snapshot.

### Subscription Lifecycle Example

```
-- 1. Probe — does this source have changes worth pulling?SELECT last_modified FROM sap_odp_get_last_modified('BW', 'VBAK$F');-- 2. First call: auto-DELTAINIT.  Registers a delta pointer under 'NIGHTLY_ETL'.SELECT * FROM sap_odp_read_delta('BW', 'VBAK$F', 'NIGHTLY_ETL');-- 3. Inspect the cursor that was createdSELECT * FROM sap_odp_show_cursors(subscriber_name => 'ERPL')WHERE subscriber_proc = 'NIGHTLY_ETL';-- 4. Subsequent runs: only changes since the previous callSELECT * FROM sap_odp_read_delta('BW', 'VBAK$F', 'NIGHTLY_ETL');-- 5. Pipeline finished — close the cursor gracefullyPRAGMA sap_odp_close_delta_cursor('BW', 'NIGHTLY_ETL', 'VBAK$F');
```

## Real-World Examples

### Daily Delta Replication Pipeline

```
-- Daily delta replication workflowWITH delta_data AS (  SELECT    *,    ODQ_CHANGEMODE,    CASE ODQ_CHANGEMODE      WHEN 'C' THEN 'INSERT'      WHEN 'U' THEN 'UPDATE'      WHEN 'D' THEN 'DELETE'      ELSE 'UNKNOWN'    END AS change_type  FROM sap_odp_read_delta('BW', 'VBAK$F', 'NIGHTLY_ETL'))SELECT  change_type,  COUNT(*) AS record_count,  MIN(ERDAT) AS earliest_date,  MAX(ERDAT) AS latest_dateFROM delta_dataGROUP BY change_typeORDER BY change_type;
```

### Multi-Context Data Integration

```
-- Combine data from multiple ODP contextsWITH bw_sales AS (  SELECT     VBELN AS sales_doc,    ERDAT AS doc_date,    KUNNR AS customer,    NETWR AS net_value,    'BW' AS source_context  FROM sap_odp_read_full('BW', 'VBAK$F')),cds_sales AS (  SELECT     SalesOrder AS sales_doc,    DocumentDate AS doc_date,    SoldToParty AS customer,    NetAmount AS net_value,    'CDS' AS source_context  FROM sap_odp_read_full('ABAP_CDS', 'I_SALESORDER'))SELECT   sales_doc,  doc_date,  customer,  net_value,  source_contextFROM bw_salesUNION ALLSELECT   sales_doc,  doc_date,  customer,  net_value,  source_contextFROM cds_salesORDER BY doc_date DESC, net_value DESC;
```

### Change Data Capture Analysis

```
-- Analyze change patternsWITH change_analysis AS (  SELECT    ODQ_CHANGEMODE,    ERDAT AS change_date,    COUNT(*) AS change_count,    SUM(NETWR) AS total_value  FROM sap_odp_read_delta('BW', 'VBAK$F', 'CDC_ANALYSIS')  GROUP BY ODQ_CHANGEMODE, ERDAT)SELECT  ODQ_CHANGEMODE,  change_date,  change_count,  total_value,  change_count * 100.0 / SUM(change_count) OVER() AS percentage_of_changesFROM change_analysisORDER BY change_date DESC, change_count DESC;
```

### Error Recovery Pattern

If a delta fetch was interrupted, try `recover => true` first to re-stream the last unconfirmed packet without advancing the pointer. If the cursor itself is wedged, fall back to `PRAGMA sap_odp_drop` to wipe the subscription so the next call performs DELTAINIT.

```
-- 1. Inspect what's thereSELECT * FROM sap_odp_show_cursors(subscriber_name => 'ERPL');-- 2a. Replay the last packet (no pointer advance)SELECT * FROM sap_odp_read_delta('BW', 'VBAK$F', 'NIGHTLY_ETL', recover => true);-- 2b. Or hard-reset if the cursor is stuckPRAGMA sap_odp_drop('BW', 'ERPL', 'NIGHTLY_ETL', 'VBAK$F');-- 3. Re-run — DELTAINIT after a drop returns the full snapshot againSELECT * FROM sap_odp_read_delta('BW', 'VBAK$F', 'NIGHTLY_ETL');
```

## Performance Optimization

### Batch Processing

```
-- Process multiple data sources in parallelSELECT 'VBAK$F' AS data_source, COUNT(*) AS record_countFROM sap_odp_read_full('BW', 'VBAK$F')UNION ALLSELECT 'VBAP$F' AS data_source, COUNT(*) AS record_countFROM sap_odp_read_full('BW', 'VBAP$F')UNION ALLSELECT 'KNA1$F' AS data_source, COUNT(*) AS record_countFROM sap_odp_read_full('BW', 'KNA1$F');
```

### Selective Field Extraction

```
-- Extract only specific fields to reduce data volumeSELECT   VBELN,  ERDAT,  KUNNR,  NETWRFROM sap_odp_read_full(  'BW',   'VBAK$F');
```

### Connection Pooling

```
-- Reuse connections for multiple extractions-- (Connection pooling is handled automatically by ERPL)SELECT * FROM sap_odp_read_full('BW', 'VBAK$F');SELECT * FROM sap_odp_read_full('BW', 'VBAP$F');SELECT * FROM sap_odp_read_full('BW', 'KNA1$F');
```

## Troubleshooting

### Common Issues

**Data Source Not Found**

```
-- Check available data sourcesSELECT * FROM sap_odp_show('BW');-- Verify data source name (case-sensitive)SELECT * FROM sap_odp_describe('BW', 'VBAK$F');
```

**Subscription Errors**

```
-- Check subscription status (ERPL-owned)SELECT * FROM sap_odp_show_subscriptions();-- Or look at *all* subscribers on a specific sourceSELECT * FROM sap_odp_get_subscriptions('BW', 'VBAK$F');-- Check cursor statusSELECT * FROM sap_odp_show_cursors();-- Reset the cursor and re-DELTAINITPRAGMA sap_odp_drop('BW', 'ERPL', 'NIGHTLY_ETL', 'VBAK$F');SELECT * FROM sap_odp_read_delta('BW', 'VBAK$F', 'NIGHTLY_ETL');
```

**Permission Denied**

```
-- Check ODP access permissionsSELECT * FROM sap_odp_show_contexts();-- Verify user permissions in SAPSELECT * FROM sap_odp_show('BW');
```

**No Delta Data**

```
-- Check if a subscription exists for this pipelineSELECT * FROM sap_odp_show_subscriptions()WHERE subscriber_proc = 'NIGHTLY_ETL';-- Probe whether anything changed since the last callSELECT last_modified FROM sap_odp_get_last_modified('BW', 'VBAK$F');-- If no subscription, the next call auto-DELTAINITsSELECT * FROM sap_odp_read_delta('BW', 'VBAK$F', 'NIGHTLY_ETL');
```

### Debugging Tips

```
-- Enable ERPL tracing to see the underlying RFC callsSET erpl_trace_enabled = TRUE;SET erpl_trace_level   = 'DEBUG';   -- TRACE | DEBUG | INFO | WARN | ERRORSET erpl_trace_output  = 'console'; -- console | file | both-- Preview data structureSELECT * FROM sap_odp_preview('BW', 'VBAK$F', max_rows => 10);-- Check subscription detailsSELECT * FROM sap_odp_show_subscriptions();-- Monitor extraction performanceEXPLAIN SELECT * FROM sap_odp_read_full(  'BW',   'VBAK$F');
```

## For SAP Experts

The wire-level ODP mechanics for SAP Basis administrators and developers — the `RODPS_REPL_*` modules, the fetch loop, the BXML payload, and the delta-queue (ODQ) semantics behind the SQL functions. Expand the deep dive below.

**ODP protocol deep dive** — RODPS\_REPL\_\* modules, fetch loop, BXML payload, delta/pointer mechanics, ODQ columns

#### Protocol details

ODP in ERPL is an **RFC interface**, not a separate web protocol. Everything runs over the SAP NetWeaver RFC SDK using your `sap_rfc` secret. The relevant RFC function modules are:

| ERPL function | RFC function module(s) | Key parameters |
| --- | --- | --- |
| `sap_odp_show_contexts` | `RODPS_REPL_CONTEXT_GET_LIST` | —   |
| `sap_odp_show` | `RODPS_REPL_ODP_GET_LIST` | `I_CONTEXT`, `I_SEARCH_PATTERN` |
| `sap_odp_describe` / `read_*` (bind) | `RODPS_REPL_ODP_GET_DETAIL` | `I_CONTEXT`, `I_ODPNAME` → `ET_FIELDS`, `E_SUPPORTS_FULL/DELTA` |
| `sap_odp_read_full` / `read_delta` | `RODPS_REPL_ODP_OPEN` → `RODPS_REPL_ODP_FETCH_XML` → `RODPS_REPL_ODP_CLOSE` | `I_EXTRACTION_MODE`, `E_POINTER` |
| `sap_odp_preview` | `RODPS_REPL_ODP_READ_DIRECT_XML` | `I_MAXIMUM_ROWS` |
| `sap_odp_get_last_modified` | `RODPS_REPL_ODP_GET_LAST_MODIF` | `IT_ODP` → `E_LAST_MODIFIED` |
| `sap_odp_get_subscriptions` | `RODPS_REPL_ODP_GET_SUBSCR` | `I_CONTEXT`, `I_ODPNAME` |
| `sap_odp_show_cursors` | `RODPS_REPL_CURSOR_GET_LIST` | `I_SUBSCRIBER_*`, `I_EXTRACTION_MODE` |
| `PRAGMA sap_odp_close_delta_cursor` | `RODPS_REPL_ODP_CLOSE` | `I_POINTER` |
| `PRAGMA sap_odp_drop` | `RODPS_REPL_ODP_RESET` | subscriber tuple |

Authentication is the RFC `sap_rfc` secret (user/password or SNC), and ODP authorizations (RODPS) govern which contexts and DataSources a user may read.

#### The OPEN → FETCH → CLOSE loop

Every extraction is an `OPEN`, a loop of `FETCH_XML` calls, and a `CLOSE`:

1.  **OPEN** — `RODPS_REPL_ODP_OPEN` is called with `I_EXTRACTION_MODE` = `'F'` (full), `'D'` (delta), or `'R'` (recover), the subscriber tuple, `I_MAXPACKAGESIZE` (≈2 MiB), and the selection/projection tables. It returns `E_POINTER` (the cursor handle / TSN) and `E_DELTA_EXTENSION`.
2.  **FETCH\_XML** — `RODPS_REPL_ODP_FETCH_XML` is called repeatedly with `I_POINTER` and a 6-digit `I_PACKAGE` counter (`000001`, `000002`, …). Each call returns one data package as `E_XML` plus `E_NO_MORE_DATA`. The loop ends when `E_NO_MORE_DATA = 'X'`.
3.  **CLOSE** — `RODPS_REPL_ODP_CLOSE` releases the cursor. **Full** reads close automatically when the scan ends; **delta** reads do **not** — you must call `PRAGMA sap_odp_close_delta_cursor` (or let SAP auto-close on `E_NO_MORE_DATA`).

ERPL can fetch packages with up to `threads => N` workers (default 5); each worker decodes its own package while the session serialises the shared cursor pointer and package counter.

#### The BXML payload

The `_XML` in `FETCH_XML` / `READ_DIRECT_XML` is **not** HTTP and **not** plain text. SAP returns a DEFLATE-compressed **BXML** (binary XML, format `ODQ_G`) stream _inside_ the RFC response. ERPL inflates it (miniz) and parses the token stream client-side:

*   The stream starts with the 4-byte magic `BXML`; a missing magic raises `"Input has no valid BXML magic number"` (the error you get when a source doesn't return a usable payload — e.g. some BW extractors under `sap_odp_preview`).
*   Row records appear as repeated `item` elements; each field tag maps to a result column.
*   SAP "initial" sentinels in date/time fields (`00000000`, `0000-00-00`, `000000`, `00:00:00`, `0000-00-00 00:00:00`) are surfaced as SQL `NULL` so DuckDB's date/time casts don't reject them. Fields whose name ends in `uuid` are decoded from binary to a UUID string.

#### Delta, subscriptions and the pointer

A subscription is identified by a **subscriber tuple**: `I_SUBSCRIBER_TYPE` (ERPL uses `SAP_BW`), `I_SUBSCRIBER_NAME` (`ERPL`), and `I_SUBSCRIBER_PROCESS` (the `subscriber_process` you pass). The server-side **pointer** is a `DECIMAL(23,9)` whose leading 14 digits are a UTC timestamp (`YYYYMMDDhhmmss`) — that is what `sap_odp_show_cursors.request_date` is parsed from.

*   **Auto-DELTAINIT** — the first `OPEN` in mode `'D'` for a new subscriber tuple has no stored pointer, so SAP returns the **full snapshot** and registers a pointer (`E_DELTA_EXTENSION = 'X'`). Subsequent `'D'` opens with the same tuple resume from that pointer and return only changes.
*   **Recover** (`recover => true`) — opens in mode `'R'`, which re-streams the **last unconfirmed package without advancing the pointer**. Use it after a crash/blip mid-fetch.
*   **Reset vs close** — `PRAGMA sap_odp_drop` calls `RODPS_REPL_ODP_RESET`, which deletes the subscription and clears the pointer (next read re-DELTAINITs from scratch). `PRAGMA sap_odp_close_delta_cursor` calls `RODPS_REPL_ODP_CLOSE`, which releases the cursor but **keeps the subscription** so the next read resumes.

> **⚠️ One consumer per `subscriber_process`** — two `sap_odp_read_delta` calls racing the same `subscriber_process` both `OPEN` the same server-side pointer; SAP's locking there is undefined and one may get a stale pointer or error. Give each parallel pipeline its own `subscriber_process`.

#### The ODQ change columns

Delta result sets carry SAP's operational delta-queue (ODQ) bookkeeping columns alongside your data fields:

| Column | Type | Meaning |
| --- | --- | --- |
| `ODQ_CHANGEMODE` | VARCHAR(1) | Operation: `C` = create/insert, `U` = update, `D` = delete (blank = unspecified) |
| `ODQ_ENTITYCNTR` | BIGINT | Entity counter — distinguishes before/after images within an update |
| `ODQ_TSN` | DECIMAL | Transaction sequence number of the delta package (matches the pointer batch) |
| `ODQ_UNITNO` | INTEGER | Logical unit number grouping related changes |
| `ODQ_RECORDNO` | INTEGER | Record number within the unit |

A full read returns these too (with `ODQ_CHANGEMODE = 'C'` for the snapshot rows).

#### Selection pushdown

The structured `filters` parameter is pushed to SAP as the `IT_SELECT` table (`RODPS_REPL_S_SELECTION` rows: `FIELDNAME`, `SIGN` `'I'`/`'E'`, `OPT` `'EQ'`/`'BT'`/…, `LOW`, `HIGH`) so the server restricts rows before they are packaged — the ODP equivalent of an ABAP SELECT-OPTIONS range. Column projection (`columns => [...]` or the SELECT list) becomes the projection table so unused fields are never serialised.

#### Cursor and subscription inspection

*   `sap_odp_show_cursors` (`RODPS_REPL_CURSOR_GET_LIST`) returns `queue_name`, `subscriber_proc`, `subscriber_id`, `pointer`, `is_closed`, `is_delta_extension`, and the derived `request_date`. `is_delta_extension` is true for delta/recover cursors; `is_closed` is set once a cursor has drained or been closed.
*   `sap_odp_show_subscriptions` reads the subscription registry; `sap_odp_get_subscriptions` (`RODPS_REPL_ODP_GET_SUBSCR`) lists _all_ subscribers on one source, adding `model_name` and the numeric `subscription_id`.

#### Security

Because ODP rides on RFC, its security model is the RFC model:

1.  **Transport security** — use SNC (or an SSH tunnel) to encrypt the RFC connection
2.  **Authentication** — the `sap_rfc` secret: user/password or SNC certificates
3.  **Authorization** — standard ODP/RODPS authorizations govern context and DataSource access
4.  **Audit logging** — the SAP security audit log (SM19/SM20) records the RFC logons

## Next Steps

### 🚀 Ready for More?

*   [RFC Protocol Guide](/docs/erpl/rfc.md) - Read SAP tables and call functions
*   [BICS Protocol Guide](/docs/erpl/bics.md) - Execute SAP BW queries
*   [Function Reference](/docs/reference/erpl-functions.md) - Complete API docs

### 🔧 Advanced Topics

*   [ODP Subscription Management](/docs/guides/advanced/odp-subscription-management.md) - Complete subscription lifecycle guide
*   [Performance Tuning](/docs/guides/advanced/performance-tuning.md) - Optimize ODP extractions
*   [Real-World Use Cases](/docs/examples/real-world-use-cases.md) - Complete scenarios

### 💡 Examples

*   [ERPL Examples](/docs/examples/erpl-examples.md) - More real-world ODP examples
*   [Integration with Python](/docs/guides/integration/python-pandas.md) - Use ODP data with Pandas
*   [Real-World Use Cases](/docs/examples/real-world-use-cases.md) - Complete scenarios

---

**Need help?** Check our [troubleshooting guide](/docs/reference/troubleshooting.md) or browse [more examples](/docs/examples/erpl-examples.md).

# RFC Protocol Deep Dive

This comprehensive guide covers the RFC (Remote Function Call) protocol in ERPL. Learn how to read SAP tables, call function modules, and handle metadata like a pro.

**What is RFC?:**

RFC (Remote Function Call) is SAP's protocol for calling functions across system boundaries. It's the foundation of SAP integration and allows you to access SAP tables and execute function modules remotely.

## RFC Architecture

## Core RFC Functions

### `sap_read_table()`

The primary function for reading SAP tables.

```
-- Basic table readingSELECT * FROM sap_read_table('KNA1', MAX_ROWS => 100);-- With field selectionSELECT KUNNR, NAME1, LAND1 FROM sap_read_table('KNA1', MAX_ROWS => 50);-- With WHERE conditionsSELECT * FROM sap_read_table('VBAK', MAX_ROWS => 100)WHERE ERDAT >= '2024-01-01';
```

#### Parameters

| Parameter | Type | Description | Example |
| --- | --- | --- | --- |
| `table_name` | VARCHAR (positional) | SAP table name | `'KNA1'` |
| `COLUMNS` | LIST(VARCHAR) | Restrict to a subset of columns | `COLUMNS => ['KUNNR', 'NAME1']` |
| `FILTER` | VARCHAR | OpenSQL-style WHERE-fragment passed straight to RFC. Use for filters too complex to push down. | `FILTER => 'LAND1 = ''DE'''` |
| `MAX_ROWS` | UINTEGER | Maximum rows to return | `MAX_ROWS => 1000` |
| `THREADS` | UINTEGER | Number of parallel RFC threads (default: 5) | `THREADS => 4` |
| `READ_TABLE_FUNCTION` | VARCHAR | Override the underlying RFC function module (e.g. `/SAPDS/RFC_READ_TABLE2` for wide rows) | `READ_TABLE_FUNCTION => '/SAPDS/RFC_READ_TABLE2'` |
| `READ_TABLE_DELIMITER` | VARCHAR | Delimiter RFC uses when packing rows. Override for tables whose values may contain whitespace. | \`READ\_TABLE\_DELIMITER => ' |
| `SECRET` | VARCHAR | Name of the DuckDB secret to authenticate with | `SECRET => 'erp_prod'` |

**Predicate and projection pushdown:**

A regular SQL `WHERE` clause is pushed down to RFC automatically — you don't need the `FILTER` named parameter for simple predicates. `COLUMNS` is inferred from the SELECT list:

```
SELECT KUNNR, NAME1 FROM sap_read_table('KNA1') WHERE LAND1 = 'DE';
```

Reach for the named parameters when you need precise control over what RFC receives — for example, complex multi-table filters that DuckDB's optimizer can't safely lower.

**Ordering:**

RFC's underlying `RFC_READ_TABLE` has no server-side sort. Apply `ORDER BY` in DuckDB after the read; it executes client-side.

### `sap_rfc_invoke()`

Call SAP function modules and BAPIs. Input parameters are passed as **positional struct arguments** after the function name — one struct per import / changing parameter the BAPI expects. DuckDB struct syntax (`{'KEY': value, ...}`) lets you express nested structures and tables naturally.

```
-- Simple function module with one import structSELECT trim(ECHOTEXT)FROM sap_rfc_invoke('STFC_CONNECTION', {'REQUTEXT': 'Hello'});-- BAPI returning a sub-table — pick it via pathSELECT *FROM sap_rfc_invoke(  'BAPI_FLIGHT_GETLIST',  {'AIRLINE': 'LH', 'DESTINATION_FROM': {'AIRPORTID': 'FRA'}},  path => '/FLIGHT_LIST');-- BAPI with nested import + path selectionSELECT *FROM sap_rfc_invoke(  'BAPI_FLIGHT_GETDETAIL',  {'AIRLINEID': 'LH', 'CONNECTIONID': '0400', 'FLIGHTDATE': '2016-11-18'::DATE});
```

#### Parameters

| Parameter | Type | Description | Example |
| --- | --- | --- | --- |
| `function_name` | VARCHAR (positional) | SAP function module / BAPI name | `'BAPI_FLIGHT_GETLIST'` |
| `parameters...` | STRUCT (positional varargs) | One struct per RFC import/changing parameter | `{'AIRLINE': 'LH'}` |
| `path` | VARCHAR (named) | Optional path into the response structure — picks a sub-table of the result | `path => '/FLIGHT_LIST'` |
| `secret` | VARCHAR (named) | Name of the DuckDB secret to authenticate with | `secret => 'erp_prod'` |

### `sap_rfc_describe_function()`

Get detailed information about RFC function parameters.

```
-- Describe function parametersSELECT * FROM sap_rfc_describe_function('BAPI_FLIGHT_GETLIST');-- Get parameter detailsSELECT   name,  import[1].name AS import_params,  export[1].name AS export_paramsFROM sap_rfc_describe_function('STFC_CONNECTION');
```

### `sap_describe_fields()`

Get table structure and field information. The result columns are `pos`, `is_key`, `field`, `text`, `sap_type`, `length`, `decimals`, `check_table`, `ref_table`, `ref_field`, and `language`.

```
-- Get table structureSELECT * FROM sap_describe_fields('KNA1');-- Get field details for a specific tableSELECT  field,  text,  sap_type,  length,  decimalsFROM sap_describe_fields('VBAK');
```

### `sap_rfc_authorizations()`

Document which RFC function module each ERPL function calls — useful when requesting RFC authorizations from SAP Basis. Returns `extension`, `duckdb_function`, `rfc_function_module`, `invocation`, and `purpose`.

```
-- Every RFC module ERPL may invokeSELECT * FROM sap_rfc_authorizations();-- Just the distinct module names used by the RFC extensionSELECT DISTINCT rfc_function_moduleFROM sap_rfc_authorizations()WHERE extension = 'erpl_rfc';
```

### `sap_rfc_show_function()`

List all available RFC functions.

```
-- List all RFC functionsSELECT * FROM sap_rfc_show_function();
```

### `sap_rfc_show_groups()`

List all RFC function groups.

```
-- List all function groupsSELECT * FROM sap_rfc_show_groups();
```

### `sap_show_tables()`

List all available SAP tables.

```
-- List all tablesSELECT * FROM sap_show_tables();
```

## Common SAP Tables

### Master Data Tables

```
-- Customer Master DataSELECT   KUNNR AS customer_number,  NAME1 AS customer_name,  LAND1 AS country,  REGIO AS region,  ORT01 AS cityFROM sap_read_table('KNA1', MAX_ROWS => 1000)WHERE LAND1 = 'DE';-- Material Master DataSELECT   MATNR AS material_number,  MTART AS material_type,  MEINS AS base_unit,  MATKL AS material_groupFROM sap_read_table('MARA', MAX_ROWS => 1000)WHERE MTART = 'FERT';  -- Finished goods-- Vendor Master DataSELECT   LIFNR AS vendor_number,  NAME1 AS vendor_name,  LAND1 AS country,  REGIO AS regionFROM sap_read_table('LFA1', MAX_ROWS => 1000);
```

### Transaction Data Tables

```
-- Sales Document HeaderSELECT   VBELN AS sales_document,  ERDAT AS document_date,  KUNNR AS customer,  NETWR AS net_value,  WAERK AS currencyFROM sap_read_table('VBAK', MAX_ROWS => 1000)WHERE ERDAT >= '2024-01-01';-- Sales Document ItemsSELECT   VBELN AS sales_document,  POSNR AS item_number,  MATNR AS material,  KWMENG AS quantity,  NETWR AS net_valueFROM sap_read_table('VBAP', MAX_ROWS => 1000)WHERE VBELN IN ('0000000001', '0000000002');
```

### Financial Tables

```
-- Accounting Document HeaderSELECT   BUKRS AS company_code,  BELNR AS document_number,  GJAHR AS fiscal_year,  BLART AS document_type,  BUDAT AS posting_dateFROM sap_read_table('BKPF', MAX_ROWS => 1000)WHERE GJAHR = '2024';-- Accounting Document ItemsSELECT   BUKRS AS company_code,  BELNR AS document_number,  GJAHR AS fiscal_year,  BUZEI AS line_item,  HKONT AS gl_account,  DMBTR AS amount_in_local_currencyFROM sap_read_table('BSEG', MAX_ROWS => 1000)WHERE HKONT LIKE '1%';  -- Asset accounts
```

## Advanced RFC Techniques

### Metadata Handling

```
-- Get table structureSELECT * FROM sap_describe_fields('KNA1');-- Get function module parametersSELECT * FROM sap_rfc_describe_function('BAPI_CUSTOMER_GETDETAIL');-- List all available functionsSELECT * FROM sap_rfc_show_function();-- List function groupsSELECT * FROM sap_rfc_show_groups();-- List all tablesSELECT * FROM sap_show_tables();
```

### Batch Processing

```
-- Process multiple tables in one callSELECT   'KNA1' AS table_name,  COUNT(*) AS record_countFROM sap_read_table('KNA1', MAX_ROWS => 10000)UNION ALLSELECT   'VBAK' AS table_name,  COUNT(*) AS record_countFROM sap_read_table('VBAK', MAX_ROWS => 10000);
```

### Reading function output and errors

`sap_rfc_invoke` returns the function module's **actual export/table parameters** as columns — there are no synthetic `return_code`/`error_message` columns. For `RFC_SYSTEM_INFO`, the export parameter is the `RFCSI_EXPORT` struct, so you read fields out of it:

```
-- System info is returned as the RFCSI_EXPORT structSELECT  RFCSI_EXPORT.rfcsysid AS system_id,  RFCSI_EXPORT.rfchost  AS host,  RFCSI_EXPORT.rfcsaprl AS releaseFROM sap_rfc_invoke('RFC_SYSTEM_INFO');
```

For BAPIs, error status lives in the BAPI's own `RETURN` table (`BAPIRET2`). Select it with the `path` parameter and filter on its `TYPE` column (`'E'` = error, `'A'` = abort):

```
SELECT *FROM sap_rfc_invoke('BAPI_SOME_METHOD', {'KEY': 'VALUE'}, path => '/RETURN')WHERE TYPE IN ('E', 'A');
```

## Performance Optimization

### Mounting SAP tables as a catalog

ERPL caches and reuses RFC connections per secret automatically — there is no separate connection-pool object to configure. If you want SAP tables to appear as a browsable schema (so you can write `sap.KNA1` instead of `sap_read_table('KNA1')`), `ATTACH` a `sap_rfc` catalog. Use the `TABLES` option to limit which tables are exposed:

```
-- Mount selected SAP tables as the "sap" catalogATTACH '' AS sap (TYPE sap_rfc, SECRET 'my_sap', TABLES 'KNA1,VBAK,MARA');-- Query them like local tables (still read over RFC, with push-down)SELECT KUNNR, NAME1 FROM sap.KNA1 WHERE LAND1 = 'DE';SELECT * FROM sap.VBAK LIMIT 100;
```

### Parallel Processing

```
-- Read multiple tables in parallelSELECT 'KNA1' AS table_name, COUNT(*) AS countFROM sap_read_table('KNA1', MAX_ROWS => 10000)UNION ALLSELECT 'VBAK' AS table_name, COUNT(*) AS countFROM sap_read_table('VBAK', MAX_ROWS => 10000)UNION ALLSELECT 'MARA' AS table_name, COUNT(*) AS countFROM sap_read_table('MARA', MAX_ROWS => 10000);
```

## Real-World Examples

### Daily Sales Report

```
-- Extract daily sales dataWITH daily_sales AS (  SELECT     ERDAT AS sales_date,    COUNT(*) AS order_count,    SUM(NETWR) AS total_value,    AVG(NETWR) AS avg_order_value  FROM sap_read_table('VBAK', MAX_ROWS => 100000)  WHERE ERDAT >= CURRENT_DATE - INTERVAL '30' DAY  GROUP BY ERDAT)SELECT   sales_date,  order_count,  total_value,  avg_order_value,  ROUND(avg_order_value, 2) AS avg_order_value_roundedFROM daily_salesORDER BY sales_date DESC;
```

### Customer Analysis

```
-- Analyze customer distribution by countrySELECT   LAND1 AS country,  COUNT(*) AS customer_count,  COUNT(*) * 100.0 / SUM(COUNT(*)) OVER() AS percentageFROM sap_read_table('KNA1', MAX_ROWS => 100000)GROUP BY LAND1ORDER BY customer_count DESCLIMIT 10;
```

### Material Master Analysis

```
-- Analyze material types and groupsSELECT   MTART AS material_type,  MATKL AS material_group,  COUNT(*) AS material_countFROM sap_read_table('MARA', MAX_ROWS => 100000)GROUP BY MTART, MATKLORDER BY material_count DESCLIMIT 20;
```

## Troubleshooting

### Common Issues

**Table Not Found**

```
-- Check table name (case-sensitive)-- Verify table exists in your SAP systemSELECT * FROM sap_read_table('KNA1', MAX_ROWS => 1);
```

**Permission Denied**

```
-- Verify RFC access rights by calling a low-privilege functionSELECT * FROM sap_rfc_invoke('RFC_SYSTEM_INFO');
```

**Connection Timeout**

Configure timeouts via the `sap_rfc` secret options when creating the secret:

```
CREATE SECRET my_sap (  TYPE sap_rfc,  ASHOST 'your-host',  SYSNR '00',  USER 'username',  PASSWD 'password',  CLIENT '100',  LANG 'EN');
```

### Debugging Tips

```
-- Ping the SAP system (pragma — no result rows; raises on failure)PRAGMA sap_rfc_ping;-- Monitor query performanceEXPLAIN SELECT * FROM sap_read_table('KNA1', MAX_ROWS => 100);-- Smoke-test the RFC pipelineSELECT * FROM sap_rfc_invoke('RFC_SYSTEM_INFO');
```

## For SAP Experts

For SAP Basis administrators and developers: the binary RFC transport, performance characteristics, and security model. Expand the internals below.

**RFC protocol internals** — transport, performance, security

#### RFC Protocol Details

ERPL uses the proprietary **SAP NetWeaver RFC SDK**, which speaks SAP's binary RFC protocol (CPIC over TCP/IP) — the same library SAP GUI and other SAP tools use. Characteristics:

*   **Connection**: direct application-server logon (`ASHOST` + `SYSNR`) via the SAP Gateway (port `33NN`, where `NN` is the instance number), or message-server/load-balanced logon (`MSHOST` + `GROUP`)
*   **Protocol**: binary RFC over TCP/IP (not HTTP)
*   **Authentication**: user/password, or SNC (Secure Network Communication) for certificate-based, encrypted logon
*   **Transport security**: SNC, or an SSH tunnel on untrusted networks
*   **Unicode**: full Unicode support

All of this is configured through the DuckDB `sap_rfc` secret — the same secret used by the [BICS](/docs/erpl/bics.md) and [ODP](/docs/erpl/odp.md) extensions, which ride on the same RFC transport.

#### Performance Considerations

1.  **Connection reuse**: ERPL caches RFC connections per secret — keep work in one session
2.  **Row cap**: bound large reads with `MAX_ROWS`, and tune `THREADS` for your network
3.  **Field selection**: use `COLUMNS => [...]` (or rely on SELECT-list projection push-down) to reduce data transfer
4.  **Filter push-down**: a plain `WHERE` clause is lowered to RFC; use `FILTER` for predicates the optimizer can't push

#### Security Best Practices

1.  **Network Security**: Use VPN or secure network connections
2.  **Authentication**: Use strong passwords or certificates
3.  **Authorization**: Implement proper RFC user roles
4.  **Audit Logging**: Enable RFC audit logs in SAP

## Next Steps

### 🚀 Ready for More?

*   [BICS Protocol Guide](/docs/erpl/bics.md) - Execute SAP BW queries
*   [ODP Protocol Guide](/docs/erpl/odp.md) - Delta replication
*   [Function Reference](/docs/reference/erpl-functions.md) - Complete API docs

### 🔧 Advanced Topics

*   [RFC Metadata Guide](/docs/guides/advanced/rfc-metadata.md) - Discover SAP functions and tables
*   [Performance Tuning](/docs/guides/advanced/performance-tuning.md) - Optimize RFC calls
*   [Real-World Use Cases](/docs/examples/real-world-use-cases.md) - Complete scenarios

### 💡 Examples

*   [ERPL Examples](/docs/examples/erpl-examples.md) - More real-world RFC examples
*   [Integration with Python](/docs/guides/integration/python-pandas.md) - Use RFC data with Pandas

---

**Need help?** Check our [troubleshooting guide](/docs/reference/troubleshooting.md) or browse [more examples](/docs/examples/erpl-examples.md).

# BW/4HANA Modeling

erpl-adt's `bw` command group talks to the SAP BW Modeling REST API. You can search the BW catalog, read InfoProvider structures, walk lineage graphs across DTPs and transformations, and export an entire InfoArea's dataflow as a Mermaid diagram you can commit next to the model.

**Service prerequisite:**

The `/sap/bw/modeling/` endpoints must be active on the SAP system. On a fresh a4h Docker trial they default to inactive — see the [activation steps](https://github.com/datazooDE/erpl-adt/blob/main/CLAUDE.md#bw-modeling-api--activating-on-the-a4h-docker-container) in the erpl-adt repo. The CLI prints an actionable hint when it sees a 404 on a `/sap/bw/modeling/` path.

---

## Search the BW catalog

`bw search` returns objects across ADSO, HCPR, DTPA, TRFN, RSDS, QUERY (and friends) with a `Status` column distinguishing active from inactive.

![erpl-adt bw search](/assets/images/bw-search-89254560048a9825d739f8a66181c81a.gif)

```
erpl-adt bw search 'ZSD_*' --max 8erpl-adt bw search '*REVENUE*' --type QUERYerpl-adt bw search 'DTP_*' --type DTPA --changed-by DEVELOPER
```

Common filters:

| Flag | Filters by |
| --- | --- |
| `--type` | TLOGO code (ADSO, HCPR, DTPA, TRFN, RSDS, QUERY, …) |
| `--subtype` | Object subtype (REP, SOB, RKF, …) |
| `--status` | `ACT`, `INA`, `OFF` |
| `--changed-by`, `--changed-from`, `--changed-to` | Change history |
| `--depends-on-name` / `--depends-on-type` | Reverse lookups |
| `--infoarea` | InfoArea assignment |

Add `--json` for a machine-readable feed:

```
erpl-adt --json bw search 'ZSD_*' | jq '.[] | {name, type, status}'
```

---

## Inspect an ADSO's structure

`bw read-adso` returns the field layout with the InfoObject each field maps to — exactly the resolution you'd otherwise click through in BW Modeling Tools.

![erpl-adt bw read-adso](/assets/images/bw-adso-4a43e16030ec1e928edbaa702523afd7.gif)

```
erpl-adt bw read-adso ZSD_SALES_ORDERerpl-adt bw read-adso ZSD_SALES_ORDER --version m   # show modified version
```

Sibling commands for other object families:

*   `bw read-dtp <name>` — DTP details: source, target, mode, runtime stats
*   `bw read-trfn <name>` — transformation definition
*   `bw read-dmod <name>` — data model topology
*   `bw read-rsds <name>` — DataSource field structure
*   `bw read-query <name>` — query family component (supports `--format mermaid`)
*   `bw read <type> <name>` — generic fallback that works for any TLOGO

---

## Export a dataflow as Mermaid

`bw export-cube` (and its sibling `bw export-area` / `bw export-query`) traverse the full provider graph rooted at one object and emit Mermaid `graph LR` syntax. Pipe it straight into `mmdc` to render, or commit the `.mmd` next to the model for review.

![erpl-adt bw export-cube --mermaid](/assets/images/bw-mermaid-d96241c7771ff20119836b203fa4115e.gif)

```
# Print Mermaid to stdouterpl-adt bw export-cube ZC_SD_SALES_CUBE --mermaid# Write catalog JSON + Mermaid side by sideerpl-adt bw export-cube ZC_SD_SALES_CUBE --mermaid --out-dir build/lineage# OpenMetadata-shaped JSON for ingestion into a catalogerpl-adt bw export-cube ZC_SD_SALES_CUBE \  --shape openmetadata --service-name bw_prod --system-id BWP
```

The emitted diagram groups objects by role — Sources, Staging\[InfoArea\], InfoCubes, MultiProviders, Queries — and draws `provider --> consumer` dataflow edges between them. Add `--iobj-edges` to include InfoObject nodes for dimensions, filters, variables, and key figures.

---

## Lineage as a typed graph

If you'd rather work with a normalized lineage graph than a diagram, `bw lineage <DTP>` returns a typed-node/typed-edge JSON document with provenance:

```
erpl-adt --json bw lineage DTP_SD_O_TO_C | jq '.nodes[] | {id, type, name, role}'
```

The graph schema is documented in [`docs/bw-lineage-contract-v3.md`](https://github.com/datazooDE/erpl-adt/blob/main/docs/bw-lineage-contract-v3.md) in the erpl-adt repo and is intended to round-trip cleanly into OpenMetadata, DataHub, or your own catalog.

---

## Where to next

*   [Command Reference](/docs/erpl-adt/reference.md#bw) — every `bw` subcommand and flag
*   [MCP server](/docs/erpl-adt/mcp.md) — expose these capabilities to AI agents
*   [erpl-adt repo](https://github.com/datazooDE/erpl-adt) — protocol specs in `docs/bw-protocol-spec.md`

# Metadata Catalog

A unified, **cross-domain** metadata catalog — ABAP, DDIC, CDS, and BW objects in one place — persisted in a single **DuckDB** file. Once built, search, lineage, and where-used run in milliseconds instead of round-tripping SAP each time.

The catalog covers:

*   **Hybrid search** — full-text _and_ semantic (Gemini embeddings), so a search for _"procurement spend"_ can hit `0PUR_VALUE` even before anyone remembers the technical name.
*   **End-to-end lineage** — stitched across domains (a CDS view → its DDIC tables → the BW ADSO that loads from it).
*   **A business-glossary overlay** — definitions, owner, line-of-business, and confidentiality layered on top of the technical metadata. Optional, and stored only in the catalog file — it never touches SAP.
*   **Incremental sync** — diffs against what's already stored and writes only the delta, with resume-after-interruption.
*   **A web explorer** — search, browse, lineage, curate, sync status, and feed export, served straight from the `erpl-adt` binary.

The catalog is available three ways: the **CLI** (`erpl-adt catalog …`), the **web UI** (`erpl-adt catalog webui`), and **MCP tools** (`catalog_*`) for AI agents.

---

## Scope: build against a package or InfoArea

Every catalog command needs an **explicit scope** (`--package` / `--infoarea`). There is no "catalog the whole system" default: SAP has no call to enumerate every BW InfoArea, and ABAP/DDIC package search has no pagination, so a silent "everything" default would quietly miss content past the result cap.

Discover packages first with the regular `search` command:

```
erpl-adt search 'Z*' --type DEVC --json   # all custom-namespace packages
```

---

## Build

`erpl-adt catalog build` always builds the feed; add `--db` to persist it, `--format` to render it, both, or neither:

```
# Just a summary — nothing is written anywhereerpl-adt catalog build --sid A4H --package ZMY_PACKAGE --infoarea ZBW_AREA# Persist into a DuckDB file (full rebuild, replaces prior content) —# this is the file that catalog search / annotate / sync / webui all read fromerpl-adt catalog build --sid A4H --package ZMY_PACKAGE --db catalog.duckdb# Also compute embeddings for semantic / hybrid search (needs GEMINI_API_KEY)erpl-adt catalog build --sid A4H --package ZMY_PACKAGE --db catalog.duckdb --embed# Render the feed instead of (or in addition to) persisting iterpl-adt catalog build --sid A4H --package ZMY_PACKAGE --format mermaiderpl-adt catalog build --sid A4H --package ZMY_PACKAGE --format openmetadata
```

`catalog build --db` is a single-shot, all-or-nothing write with no progress output and no resume — fine for a small scope. For anything large (thousands of packages, an hour-plus run), use `catalog sync` instead, **even for the very first build**.

---

## Sync (incremental, resumable)

`catalog sync` diffs against what's already stored and writes only the delta, printing one progress line per item on stderr:

```
erpl-adt catalog sync catalog.duckdb --sid A4H --package ZMY_PACKAGE# [1/1] package ZMY_PACKAGE (elapsed 0m4s, ETA 0s)# Interrupted (connection drop, auth expiry, Ctrl-C)? Everything already synced is# durably committed — pick up exactly where it left off instead of starting over:erpl-adt catalog sync catalog.duckdb --sid A4H --package ZMY_PACKAGE --resume
```

Checkpoint/audit state (which items are done, whether the last attempt was interrupted) lives in the **same DuckDB file** as the catalog data — one artifact, no sidecar file to lose track of.

**Removal detection:**

Deleting entities that disappeared from the scope only runs on a plain, **non-resumed** sync. A resumed run only sees the items it personally processed, not the whole scope, so it skips removal rather than risk flagging a still-valid item as gone.

---

## Curate (optional business overlay)

Technical metadata gives you the _what_ (a table's fields); curation adds the _why_ a human would write — what an entity means, who owns it, how sensitive it is. It never touches SAP; it only writes to the catalog file's overlay columns.

```
# A single entityerpl-adt catalog annotate catalog.duckdb --id <entity_id> \    --definition "Total procurement value" --owner "jane@example.com" --lob Procurement# Bulk, keyed by entity_iderpl-adt catalog annotate catalog.duckdb --file overlay.yaml
```

---

## View the catalog

Three read paths, all served from the same DuckDB cache — no SAP round-trip:

```
# CLI — fast, cache-onlyerpl-adt catalog search catalog.duckdb "procurement value" --mode hybrid# Web UI — search, browse, lineage, curate, sync status, feed exporterpl-adt catalog webui catalog.duckdb --port 8383   # then open http://127.0.0.1:8383/# MCP — catalog_search / catalog_get / catalog_lineage / catalog_where_used / … for AI agentserpl-adt mcp --catalog-db catalog.duckdb          # stdioerpl-adt mcp --catalog-db catalog.duckdb --http   # JSON-RPC over HTTP
```

### The web explorer

![The erpl-adt catalog explorer — search, browse, and lineage over a DuckDB catalog](/assets/images/catalog-explorer-3218a3308fd3315f982f4c44c75f6215.png)

The web UI (a Flutter client compiled and embedded straight into the `erpl-adt` binary) is **read-only against the cache except for curation**. Search, Browse, Entity Detail, Lineage, and Driver Tree all query the same fast `catalog_*` tools the CLI and AI agents use; the **Curate** screen is the only one that writes, via `catalog_annotate`.

There is no build/sync button — `catalog webui` doesn't hold a live SAP connection, so building, exporting, and syncing stay CLI-only operations. **Sync Status** shows past sync runs and cache health, and **Feed Export** surfaces the exact `erpl-adt catalog build --format …` command to run for each format.

Every Explorer view — a search, an entity, a lineage graph, a driver tree — has a stable, copy-pastable URL, so you can share a deep link to exactly what you're looking at.

### Serving it to more than yourself

On the default `127.0.0.1` binding there is nothing to configure. Beyond that, `catalog webui` and `mcp --http` share one access-control model, described in full in the [MCP guide](/docs/erpl-adt/mcp.md#access-control):

```
# reachable on the LAN by IP — works as-is, an IP has no name to rebinderpl-adt catalog webui catalog.duckdb --host 0.0.0.0# reached in a browser by a DNS name — name it, or the browser is refused with 403erpl-adt catalog webui catalog.duckdb --host 0.0.0.0 --allowed-hosts catalog.internal.example# and require a token, since the Curate screen writes through catalog_annotateerpl-adt catalog webui catalog.duckdb --host 0.0.0.0 \    --allowed-hosts catalog.internal.example --auth-token-env ERPL_ADT_WEBUI_TOKEN
```

Binding beyond loopback without a token exposes the catalog API — including those curation writes — to everyone who can reach the port.

---

## Catalog MCP tools

When you start the MCP server with `--catalog-db`, the `catalog_*` tools are available to the agent alongside the live ADT tools:

| MCP tool | What it does |
| --- | --- |
| `catalog_search` | Hybrid (full-text + semantic) search over the cache |
| `catalog_get` | Fetch a single entity by id, with fields and overlay |
| `catalog_lineage` | Upstream/downstream lineage for an entity |
| `catalog_where_used` | Reverse dependencies — what references this entity |
| `catalog_driver_tree` | Expand an entity's driver tree |
| `catalog_object_types` / `catalog_object_subtypes` | Enumerate the type/subtype facets present in the cache |
| `catalog_stats` | Cache size, counts, and coverage |
| `catalog_sync_status` | Last sync runs and cache health |
| `catalog_annotate` | Write a business-overlay annotation (the only writing tool) |
| `catalog_build` / `catalog_export` | Build the feed / export it in a chosen format |

See the [MCP guide](/docs/erpl-adt/mcp.md) for client wiring.

---

## Where to next

*   [MCP Server](/docs/erpl-adt/mcp.md) — wire the `catalog_*` tools into Claude Code, Cursor, or Gemini
*   [Command Reference](/docs/erpl-adt/reference.md) — every `catalog` flag
*   [BW/4HANA Modeling](/docs/erpl-adt/bw.md) — the BW objects the catalog stitches lineage across

# MCP Server

erpl-adt doubles as a [Model Context Protocol](https://modelcontextprotocol.io/) server. Every CLI command is exposed as an MCP tool over JSON-RPC 2.0 on stdin/stdout. Plug it into any MCP-compatible agent — Claude Code, Cursor, Gemini CLI — and the agent decides which tools to call and in what order.

Start the server directly:

```
erpl-adt mcp --host sap.example.com --port 44300 --https
```

The process talks JSON-RPC on stdio, so you don't run this yourself; the MCP client launches it for you.

---

## Client configuration

### Claude Code

Add an entry to your `~/.claude/mcp.json` (or per-project `.claude/mcp.json`):

```
{  "mcpServers": {    "sap": {      "command": "erpl-adt",      "args": ["mcp", "--host", "sap.example.com", "--port", "44300", "--https"],      "env": { "SAP_PASSWORD": "..." }    }  }}
```

Restart Claude Code and the `sap` tools appear in the tool list.

### Cursor

`Settings → MCP → + Add new MCP server` and paste the same shape as above.

### Gemini CLI

Add to `~/.config/gemini/config.yaml` under `mcp_servers:`:

```
mcp_servers:  sap:    command: erpl-adt    args: ["mcp", "--host", "sap.example.com", "--port", "44300", "--https"]    env:      SAP_PASSWORD: "${SAP_PASSWORD}"
```

---

## Tools the agent gets

The MCP tool names are derived from the CLI command paths. The most-used ones:

Tool names are verb-first: `adt_read_source`, not `adt_source_read`.

| MCP tool | CLI equivalent | What it does |
| --- | --- | --- |
| `adt_search` | `search` | Search ABAP objects by pattern and type |
| `adt_read_object` | `object read` | Read an object's metadata (URI, includes, source URIs) |
| `adt_read_source` | `source read` | Read source for a class, program, or include |
| `adt_write_source` | `source write` | Write source (auto-lock, transport, optional activate) |
| `adt_activate` | `activate` | Activate an inactive object |
| `adt_run_tests` | `test run` | Run ABAP Unit tests |
| `adt_run_atc` | `check run` | Run ATC quality checks |
| `adt_check_syntax` | `source check` | Syntax-check an object |
| `adt_run_class` | `object run` | Execute a class implementing `IF_OO_ADT_CLASSRUN` |
| `adt_create_transport` / `adt_list_transports` / `adt_release_transport` | `transport …` | Transport lifecycle |
| `adt_read_table` | `ddic table` | Inspect a transparent table with check tables and ABAP types resolved |
| `adt_read_cds` | `ddic cds` | Read a CDS view's source |
| `adt_list_package` / `adt_package_tree` / `adt_package_exists` | `package …` | Browse package contents |
| `adt_lock` / `adt_unlock` | `object lock` / `object unlock` | Explicit lock handling for multi-step edits |
| `adt_create_object` / `adt_delete_object` | `object create` / `object delete` | Object lifecycle |
| `adt_discover` | `discover` | Discover available ADT services |
| `bw_search`, `bw_read_adso`, `bw_lineage_graph`, `bw_read_dataflow`, … | `bw …` | Full BW/4HANA toolkit, 43 tools (see [BW page](/docs/erpl-adt/bw.md)) |
| `catalog_search`, `catalog_get`, `catalog_lineage`, `catalog_where_used`, … | `catalog …` | Metadata-catalog tools, 12 in total — available when started with `--catalog-db` (see [Catalog page](/docs/erpl-adt/catalog.md)) |

77 tools are registered in total — 22 `adt_*`, 43 `bw_*` and 12 `catalog_*`. Call the standard MCP `tools/list` method from your client to print the current set, with each tool's title, annotations and output schema. Pass `--tools adt,bw` (or any subset) to register only the families you want, which keeps the agent's prompt smaller.

The `catalog_*` family is only exposed when you start the server against a pre-built catalog file:

```
erpl-adt mcp --catalog-db catalog.duckdb --host sap.example.com --port 44300 --https
```

These answer from the DuckDB cache in milliseconds without touching SAP. See the [Catalog guide](/docs/erpl-adt/catalog.md) for building the file.

---

## Example agent loop

A typical session inside Claude Code looks like this — three CLI commands the agent picks itself, from one natural-language question.

> **You:** What flight-related classes exist in this SAP system?

Claude calls `adt_search` with pattern `ZCL_FLIGHT*`:

```
Found 4 matching objects:  Name              Type      Package       Description  ────────────────  ────────  ────────────  ──────────────────────────  ZCL_FLIGHT_CTRL   CLAS/OC   ZFLIGHT_APP   Flight booking controller  ZCL_FLIGHT_MODEL  CLAS/OC   ZFLIGHT_APP   Flight data model  ZCL_FLIGHT_TEST   CLAS/OC   ZFLIGHT_APP   Flight module unit tests  ZCL_FLIGHT_API    CLAS/OC   ZFLIGHT_APP   REST API wrapper
```

> **You:** Show me the booking controller and tell me what's broken.

Claude calls `adt_read_object` → `adt_read_source` → `adt_run_tests` and writes back a diagnosis. The end-to-end narrative is in [Your AI Coding Agent Just Learned ABAP](/blog/introducing-erpl-adt) and the follow-ups [Real-Time SAP for AI, Part 1](/blog/real-time-sap-for-ai-part-1) and [Part 2](/blog/real-time-sap-for-ai-part-2).

---

## HTTP transport

Most clients speak stdio, which is the default and needs none of this. `--http` serves the same tool contract as JSON-RPC over HTTP at `POST /mcp` — one implementation, two transports — for clients that cannot spawn a process.

```
erpl-adt mcp --http                                     # 127.0.0.1:8383erpl-adt mcp --http --mcp-host 0.0.0.0 --mcp-port 9000
```

| Flag | Effect |
| --- | --- |
| `--mcp-host <addr>` | Address to bind (default: `127.0.0.1`) |
| `--mcp-port <n>` | Port to bind (default: `8383`) |
| `--cors-origin <list>` | Extra browser origins allowed to call `/mcp`. `*` allows every origin |
| `--allowed-hosts <list>` | `Host` header values this server answers to. `*` allows every host |
| `--auth-token <tok>` | Require `Authorization: Bearer <tok>`; requests without it get 401 and run nothing |
| `--auth-token-env <var>` | Read that token from an environment variable |
| `-c, --config <path>` | YAML file whose `http:` block supplies any of the above |

### Access control

The tools behind this endpoint write to a live SAP system — `adt_write_source`, `adt_delete_object`, `adt_activate`, `adt_release_transport` — so who may call it is checked, and binding to `127.0.0.1` is not by itself a boundary: your browser runs inside it.

**Origin.** Requests with no `Origin` header (curl, native MCP clients) are allowed, as are same-origin and loopback origins. Any other browser origin is refused with **403** unless named with `--cors-origin`. That stops a page you happen to visit from posting writes to your SAP system.

**Host.** Origin checking alone cannot see DNS rebinding: a page that points its own hostname at `127.0.0.1` arrives with a `Host` it controls and an `Origin` to match, so both sides of the same-origin comparison belong to the attacker. The `Host` header is the half they cannot launder, so it is checked separately. Loopback names, IP literals and the address bound are always served; **a browser asking for any other host is refused with 403**.

```
# a browser reaching this server by a DNS name needs it allowederpl-adt mcp --http --mcp-host 0.0.0.0 --allowed-hosts mcp.internal.example# or, for containers and unit files, the same thing as an env varexport ERPL_ADT_ALLOWED_HOSTS=mcp.internal.exampleerpl-adt mcp --http --mcp-host 0.0.0.0
```

```
# erpl.yaml, passed with -c — also read by `catalog webui`http:  allowed_hosts: [mcp.internal.example, buildbox.corp]  cors_origin: [https://catalog.example]  # the variable's name — a raw auth_token key is deliberately not read  auth_token_env: ERPL_ADT_MCP_TOKEN
```

An **IP literal is always allowed** — an IP address has no DNS name to rebind — so `--mcp-host 0.0.0.0` reached at `http://192.168.1.5:8383` needs no configuration. Non-browser clients (no `Origin`, no `Sec-Fetch-*`) are unaffected whatever hostname they use, because rebinding is a browser attack by construction.

**Authentication** is off unless a token is configured. Binding beyond loopback without one warns on stderr; `/healthz` never requires the token, so liveness probes keep working.

Each server prints its posture at startup, so what is enforced is visible where the refusals appear:

```
erpl-adt MCP HTTP server listening on http://127.0.0.1:8383/mcp  hosts:   loopback, IP literals (add with --allowed-hosts)  origins: same-origin, loopback (add with --cors-origin)  auth:    none (--auth-token to require a bearer token)
```

---

## Authentication

Credentials are resolved in this order: CLI flags → `--password-env` env var → `~/.adt.creds` (saved by `erpl-adt login`) → the `SAP_PASSWORD` environment variable.

For MCP usage, the `env` block in the client config is the cleanest path — the agent never sees the password, the password never appears in shell history, and you can rotate it without touching the client config.

This is SAP authentication. Authenticating callers of the **HTTP transport** is a separate control — see ([Access control](#access-control)) above.

---

## Where to next

*   [Catalog guide](/docs/erpl-adt/catalog.md) — build a DuckDB metadata catalog and expose the `catalog_*` tools
*   [BW guide](/docs/erpl-adt/bw.md) — the BW modeling tools available over MCP
*   [Command Reference](/docs/erpl-adt/reference.md) — every operation, also reachable via MCP
*   [Announcement post](/blog/introducing-erpl-adt) — full context, demo recording, and motivation

# Command Reference

Run `erpl-adt --help` for the live listing; this page documents the same surface offline. The CLI is organised as `erpl-adt [global-flags] <group> <action> [args]`. Every command accepts `--json` for machine-readable output.

---

## SEARCH

```
search <pattern>                        Search for ABAP objects    --type <type>                       Object type: CLAS, PROG, TABL, INTF, FUGR, DTEL, …    --max <n>                           Maximum number of results
```

---

## OBJECT

Read, create, delete, lock/unlock ABAP objects.

```
object create                           Create an ABAP object    --type <type>                       Object type (required, e.g. CLAS/OC, PROG/P)    --name <name>                       Object name (required)    --package <pkg>                     Target package (required)    --description <text>                Object description    --transport <id>                    Transport request numberobject delete <uri>                     Delete an ABAP object    --handle <handle>                   Lock handle (skips auto-lock if provided)    --transport <id>                    Transport request numberobject lock <uri>                       Lock an object for editing    --session-file <path>               Save session for later unlockobject read <name-or-uri>               Read object structureobject run <class-name-or-uri>          Run an ABAP console class (IF_OO_ADT_CLASSRUN)object unlock <uri>                     Unlock an object    --handle <handle>                   Lock handle (required)    --session-file <path>               Session file for stateful workflow
```

---

## SOURCE

Read, write, and check ABAP source code.

```
source check <name-or-uri>              Check syntaxsource edit <name-or-uri>               Open in $EDITOR and write back    --type <type>                       Object type for name resolution    --section <section>                 main | localdefinitions | localimplementations | testclasses    --transport <id>                    Transport request number    --activate                          Activate after writing    --no-write                          Open editor but do not write backsource read <name-or-uri>               Read source code    --version <version>                 active (default) | inactive    --section <section>                 main (default) | localdefinitions | localimplementations | testclasses | all    --type <type>                       Disambiguate name resolution    --color / --no-color                ANSI syntax highlighting    --editor                            Open in $VISUAL/$EDITOR (plain text)source write <name-or-uri>              Write source code    --file <path>                       Path to local source file (required)    --section <section>                 main (default) | localdefinitions | localimplementations | testclasses    --handle <handle>                   Lock handle (skips auto-lock if provided)    --transport <id>                    Transport request number    --session-file <path>               Session file for stateful workflow    --activate                          Activate after writing    --optimistic                        Try lockless write first (pre-7.51 SAP)
```

---

## ACTIVATE

```
activate <name-or-uri>                  Activate an inactive ABAP object
```

---

## TEST / CHECK

```
test <name-or-uri>                      Run ABAP Unit testscheck <name-or-uri>                     Run ATC quality checks    --variant <name>                    ATC variant (default: DEFAULT)
```

---

## TRANSPORT

```
transport create                        Create a transport    --desc <text>                       Transport description (required)    --package <pkg>                     Target package (required)transport list                          List transports    --user <user>                       Filter by user (default: DEVELOPER)transport release <number>              Release a transport
```

---

## DATA DICTIONARY

```
ddic cds <name>                         Get CDS sourceddic table <name>                       Get table definition (fetches abap_type + check_table by default)    --no-resolve-types                  Skip data-element lookup; show field names and types only (fast)    --raw                               Print raw SAP XML response
```

---

## PACKAGE

```
package exists <name>                   Check if package existspackage list <name>                     List package contents (one level)package tree <name>                     List package contents recursively (BFS)    --type <type>                       Filter: CLAS, PROG, TABL, INTF, FUGR, …    --max-depth <n>                     Maximum recursion depth (default: 50)
```

---

## DISCOVER

```
discover services                       Discover ADT services    --workspace <name>                  Filter by workspace (e.g. "Object Repository")
```

---

## CATALOG

Unified, cross-domain metadata catalog (ABAP + DDIC + CDS + BW) persisted in a single DuckDB file. See the [Catalog guide](/docs/erpl-adt/catalog.md) for the full walkthrough. Every command needs an explicit scope (`--package` / `--infoarea`).

```
catalog build                           Build a unified cross-domain catalog (optionally persist/render it)    --sid <sid>                         SAP system id (labels the catalog)    --package <name>                    Scope: an ABAP/DDIC package (repeatable)    --infoarea <name>                   Scope: a BW InfoArea (repeatable)    --db <path>                         Persist into a DuckDB file (full rebuild)    --embed                             Also compute embeddings for semantic search (needs GEMINI_API_KEY)    --format <catalog|mermaid|openmetadata>   Render the feed in this shapecatalog sync <db>                       Incrementally sync a scope into an existing DuckDB file    --sid <sid>                         SAP system id    --package <name> / --infoarea <name>   Scope (repeatable)    --resume                            Resume an interrupted sync (skips removal detection)catalog search <db> <query>             Search a built catalog (cache-only, no SAP call)    --mode <fts|semantic|hybrid>        Search mode (default: hybrid)catalog annotate <db>                   Curate the business-overlay fields (never touches SAP)    --id <entity_id>                    Entity to annotate    --definition <text> / --owner <who> / --lob <name>   Overlay fields    --file <overlay.yaml>               Bulk annotate, keyed by entity_idcatalog webui <db>                      Serve the embedded web catalog client + API (blocking)    --port <n>                          Listen port (default: 8383)    --host <addr>                       Address to bind (default: 127.0.0.1)    --cors-origin <list>                Extra browser origins allowed to call the API    --allowed-hosts <list>              Host headers this server answers to beyond loopback and                                        IP literals; a browser using any other is refused    --auth-token <tok>                  Require 'Authorization: Bearer <tok>' on the API    --auth-token-env <var>              Read that token from an environment variable    -c, --config <path>                 YAML file whose http: block supplies these settings
```

`catalog webui` and `mcp --http` share one access-control model — see the [MCP guide](/docs/erpl-adt/mcp.md#access-control).

---

## BW

SAP BW/4HANA Modeling operations. See the [BW guide](/docs/erpl-adt/bw.md) for usage. Each subcommand below is documented as a top-level command; flags are listed exactly as `erpl-adt bw <action> --help` prints them.

### bw activate

Activate BW objects.

```
bw activate <type> <name>    --validate                          Pre-check only, don't activate    --simulate                          Dry run of activation    --background                        Run as background job    --force                             Force activation even with warnings    --exec-check                        Set execChk=true in activation payload    --with-cto                          Set withCTO=true in activation payload    --sort                              Validate mode: sort dependency order    --only-ina                          Validate mode: only inactive objects    --transport <corrnr>                Transport request
```

### bw adturi

Show BW-to-ADT URI mappings.

```
bw adturi
```

### bw applog

Read BW repository application logs.

```
bw applog    --username <user>                   Filter by user    --start <timestamp>                 Filter by start timestamp    --end <timestamp>                   Filter by end timestamp
```

### bw changeability

Show BW changeability settings.

```
bw changeability
```

### bw create

Create a BW object.

```
bw create <type> <name>    --package <pkg>                     Target package    --copy-from-name <name>             Copy source object name    --copy-from-type <type>             Copy source object type    --file <path>                       Optional XML payload file for create request body
```

### bw datavolumes

Read BW data volumes.

```
bw datavolumes    --infoprovider <name>               InfoProvider filter    --max <n>                           Max rows
```

### bw dbinfo

Show HANA database info.

```
bw dbinfo
```

### bw delete

Delete a BW object.

```
bw delete <type> <name>    --lock-handle <handle>              Lock handle (required)    --transport <corrnr>                Transport request number    --transport-lock-holder <corrnr>    Explicit Transport-Lock-Holder header    --foreign-objects <value>           Foreign-Objects header    --foreign-object-locks <value>      Foreign-Object-Locks header    --foreign-correction-number <corrnr>  Foreign-Correction-Number header    --foreign-package <pkg>             Foreign-Package header
```

### bw discover

Discover BW modeling services.

```
bw discover
```

### bw export-area

Export all objects in a BW InfoArea to JSON or Mermaid.

```
bw export-area <infoarea>    --mermaid                           Output Mermaid dataflow diagram instead of JSON    --shape <catalog|openmetadata>      JSON output shape: catalog (default) or openmetadata    --max-depth <n>                     Max recursion depth for nested infoareas (default: 10)    --types <T1,T2,...>                 Comma-separated TLOGO type filter (e.g. ADSO,DTPA). Default: all    --no-lineage                        Skip DTP lineage graph collection (faster)    --no-queries                        Skip query graph collection    --no-search                         Skip search supplement, use BFS tree only (faster)    --no-xref-edges                     Skip xref-based INFOPROVIDER→QUERY edge collection    --no-elem-edges                     Skip orphan ELEM XML parsing for provider edge recovery    --iobj-edges                        Show InfoObject nodes (dimensions, filters, variables) in Mermaid    --version <a|m>                     Object version: a (active, default) or m (modified)    --out-dir <dir>                     Save {name}_catalog.json and {name}_dataflow.mmd to directory    --service-name <name>               Service name for openmetadata FQN (default: erpl_adt)    --system-id <id>                    System ID for openmetadata FQN prefix
```

### bw export-cube

Export a single BW InfoProvider and its connected graph to JSON or Mermaid.

```
bw export-cube <cube-name>    --mermaid                           Output Mermaid dataflow diagram instead of JSON    --shape <catalog|openmetadata>      JSON output shape: catalog (default) or openmetadata    --no-lineage                        Skip DTP lineage graph collection (faster)    --no-xref-edges                     Skip xref-based edge collection (faster, fewer API calls)    --no-elem-edges                     Skip orphan ELEM XML parsing for provider edge recovery    --iobj-edges                        Show InfoObject nodes (dimensions, filters, variables) in Mermaid    --version <a|m>                     Object version: a (active, default) or m (modified)    --out-dir <dir>                     Save {name}_catalog.json and {name}_dataflow.mmd to directory    --service-name <name>               Service name for openmetadata FQN (default: erpl_adt)    --system-id <id>                    System ID for openmetadata FQN prefix
```

### bw export-query

Export a single BW query and its connected graph to JSON or Mermaid.

```
bw export-query <query-name>    --mermaid                           Output Mermaid dataflow diagram instead of JSON    --shape <catalog|openmetadata>      JSON output shape: catalog (default) or openmetadata    --no-lineage                        Skip DTP lineage graph collection (faster)    --no-queries                        Skip query graph collection    --no-xref-edges                     Skip xref-based edge collection (faster, fewer API calls)    --no-elem-edges                     Skip orphan ELEM XML parsing for provider edge recovery    --iobj-edges                        Show InfoObject nodes (dimensions, filters, variables) in Mermaid    --version <a|m>                     Object version: a (active, default) or m (modified)    --out-dir <dir>                     Save {name}_catalog.json and {name}_dataflow.mmd to directory    --service-name <name>               Service name for openmetadata FQN (default: erpl_adt)    --system-id <id>                    System ID for openmetadata FQN prefix
```

### bw favorites

List or clear BW backend favorites.

```
bw favorites
```

### bw job

BW background job operations.

```
bw job <action>
```

### bw lineage

Build a canonical BW lineage graph rooted at a DTP. Returns typed nodes, typed edges, and per-call provenance. JSON output is intended to round-trip into OpenMetadata, DataHub, or a custom catalog.

```
bw lineage <dtp_name>    --trfn <name>                       Optional explicit transformation name    --version <v>                       Version: a (active, default), m (modified), d (delivery)    --max-xref <n>                      Maximum xref neighbors to include (default: 100)    --no-xref                           Disable xref expansion for a strict DTP/TRFN graph
```

### bw lock

Lock a BW object for editing. Stateful — pair with `bw save` and `bw unlock`.

```
bw lock <type> <name>    --activity <code>                   Activity: CHAN (default), DELE, MAIN    --parent-name <name>                Parent object name (lock context)    --parent-type <type>                Parent object type (lock context)    --transport-lock-holder <corrnr>    Explicit Transport-Lock-Holder header    --foreign-objects <value>           Foreign-Objects header    --foreign-object-locks <value>      Foreign-Object-Locks header    --foreign-correction-number <corrnr>  Foreign-Correction-Number header    --foreign-package <pkg>             Foreign-Package header    --session-file <path>               Save session state for multi-step workflow
```

### bw locks

Monitor and break BW object locks.

```
bw locks <list|delete>    --user <name>                       Filter / specify lock owner user    --search <pattern>                  Search pattern for list    --max <n>                           Maximum results (default: 100)    --table-name <name>                 Table name from list (for delete)    --arg <base64>                      Encoded arg from list (for delete)    --mode <code>                       Lock mode, default: E (for delete)    --scope <n>                         Lock scope, default: 1 (for delete)    --owner1 <base64>                   Owner1 from list (for delete)    --owner2 <base64>                   Owner2 from list (for delete)
```

### bw message

Resolve a BW message text.

```
bw message <identifier> <textype>    --msgv1 <value>                     Message variable 1    --msgv2 <value>                     Message variable 2    --msgv3 <value>                     Message variable 3    --msgv4 <value>                     Message variable 4
```

### bw move

List BW move requests.

```
bw move
```

### bw nodepath

Resolve a BW object node path.

```
bw nodepath    --object-uri <uri>                  BW object URI (e.g. /sap/bw/modeling/adso/…) (required)
```

### bw nodes

Show BW object node structure.

```
bw nodes <type> <name>    --datasource                        Use DataSource structure path instead of InfoProvider    --child-name <name>                 Filter by child name    --child-type <type>                 Filter by child type
```

### bw qprops

Read BW query properties rules.

```
bw qprops
```

### bw read

Generic BW object reader. Use the family-specific readers (`bw read-adso`, `bw read-dtp`, …) when you want a parsed view.

```
bw read <type> <name>    --version <v>                       Version: a (active, default), m (modified), d (delivery)    --source-system <name>              Source system (required for RSDS, APCO)    --uri <path>                        Direct URI from search results (overrides type/name path)    --raw                               Output raw XML
```

### bw read-adso

Read BW ADSO field structure.

```
bw read-adso <name>    --version <v>                       Version: a (active, default), m (modified), d (delivery)
```

### bw read-dmod

Read BW DMOD topology.

```
bw read-dmod <name>    --version <v>                       Version: a (active, default), m (modified), d (delivery)
```

### bw read-dtp

Read BW DTP connection details (source, target, mode, runtime settings).

```
bw read-dtp <name>    --version <v>                       Version: a (active, default), m (modified), d (delivery)
```

### bw read-query

Read a BW query-family component. Defaults to a Mermaid-rendered graph; switch to `--format table` or `--json` for structured output.

```
bw read-query <name>    --version <v>                       Version: a (active, default), m (modified), d (delivery)    --format <f>                        Non-JSON output format: mermaid (default) or table    --layout <l>                        Mermaid layout: detailed (default) or compact    --direction <d>                     Mermaid direction: TD (default) or LR    --max-nodes-per-role <n>            Reduce fan-out: keep at most n nodes per role; add summary nodes    --focus-role <r>                    Limit reduction to a specific role (rows|columns|free|filter|member|subcomponent|component)    --json-shape <s>                    JSON output shape: legacy (default), catalog (flat), or truth (lineage v3)    --upstream <m>                      Upstream resolution mode: explicit (default) or auto    --upstream-dtp <name>               Compose query graph with upstream BW lineage rooted at DTP    --upstream-no-xref                  Disable xref expansion for upstream lineage composition    --upstream-max-xref <n>             Maximum xref neighbors in upstream lineage composition (default: 100)    --lineage-max-steps <n>             Maximum auto-upstream planner expansion steps (default: 4)    --lineage-strict                    Fail when auto-upstream resolution is ambiguous or incomplete    --lineage-explain                   Emit upstream resolution warnings/decisions for troubleshooting
```

### bw read-rsds

Read BW RSDS field structure.

```
bw read-rsds <name>    --source-system <logsys>            Source system (required, e.g. ECLCLNT100)    --version <v>                       Version: a (active, default), m (modified), d (delivery)
```

### bw read-trfn

Read BW transformation definition.

```
bw read-trfn <name>    --version <v>                       Version: a (active, default), m (modified), d (delivery)
```

### bw reporting

Read BW reporting metadata.

```
bw reporting <compid>    --dbgmode                           Set dbgmode=true query parameter    --metadata-only                     MetadataOnly header    --incl-metadata                     InclMetadata header    --incl-object-values                InclObjectValues header    --incl-except-def                   InclExceptDef header    --compact-mode                      CompactMode header    --from-row <n>                      FromRow header    --to-row <n>                        ToRow header
```

### bw save

Save a modified BW object. Pair with `bw lock` (to acquire the handle) and `bw unlock` (after save).

```
bw save <type> <name>    --lock-handle <handle>              Lock handle from bw lock (required)    --transport <corrnr>                Transport request number    --timestamp <ts>                    Server timestamp from lock response    --transport-lock-holder <corrnr>    Explicit Transport-Lock-Holder header    --foreign-objects <value>           Foreign-Objects header    --foreign-object-locks <value>      Foreign-Object-Locks header    --foreign-correction-number <corrnr>  Foreign-Correction-Number header    --foreign-package <pkg>             Foreign-Package header
```

### bw search

Search BW objects across the catalog. Supports rich metadata filtering by type, subtype, status, change history, InfoArea assignment, and reverse dependency lookup.

```
bw search <pattern>    --type <code>                       Object type (ADSO, HCPR, IOBJ, TRFN, DTPA, RSDS, …)    --subtype <code>                    Object subtype (REP, SOB, RKF, …)    --max <n>                           Maximum results (default: 100)    --status <code>                     Object status: ACT, INA, OFF    --changed-by <user>                 Last changed by filter    --changed-from <date>               Changed on or after date    --changed-to <date>                 Changed on or before date    --created-by <user>                 Created by filter    --created-from <date>               Created on or after date    --created-to <date>                 Created on or before date    --depends-on-name <name>            Filter by dependency object name    --depends-on-type <type>            Filter by dependency object type    --infoarea <name>                   Filter by InfoArea assignment (e.g. 0D_NW_DEMO)    --search-desc                       Also search in descriptions    --search-name                       Search in names (default: true)
```

### bw search-md

Show BW search metadata.

```
bw search-md
```

### bw sysinfo

Show BW system properties.

```
bw sysinfo
```

### bw transport

Transport lifecycle for BW objects.

```
bw transport <check|write|list|collect>    --transport <corrnr>                Transport number (for write/collect)    --package <pkg>                     Package name (for write)    --own-only                          Show only own transport requests    --rddetails <mode>                  Check/list detail mode: off|objs|all (default: all)    --rdprops                           Check/list include properties section    --allmsgs                           Include all messages where supported    --simulate                          Dry run (write only)    --mode <code>                       Collection mode (e.g. 000, 001, 002, 003, 004, 005, 033)    --transport-lock-holder <corrnr>    Explicit Transport-Lock-Holder header    --foreign-objects <value>           Foreign-Objects header    --foreign-object-locks <value>      Foreign-Object-Locks header    --foreign-correction-number <corrnr>  Foreign-Correction-Number header    --foreign-package <pkg>             Foreign-Package header
```

### bw unlock

Release a BW object lock.

```
bw unlock <type> <name>
```

### bw validate

Validate (lint) a BW object.

```
bw validate <type> <name>    --action <name>                     Validation action name (default: validate)
```

### bw valuehelp

BW value-help lookup.

```
bw valuehelp <domain>    --query <qs>                        Raw query string (k=v&k2=v2)    --max <n>                           Max rows    --pattern <text>                    Pattern filter    --type <code>                       Object type filter    --infoprovider <name>               InfoProvider filter
```

### bw virtualfolders

Read BW virtual folders.

```
bw virtualfolders    --package <pkg>                     Package filter    --type <type>                       Object type filter    --user <user>                       User filter
```

### bw xref

Show BW cross-references.

```
bw xref <type> <name>    --version <v>                       Object version: A (active), M (modified)    --association <code>                Filter by association code (001, 002, 003, …)    --assoc-type <type>                 Filter by associated object type (IOBJ, ADSO, …)    --max <n>                           Maximum number of results to return
```

---

## MCP SERVER

```
mcp                                     Start MCP server (JSON-RPC over stdio)    --catalog-db <path>                 Also expose the catalog_* tools over a DuckDB cache    --tools <families>                  Expose only these tool families (adt, bw, catalog;                                        comma-separated). Default: all registered families  Transport    --http                              Serve JSON-RPC over HTTP at POST /mcp instead of stdio    --mcp-host <addr>                   Address to bind with --http (default: 127.0.0.1)    --mcp-port <n>                      Port to bind with --http (default: 8383)  HTTP access control (--http only)    --cors-origin <list>                Extra browser origins allowed to call /mcp. Same-origin,                                        loopback and non-browser requests are always allowed;                                        anything else is refused with 403. '*' allows every origin    --allowed-hosts <list>              Host header values this server answers to, beyond loopback,                                        IP literals and the bound address. A browser request for any                                        other Host is refused with 403 — that is what DNS rebinding                                        looks like. Non-browser clients are unaffected.                                        '*' allows every Host    --auth-token <tok>                  Require 'Authorization: Bearer <tok>' on /mcp    --auth-token-env <var>              Read that token from an environment variable    -c, --config <path>                 YAML file whose http: block supplies these settings
```

`--allowed-hosts`, `--cors-origin` and `--auth-token` are also settable as `ERPL_ADT_ALLOWED_HOSTS`, `ERPL_ADT_CORS_ORIGIN` and `ERPL_ADT_AUTH_TOKEN` (the `SAP_*` prefix works too). Precedence: flag > environment > config file.

See the [MCP guide](/docs/erpl-adt/mcp.md) for client wiring, the access-control model, and the full tool catalogue.

---

## CREDENTIALS

```
login                                   Save connection credentials (~/.adt.creds, chmod 600)logout                                  Remove saved credentials
```

---

## GLOBAL FLAGS

```
--host <host>                           SAP hostname (default: localhost)--port <port>                           SAP port (default: 50000)--user <user>                           SAP username (default: DEVELOPER)--password <pass>                       SAP password--password-env <var>                    Read password from env var (default: SAP_PASSWORD)--client <num>                          SAP client (default: 001)--language <iso>                        SAP logon language (ISO, e.g. EN, DE; default: EN)--https                                 Use HTTPS--insecure                              Skip TLS verification--json                                  Machine-readable JSON output--timeout <sec>                         Per-request read timeout in seconds (default: 600)--session-file <path>                   Persist session for lock/write/unlock workflows--color / --no-color                    ANSI colored output (auto-detects TTY)-v                                      Verbose logging (INFO level)-vv                                     Debug logging (DEBUG level)
```

Credential resolution: explicit `--password` flag > `--password-env` > `.adt.creds` (via `login`) > `SAP_PASSWORD` env var.

Every connection setting can also come from the environment under two spellings, project prefix first: `ERPL_ADT_HOST` / `SAP_HOST`, `ERPL_ADT_PASSWORD` / `SAP_PASSWORD`, and so on.

---

## EXIT CODES

| Code | Meaning |
| --- | --- |
| `0` | Success |
| `1` | Connection / authentication error |
| `2` | Not found (object, package, etc.) |
| `3` | Clone error (deploy workflow) |
| `4` | Pull error (deploy workflow) |
| `5` | Activation error |
| `6` | Lock conflict |
| `7` | Test failure |
| `8` | ATC check error |
| `9` | Transport error |
| `10` | Timeout |
| `99` | Internal error |

---

Tip: `erpl-adt <group> <action> --help` prints command-specific examples and the latest flag listing — useful when a release adds something this page hasn't picked up yet.

# The Segment Dictionary

Typed mode needs to know each segment's field layout (name, offset, length, type). That **segment dictionary** is just a **relation** with these columns:

```
idoctyp, cimtyp, release, segnam, segdef, field_pos, field_name, offset, length, datatype, ...
```

Its origin is irrelevant to the parser — a file, a table, a view, or a query all work. The readers that take a `dict` argument ([`sap_idoc_read_segment`, `sap_idoc_read_fields`](/docs/erpl-idoc/functions.md), and the converters) accept any of them.

## Option A — Offline / hand-authored

Write the fields (order + width) and let `sap_idoc_dict_offsets` compute the offsets, then check it with `sap_idoc_dict_validate`:

```
-- compute field offsets from lengthsSELECT * FROM sap_idoc_dict_offsets('mytype.fields.csv');-- list structural problems — an empty result means the dictionary is soundSELECT * FROM sap_idoc_dict_validate('mytype.dict.csv');
```

`sap_idoc_dict_validate` catches bad offsets, overlaps, and missing fields early — run it before you rely on a hand-authored dictionary.

## Option B — Online, from a live system

Requires [`erpl_rfc`](/docs/erpl/rfc.md) loaded. The extension ships two SQL macros so it's a one-liner:

```
LOAD erpl_rfc;LOAD erpl_idoc;CREATE SECRET sap (  TYPE sap_rfc, ASHOST '…', SYSNR '00', CLIENT '100', USER '…', PASSWD '…');-- fetch + normalize the dictionary for a basic typeSELECT * FROM sap_idoc_dictionary(sap_idoc_params('ORDERS05'));
```

### Persist once → reuse forever offline

This is the **connected → detached bridge**: fetch the dictionary once on a machine that can reach SAP, persist it to Parquet, then decode IDocs anywhere with **no SAP and no `erpl_rfc`**.

```
COPY (SELECT * FROM sap_idoc_dictionary(sap_idoc_params('ORDERS05')))     TO 'orders.dict.parquet' (FORMAT parquet);
```

```
-- later, on a SAP-less host — only erpl_idoc + the dictionary fileSELECT * FROM sap_idoc_read_segment('doc.idoc', 'E1BPSBONEW', 'orders.dict.parquet');
```

## Normalizing a raw field list

If you already have a raw `IDOCTYPE_READ_COMPLETE` field list, normalize it to the dictionary schema:

```
SELECT * FROM sap_idoc_dict_from_fields(fields, 'ORDERS05', '', '740');
```

## See also

*   [Quick Start](/docs/erpl-idoc/quickstart.md) — typed decode end-to-end.
*   [Function reference](/docs/erpl-idoc/functions.md#dictionary-tooling) — the dictionary tooling functions.
*   [`erpl_rfc`](/docs/erpl/rfc.md) — the RFC extension used for the online dictionary fetch.

# ERPL-IDoc Function Reference

Every function is self-documenting — run `SELECT * FROM duckdb_functions() WHERE function_name LIKE 'sap_idoc_%'` to see a description and an example for each.

## Reading

| Function | What you get |
| --- | --- |
| `sap_idoc_read(path [, framing, lenient, encoding])` | generic long rows: `document_key, docnum, segnum, segnam, psgnum, hlevel, mandt, sdata` |
| `sap_idoc_read_control(path [, …])` | the control record — all 36 `EDI_DC40` fields, typed (flat **or** XML) |
| `sap_idoc_read_segment(path, segnam, dict [, …])` | typed columns for one segment type, sliced from `SDATA` per the dictionary |
| `sap_idoc_read_fields(path, dict [, …])` | **every field of every record** in one call — long rows: `document_key, segnum, psgnum, hlevel, segnam, field_pos, field_name, datatype, value` |
| `sap_idoc_read_raw(path [, …])` | one row per physical record with exact bytes — the byte-exact writer source |
| `sap_idoc_read_xml(path)` | generic long rows from an IDoc-XML file (self-describing; no dictionary) |

Every reader accepts a **single path, a glob, or a `LIST` of paths**, resolved through DuckDB's virtual filesystem — so a whole directory works, including remote stores (`s3://…`, `http(s)://…`, `gs://…`) once the matching extension is loaded (`INSTALL httpfs; LOAD httpfs;`) and a `CREATE SECRET` is set for credentials:

```
SELECT * FROM sap_idoc_read(['a.idoc', 'b.idoc']);          -- explicit list
```

```
SELECT filename, idoctypFROM sap_idoc_read_control('s3://bucket/idocs/*.idoc', filename => true);
```

### Reader parameters

All readers accept:

*   `framing` = `'fixed'` (default) | `'lf'` | `'crlf'` — auto-detected when omitted.
*   `lenient := true` — salvage complete records from a truncated file.
*   `encoding` = `'utf-8'` (default) | `'latin-1'`.
*   `filename := true` — add a source-file column (handy across a glob).

`sap_idoc_read_fields` also takes `include_unknown := false` to drop segments absent from the dictionary (default keeps them as one row with the raw trimmed `SDATA`).

**Streaming & parallel:**

The readers are **streaming and parallel**: each file is parsed record-by-record in constant memory (never fully buffered), and a glob/`LIST` is read with one thread per file. Rows are therefore **unordered across files** (order within a file is preserved) — add `ORDER BY` if you need a stable order, exactly as with `read_csv`/`read_parquet`.

## Writing

```
COPY (<single BLOB/VARCHAR column of raw records>)  TO 'file.idoc' (FORMAT sap_idoc [, framing 'fixed'|'lf'|'crlf', validate true]);
```

Build the records with the pure encoders when composing from scratch:

| Encoder | Produces |
| --- | --- |
| `sap_idoc_encode_sdata(offsets, lengths, values)` | a 1000-byte `SDATA` payload |
| `sap_idoc_encode_data_record(segnam, mandt, docnum, segnum, psgnum, hlevel, sdata)` | a 1063-byte `EDI_DD40` record |
| `sap_idoc_encode_control(values)` | a 524-byte `EDI_DC40` control record |

## Converting (flat ⇄ XML)

| Function | Direction |
| --- | --- |
| `sap_idoc_to_xml(flat_path, dict)` | flat → IDoc-XML text |
| `sap_idoc_xml_to_records(xml_path, dict)` | IDoc-XML → flat records (for `COPY … (FORMAT sap_idoc)`) |

## Dictionary tooling

| Function | Purpose |
| --- | --- |
| `sap_idoc_dict_offsets(src)` | compute field offsets from lengths (author a dict from field order + width) |
| `sap_idoc_dict_validate(src)` | list structural problems; empty = sound |
| `sap_idoc_dict_from_fields(fields, idoctyp, cimtyp, release)` | normalize a raw `IDOCTYPE_READ_COMPLETE` field list to the dictionary schema |

See [The segment dictionary](/docs/erpl-idoc/dictionary.md) for the dictionary schema and how to author or fetch one.

# ERPL-IDoc Quick Start

In this 5-minute tutorial you'll read an IDoc file, decode its opaque `SDATA` into typed columns, write a byte-valid IDoc back, and convert flat ⇄ IDoc-XML — **without a SAP connection**.

**Prerequisites:**

*   DuckDB installed (version 1.5.4+)
*   One or more IDoc files (flat or IDoc-XML) — no SAP system needed
*   For _typed_ decode: a [segment dictionary](/docs/erpl-idoc/dictionary.md) (a CSV/Parquet file, table, or view). Generic and XML reads need no dictionary.

## Step 1: Install ERPL-IDoc

```
INSTALL erpl_idoc FROM community;LOAD erpl_idoc;
```

**Success check:**

If installation worked you'll see no errors. `erpl_idoc` links no SAP libraries and makes no network calls — it's a pure, offline file engine.

## Step 2: Read an IDoc file

```
-- one row per data record: segment name, hierarchy, and the raw SDATA payloadSELECT segnam, hlevel, sdataFROM sap_idoc_read('orders.idoc');-- the envelope (control record) as 36 typed columnsSELECT idoctyp, mestyp, sndprn, rcvprn, credatFROM sap_idoc_read_control('orders.idoc');
```

Every reader accepts a **single path, a glob, or a `LIST` of paths**, resolved through DuckDB's virtual filesystem — so a whole directory works, including remote stores (`s3://…`, `http(s)://…`, `gs://…`) once `httpfs` is loaded and a `CREATE SECRET` is set.

```
SELECT filename, idoctypFROM sap_idoc_read_control('s3://bucket/idocs/*.idoc', filename => true);
```

## Step 3: Decode SDATA into typed columns

`SDATA` is opaque fixed-width until you apply a **segment dictionary**. Point at one and get named, typed columns:

```
SELECT airlineid, flightdate, customerid, class, passnameFROM sap_idoc_read_segment('booking.idoc', 'E1BPSBONEW', 'flightbooking.dict.parquet');-- LH | 20260715 | 00000042 | Y | MUELLER
```

Decode a whole IDoc — all segments, all fields — in one call:

```
SELECT segnam, field_name, valueFROM sap_idoc_read_fields('order.idoc', 'order_dict.csv');
```

See [The segment dictionary](/docs/erpl-idoc/dictionary.md) for where a dictionary comes from (hand-authored offline, or fetched from a live system with `erpl_rfc`).

## Step 4: Generate an IDoc file from SQL

Compose records from your data and write a byte-valid IDoc:

```
COPY (  SELECT raw_record  FROM sap_idoc_read_raw('template.idoc')   -- or build records with the encoders  ORDER BY record_index) TO 'outbound.idoc' (FORMAT sap_idoc);
```

The writer recomputes derived fields (`SEGNUM`, `PSGNUM`, `HLEVEL`, lengths) for you. `sap_idoc_read_raw → COPY (FORMAT sap_idoc)` reproduces the input **byte-for-byte**.

## Step 5: Convert flat ⇄ IDoc-XML

```
-- flat → self-describing XMLSELECT xml FROM sap_idoc_to_xml('orders.idoc', 'orders.dict.parquet');-- XML → flat (write it out)COPY (  SELECT raw_record  FROM sap_idoc_xml_to_records('orders.xml', 'orders.dict.parquet')  ORDER BY record_index) TO 'orders.idoc' (FORMAT sap_idoc);
```

`flat → xml → flat` is byte-exact with the dictionary.

## Next steps

*   [Function reference](/docs/erpl-idoc/functions.md) — every reader, writer, encoder, and converter, with parameters.
*   [The segment dictionary](/docs/erpl-idoc/dictionary.md) — author one offline or fetch it from a live system.
*   [ERPL-IDoc overview](/docs/erpl-idoc.md) — use cases, scope, and round-trip guarantees.

# SAP Datasphere

ERPL-Web is the **first native DuckDB client for SAP Datasphere**. Discover, explore, and query your Datasphere data directly from SQL with full OAuth2 security.

## Overview

SAP Datasphere (formerly Data Warehouse Cloud) is SAP's cloud data platform. ERPL-Web integrates with both the DWAAS Core APIs and the Catalog OData service to provide:

**What you get:**

*   ✅ OAuth2 authentication (authorization code & client credentials)
*   ✅ Discover spaces and assets automatically
*   ✅ Rich metadata for relational and analytical datasets
*   ✅ Query relational data with full SQL pushdown
*   ✅ Query analytical data with metrics and dimensions
*   ✅ Secure secret management
*   ✅ Automatic token refresh

---

## Prerequisites

Before you start, you'll need:

1.  **SAP Datasphere Tenant** - Active Datasphere instance
2.  **OAuth2 App Registration** - Client ID and secret (or use pre-delivered credentials)
3.  **Permissions** - Access to spaces and views you want to query
4.  **Network Access** - Ability to reach your Datasphere tenant

**Tip:**

For quick testing, ERPL-Web can use Datasphere's pre-delivered OAuth2 client. You only need your tenant name and data center!

---

## Quick Start

### Step 1: Create OAuth2 Secret

```
-- Load the extensionLOAD erpl_web;-- Create OAuth2 secret (minimal configuration)CREATE SECRET datasphere (  TYPE datasphere,  PROVIDER oauth2,  tenant_name 'your-tenant',  data_center 'eu10',  scope 'default');
```

This opens your browser for authentication. After you log in, the tokens are stored securely.

### Step 2: Discover Your Spaces

```
-- List all accessible spacesSELECT * FROM datasphere_show_spaces();
```

**Example Output:**

```
┌──────────────┐│     name     │├──────────────┤│ SALES        ││ FINANCE      ││ SUPPLY_CHAIN │└──────────────┘
```

### Step 3: Discover Assets in a Space

```
-- List assets in SALES spaceSELECT name, object_type, technical_nameFROM datasphere_show_assets('SALES');
```

**Example Output:**

```
┌─────────────────────┬──────────────┬────────────────────┐│        name         │ object_type  │  technical_name    │├─────────────────────┼──────────────┼────────────────────┤│ Sales Analytics     │ View         │ SALES_ANALYTICS_V  ││ Customer Master     │ View         │ CUSTOMER_MASTER_V  ││ Revenue by Region   │ View         │ REVENUE_REGION_V   │└─────────────────────┴──────────────┴────────────────────┘
```

### Step 4: Query Your Data

```
-- Read relational dataSELECT * FROM datasphere_read_relational('SALES', 'CUSTOMER_MASTER_V')LIMIT 10;
```

Done! You're now querying SAP Datasphere from DuckDB.

---

## OAuth2 Authentication

ERPL-Web supports multiple OAuth2 flows for different use cases.

### Authorization Code Flow (Interactive)

Best for **interactive use** - opens browser for login:

```
CREATE SECRET datasphere (  TYPE datasphere,  PROVIDER oauth2,  tenant_name 'mytenant',  data_center 'eu10',  scope 'default',  REDIRECT_URI 'http://localhost:65000'  -- Default);
```

**What happens:**

1.  A local server starts on port 65000
2.  Browser opens to Datasphere login
3.  You authenticate with your credentials
4.  Tokens are saved in the secret
5.  Tokens auto-refresh when expired

### Client Credentials Flow (Service Accounts)

Best for **automation and scripts** - no browser needed:

```
CREATE SECRET datasphere_svc (  TYPE datasphere,  PROVIDER oauth2,  tenant_name 'mytenant',  data_center 'eu10',  client_id 'your-client-id',  client_secret 'your-client-secret',  GRANT_TYPE 'client_credentials',  scope 'default');
```

**Warning:**

Keep client\_secret secure! Never commit it to version control. Use environment variables or secure vaults.

### Using Pre-Delivered Client

If you don't have custom OAuth2 app, use Datasphere's built-in client:

```
CREATE SECRET datasphere (  TYPE datasphere,  PROVIDER oauth2,  tenant_name 'mytenant',  data_center 'eu10'  -- No client_id or client_secret needed!);
```

### Data Centers

Specify your Datasphere data center:

| Data Center | Location |
| --- | --- |
| `eu10` | Europe (Frankfurt) |
| `us10` | US East |
| `ap10` | Asia Pacific (Sydney) |
| `jp10` | Japan (Tokyo) |

---

## Discovery Functions

ERPL-Web provides comprehensive discovery to explore your Datasphere environment.

### datasphere\_show\_spaces

List all accessible spaces.

**Signature:**

```
datasphere_show_spaces([secret VARCHAR])
```

**Returns:** Table with column:

*   `name` (VARCHAR): Space ID

**Example:**

```
SELECT * FROM datasphere_show_spaces();
```

---

### datasphere\_show\_assets

List assets in a space or across all spaces.

**Signatures:**

```
-- Assets in specific spacedatasphere_show_assets(space_id VARCHAR [, secret VARCHAR])-- Assets across all spacesdatasphere_show_assets([secret VARCHAR])
```

**Returns:** Table with columns:

*   `name` (VARCHAR): Asset name (label)
*   `object_type` (VARCHAR): Type (View, Table, etc.)
*   `technical_name` (VARCHAR): Technical identifier
*   `space_name` (VARCHAR): Space ID (only when querying all spaces)

**Examples:**

```
-- Assets in one spaceSELECT * FROM datasphere_show_assets('SALES');-- All accessible assetsSELECT * FROM datasphere_show_assets();-- Filter by typeSELECT * FROM datasphere_show_assets('SALES')WHERE object_type = 'View';
```

---

### datasphere\_describe\_space

Get detailed space metadata.

**Signature:**

```
datasphere_describe_space(space_id VARCHAR [, secret VARCHAR])
```

**Returns:** Table with columns:

*   `name` (VARCHAR): Space ID
*   `label` (VARCHAR): Display label

**Example:**

```
SELECT * FROM datasphere_describe_space('SALES');
```

---

### datasphere\_describe\_asset

Get comprehensive asset metadata including schema details.

**Signature:**

```
datasphere_describe_asset(  space_id VARCHAR,   asset_id VARCHAR   [, secret VARCHAR])
```

**Returns:** Table with 15 columns:

**Basic Metadata:**

*   `name` (VARCHAR): Technical name
*   `space_name` (VARCHAR): Space ID
*   `label` (VARCHAR): Display label
*   `asset_type` (VARCHAR): Asset type

**Access URLs:**

*   `asset_relational_metadata_url` (VARCHAR)
*   `asset_relational_data_url` (VARCHAR)
*   `asset_analytical_metadata_url` (VARCHAR)
*   `asset_analytical_data_url` (VARCHAR)

**Capabilities:**

*   `supports_analytical_queries` (BOOLEAN)
*   `has_relational_access` (BOOLEAN)
*   `has_analytical_access` (BOOLEAN)

**Schema Information:**

*   `relational_schema` (STRUCT): Relational columns definition
*   `analytical_schema` (STRUCT): Metrics, dimensions, variables
*   `odata_context` (VARCHAR)
*   `odata_metadata_etag` (VARCHAR)

**Example:**

```
SELECT   name,  label,  supports_analytical_queries,  relational_schema,  analytical_schemaFROM datasphere_describe_asset('SALES', 'REVENUE_REGION_V');
```

---

## Reading Data

### datasphere\_read\_relational

Query relational (table-like) data.

**Signature:**

```
datasphere_read_relational(  space_id VARCHAR,  asset_id VARCHAR  [, secret VARCHAR]  [, top BIGINT]  [, skip BIGINT]  [, params MAP(VARCHAR, VARCHAR)])
```

**Parameters:**

*   `space_id`: Space identifier
*   `asset_id`: Asset technical name
*   `secret`: Optional secret name (uses default if not specified)
*   `top`: Limit number of rows (OData $top)
*   `skip`: Skip rows (OData $skip)
*   `params`: Input parameters for parameterized views

**Example:**

```
-- Read all dataSELECT * FROM datasphere_read_relational('SALES', 'CUSTOMER_MASTER_V');-- With limitSELECT * FROM datasphere_read_relational('SALES', 'CUSTOMER_MASTER_V', top := 100);-- With paginationSELECT * FROM datasphere_read_relational(  'SALES',   'CUSTOMER_MASTER_V',  top := 50,  skip := 100);-- With parametersSELECT * FROM datasphere_read_relational(  'SALES',  'PARAMETERIZED_VIEW',  params := {'YEAR': '2024', 'REGION': 'EMEA'});
```

---

### datasphere\_read\_analytical

Query analytical (multidimensional) data.

**Signature:**

```
datasphere_read_analytical(  space_id VARCHAR,  asset_id VARCHAR  [, secret VARCHAR]  [, top BIGINT]  [, skip BIGINT]  [, params MAP(VARCHAR, VARCHAR)]  [, metrics LIST(VARCHAR)]  [, dimensions LIST(VARCHAR)])
```

**Parameters:**

*   `space_id`: Space identifier
*   `asset_id`: Asset technical name
*   `secret`: Optional secret name
*   `top`: Limit rows
*   `skip`: Skip rows
*   `params`: Input parameters
*   `metrics`: List of measures to retrieve
*   `dimensions`: List of dimensions to retrieve

**Tip:**

The `metrics` and `dimensions` parameters automatically generate an optimal OData $select clause for analytical queries.

**Examples:**

```
-- Read all metrics and dimensionsSELECT * FROM datasphere_read_analytical('SALES', 'REVENUE_ANALYTICS');-- Select specific metrics and dimensionsSELECT * FROM datasphere_read_analytical(  'SALES',  'REVENUE_ANALYTICS',  metrics := ['TotalRevenue', 'TotalCost', 'Profit'],  dimensions := ['Region', 'ProductCategory', 'Year']);-- With parameters and limitSELECT * FROM datasphere_read_analytical(  'SALES',  'REVENUE_ANALYTICS',  params := {'FISCAL_YEAR': '2024'},  metrics := ['TotalRevenue'],  dimensions := ['Quarter', 'Region'],  top := 100);
```

---

## Complete Workflow Example

Here's a full end-to-end workflow:

```
-- 1. SetupLOAD erpl_web;CREATE SECRET datasphere (  TYPE datasphere,  PROVIDER oauth2,  tenant_name 'acme-corp',  data_center 'eu10');-- 2. Discover spacesSELECT * FROM datasphere_show_spaces();-- Result: SALES, FINANCE, HR-- 3. Explore SALES spaceSELECT name, object_type, technical_nameFROM datasphere_show_assets('SALES')ORDER BY name;-- 4. Get metadata for specific assetSELECT   label,  supports_analytical_queries,  has_relational_access,  has_analytical_accessFROM datasphere_describe_asset('SALES', 'Q1_REVENUE_VIEW');-- 5. Query relational dataCREATE TABLE local_customers ASSELECT * FROM datasphere_read_relational('SALES', 'CUSTOMER_MASTER')WHERE country_code = 'US';-- 6. Query analytical dataCREATE TABLE revenue_by_region ASSELECT *FROM datasphere_read_analytical(  'SALES',  'REVENUE_ANALYTICS',  metrics := ['Revenue', 'Margin'],  dimensions := ['Region', 'Quarter']);-- 7. Analyze locally with DuckDBSELECT   Region,  SUM(Revenue) as total_revenue,  AVG(Margin) as avg_marginFROM revenue_by_regionGROUP BY RegionORDER BY total_revenue DESC;
```

---

## Advanced Features

### Named Parameters for Parameterized Views

Some Datasphere views require input parameters:

```
SELECT *FROM datasphere_read_relational(  'SALES',  'MONTHLY_SALES_VIEW',  params := {    'P_YEAR': '2024',    'P_MONTH': '03',    'P_REGION': 'EMEA'  });
```

### Snake\_Case Column Names

ERPL-Web converts column names to snake\_case for consistency:

```
-- Datasphere column: TotalRevenue-- DuckDB column: total_revenueSELECT total_revenue, product_categoryFROM datasphere_read_analytical('SALES', 'REVENUE_VIEW');
```

### Combine with DuckDB Features

```
-- Export to ParquetCOPY (  SELECT * FROM datasphere_read_relational('SALES', 'CUSTOMERS')) TO 'customers.parquet' (FORMAT PARQUET);-- Join Datasphere data with local dataSELECT   c.customer_name,  l.local_dataFROM datasphere_read_relational('SALES', 'CUSTOMERS') cJOIN local_table l ON c.customer_id = l.id;-- Window functionsSELECT   region,  revenue,  AVG(revenue) OVER (PARTITION BY region) as avg_regional_revenueFROM datasphere_read_analytical('SALES', 'REVENUE_VIEW');
```

---

## Secret Management

### Using Config Files

Store credentials in a config file:

```
# ~/.datasphere/config.ini[datasphere]tenant_name=acme-corpdata_center=eu10client_id=abc123client_secret=secret456
```

```
CREATE SECRET datasphere_cfg (  TYPE datasphere,  PROVIDER config,  CONFIG_FILE '/home/user/.datasphere/config.ini');
```

### Using File Provider

Store credentials in JSON:

```
{  "tenant_name": "acme-corp",  "data_center": "eu10",  "access_token": "...",  "refresh_token": "..."}
```

```
CREATE SECRET datasphere_file (  TYPE datasphere,  PROVIDER file,  FILEPATH '/secure/path/datasphere-creds.json');
```

### Multiple Secrets for Multiple Tenants

```
-- Production tenantCREATE SECRET datasphere_prod (  TYPE datasphere,  PROVIDER oauth2,  tenant_name 'acme-prod',  data_center 'eu10');-- Development tenantCREATE SECRET datasphere_dev (  TYPE datasphere,  PROVIDER oauth2,  tenant_name 'acme-dev',  data_center 'eu10');-- Use specific secretSELECT * FROM datasphere_read_relational(  'SALES',   'CUSTOMERS',  secret := 'datasphere_prod');
```

---

## Performance Tips

### 1\. Use Column Selection

Select only columns you need for better performance:

```
-- Good: Specific columnsSELECT customer_id, customer_name, countryFROM datasphere_read_relational('SALES', 'CUSTOMERS');-- Less efficient: All columnsSELECT * FROM datasphere_read_relational('SALES', 'CUSTOMERS');
```

### 2\. Use top Parameter for Sampling

```
-- Quick sample for explorationSELECT * FROM datasphere_read_relational('SALES', 'LARGE_VIEW', top := 100);
```

### 3\. Filter in DuckDB After Loading

OData filters can be limited. Sometimes it's faster to load data and filter locally:

```
-- Load once, filter multiple timesCREATE TABLE customer_cache ASSELECT * FROM datasphere_read_relational('SALES', 'CUSTOMERS');-- Fast local filteringSELECT * FROM customer_cache WHERE country = 'US';SELECT * FROM customer_cache WHERE revenue > 100000;
```

### 4\. For Analytical Queries, Specify Metrics/Dimensions

```
-- Efficient: Only requested columns are transferredSELECT *FROM datasphere_read_analytical(  'SALES',  'REVENUE_VIEW',  metrics := ['Revenue'],  dimensions := ['Region']);
```

---

## Troubleshooting

### OAuth2 Browser Not Opening

**Issue:** Browser doesn't open for authentication

**Solution:**

```
-- Check redirect_uri and portCREATE SECRET datasphere (  TYPE datasphere,  PROVIDER oauth2,  tenant_name 'mytenant',  data_center 'eu10',  REDIRECT_URI 'http://localhost:65000'  -- Try different port if blocked);
```

### Token Expired

**Issue:** `401 Unauthorized` after some time

**Solution:** Tokens are auto-refreshed. If refresh fails, recreate the secret:

```
DROP SECRET datasphere;CREATE SECRET datasphere (...);  -- Re-authenticate
```

### Asset Not Found

**Issue:** `404 Not Found` when querying asset

**Solution:**

```
-- Verify asset existsSELECT * FROM datasphere_show_assets('SPACE_NAME');-- Check exact technical_name (case-sensitive)SELECT technical_name FROM datasphere_show_assets('SPACE_NAME')WHERE name LIKE '%search_term%';
```

### Permission Denied

**Issue:** `403 Forbidden`

**Solution:**

*   Verify your user has access to the space
*   Check space permissions in Datasphere UI
*   Ensure OAuth2 scopes include necessary permissions

### Enable Tracing

For detailed diagnostics:

```
SET erpl_trace_enabled = TRUE;SET erpl_trace_level = 'DEBUG';-- Run your querySELECT * FROM datasphere_show_spaces();-- Check trace logs for details
```

See [Tracing & Diagnostics](/docs/erpl-web/tracing.md) for more information.

---

## Integration Patterns

### ETL to Data Lake

```
-- Extract Datasphere data to ParquetCOPY (  SELECT * FROM datasphere_read_relational('SALES', 'TRANSACTIONS')  WHERE transaction_date >= '2024-01-01') TO 's3://datalake/sales/transactions.parquet' (FORMAT PARQUET);
```

### Federated Analytics

```
-- Combine Datasphere with other sourcesSELECT   ds.customer_id,  ds.customer_name,  pg.order_count,  sf.crm_scoreFROM datasphere_read_relational('SALES', 'CUSTOMERS') dsLEFT JOIN postgres_customers pg ON ds.customer_id = pg.idLEFT JOIN snowflake_crm sf ON ds.customer_id = sf.customer_id;
```

### Incremental Sync

```
-- Track last sync timeCREATE TABLE sync_metadata (  table_name VARCHAR,  last_sync_time TIMESTAMP);-- Initial loadINSERT INTO sync_metadata VALUES ('CUSTOMERS', CURRENT_TIMESTAMP);-- Incremental updates (if Datasphere view supports it)INSERT INTO local_customersSELECT * FROM datasphere_read_relational(  'SALES',  'CUSTOMERS',  params := {'LAST_MODIFIED': (    SELECT last_sync_time FROM sync_metadata WHERE table_name = 'CUSTOMERS'  )::VARCHAR});UPDATE sync_metadata SET last_sync_time = CURRENT_TIMESTAMP WHERE table_name = 'CUSTOMERS';
```

---

## Next Steps

*   Learn about [OData integration](/docs/erpl-web/odata.md) for general OData services
*   Explore [Secrets Management](/docs/erpl-web/secrets.md) for security best practices
*   See [Tracing & Diagnostics](/docs/erpl-web/tracing.md) for debugging
*   Check [Examples](/docs/examples/erpl-web-examples.md) for more patterns

---

## Summary

ERPL-Web's Datasphere integration provides:

✅ **Native Integration** - First DuckDB client for Datasphere  
✅ **Secure** - Full OAuth2 with automatic token refresh  
✅ **Discoverable** - Explore spaces and assets easily  
✅ **Powerful** - Query relational and analytical data  
✅ **Flexible** - Combine with all DuckDB features

Connect your SAP Datasphere to DuckDB analytics today!

# functions

# HTTP Functions

ERPL-Web turns DuckDB into a powerful HTTP client. Call any REST API, query web services, or integrate with external systems directly from SQL.

## Overview

The HTTP functions let you make web requests and process responses as regular table rows. Think of every API endpoint as a queryable table.

**What you can do:**

*   ✅ Call REST APIs with GET, POST, PUT, PATCH, DELETE
*   ✅ Send custom headers and authentication
*   ✅ Handle JSON, XML, or plain text responses
*   ✅ Set timeouts and retry logic
*   ✅ Process API responses with SQL

## Quick Start

```
-- Load the extensionLOAD erpl_web;-- Make your first API callSELECT contentFROM http_get('https://httpbun.com/ip');-- Parse JSON responseSELECT content::JSON->>'ip' AS ip_addressFROM http_get('https://httpbun.com/ip');
```

That's it! You just called an API from SQL.

---

## HTTP Functions Reference

### Common named parameters

Every HTTP function accepts these named parameters:

| Parameter | Type | Description |
| --- | --- | --- |
| `headers` | MAP(VARCHAR, VARCHAR) | Custom request headers, e.g. `{'X-API-Key': 'k'}` |
| `accept` | VARCHAR | `Accept` header — e.g. `'application/json'` |
| `content_type` | VARCHAR | `Content-Type` header (mutating verbs only — overload 2; the JSON overload sets it automatically) |
| `auth` | VARCHAR | Credential string (`'user:pass'` for BASIC, raw token for BEARER) |
| `auth_type` | ENUM | `'BASIC'` or `'BEARER'` |
| `timeout` | INTEGER | Request timeout in milliseconds |
| `url_encode` | BOOLEAN | Auto-URL-encode the URL (default: depends on URL contents) |

**Returns** (every HTTP function): a single-row table with columns

| Column | Type | Description |
| --- | --- | --- |
| `method` | VARCHAR | HTTP method used |
| `status` | INTEGER | HTTP status code |
| `url` | VARCHAR | Final request URL after redirects |
| `headers` | MAP(VARCHAR, VARCHAR) | Response headers |
| `content_type` | VARCHAR | Response content type |
| `content` | VARCHAR | Response body |

---

### http\_get / http\_head

Fetch a URL. `http_head` returns only headers.

**Signature:**

```
http_get(url VARCHAR)http_head(url VARCHAR)
```

**Example:**

```
SELECT status, contentFROM http_get('https://api.github.com/repos/duckdb/duckdb');-- With named parametersSELECT status, contentFROM http_get(  'https://api.example.com/data',  accept => 'application/json',  auth => 'token-xyz',  auth_type => 'BEARER',  timeout => 10000);
```

---

### http\_post / http\_put / http\_patch / http\_delete

All four mutating verbs share the same shape: **two overloads**.

**Overload 1 — JSON body:**

```
http_post(url VARCHAR, body JSON)http_put(url VARCHAR, body JSON)http_patch(url VARCHAR, body JSON)http_delete(url VARCHAR, body JSON)
```

`Content-Type` is fixed to `application/json` on this overload.

**Overload 2 — arbitrary body with explicit content type:**

```
http_post(url VARCHAR, body VARCHAR, content_type VARCHAR)http_put(url VARCHAR, body VARCHAR, content_type VARCHAR)http_patch(url VARCHAR, body VARCHAR, content_type VARCHAR)http_delete(url VARCHAR, body VARCHAR, content_type VARCHAR)
```

Use this for form-encoded, XML, plain text, or any non-JSON payload.

**Example — overload 1 (JSON):**

```
SELECT status, contentFROM http_post(  'https://httpbin.org/post',  '{"name": "DuckDB", "type": "database"}'::JSON);
```

**Example — overload 2 (form data):**

```
SELECT status, contentFROM http_post(  'https://api.example.com/forms',  'name=John&email=john@example.com',  'application/x-www-form-urlencoded');
```

**Example — `http_delete` with a JSON body** (some APIs require it):

```
SELECT statusFROM http_delete(  'https://api.example.com/items/bulk',  '{"ids": [1, 2, 3]}'::JSON);
```

Both overloads accept the ([common named parameters](#common-named-parameters)) above.

---

## Authentication

ERPL-Web supports multiple authentication methods.

### Basic Authentication

Use username and password:

```
SELECT contentFROM http_get(  'https://api.example.com/data',  auth := 'username:password',  auth_type := 'BASIC');
```

### Bearer Token

Use an API token:

```
SELECT contentFROM http_get(  'https://api.example.com/data',  auth := 'your-api-token-here',  auth_type := 'BEARER');
```

### Custom Headers

For other authentication schemes, use headers:

```
SELECT contentFROM http_get(  'https://api.example.com/data',  headers := {'X-API-Key': 'your-api-key'});
```

### Using DuckDB Secrets

Store credentials securely with DuckDB secrets:

```
-- Create a secretCREATE SECRET api_token (  TYPE http_bearer,  token 'your-secret-token');-- Use it in requests (automatically applied)SELECT contentFROM http_get('https://api.example.com/data');
```

**Tip:**

Always use secrets for production deployments. Never hardcode credentials in SQL queries!

---

## Advanced Examples

### POST JSON with Headers

Cast the body to `JSON` to pick the JSON overload (content type is fixed to `application/json`):

```
SELECT status, contentFROM http_post(  'https://api.example.com/webhooks',  '{"event": "user.created", "user_id": 123}'::JSON,  headers => {    'X-Webhook-Secret': 'secret123',    'X-Request-ID': 'req-456'  });
```

### Form Data Submission

Use the 3-positional overload to set a custom content type:

```
SELECT status, contentFROM http_post(  'https://api.example.com/forms',  'name=John&email=john@example.com',  'application/x-www-form-urlencoded');
```

### API Pagination

Query paginated APIs using LATERAL joins:

```
-- Fetch first pageWITH first_page AS (  SELECT content::JSON AS data  FROM http_get('https://api.example.com/users?page=1'))SELECT data->>'name' AS name, data->>'email' AS emailFROM first_page, LATERAL UNNEST(data->'users') AS data;
```

### Handle Timeouts

Set custom timeout for slow APIs:

```
SELECT status, contentFROM http_get(  'https://slow-api.example.com/data',  timeout := 30000  -- 30 seconds);
```

### Error Handling

Check status codes and handle errors:

```
WITH api_response AS (  SELECT status, content  FROM http_get('https://api.example.com/data'))SELECT   CASE     WHEN status = 200 THEN content::JSON    WHEN status = 404 THEN '{"error": "Not found"}'::JSON    ELSE '{"error": "Request failed"}'::JSON  END AS resultFROM api_response;
```

---

## Working with JSON APIs

Most modern APIs return JSON. Here's how to work with it:

### Parse JSON Response

```
SELECT   content::JSON->>'name' AS name,  content::JSON->>'email' AS email,  content::JSON->'address'->>'city' AS cityFROM http_get('https://api.example.com/user/123');
```

### Extract Arrays

```
SELECT   item->>'id' AS id,  item->>'title' AS titleFROM http_get('https://api.example.com/items'),LATERAL UNNEST(content::JSON->'items') AS item;
```

### Build API Requests from Data

```
-- Create POST requests from a tableCREATE TABLE users (name VARCHAR, email VARCHAR);INSERT INTO users VALUES ('Alice', 'alice@example.com'), ('Bob', 'bob@example.com');SELECT   name,  response.statusFROM users,LATERAL (  SELECT status   FROM http_post(    'https://api.example.com/users',    json_object('name', name, 'email', email)::JSON  )) AS response;
```

---

## Integration Patterns

### ETL from API to Parquet

```
-- Extract data from API and save to ParquetCOPY (  SELECT     data->>'id' AS id,    data->>'name' AS name,    data->>'created_at' AS created_at  FROM http_get('https://api.example.com/export'),  LATERAL UNNEST(content::JSON->'records') AS data) TO 'output.parquet' (FORMAT PARQUET);
```

### API Gateway Pattern

```
-- Create a view that wraps an APICREATE VIEW github_duckdb_info ASSELECT   content::JSON->>'name' AS name,  content::JSON->>'stargazers_count' AS stars,  content::JSON->>'forks_count' AS forksFROM http_get('https://api.github.com/repos/duckdb/duckdb');-- Query it like a normal tableSELECT * FROM github_duckdb_info;
```

### Webhook Testing

```
-- Send a test webhookSELECT   status,  content::JSON->>'message' AS response_messageFROM http_post(  'https://your-webhook-url.com/endpoint',  json_object(    'event', 'test',    'timestamp', current_timestamp,    'data', json_object('test', true)  )::JSON);
```

---

## Best Practices

### Performance

1.  **Cache responses** - Store API results in tables for repeated queries
2.  **Use LIMIT** - Don't fetch more data than you need
3.  **Batch requests** - Combine multiple operations when possible
4.  **Set timeouts** - Prevent hanging on slow APIs

```
-- Good: Cache API resultsCREATE TABLE api_cache ASSELECT content::JSON AS dataFROM http_get('https://api.example.com/large-dataset');-- Query the cacheSELECT * FROM api_cache;
```

### Security

1.  **Use secrets** - Never hardcode credentials
2.  **HTTPS only** - Avoid HTTP for sensitive data
3.  **Validate responses** - Check status codes
4.  **Rate limiting** - Respect API limits

```
-- Good: Use secretsCREATE SECRET my_api (  TYPE http_bearer,  token 'secret-token');-- Bad: Hardcoded token-- SELECT * FROM http_get('url', auth := 'secret-token');
```

### Error Handling

Always check HTTP status codes:

```
WITH response AS (  SELECT status, content  FROM http_get('https://api.example.com/data'))SELECT   CASE     WHEN status BETWEEN 200 AND 299 THEN 'Success'    WHEN status BETWEEN 400 AND 499 THEN 'Client Error'    WHEN status BETWEEN 500 AND 599 THEN 'Server Error'    ELSE 'Unknown'  END AS result_type,  contentFROM response;
```

---

## Troubleshooting

### Connection Refused

```
-- Error: Connection refused-- Solution: Check URL and network connectivitySELECT * FROM http_get('http://localhost:8080');
```

**Fixes:**

*   Verify the URL is correct
*   Check if the service is running
*   Ensure firewall rules allow connections

### Timeout Errors

```
-- Error: Request timeout-- Solution: Increase timeoutSELECT * FROM http_get(  'https://slow-api.example.com',  timeout := 60000  -- 60 seconds);
```

### SSL Certificate Errors

```
-- Error: SSL certificate verification failed-- For development only, consider using HTTP instead of HTTPS-- Or ensure SSL certificates are properly configured
```

### JSON Parsing Errors

```
-- Error: Invalid JSON-- Solution: Check content_type and validate JSONSELECT   status,  content,  content_type,  TRY_CAST(content AS JSON) AS parsed_jsonFROM http_get('https://api.example.com/data');
```

---

## Next Steps

*   Learn about [OData integration](/docs/erpl-web/odata.md) for structured APIs
*   Explore [Secrets Management](/docs/erpl-web/secrets.md) for secure authentication
*   See [Tracing & Diagnostics](/docs/erpl-web/tracing.md) for debugging
*   Check [Examples](/docs/examples/erpl-web-examples.md) for real-world use cases

---

## Summary

The HTTP functions in ERPL-Web transform DuckDB into a powerful API client:

✅ **Simple** - Query APIs like database tables  
✅ **Flexible** - Support for all HTTP methods and authentication  
✅ **Powerful** - Combine with SQL for data transformation  
✅ **Secure** - Built-in secrets management

Start making HTTP requests from SQL today!

# OData V2/V4

ERPL-Web provides a universal OData reader that works seamlessly with both OData V2 and V4 services. Query any OData API like it's a native DuckDB table.

## Overview

OData (Open Data Protocol) is a standard for building and consuming RESTful APIs. ERPL-Web makes OData services feel like regular databases.

**What you get:**

*   ✅ Automatic version detection (V2 or V4)
*   ✅ ATTACH services as databases
*   ✅ Predicate pushdown ($filter, $select, $top, $skip)
*   ✅ Expand navigation properties
*   ✅ Type mapping from EDM to DuckDB
*   ✅ Automatic pagination handling

---

## Quick Start

```
-- Load the extensionLOAD erpl_web;-- Attach an OData serviceATTACH 'https://services.odata.org/TripPinRESTierService'   AS trippin (TYPE odata);-- Query it like a normal tableSELECT UserName, FirstName, LastName FROM trippin.People WHERE FirstName = 'Russell';
```

That's it! The OData service is now a queryable database.

---

## Two Ways to Use OData

### 1\. ATTACH as Database (Recommended)

Attach the service once, query any entity set:

```
-- Attach OData V4 serviceATTACH 'https://services.odata.org/TripPinRESTierService'   AS trippin (TYPE odata);-- List available tablesSHOW TABLES;-- Query any entitySELECT * FROM trippin.People LIMIT 10;SELECT * FROM trippin.Airlines;SELECT * FROM trippin.Airports;
```

### 2\. Direct Read with odata\_read

Query a specific entity set directly:

```
SELECT UserName, FirstName, LastNameFROM odata_read('https://services.odata.org/TripPinRESTierService/People')WHERE UserName = 'russellwhyte';
```

**Tip:**

Use **ATTACH** for exploring services or querying multiple entities. Use **odata\_read** for one-off queries or when you know exactly what you need.

---

## Version Support

ERPL-Web automatically detects and handles both OData V2 and V4.

### OData V4 Example

```
-- TripPin V4 ServiceATTACH 'https://services.odata.org/TripPinRESTierService'   AS trippin (TYPE odata);SELECT   UserName,  FirstName,  LastName,  GenderFROM trippin.PeopleWHERE Gender = 'Female'LIMIT 5;
```

### OData V2 Example

```
-- Northwind V2 ServiceATTACH 'https://services.odata.org/V2/Northwind/Northwind.svc'   AS northwind (TYPE odata);SELECT   CustomerID,  CompanyName,  CountryFROM northwind.CustomersWHERE Country = 'Germany';
```

No configuration needed - ERPL-Web figures out the version automatically!

---

## Predicate Pushdown

ERPL-Web translates SQL WHERE clauses into OData $filter queries, sending the filtering to the server. This dramatically improves performance.

### Automatic $filter Translation

```
-- This SQL query...SELECT * FROM trippin.PeopleWHERE FirstName = 'Russell' AND LastName = 'Whyte';-- ...becomes this OData request:-- GET /People?$filter=FirstName eq 'Russell' and LastName eq 'Whyte'
```

### Supported Operators

| SQL Operator | OData Translation |
| --- | --- |
| `=` | `eq` |
| `!=` or `<>` | `ne` |
| `>` | `gt` |
| `>=` | `ge` |
| `<` | `lt` |
| `<=` | `le` |
| `AND` | `and` |
| `OR` | `or` |
| `NOT` | `not` |

### $top and $skip (LIMIT and OFFSET)

```
-- Get first 10 recordsSELECT * FROM trippin.People LIMIT 10;-- Translates to: $top=10-- Skip first 20, get next 10SELECT * FROM trippin.People LIMIT 10 OFFSET 20;-- Translates to: $skip=20&$top=10
```

### $select (Column Selection)

```
-- Only fetch specific columnsSELECT UserName, FirstName FROM trippin.People;-- Translates to: $select=UserName,FirstName
```

### $orderby (Sorting)

```
-- Sort by last nameSELECT * FROM trippin.People ORDER BY LastName;-- Translates to: $orderby=LastName-- Sort descendingSELECT * FROM trippin.People ORDER BY LastName DESC;-- Translates to: $orderby=LastName desc
```

---

## Expand Navigation Properties

OData services often have related entities. Use expand to fetch them in one query.

### Basic Expand

```
-- Fetch people with their tripsSELECT   UserName,  FirstName,  TripsFROM odata_read(  'https://services.odata.org/TripPinRESTierService/People',  expand := 'Trips');
```

### Nested Expand

```
-- Expand multiple levelsSELECT UserName, FriendsFROM odata_read(  'https://services.odata.org/TripPinRESTierService/People',  expand := 'Friends($expand=Trips)');
```

### Expand with Filter

```
-- Expand with filteringSELECT UserName, TripsFROM odata_read(  'https://services.odata.org/TripPinRESTierService/People',  expand := 'Trips($filter=Name eq ''Trip to France'')');
```

---

## Authentication

### Using Secrets

Most production OData services require authentication:

```
-- Create a secret for Basic AuthCREATE SECRET sap_odata (  TYPE http_basic,  username 'your_username',  password 'your_password');-- Attach with authentication (secret is auto-used)ATTACH 'https://your-sap-server:port/sap/opu/odata/sap/SERVICE_NAME'   AS sap_service (TYPE odata);SELECT * FROM sap_service.EntitySet;
```

### Bearer Token

```
-- Create Bearer token secretCREATE SECRET api_token (  TYPE http_bearer,  token 'your-bearer-token');-- Use it automaticallyATTACH 'https://api.example.com/odata'   AS myservice (TYPE odata);
```

### Selecting a Secret

`odata_read()` uses DuckDB's automatic secret lookup — the secret whose `SCOPE` best matches the request URL is chosen. To force a specific secret, use `ATTACH`:

```
ATTACH 'https://your-server/odata' AS my_service (TYPE odata, SECRET sap_odata);SELECT * FROM my_service.EntitySet;
```

---

## Type Mapping

ERPL-Web maps OData EDM types to DuckDB types automatically.

| EDM Type | DuckDB Type |
| --- | --- |
| Edm.String | VARCHAR |
| Edm.Int16 | SMALLINT |
| Edm.Int32 | INTEGER |
| Edm.Int64 | BIGINT |
| Edm.Decimal | DECIMAL |
| Edm.Double | DOUBLE |
| Edm.Boolean | BOOLEAN |
| Edm.DateTime | TIMESTAMP |
| Edm.DateTimeOffset | TIMESTAMP WITH TIME ZONE |
| Edm.Guid | VARCHAR |
| Edm.Binary | BLOB |

Complex types and collections are mapped to DuckDB STRUCT and LIST types.

---

## Pagination

ERPL-Web handles pagination automatically. If a service returns paginated results, the extension follows the `@odata.nextLink` to fetch all pages.

```
-- This might fetch multiple pages automaticallySELECT COUNT(*) FROM trippin.People;
```

### Manual Pagination Control

```
-- Fetch specific page sizeSELECT * FROM trippin.People LIMIT 100;-- Skip to specific offsetSELECT * FROM trippin.People LIMIT 50 OFFSET 200;
```

---

## Advanced Examples

### SAP OData Services

```
-- Connect to SAP Gateway OData serviceCREATE SECRET sap_gateway (  TYPE http_basic,  username 'SAP_USER',  password 'SAP_PASS');ATTACH 'https://sap-server:port/sap/opu/odata/sap/ZSERVICE_SRV'   AS sap (TYPE odata);-- Query SAP dataSELECT * FROM sap.Customers WHERE Country = 'US';
```

### Complex Filters

```
-- Multiple conditionsSELECT * FROM northwind.OrdersWHERE   ShipCountry = 'USA'   AND OrderDate >= '1997-01-01'  AND Freight > 100ORDER BY OrderDate DESC;
```

### Joins Across Entity Sets

```
-- Join customers and ordersSELECT   c.CompanyName,  o.OrderID,  o.OrderDateFROM northwind.Customers cJOIN northwind.Orders o ON c.CustomerID = o.CustomerIDWHERE c.Country = 'Germany'LIMIT 10;
```

### Export to Parquet

```
-- Extract OData to Parquet fileCOPY (  SELECT * FROM trippin.People) TO 'people.parquet' (FORMAT PARQUET);
```

### Use with DuckDB Features

```
-- Aggregate OData dataSELECT   Country,  COUNT(*) as customer_count,  AVG(Freight) as avg_freightFROM northwind.Orders oJOIN northwind.Customers c ON o.CustomerID = c.CustomerIDGROUP BY CountryORDER BY customer_count DESC;
```

---

## Performance Tips

### 1\. Use Predicate Pushdown

```
-- Good: Filter is pushed to OData serviceSELECT * FROM trippin.People WHERE Gender = 'Female';-- Bad: Fetches all data, then filters locally-- (Avoid if possible)
```

### 2\. Select Only Needed Columns

```
-- Good: Only fetches 2 columnsSELECT UserName, FirstName FROM trippin.People;-- Bad: Fetches all columnsSELECT * FROM trippin.People;
```

### 3\. Use LIMIT for Exploration

```
-- Good: Quick data sampleSELECT * FROM trippin.People LIMIT 10;-- Be careful with large datasets-- SELECT * FROM large_entity_set;  -- Could take a long time
```

### 4\. Cache Results for Repeated Queries

```
-- Cache OData results in a tableCREATE TABLE people_cache ASSELECT * FROM trippin.People;-- Query the cacheSELECT * FROM people_cache WHERE FirstName = 'Russell';
```

---

## Troubleshooting

### Service Not Found

```
Error: OData service not found
```

**Fixes:**

*   Verify the service URL is correct
*   Check if service requires authentication
*   Ensure the service is accessible from your network

### Entity Set Not Found

```
Error: Entity set 'XYZ' not found
```

**Fixes:**

*   Use `SHOW TABLES` to list available entity sets
*   Check capitalization (OData is case-sensitive)
*   Verify you're using the correct version (V2 vs V4)

### Authentication Failed

```
Error: 401 Unauthorized
```

**Fixes:**

*   Verify credentials in your secret
*   Check if credentials have expired
*   Ensure user has permissions to access the service

### Slow Queries

**Solutions:**

*   Use WHERE clauses to push filters to server
*   Select only needed columns
*   Use LIMIT for large datasets
*   Check OData service performance

### Enable Tracing

```
-- Enable tracing to see what's happeningSET erpl_trace_enabled = TRUE;SET erpl_trace_level = 'DEBUG';-- Run your querySELECT * FROM trippin.People LIMIT 5;-- Check trace output for details
```

See [Tracing & Diagnostics](/docs/erpl-web/tracing.md) for more details.

---

## Known Limitations

*   **Write operations**: ERPL-Web currently supports read-only operations (GET). POST/PUT/PATCH/DELETE are not yet supported.
*   **Function imports**: Custom OData functions may have limited support.
*   **Batch requests**: Not currently supported.

---

## Examples by Use Case

### Data Integration

```
-- Sync OData service to local DuckDBCREATE TABLE local_customers ASSELECT * FROM odata_read('https://api.example.com/odata/Customers');-- Incremental updatesINSERT INTO local_customersSELECT * FROM odata_read('https://api.example.com/odata/Customers')WHERE ModifiedDate > (SELECT MAX(ModifiedDate) FROM local_customers);
```

### Analytics

```
-- Analyze OData data with DuckDBWITH monthly_sales AS (  SELECT     DATE_TRUNC('month', OrderDate) as month,    SUM(Freight) as total_freight  FROM northwind.Orders  GROUP BY month)SELECT * FROM monthly_salesORDER BY month;
```

### Data Export

```
-- Export filtered data to CSVCOPY (  SELECT * FROM trippin.People  WHERE Gender = 'Female') TO 'female_travelers.csv' (HEADER, DELIMITER ',');
```

---

## Next Steps

*   Explore [SAP Datasphere](/docs/erpl-web/datasphere.md) for SAP-specific OData integration
*   Learn about [ODP via OData](/docs/erpl-web/odp-web.md) for delta replication
*   See [Secrets Management](/docs/erpl-web/secrets.md) for secure authentication
*   Check [Examples](/docs/examples/erpl-web-examples.md) for more real-world patterns

---

## Summary

ERPL-Web's OData integration gives you:

✅ **Universal** - Works with V2 and V4 automatically  
✅ **Powerful** - Predicate pushdown and expand support  
✅ **Simple** - ATTACH and query like normal SQL  
✅ **Fast** - Server-side filtering and pagination

Turn any OData service into a DuckDB table in seconds!

# ODP via OData (Delta Replication)

Extract SAP data with automatic delta replication using ODP (Operational Data Provisioning) via the OData protocol. Keep your data warehouse synchronized with minimal network overhead.

## Overview

**ODP (Operational Data Provisioning)** is SAP's framework for extracting data with **delta replication** support. Since SAP OSS Note 3255746 banned RFC use for external data extraction, ODP via OData is now the recommended approach.

**What you get:**

*   ✅ Automatic delta replication (only changed records)
*   ✅ Subscription management (persistent state tracking)
*   ✅ Change type detection (Insert/Update/Delete)
*   ✅ Audit logging for monitoring
*   ✅ Initial full load + incremental updates
*   ✅ Real-time data synchronization

---

## Why Use ODP OData?

### Traditional RFC Limitations

*   ❌ **Banned by SAP** - RFC extraction prohibited per OSS Note 3255746
*   ❌ **No delta support** - Must extract full datasets
*   ❌ **Firewall issues** - Requires NetWeaver connectivity

### ODP OData Benefits

*   ✅ **SAP Approved** - Official method for data extraction
*   ✅ **Delta replication** - Only changed records transferred
*   ✅ **Web-based** - Works through HTTPS
*   ✅ **Real-time** - Incremental updates as data changes
*   ✅ **Scalable** - Handles large datasets efficiently

---

## Key Concepts

| Term | Description | Example |
| --- | --- | --- |
| **Entity Set** | A collection of data (like a table) | `FactsOf0D_NW_C01` (sales data) |
| **Delta Token** | A bookmark for incremental updates | `'D20250914154609_000019000'` |
| **Subscription** | A persistent connection for delta updates | Automatically managed by ERPL-Web |
| **RECORD\_MODE** | Change type indicator | `''` (update), `'N'` (insert), `'D'` (delete) |
| **OData Service** | The API endpoint providing the data | `/sap/opu/odata/sap/Z_ODP_BW_1_SRV/` |

---

## Prerequisites

Before starting, ensure you have:

1.  **SAP System with ODP enabled** - S/4HANA, BW, ECC with ODP support
2.  **ICF Services activated** - OData services must be active in SAP
3.  **User Permissions** - Access to ODP providers and data sources
4.  **Network Access** - HTTPS connectivity to SAP system

**Tip:**

Check if ICF services are activated using the `activate_icf_services.abap` script in the erpl-web repository.

---

## Quick Start

### Step 1: Create Authentication Secret

```
-- Load the extensionLOAD erpl_web;-- Create HTTP Basic authentication secretCREATE SECRET sap_system (  TYPE http_basic,  username 'YOUR_SAP_USERNAME',  password 'YOUR_SAP_PASSWORD');
```

### Step 2: Discover Available Data

```
-- Find ODP OData servicesSELECT   service_name,  entity_set_name,  full_entity_set_url,  descriptionFROM odp_odata_show('https://your-sap-server:port', secret='sap_system')WHERE entity_set_name LIKE '%SALES%'LIMIT 10;
```

**Example Output:**

```
┌─────────────────┬──────────────────┬────────────────────────────────┬──────────────────┐│  service_name   │ entity_set_name  │     full_entity_set_url        │   description    │├─────────────────┼──────────────────┼────────────────────────────────┼──────────────────┤│ Z_ODP_BW_1_SRV  │ FactsOf0D_NW_C01 │ https://server/sap/opu/odata...│ Sales Facts Data │└─────────────────┴──────────────────┴────────────────────────────────┴──────────────────┘
```

### Step 3: Initial Load (Creates Subscription)

The first call to `odp_odata_read()` automatically:

*   Creates a subscription
*   Performs a full initial load
*   Stores a delta token for future updates

**Signature:**

```
odp_odata_read(entity_set_url VARCHAR)
```

**Named parameters:**

| Parameter | Type | Description |
| --- | --- | --- |
| `secret` | VARCHAR | DuckDB secret name for authentication |
| `force_full_load` | BOOLEAN | Bypass the delta token and re-read everything (default: `false`) |
| `import_delta_token` | VARCHAR | Resume from a specific delta token (e.g. from another environment) |
| `max_page_size` | UINTEGER | Override page size for the underlying OData fetch |

```
-- Create subscription and load all dataSELECT COUNT(*) as total_recordsFROM odp_odata_read(  'https://your-sap-server:port/sap/opu/odata/sap/Z_ODP_BW_1_SRV/FactsOf0D_NW_C01',  secret='sap_system');-- Re-load everything (e.g. after a schema change)SELECT * FROM odp_odata_read(  'https://.../FactsOf0D_NW_C01',  secret => 'sap_system',  force_full_load => true);-- Resume from a known delta tokenSELECT * FROM odp_odata_read(  'https://.../FactsOf0D_NW_C01',  secret => 'sap_system',  import_delta_token => 'D20260101120000000000000');
```

**Example Output:**

```
┌───────────────┐│ total_records │├───────────────┤│        15420  │└───────────────┘
```

### Step 4: Delta Load (Only Changes)

Subsequent calls fetch only changes since last extraction:

```
-- Fetch delta updatesSELECT RECORD_MODE, COUNT(*) as countFROM odp_odata_read(  'https://your-sap-server:port/sap/opu/odata/sap/Z_ODP_BW_1_SRV/FactsOf0D_NW_C01',  secret='sap_system')GROUP BY RECORD_MODE;
```

**Example Output:**

```
┌─────────────┬───────┐│ RECORD_MODE │ count │├─────────────┼───────┤│             │   342 │  -- Updated records│ N           │    87 │  -- New records│ D           │    23 │  -- Deleted records└─────────────┴───────┘
```

---

## Understanding RECORD\_MODE

The `RECORD_MODE` column tells you what action to take for each record:

| RECORD\_MODE | Meaning | SQL Action |
| --- | --- | --- |
| `''` (empty string) | **Updated Record** | `UPDATE` in your target |
| `'N'` | **New Record** | `INSERT` in your target |
| `'D'` | **Deleted Record** | `DELETE` from your target |

### Example: Processing Changes

```
-- Get all changesWITH delta_data AS (  SELECT * FROM odp_odata_read(    'https://sap-server/odata/SALES_DATA',    secret='sap_system'  ))-- Process insertsINSERT INTO target_tableSELECT * FROM delta_data WHERE RECORD_MODE = 'N';-- Process updatesUPDATE target_table tSET   field1 = d.field1,  field2 = d.field2FROM delta_data dWHERE t.key = d.key AND d.RECORD_MODE = '';-- Process deletesDELETE FROM target_tableWHERE key IN (  SELECT key FROM delta_data WHERE RECORD_MODE = 'D');
```

---

## Subscription Management

ERPL-Web automatically manages subscriptions, but you can monitor and control them.

### List Active Subscriptions

```
SELECT   subscription_id,  entity_set_name,  entity_set_url,  subscription_status,  last_delta_token,  created_atFROM odp_odata_list_subscriptions();
```

**Example Output:**

```
┌─────────────────┬──────────────────┬─────────────────────┬────────────────────┬──────────────────────────┬─────────────────────┐│ subscription_id │ entity_set_name  │  entity_set_url     │ subscription_status│    last_delta_token      │     created_at      │├─────────────────┼──────────────────┼─────────────────────┼────────────────────┼──────────────────────────┼─────────────────────┤│ sub_001         │ FactsOf0D_NW_C01 │ https://sap/odata...│ active             │ D20250914154609_000019000│ 2024-01-15 10:30:00 │└─────────────────┴──────────────────┴─────────────────────┴────────────────────┴──────────────────────────┴─────────────────────┘
```

### Remove a Subscription

```
-- Remove subscription from local trackingPRAGMA odp_odata_remove_subscription('sub_001', false);-- Remove from local AND delete on SAP serverPRAGMA odp_odata_remove_subscription('sub_001', true);
```

**Warning:**

Removing a subscription resets delta tracking. The next extraction will be a full load.

### Subscription Lifecycle

1.  **Creation** - First call to `odp_odata_read()` creates subscription
2.  **Active** - Subscription tracks delta token after each extraction
3.  **Delta Updates** - Subsequent calls use stored delta token
4.  **Removal** - Manually remove when no longer needed

---

## Monitoring & Auditing

ERPL-Web logs all ODP operations to an audit table for monitoring and troubleshooting.

### View Audit Logs

```
SELECT   subscription_id,  request_timestamp,  request_type,  package_count,  records_received,  has_more_data,  new_delta_token,  execution_time_msFROM erpl_web.odp_subscription_auditORDER BY request_timestamp DESCLIMIT 20;
```

**Columns Explained:**

*   `subscription_id`: Which subscription made the request
*   `request_timestamp`: When the extraction occurred
*   `request_type`: `'INITIAL'` or `'DELTA'`
*   `package_count`: Number of packages received
*   `records_received`: Total records extracted
*   `has_more_data`: Whether more data is available
*   `new_delta_token`: Delta token for next extraction
*   `execution_time_ms`: Query duration

### Performance Analysis

```
-- Average extraction time by subscriptionSELECT   subscription_id,  entity_set_name,  AVG(execution_time_ms) as avg_time_ms,  AVG(records_received) as avg_records,  COUNT(*) as extraction_countFROM erpl_web.odp_subscription_auditGROUP BY subscription_id, entity_set_nameORDER BY avg_time_ms DESC;
```

### Monitor Data Freshness

```
-- Check when data was last updatedSELECT   subscription_id,  entity_set_name,  MAX(request_timestamp) as last_extraction,  DATEDIFF('hour', MAX(request_timestamp), CURRENT_TIMESTAMP) as hours_since_updateFROM erpl_web.odp_subscription_auditGROUP BY subscription_id, entity_set_name;
```

---

## Delta Replication Workflow

### Complete ETL Pattern

```
-- 1. Create target table (first time only)CREATE TABLE sales_facts (  sales_order VARCHAR,  customer_id VARCHAR,  revenue DECIMAL(15,2),  order_date DATE,  RECORD_MODE VARCHAR  -- Track change type);-- 2. Initial LoadINSERT INTO sales_factsSELECT * FROM odp_odata_read(  'https://sap/odata/Z_ODP_BW_1_SRV/FactsOf0D_NW_C01',  secret='sap_system');-- 3. Incremental Updates (run periodically)WITH delta AS (  SELECT * FROM odp_odata_read(    'https://sap/odata/Z_ODP_BW_1_SRV/FactsOf0D_NW_C01',    secret='sap_system'  ))-- Handle insertsINSERT INTO sales_factsSELECT * FROM delta WHERE RECORD_MODE = 'N';-- Handle updatesUPDATE sales_facts tSET   revenue = d.revenue,  order_date = d.order_dateFROM delta dWHERE t.sales_order = d.sales_order   AND d.RECORD_MODE = '';-- Handle deletesDELETE FROM sales_factsWHERE sales_order IN (  SELECT sales_order FROM delta WHERE RECORD_MODE = 'D');
```

### Export to Parquet Data Lake

```
-- Extract to Parquet with partitioningCOPY (  SELECT * FROM odp_odata_read(    'https://sap/odata/SALES_DATA',    secret='sap_system'  )) TO 'sales_data.parquet' (  FORMAT PARQUET,  PARTITION_BY (order_date));
```

### Incremental File Updates

```
-- Daily delta extraction to dated filesCOPY (  SELECT * FROM odp_odata_read(    'https://sap/odata/SALES_DATA',    secret='sap_system'  )) TO 'sales_delta_2024-01-15.parquet' (FORMAT PARQUET);
```

---

## Advanced Usage

### Batch Processing Multiple Entities

```
-- Create a list of entity sets to extractCREATE TABLE entity_sets (entity_url VARCHAR);INSERT INTO entity_sets VALUES   ('https://sap/odata/Z_ODP_BW_1_SRV/FactsOf0D_NW_C01'),  ('https://sap/odata/Z_ODP_BW_1_SRV/CustomerData'),  ('https://sap/odata/Z_ODP_BW_1_SRV/ProductData');-- Extract all entitiesSELECT   e.entity_url,  COUNT(*) as record_countFROM entity_sets e,LATERAL (  SELECT * FROM odp_odata_read(e.entity_url, secret := 'sap_system')) dataGROUP BY e.entity_url;
```

### Custom Subscription IDs

By default, subscriptions are auto-generated. For persistent tracking:

```
-- The subscription_id is derived from the entity_set_url-- and stored automatically in the audit table
```

### Error Handling and Retries

```
-- Wrap extraction in error handlingCREATE OR REPLACE MACRO extract_with_retry(url, max_retries := 3) AS (  -- Implementation would require scripting/procedural logic  -- For now, check audit logs for failures and re-run);
```

---

## Troubleshooting

### Authentication Failures

**Issue:** `401 Unauthorized`

**Solutions:**

```
-- 1. Verify credentialsCREATE SECRET sap_system (  TYPE http_basic,  username 'correct_username',  password 'correct_password');-- 2. Test with a simple HTTP request firstSELECT status FROM http_get(  'https://sap-server:port',  auth := 'user:pass',  auth_type := 'BASIC');
```

### Subscription State Issues

**Issue:** Subscription not found or corrupt

**Solution:**

```
-- Remove and recreate subscriptionPRAGMA odp_odata_remove_subscription('subscription_id', false);-- Next extraction will create new subscriptionSELECT * FROM odp_odata_read(  'https://sap/odata/ENTITY_SET',  secret='sap_system');
```

### Network Timeouts

**Issue:** Request timeout

**Solution:**

```
-- ODP extractions can be large. Check audit logs for progress.-- If timeout occurs, subscription state is preserved and can resume.-- Check last successful extractionSELECT * FROM erpl_web.odp_subscription_auditWHERE subscription_id = 'your_sub_id'ORDER BY request_timestamp DESC LIMIT 1;
```

### No Delta Data Returned

**Issue:** Delta extraction returns 0 records

**Explanation:** This is normal! It means no data changed since the last extraction.

```
-- This is expected if no changes occurredSELECT COUNT(*) FROM odp_odata_read(...);-- Returns: 0
```

### Memory Issues

**Issue:** Out of memory during large extractions

**Solutions:**

1.  **Check package sizes** in audit logs
2.  **Increase DuckDB memory** settings
3.  **Extract to files** instead of memory:

```
-- Stream directly to ParquetCOPY (  SELECT * FROM odp_odata_read(...)) TO 'output.parquet' (FORMAT PARQUET);
```

### Enable Tracing

For detailed debugging:

```
SET erpl_trace_enabled = TRUE;SET erpl_trace_level = 'DEBUG';-- Run extractionSELECT * FROM odp_odata_read(...);-- Check trace logs
```

---

## Performance Optimization

### 1\. Monitor Extraction Times

```
SELECT   entity_set_name,  AVG(execution_time_ms) / 1000.0 as avg_time_seconds,  AVG(records_received) as avg_recordsFROM erpl_web.odp_subscription_auditWHERE request_type = 'DELTA'GROUP BY entity_set_name;
```

### 2\. Reduce Extraction Frequency

If extractions are slow, consider less frequent updates:

*   Real-time: Every 5 minutes
*   Near real-time: Every hour
*   Batch: Daily/weekly

### 3\. Use Parquet for Large Datasets

```
-- Don't load large datasets into memory-- Stream directly to Parquet filesCOPY (SELECT * FROM odp_odata_read(...))TO 'data.parquet' (FORMAT PARQUET, COMPRESSION 'ZSTD');
```

### 4\. Parallel Extraction

Extract multiple entity sets in parallel (using multiple DuckDB sessions):

```
# Session 1duckdb -c "SELECT * FROM odp_odata_read('url1', ...)"# Session 2 (separate process)duckdb -c "SELECT * FROM odp_odata_read('url2', ...)"
```

---

## Integration Patterns

### Scheduled Delta Sync

```
#!/bin/bash# daily_sync.shduckdb analytics.db <<EOFLOAD erpl_web;-- Extract delta changesINSERT INTO sales_factsSELECT * FROM odp_odata_read(  'https://sap/odata/SALES_DATA',  secret='sap_system')WHERE RECORD_MODE = 'N';  -- Only inserts for simplicity-- Log completionINSERT INTO sync_log VALUES (CURRENT_TIMESTAMP, 'SALES_FACTS', 'SUCCESS');EOF
```

### Data Lake Pattern

```
-- Daily partitioned extractionsCOPY (  SELECT *, CURRENT_DATE as extraction_date  FROM odp_odata_read(    'https://sap/odata/SALES_DATA',    secret='sap_system'  )) TO 'datalake/sales/year=2024/month=01/day=15/data.parquet'(FORMAT PARQUET);
```

### Change Data Capture (CDC)

```
-- Track all changes with timestampsCREATE TABLE cdc_log (  change_timestamp TIMESTAMP DEFAULT CURRENT_TIMESTAMP,  source_table VARCHAR,  change_type VARCHAR,  key_value VARCHAR,  old_value JSON,  new_value JSON);-- Log changes from ODPWITH changes AS (  SELECT * FROM odp_odata_read(...))INSERT INTO cdc_logSELECT   CURRENT_TIMESTAMP,  'SALES_FACTS',  RECORD_MODE,  sales_order,  NULL,  -- Would need to query old value  to_json(changes)FROM changes;
```

---

## Best Practices

### 1\. Always Use Secrets

```
-- GoodCREATE SECRET sap_prod (TYPE http_basic, ...);-- Bad (hardcoded credentials)-- SELECT * FROM odp_odata_read(..., auth := 'user:pass');
```

### 2\. Monitor Audit Logs Regularly

```
-- Weekly audit reviewSELECT   DATE_TRUNC('day', request_timestamp) as day,  COUNT(*) as extraction_count,  SUM(records_received) as total_recordsFROM erpl_web.odp_subscription_auditWHERE request_timestamp >= CURRENT_DATE - INTERVAL 7 DAYSGROUP BY day;
```

### 3\. Handle All RECORD\_MODE Types

Always process inserts, updates, AND deletes:

```
-- Don't forget deletes!DELETE FROM target WHERE key IN (  SELECT key FROM delta WHERE RECORD_MODE = 'D');
```

### 4\. Archive Audit Logs

```
-- Prevent audit table from growing too largeDELETE FROM erpl_web.odp_subscription_auditWHERE request_timestamp < CURRENT_DATE - INTERVAL 90 DAYS;
```

### 5\. Test with Small Datasets First

```
-- Start with a small entity setSELECT * FROM odp_odata_show(...)WHERE entity_set_name LIKE '%TEST%';
```

---

## Complete Reference Example

Here's a production-ready delta replication workflow:

```
-- ============================================-- Complete ODP Delta Replication Example-- ============================================-- 1. Setup (once)LOAD erpl_web;CREATE SECRET sap_prod (  TYPE http_basic,  username 'SAP_USER',  password 'SAP_PASS');-- 2. Discover data sourcesCREATE TABLE available_sources ASSELECT * FROM odp_odata_show(  'https://sap-prod:8001',  secret='sap_prod');-- 3. Create target tableCREATE TABLE sales_data (  sales_order VARCHAR PRIMARY KEY,  customer_id VARCHAR,  product_id VARCHAR,  quantity INTEGER,  revenue DECIMAL(15,2),  order_date DATE,  last_modified TIMESTAMP DEFAULT CURRENT_TIMESTAMP);-- 4. Initial loadINSERT INTO sales_dataSELECT   sales_order,  customer_id,  product_id,  quantity,  revenue,  order_dateFROM odp_odata_read(  'https://sap-prod:8001/sap/opu/odata/sap/Z_SALES_SRV/SalesOrders',  secret='sap_prod');-- 5. Delta sync function (run periodically)CREATE OR REPLACE MACRO sync_sales_data() AS TABLE (  WITH delta AS (    SELECT * FROM odp_odata_read(      'https://sap-prod:8001/sap/opu/odata/sap/Z_SALES_SRV/SalesOrders',      secret='sap_prod'    )  )  -- Return summary  SELECT     COUNT(CASE WHEN RECORD_MODE = 'N' THEN 1 END) as inserts,    COUNT(CASE WHEN RECORD_MODE = '' THEN 1 END) as updates,    COUNT(CASE WHEN RECORD_MODE = 'D' THEN 1 END) as deletes  FROM delta);-- 6. Monitor performanceSELECT * FROM erpl_web.odp_subscription_auditWHERE entity_set_name LIKE '%SalesOrders%'ORDER BY request_timestamp DESC LIMIT 10;
```

---

## Next Steps

*   Learn about [OData](/docs/erpl-web/odata.md) for general OData services
*   Explore [Secrets Management](/docs/erpl-web/secrets.md) for credential security
*   See [Tracing & Diagnostics](/docs/erpl-web/tracing.md) for debugging
*   Check [Examples](/docs/examples/erpl-web-examples.md) for more patterns

---

## Summary

ODP via OData in ERPL-Web provides:

✅ **SAP-Approved** - Official method replacing RFC extraction  
✅ **Delta Replication** - Only transfer changed data  
✅ **Automatic** - Subscription and token management built-in  
✅ **Auditable** - Complete logging for monitoring  
✅ **Scalable** - Handle large SAP datasets efficiently

Start synchronizing your SAP data with delta replication today!

# SAP Analytics Cloud (SAC)

ERPL-Web reads **SAP Analytics Cloud** models and stories directly from DuckDB SQL. Authenticate once with OAuth2, then list, describe, and query your SAC assets.

## Authenticate

Create a `sac` secret with an OAuth2 client provisioned in your SAC tenant.

```
CREATE SECRET sac (  TYPE sac,  PROVIDER oauth2,  tenant_name 'mytenant',  region 'eu10');
```

The OAuth2 flow refreshes tokens automatically — you do not need to wire access tokens by hand.

## Discover assets

List the models and stories in your tenant:

```
SELECT * FROM sac_show_models();SELECT * FROM sac_show_stories();
```

## Describe a model or story

Pull the full metadata for a specific asset by ID:

```
SELECT * FROM sac_describe_model('YOUR_MODEL_ID');SELECT * FROM sac_describe_story('YOUR_STORY_ID');
```

## Related

*   [SAP Datasphere](/docs/erpl-web/datasphere.md) — the warehousing layer beneath SAC.
*   [ODP via OData](/docs/erpl-web/odp-web.md) — operational extractors over the SAP OData channel.
*   [Secrets Management](/docs/erpl-web/secrets.md) — secret types, OAuth2 flows, and rotation.

# Secrets Management

ERPL-Web integrates with DuckDB's secrets management system to keep your credentials secure. Never hardcode passwords or API tokens again.

## Overview

DuckDB secrets provide a secure way to store and manage authentication credentials. ERPL-Web extends this with custom secret types and providers for SAP and cloud services.

**What you get:**

*   ✅ Encrypted storage of credentials
*   ✅ OAuth2 flows with automatic token refresh
*   ✅ Multiple provider types (oauth2, config, file)
*   ✅ Secret scoping and isolation
*   ✅ No plaintext passwords in SQL
*   ✅ Integration with all ERPL-Web functions

---

## Quick Start

```
-- Create a simple HTTP basic auth secretCREATE SECRET my_api (  TYPE http_basic,  username 'api_user',  password 'api_password');-- Use it automatically in requestsSELECT * FROM http_get('https://api.example.com/data');-- Or specify explicitlySELECT * FROM http_get(  'https://api.example.com/data',  secret := 'my_api');
```

---

## Secret Types

### http\_basic

For HTTP Basic Authentication (username/password).

**Usage:**

```
CREATE SECRET sap_system (  TYPE http_basic,  username 'SAP_USER',  password 'SAP_PASSWORD');-- Used in ODP, OData, HTTP functionsSELECT * FROM odp_odata_read('https://sap/odata/...', secret='sap_system');
```

---

### http\_bearer

For HTTP Bearer Token Authentication (API tokens).

**Usage:**

```
CREATE SECRET api_token (  TYPE http_bearer,  token 'your-bearer-token-here');-- Used in HTTP requestsSELECT * FROM http_get('https://api.example.com/data');
```

---

### datasphere

For SAP Datasphere OAuth2 authentication (custom type).

**Usage:**

```
CREATE SECRET datasphere (  TYPE datasphere,  PROVIDER oauth2,  tenant_name 'your-tenant',  data_center 'eu10',  scope 'default');-- Used in Datasphere functionsSELECT * FROM datasphere_show_spaces();
```

---

## Secret Providers

Providers control how secrets are created and managed.

### oauth2 Provider

Interactive OAuth2 flows for cloud services.

**Features:**

*   Opens browser for authentication
*   Stores access and refresh tokens
*   Automatic token refresh
*   Supports authorization\_code and client\_credentials flows

**Authorization Code Flow (Interactive):**

```
CREATE SECRET datasphere (  TYPE datasphere,  PROVIDER oauth2,  tenant_name 'acme-corp',  data_center 'eu10',  scope 'default',  REDIRECT_URI 'http://localhost:65000'  -- Optional, default shown);
```

**What happens:**

1.  Local server starts on port 65000
2.  Browser opens to OAuth2 provider
3.  You log in with your credentials
4.  Authorization code is captured
5.  Tokens are exchanged and stored
6.  Secret is ready to use

**Client Credentials Flow (Service Account):**

```
CREATE SECRET datasphere_svc (  TYPE datasphere,  PROVIDER oauth2,  tenant_name 'acme-corp',  data_center 'eu10',  client_id 'service-account-id',  client_secret 'service-account-secret',  GRANT_TYPE 'client_credentials',  scope 'default');
```

No browser needed - uses service account credentials directly.

---

### config Provider

Load secrets from INI-style configuration files.

**Config File Format:**

```
# ~/.erpl/sap-prod.conf[sap_system]username=SAP_USERpassword=SAP_PASSWORD[datasphere]tenant_name=acme-corpdata_center=eu10client_id=abc123client_secret=secret456
```

**Create Secret from Config:**

```
CREATE SECRET sap_prod (  TYPE http_basic,  PROVIDER config,  CONFIG_FILE '/home/user/.erpl/sap-prod.conf',  SECTION 'sap_system'  -- Optional, defaults to secret name);
```

**Benefits:**

*   Separate credentials from code
*   Easy to manage multiple environments
*   Version control friendly (exclude config files)
*   Rotate credentials without changing SQL

---

### file Provider

Load secrets from JSON or text files.

**JSON File Format:**

```
{  "tenant_name": "acme-corp",  "data_center": "eu10",  "access_token": "eyJhbGc...",  "refresh_token": "eyJhbGc...",  "expires_at": "2024-01-15T15:30:00Z"}
```

**Create Secret from File:**

```
CREATE SECRET datasphere_prod (  TYPE datasphere,  PROVIDER file,  FILEPATH '/secure/credentials/datasphere-prod.json');
```

**Use Cases:**

*   Store pre-acquired tokens
*   Integration with secret managers (Vault, AWS Secrets Manager)
*   Share credentials across team
*   Backup/restore secrets

---

## Managing Secrets

### List Secrets

```
-- Show all secretsSELECT * FROM duckdb_secrets();
```

**Output:**

```
┌─────────────┬──────────────┬───────────┬─────────┐│    name     │     type     │  provider │  scope  │├─────────────┼──────────────┼───────────┼─────────┤│ my_api      │ http_basic   │ inline    │ user    ││ datasphere  │ datasphere   │ oauth2    │ user    │└─────────────┴──────────────┴───────────┴─────────┘
```

### Update a Secret

```
-- Drop and recreateDROP SECRET my_api;CREATE SECRET my_api (  TYPE http_basic,  username 'new_user',  password 'new_password');
```

### Remove a Secret

```
DROP SECRET my_api;
```

### Temporary vs Persistent Secrets

```
-- Temporary (session only)CREATE TEMPORARY SECRET temp_api (  TYPE http_basic,  username 'user',  password 'pass');-- Persistent (saved to database)CREATE SECRET persistent_api (  TYPE http_basic,  username 'user',  password 'pass');
```

---

## OAuth2 Flows in Detail

### Authorization Code Flow

Best for interactive use (desktop, notebooks).

```
CREATE SECRET interactive_ds (  TYPE datasphere,  PROVIDER oauth2,  tenant_name 'mytenant',  data_center 'eu10',  scope 'default');
```

**Step-by-Step:**

1.  Secret creation starts OAuth2 flow
2.  Local callback server starts (default: `http://localhost:65000`)
3.  Browser opens to authorization URL
4.  User authenticates with username/password
5.  User grants permissions
6.  Browser redirects to callback with authorization code
7.  Extension exchanges code for tokens
8.  Tokens stored in secret (encrypted)

### Token Refresh

Tokens expire (typically after 1 hour). ERPL-Web handles refresh automatically:

```
-- First use after token expirySELECT * FROM datasphere_show_spaces();-- Automatically refreshes token if expired-- You never need to manually refresh!
```

### Client Credentials Flow

Best for automation (scripts, services, CI/CD).

```
CREATE SECRET service_ds (  TYPE datasphere,  PROVIDER oauth2,  tenant_name 'mytenant',  data_center 'eu10',  client_id 'service-id',  client_secret 'service-secret',  GRANT_TYPE 'client_credentials',  TOKEN_URL 'https://mytenant.authentication.eu10.hana.ondemand.com/oauth/token',  -- Optional  scope 'default');
```

No browser interaction - ideal for:

*   Scheduled jobs
*   Docker containers
*   CI/CD pipelines
*   Backend services

---

## Security Best Practices

### 1\. Never Commit Secrets

```
# .gitignore*.conf*-credentials.json.envsecrets/
```

### 2\. Use Environment Variables

```
-- Good: Load from environmentCREATE SECRET api_key (  TYPE http_bearer,  token getenv('API_TOKEN'));-- Bad: Hardcoded-- CREATE SECRET api_key (TYPE http_bearer, token 'abc123');
```

### 3\. Restrict File Permissions

```
# Only owner can read secret fileschmod 600 ~/.erpl/credentials.confchmod 600 /secure/datasphere.json
```

### 4\. Use Different Secrets Per Environment

```
-- DevelopmentCREATE SECRET datasphere_dev (  TYPE datasphere,  PROVIDER oauth2,  tenant_name 'acme-dev',  data_center 'eu10');-- ProductionCREATE SECRET datasphere_prod (  TYPE datasphere,  PROVIDER oauth2,  tenant_name 'acme-prod',  data_center 'eu10');-- Use explicitlySELECT * FROM datasphere_show_spaces(secret := 'datasphere_prod');
```

### 5\. Rotate Credentials Regularly

```
-- Update secret with new credentialsDROP SECRET old_api;CREATE SECRET new_api (...);-- Update all queries to use new secret
```

### 6\. Use Temporary Secrets for Testing

```
-- Temporary secret (not saved to disk)CREATE TEMPORARY SECRET test_api (  TYPE http_basic,  username 'test',  password 'test123');-- Automatically dropped at end of session
```

---

## Advanced Patterns

### Multiple Tenants

```
-- Create secrets for each tenantCREATE SECRET tenant_a (  TYPE datasphere,  PROVIDER oauth2,  tenant_name 'tenant-a',  data_center 'eu10');CREATE SECRET tenant_b (  TYPE datasphere,  PROVIDER oauth2,  tenant_name 'tenant-b',  data_center 'us10');-- Query specific tenantSELECT * FROM datasphere_show_spaces(secret := 'tenant_a');SELECT * FROM datasphere_show_spaces(secret := 'tenant_b');
```

### Secret Rotation Without Downtime

```
-- Create new secret with new credentialsCREATE SECRET api_v2 (  TYPE http_basic,  username 'new_user',  password 'new_pass');-- Test new secretSELECT * FROM http_get('https://api.example.com/test', secret := 'api_v2');-- If successful, make it the defaultDROP SECRET api_v1;ALTER SECRET api_v2 RENAME TO api_v1;  -- If supported-- Or update queries to use api_v2
```

### Config File with Multiple Environments

```
# config.conf[dev]tenant_name=dev-tenantdata_center=eu10client_id=dev-client[staging]tenant_name=staging-tenantdata_center=eu10client_id=staging-client[prod]tenant_name=prod-tenantdata_center=eu10client_id=prod-client
```

```
-- Load different environmentsCREATE SECRET datasphere_dev (  TYPE datasphere,  PROVIDER config,  CONFIG_FILE 'config.conf',  SECTION 'dev');CREATE SECRET datasphere_prod (  TYPE datasphere,  PROVIDER config,  CONFIG_FILE 'config.conf',  SECTION 'prod');
```

### Integration with External Secret Managers

```
#!/bin/bash# fetch_secret.sh - Retrieve from AWS Secrets Manageraws secretsmanager get-secret-value \  --secret-id datasphere/prod \  --query SecretString \  --output text > /tmp/datasphere-creds.json
```

```
-- Use the fetched secretCREATE SECRET datasphere (  TYPE datasphere,  PROVIDER file,  FILEPATH '/tmp/datasphere-creds.json');-- Clean up after use!rm /tmp/datasphere-creds.json
```

---

## Troubleshooting

### Secret Not Found

**Issue:** `Error: Secret 'xyz' not found`

**Solutions:**

```
-- List all secretsSELECT name FROM duckdb_secrets();-- Create the missing secretCREATE SECRET xyz (...);
```

### OAuth2 Flow Fails

**Issue:** Browser doesn't open or redirect fails

**Solutions:**

```
-- Try different redirect portCREATE SECRET datasphere (  TYPE datasphere,  PROVIDER oauth2,  tenant_name 'mytenant',  data_center 'eu10',  REDIRECT_URI 'http://localhost:8080'  -- Try 8080, 3000, etc.);-- Check firewall settings-- Ensure localhost can bind to the port
```

### Token Expired

**Issue:** `401 Unauthorized` after period of inactivity

**Solution:**

```
-- Tokens auto-refresh, but if refresh fails:DROP SECRET datasphere;CREATE SECRET datasphere (...);  -- Re-authenticate
```

### Config File Not Found

**Issue:** `Error: Could not read config file`

**Solutions:**

```
# Check file existsls -la /path/to/config.conf# Check permissionschmod 600 /path/to/config.conf# Use absolute pathCREATE SECRET (..., CONFIG_FILE '/home/user/.erpl/config.conf');
```

### Wrong Credentials in Config

**Issue:** Authentication fails with config-based secret

**Solutions:**

```
# Verify config file format[section_name]key=value  # No spaces around =key2=value2# Not this:# key = value  (spaces cause issues in some parsers)
```

---

## Examples by Use Case

### Local Development

```
-- Simple inline secret for local testingCREATE TEMPORARY SECRET local_sap (  TYPE http_basic,  username 'DEV_USER',  password 'dev123');
```

### Team Shared Config

```
# team-config.conf (committed to repo)[sap_dev]base_url=https://sap-dev.example.com# Sensitive values in environment variables[datasphere_dev]tenant_name=team-dev-tenantdata_center=eu10# Client ID/secret in environment
```

```
CREATE SECRET sap_dev (  TYPE http_basic,  PROVIDER config,  CONFIG_FILE 'team-config.conf',  SECTION 'sap_dev',  username getenv('SAP_DEV_USER'),  password getenv('SAP_DEV_PASS'));
```

### Production Deployment

```
# Use secret managerexport DATASPHERE_CREDS=$(vault kv get -field=json secret/datasphere/prod)echo "$DATASPHERE_CREDS" > /run/secrets/datasphere.json
```

```
-- Load from mounted secretCREATE SECRET datasphere (  TYPE datasphere,  PROVIDER file,  FILEPATH '/run/secrets/datasphere.json');
```

---

## Next Steps

*   Apply secrets in [HTTP Functions](/docs/erpl-web/http-functions.md)
*   Use with [OData](/docs/erpl-web/odata.md) services
*   Secure [Datasphere](/docs/erpl-web/datasphere.md) connections
*   Protect [ODP](/docs/erpl-web/odp-web.md) extractions

---

## Summary

ERPL-Web's secrets management provides:

✅ **Secure** - Encrypted storage, no plaintext passwords  
✅ **Flexible** - Multiple providers (oauth2, config, file)  
✅ **Automatic** - OAuth2 token refresh built-in  
✅ **DuckDB Native** - Integrated with DuckDB secrets system  
✅ **Production Ready** - Environment separation and rotation

Keep your credentials secure while querying APIs and SAP systems!

# Tracing & Diagnostics

ERPL-Web includes powerful tracing capabilities to help you debug network calls, optimize performance, and troubleshoot issues. See exactly what's happening under the hood.

## Overview

Tracing captures detailed information about:

*   HTTP requests and responses
*   URL construction and query parameters
*   Request/response timing
*   Pagination and chunking
*   Retry logic and error handling
*   OData metadata extraction
*   Datasphere endpoint selection
*   ODP subscription state changes

**When to use tracing:**

*   ✅ Debugging authentication issues
*   ✅ Understanding slow queries
*   ✅ Troubleshooting API errors
*   ✅ Optimizing data extraction
*   ✅ Learning how features work

---

## Quick Start

```
-- Enable tracingSET erpl_trace_enabled = TRUE;-- Run your querySELECT * FROM http_get('https://api.example.com/data');-- Check console output for trace logs
```

That's it! Trace information appears in your console.

---

## Configuration

### Enable/Disable Tracing

```
-- Enable tracingSET erpl_trace_enabled = TRUE;-- Disable tracingSET erpl_trace_enabled = FALSE;-- Check current settingSELECT current_setting('erpl_trace_enabled');
```

---

### Trace Levels

Control the verbosity of trace output:

```
SET erpl_trace_level = 'DEBUG';
```

**Available Levels:**

*   `TRACE` - Most verbose, every detail
*   `DEBUG` - Detailed information for debugging
*   `INFO` - General informational messages
*   `WARN` - Warnings and potential issues
*   `ERROR` - Only errors

**Example:**

```
-- Maximum detailSET erpl_trace_level = 'TRACE';-- Moderate detail (recommended)SET erpl_trace_level = 'DEBUG';-- Minimal outputSET erpl_trace_level = 'ERROR';
```

---

### Trace Output Destination

Choose where trace logs appear:

```
SET erpl_trace_output = 'both';
```

**Options:**

*   `console` - Print to console/terminal only
*   `file` - Write to log file only
*   `both` - Print to console AND write to file

**Example:**

```
-- Console only (default)SET erpl_trace_output = 'console';-- File only (for production)SET erpl_trace_output = 'file';-- Both (for debugging)SET erpl_trace_output = 'both';
```

---

### File Configuration

When using file output, configure the log file:

```
-- Set log file pathSET erpl_trace_file_path = './erpl_trace.log';-- Set maximum file size (bytes)SET erpl_trace_max_file_size = 10485760;  -- 10MB-- Enable log rotationSET erpl_trace_rotation = TRUE;
```

**File Settings Explained:**

*   `erpl_trace_file_path`: Where to write log file (default: `./erpl_trace.log`)
*   `erpl_trace_max_file_size`: Max file size before rotation (default: 10MB)
*   `erpl_trace_rotation`: Enable automatic rotation when file reaches max size

**Example with all file settings:**

```
SET erpl_trace_enabled = TRUE;SET erpl_trace_level = 'DEBUG';SET erpl_trace_output = 'file';SET erpl_trace_file_path = '/var/log/erpl/trace.log';SET erpl_trace_max_file_size = 52428800;  -- 50MBSET erpl_trace_rotation = TRUE;
```

---

## What Gets Traced

### HTTP Functions

```
SET erpl_trace_enabled = TRUE;SET erpl_trace_level = 'DEBUG';SELECT * FROM http_get('https://httpbun.com/json');
```

**Trace Output:**

```
[DEBUG] HTTP GET Request  URL: https://httpbun.com/json  Headers: {Accept: application/json}  Timeout: 30000ms[DEBUG] HTTP Response Received  Status: 200 OK  Content-Type: application/json  Content-Length: 429 bytes  Response Time: 245ms
```

---

### OData Functions

```
SET erpl_trace_enabled = TRUE;SET erpl_trace_level = 'DEBUG';SELECT * FROM odata_read('https://services.odata.org/TripPinRESTierService/People')LIMIT 5;
```

**Trace Output:**

```
[DEBUG] OData Request Initiated  URL: https://services.odata.org/TripPinRESTierService/People  Method: GET[DEBUG] OData Metadata Fetch  Metadata URL: https://services.odata.org/TripPinRESTierService/$metadata  Version: OData V4[DEBUG] OData Predicate Pushdown  $top: 5  Final URL: https://services.odata.org/TripPinRESTierService/People?$top=5[DEBUG] OData Type Mapping  UserName: Edm.String -> VARCHAR  FirstName: Edm.String -> VARCHAR  Age: Edm.Int32 -> INTEGER[DEBUG] OData Pagination  Current Page: 1  Records Fetched: 5  Has Next: false[DEBUG] OData Request Complete  Total Records: 5  Execution Time: 1247ms
```

---

### Datasphere Functions

```
SET erpl_trace_enabled = TRUE;SET erpl_trace_level = 'DEBUG';SELECT * FROM datasphere_show_spaces();
```

**Trace Output:**

```
[DEBUG] Datasphere Request  Function: datasphere_show_spaces  Tenant: acme-corp  Data Center: eu10[DEBUG] OAuth2 Token Check  Token Valid: true  Expires In: 2847 seconds  [DEBUG] DWAAS API Request  Endpoint: /dwaas-core/v1/spaces  Method: GET  [DEBUG] Datasphere Response  Status: 200  Spaces Found: 3  Response Time: 456ms
```

---

### ODP Functions

```
SET erpl_trace_enabled = TRUE;SET erpl_trace_level = 'DEBUG';SELECT * FROM odp_odata_read(  'https://sap/odata/Z_ODP_BW_1_SRV/FactsOf0D_NW_C01',  secret='sap_system');
```

**Trace Output:**

```
[DEBUG] ODP OData Request  Entity Set: FactsOf0D_NW_C01  Service: Z_ODP_BW_1_SRV[DEBUG] Subscription Check  Subscription ID: sub_abc123  Status: active  Last Delta Token: D20250914154609_000019000[DEBUG] ODP Delta Request  URL: https://sap/odata/Z_ODP_BW_1_SRV/FactsOf0D_NW_C01?$deltatoken=D20250914154609_000019000  [DEBUG] ODP Response Processing  Package Count: 3  Records Received: 452  New Records (N): 87  Updated Records: 342  Deleted Records (D): 23  New Delta Token: D20250914161234_000020000  [DEBUG] Subscription Update  Updated Delta Token in Database  [DEBUG] ODP Audit Log Written  Execution Time: 3456ms
```

---

## Reading Trace Files

### Tail Log File

```
# Watch logs in real-timetail -f ./erpl_trace.log# Last 100 linestail -n 100 ./erpl_trace.log# Filter for errorsgrep 'ERROR' ./erpl_trace.log
```

### Search for Specific Patterns

```
# Find all HTTP requestsgrep 'HTTP.*Request' ./erpl_trace.log# Find slow queries (> 5 seconds)grep 'Execution Time: [5-9][0-9][0-9][0-9]ms' ./erpl_trace.log# Find authentication issuesgrep -i 'auth\|401\|403' ./erpl_trace.log
```

### Analyze with DuckDB

```
-- Load log file into DuckDBCREATE TABLE trace_logs ASSELECT * FROM read_csv_auto('erpl_trace.log',   delim='|',  header=false,  columns={'timestamp': 'VARCHAR', 'level': 'VARCHAR', 'message': 'VARCHAR'});-- Find slowest operationsSELECT message, COUNT(*) as occurrencesFROM trace_logsWHERE message LIKE '%Execution Time:%'ORDER BY occurrences DESC;-- Errors by hourSELECT   DATE_TRUNC('hour', TRY_CAST(timestamp AS TIMESTAMP)) as hour,  COUNT(*) as error_countFROM trace_logsWHERE level = 'ERROR'GROUP BY hourORDER BY hour;
```

---

## Performance Analysis

### Identify Slow Queries

Enable tracing and look for timing information:

```
SET erpl_trace_enabled = TRUE;SET erpl_trace_level = 'DEBUG';-- Run your querySELECT * FROM datasphere_read_relational('SALES', 'LARGE_VIEW');-- Look for "Execution Time" in output
```

**Optimization Tips Based on Trace:**

*   High network time → Check network latency
*   High metadata fetch time → Metadata is cached after first call
*   High parsing time → Simplify JSON/XML structure
*   Multiple roundtrips → Use predicate pushdown

---

### Monitor Request Patterns

```
SET erpl_trace_enabled = TRUE;-- See pagination in actionSELECT * FROM odata_read('https://services.odata.org/V2/Northwind/Northwind.svc/Orders');
```

**Trace shows:**

```
[DEBUG] OData Pagination  Page 1: 100 records  Page 2: 100 records  Page 3: 100 records  Page 4: 30 records  Total: 330 records
```

**Optimization:** Use LIMIT to avoid fetching all pages:

```
SELECT * FROM odata_read(...) LIMIT 100;  -- Only fetch first page
```

---

### Analyze Retry Logic

```
SET erpl_trace_enabled = TRUE;SET erpl_trace_level = 'DEBUG';-- Simulate flaky networkSELECT * FROM http_get('https://unstable-api.example.com/data');
```

**Trace shows retries:**

```
[DEBUG] HTTP Request Failed  Status: 503 Service Unavailable  Attempt: 1/3  Retry in: 1000ms[DEBUG] HTTP Retry Attempt  Attempt: 2/3  [DEBUG] HTTP Request Succeeded  Status: 200 OK  Attempts: 2
```

---

## Troubleshooting with Traces

### Authentication Issues

**Enable tracing:**

```
SET erpl_trace_enabled = TRUE;SET erpl_trace_level = 'DEBUG';SELECT * FROM http_get('https://api.example.com/data');
```

**Look for:**

*   `401 Unauthorized` → Wrong credentials
*   `403 Forbidden` → Insufficient permissions
*   `Token Expired` → Need to refresh OAuth2 token
*   `Invalid Client` → Wrong CLIENT\_ID or CLIENT\_SECRET

---

### Network Issues

**Trace shows:**

```
[ERROR] Connection Refused  URL: https://api.example.com  Error: Connection timeout after 30000ms
```

**Solutions:**

*   Check network connectivity
*   Verify URL is correct
*   Check firewall rules
*   Increase timeout: `timeout := 60000`

---

### OData Metadata Issues

**Trace shows:**

```
[ERROR] OData Metadata Parse Error  URL: https://api.example.com/odata/$metadata  Error: Invalid XML
```

**Solutions:**

*   Verify service supports OData
*   Check if service is V2 or V4
*   Ensure metadata endpoint is accessible

---

### ODP Subscription Issues

**Trace shows:**

```
[WARN] ODP Subscription Not Found  Entity: FactsOf0D_NW_C01  Creating New Subscription...
```

**This is normal for first extraction. If it happens repeatedly:**

*   Check subscription storage
*   Verify database permissions
*   Check for subscription cleanup scripts

---

## Advanced Trace Analysis

### Custom Log Parser

```
import refrom datetime import datetimedef parse_trace_log(filepath):    """Parse ERPL trace log file"""    with open(filepath, 'r') as f:        for line in f:            match = re.match(r'\[(\w+)\] (.+)', line)            if match:                level, message = match.groups()                if 'Execution Time:' in message:                    time_ms = int(re.search(r'(\d+)ms', message).group(1))                    if time_ms > 5000:  # Slow query                        print(f"SLOW: {message}")parse_trace_log('erpl_trace.log')
```

### Trace Aggregation

```
-- Parse trace logs with DuckDBCREATE TABLE trace_parsed ASSELECT   regexp_extract(line, '\[(\w+)\]', 1) as level,  regexp_extract(line, 'Execution Time: (\d+)ms', 1)::INTEGER as exec_time_ms,  lineFROM read_csv_auto('erpl_trace.log', header=false, columns={'line': 'VARCHAR'});-- StatisticsSELECT   level,  COUNT(*) as count,  AVG(exec_time_ms) as avg_time,  MAX(exec_time_ms) as max_timeFROM trace_parsedWHERE exec_time_ms IS NOT NULLGROUP BY level;
```

---

## Production Considerations

### Log Rotation

```
-- Enable rotation to prevent disk space issuesSET erpl_trace_rotation = TRUE;SET erpl_trace_max_file_size = 52428800;  -- 50MB
```

**How rotation works:**

*   When log file reaches max size
*   Current file renamed to `erpl_trace.log.1`
*   New `erpl_trace.log` created
*   Old rotated files eventually purged (keep last 5)

---

### Performance Impact

Tracing has minimal overhead, but for production:

```
-- Production: Only log errorsSET erpl_trace_enabled = TRUE;SET erpl_trace_level = 'ERROR';SET erpl_trace_output = 'file';-- Development: Full debuggingSET erpl_trace_enabled = TRUE;SET erpl_trace_level = 'TRACE';SET erpl_trace_output = 'both';-- Optimal: Disable in production unless troubleshootingSET erpl_trace_enabled = FALSE;
```

---

### Centralized Logging

For production deployments, send logs to centralized system:

```
# Ship logs to ELK, Splunk, etc.tail -f /var/log/erpl/trace.log | logstash -f config.conf# Or use file rotation + log shipper# Filebeat, Fluentd, etc.
```

---

## Examples by Scenario

### Debug Authentication

```
SET erpl_trace_enabled = TRUE;SET erpl_trace_level = 'DEBUG';-- Test authenticationSELECT * FROM http_get(  'https://api.example.com/test',  auth := 'user:pass',  auth_type := 'BASIC');-- Check trace for:-- - Auth header construction-- - 401/403 responses-- - Token expiry messages
```

### Optimize Slow Query

```
SET erpl_trace_enabled = TRUE;SET erpl_trace_level = 'DEBUG';-- Run slow querySELECT * FROM datasphere_read_analytical('SALES', 'BIG_VIEW');-- Look for:-- - Network time-- - Metadata fetch time-- - Row processing time-- - Pagination count-- Then optimize based on findings
```

### Monitor API Rate Limits

```
SET erpl_trace_enabled = TRUE;-- Make multiple requestsSELECT * FROM http_get('https://api.example.com/data');SELECT * FROM http_get('https://api.example.com/more');-- Check trace for:-- - 429 Too Many Requests-- - Retry-After headers-- - Rate limit warnings
```

---

## Best Practices

### 1\. Enable for Debugging, Disable for Production

```
-- Debug modeSET erpl_trace_enabled = TRUE;SET erpl_trace_level = 'DEBUG';-- Production modeSET erpl_trace_enabled = FALSE;
```

### 2\. Use Appropriate Trace Levels

```
-- Too verbose (slow)SET erpl_trace_level = 'TRACE';-- Good for debuggingSET erpl_trace_level = 'DEBUG';-- ProductionSET erpl_trace_level = 'ERROR';
```

### 3\. Rotate Logs Regularly

```
SET erpl_trace_rotation = TRUE;SET erpl_trace_max_file_size = 10485760;  -- 10MB
```

### 4\. Sanitize Sensitive Data

Traces may contain URLs with parameters. Review logs before sharing:

```
[DEBUG] HTTP GET https://api.example.com/data?api_key=SECRET123
```

Consider filtering sensitive params from logs.

---

## Next Steps

*   Apply tracing to [HTTP Functions](/docs/erpl-web/http-functions.md)
*   Debug [OData](/docs/erpl-web/odata.md) queries
*   Monitor [Datasphere](/docs/erpl-web/datasphere.md) performance
*   Troubleshoot [ODP](/docs/erpl-web/odp-web.md) extractions

---

## Summary

ERPL-Web's tracing provides:

✅ **Detailed** - See every HTTP request and response  
✅ **Configurable** - Control verbosity and output  
✅ **Production Ready** - File rotation and log levels  
✅ **Actionable** - Performance metrics for optimization  
✅ **Comprehensive** - Covers all ERPL-Web features

Debug smarter, not harder with ERPL-Web tracing!

# Business Central

ERPL-Web turns a Business Central company into a DuckDB catalog. You can list companies and entities, pull data with predicate pushdown, or attach the whole company as a database.

## Discover

```
SELECT * FROM bc_show_companies();SELECT * FROM bc_show_entities(company => 'CRONUS Germany AG');SELECT * FROM bc_describe(  company => 'CRONUS Germany AG',  entity => 'customer');
```

## Read

```
SELECT no, name, balance_dueFROM bc_read(  company => 'CRONUS Germany AG',  entity => 'customer')WHERE balance_due > 0;
```

Company names are resolved to GUIDs automatically — no manual lookup required.

## Attach a company as a catalog

The most powerful pattern: turn a BC company into a queryable DuckDB database.

```
ATTACH 'CRONUS Germany AG' AS bc (TYPE business_central);SELECT no, name FROM bc.customer;SELECT * FROM bc.salesInvoice WHERE document_date > date '2026-01-01';SELECT c.name, COUNT(s.id) AS invoice_countFROM bc.customer c LEFT JOIN bc.salesInvoice s ON s.customerId = c.idGROUP BY c.name;
```

Predicate pushdown, `$expand` support, and OData V4 metadata discovery are all built in.

# Dataverse / Dynamics CRM

Read accounts, contacts, opportunities, and any custom entities from Microsoft Dataverse (the data backbone of Dynamics 365 Customer Engagement).

## Discover

```
SELECT * FROM crm_show_entities();SELECT * FROM crm_describe(entity => 'account');
```

## Read

```
SELECT name, revenue, industrycodeFROM crm_read(entity => 'account')WHERE statecode = 0;  -- Active accounts
```

## Expand related records

`$expand` is supported for related-entity navigation:

```
SELECT name, primarycontactid_valueFROM crm_read(  entity => 'account',  expand => 'primarycontactid');
```

Predicate pushdown applies to all standard scalar columns.

# Entra ID

Microsoft renamed Azure AD to Entra ID; ERPL-Web exposes its directory data through Microsoft Graph.

## Users

```
SELECT id, displayName, mail, userPrincipalName, departmentFROM graph_users()WHERE department = 'Finance';
```

## Sign-in logs

Useful for audit and security analytics:

```
SELECT userPrincipalName, ipAddress, status, createdDateTimeFROM graph_user_signin_logs()WHERE createdDateTime > now() - INTERVAL '24 hours'  AND status.errorCode != 0;
```

Both functions support predicate pushdown where the Graph API permits it.

# Excel Workbooks

ERPL-Web reads and writes Excel workbooks living in OneDrive or SharePoint — typed columns (dates inferred from number formats, doubles, timestamps, booleans), inline ranges, and full table-level operations.

## Read a worksheet

```
SELECT *FROM graph_excel_read(  site => 'finance.sharepoint.com',  workbook => 'sales-2026.xlsx',  worksheet => 'Pipeline');
```

## Attach a workbook by name

No GUIDs required — just the site and file name:

```
ATTACH 'sales-2026.xlsx' AS book (  TYPE excel_workbook,  SITE 'finance.sharepoint.com');SELECT * FROM book.Pipeline;SELECT region, SUM(amount) FROM book.Pipeline GROUP BY region;
```

## Write back

```
-- Append rows to a named tableSELECT graph_excel_write(  site => 'finance.sharepoint.com',  workbook => 'sales-2026.xlsx',  worksheet => 'Pipeline',  rows => (SELECT * FROM new_deals));-- Or use COPY TO for batch writesCOPY (SELECT * FROM new_deals)TO 'finance.sharepoint.com/sales-2026.xlsx/Pipeline' (FORMAT graph_excel_table);
```

See the [function reference](/docs/erpl-web/functions.md) for `graph_excel_tables`, `graph_excel_worksheets`, `graph_excel_range`, `graph_excel_read`, `graph_excel_write`, `graph_excel_delete_rows`.

# Outlook

The complete Outlook trio is exposed to DuckDB: emails, calendar events, and contacts. Lazy streaming and folder pagination mean tenant-scale reads stay efficient.

## Emails

```
-- All mail foldersSELECT * FROM graph_outlook_mail_folders();-- Messages from a specific folderSELECT subject, "from", receivedDateTimeFROM graph_outlook_emails(folder_name => 'Inbox')WHERE receivedDateTime > now() - INTERVAL '7 days';
```

## Calendar

```
SELECT subject, start_dateTime, end_dateTime, organizerFROM graph_calendar_events()WHERE start_dateTime > now()ORDER BY start_dateTime;
```

## Contacts

```
SELECT displayName, emailAddresses, companyNameFROM graph_contacts()WHERE companyName ILIKE '%acme%';
```

All three functions return strongly-typed columns and support filtering pushdown where the Graph API allows it.

# Microsoft Planner

Read your Planner data — plans, buckets, tasks — and create tasks in bulk straight from a SQL query.

## Read

```
SELECT * FROM graph_planner_plans();SELECT * FROM graph_planner_buckets(plan => 'Q2 Launch');SELECT title, percentComplete, due_dateFROM graph_planner_tasks(plan => 'Q2 Launch');
```

## Bulk-create tasks

Useful for backfilling work from a spreadsheet or upstream system:

```
SELECT graph_planner_create_task(  plan => 'Q2 Launch',  bucket => 'Engineering',  -- resolved by name, no GUID needed  title => task_name,  due_date => due_date)FROM new_tasks;
```

Bucket name resolution is built in — no need to fetch IDs first.

# SharePoint Lists

ERPL-Web exposes SharePoint lists as if they were SQL tables — with typed columns, predicate pushdown, and full read/write support. You can reference sites and lists by their human-readable names rather than GUIDs.

## Discover what's there

```
SELECT * FROM graph_show_sites();SELECT * FROM graph_show_lists(site => 'finance.sharepoint.com');SELECT * FROM graph_describe_list(  site => 'finance.sharepoint.com',  list => 'Contracts');
```

## Read a list

```
SELECT *FROM graph_sharepoint_list_read(  site => 'finance.sharepoint.com',  list => 'Contracts')WHERE Status = 'Active';
```

## Attach a site as a catalog

```
ATTACH 'finance.sharepoint.com' AS finance (TYPE sharepoint_lists);SELECT * FROM finance.Contracts;SELECT COUNT(*) FROM finance.Vendors WHERE Country = 'DE';
```

## Write back

```
-- Insert a new itemSELECT graph_sharepoint_create_item(  site => 'finance.sharepoint.com',  list => 'Contracts',  fields => { 'Title': 'ACME 2026', 'Status': 'Draft' });-- Bulk insert from a queryCOPY (SELECT name AS Title, 'Draft' AS Status FROM new_contracts)TO 'finance.sharepoint.com/Contracts' (FORMAT graph_sharepoint_list);
```

See the [function reference](/docs/erpl-web/functions.md) for the full SharePoint surface (`graph_show_lists`, `graph_describe_list`, `graph_sharepoint_list_read`, `graph_sharepoint_create_item`, `graph_sharepoint_update_items`, `graph_sharepoint_delete_rows`).

# Microsoft Teams

Read your Teams data — teams, channels, members, and channel messages — by human-readable names rather than GUIDs.

```
-- Teams you belong toSELECT * FROM graph_my_teams();-- Channels in a specific teamSELECT * FROM graph_team_channels(team => 'Sales EU');-- Members of a channelSELECT * FROM graph_team_members(team => 'Sales EU', channel => 'General');-- Recent channel messagesSELECT createdDateTime, "from", bodyFROM graph_channel_messages(  team => 'Sales EU',  channel => 'Announcements')WHERE createdDateTime > now() - INTERVAL '7 days';
```

Useful for collaboration analytics, audit trails, and lightweight reporting on team activity.

# ERPL Examples

Real-world examples using ERPL for on-premise SAP integration.

## RFC Examples

### Read Customer Data

```
SELECT * FROM sap_read_table('KNA1', MAX_ROWS => 100);
```

### Read Sales Orders

```
SELECT * FROM sap_read_table('VBAK', MAX_ROWS => 100);
```

### Call BAPI Function

```
SELECT * FROM sap_rfc_invoke(    'BAPI_FLIGHT_GETLIST',    path => '/FLIGHT_LIST');
```

### Get Function Metadata

```
SELECT * FROM sap_rfc_describe_function('BAPI_FLIGHT_GETLIST');
```

### Get Table Structure

```
SELECT * FROM sap_describe_fields('KNA1');
```

## BICS Examples

### Execute BW Query

```
-- Open a state, place a characteristic on rows, filter, then read by idSELECT state_id FROM sap_bics_begin('0D_NW_C01', id => 'q1');SELECT state_id FROM sap_bics_rows('q1', '0D_NW_PROD', op => 'SET');SELECT state_id FROM sap_bics_filter('q1', '0CALMONTH', '202401', op => 'SET');SELECT * FROM sap_bics_result('q1');
```

### List Available Cubes

```
SELECT * FROM sap_bics_show(obj_type => 'CUBE');
```

### Get Query Lineage

```
-- All lineage edges touching one BEx querySELECT *FROM sap_bics_lineage_edges()WHERE tgt_name = '0D_FC_NW_C01_Q0008' OR src_name = '0D_FC_NW_C01_Q0008';
```

## ODP Examples

### Delta Replication

```
-- First call: auto-DELTAINIT (full snapshot + delta-pointer registered).SELECT * FROM sap_odp_read_delta('BW', 'VBAK$F', 'NIGHTLY_ETL');-- Subsequent calls: only the changes since the previous call.SELECT * FROM sap_odp_read_delta('BW', 'VBAK$F', 'NIGHTLY_ETL');-- Release the cursor when the pipeline finishes (resumable on next run).PRAGMA sap_odp_close_delta_cursor('BW', 'NIGHTLY_ETL', 'VBAK$F');
```

### Full Replication

```
-- One-shot: opens a FULL cursor, streams, auto-closes.SELECT * FROM sap_odp_read_full('BW', 'VBAK$F');
```

### Check Subscriptions

```
-- ERPL-owned subscriptionsSELECT * FROM sap_odp_show_subscriptions();-- Every subscriber on a given source (cross-team visibility)SELECT * FROM sap_odp_get_subscriptions('BW', 'VBAK$F');
```

### Probe Before Extract

```
-- Cheap last-modified probe (no cursor side effects)SELECT * FROM sap_odp_get_last_modified('BW', 'VBAK$F');
```

### Preview Data

```
SELECT * FROM sap_odp_preview('BW', 'VBAK$F');
```

## Advanced Examples

### Cross-Protocol Data Integration

```
-- Combine RFC and BICS dataWITH erp_customers AS (  SELECT KUNNR AS customer_id, NAME1 AS customer_name  FROM sap_read_table('KNA1', MAX_ROWS => 1000)),bw_sales AS (  -- State 'q1' built above with sap_bics_begin/rows/filter  SELECT    "0D_NW_PROD" AS product,    "0D_NW_NETV" AS net_value  FROM sap_bics_result('q1'))SELECT  e.customer_id,  e.customer_name,  b.net_valueFROM erp_customers eJOIN bw_sales b ON e.customer_name = b.product;  -- illustrative join
```

### ODP Delta Pipeline

```
-- Daily delta replication pipeline (run on schedule from your orchestrator)WITH delta_data AS (  SELECT    *,    ODQ_CHANGEMODE,    CASE ODQ_CHANGEMODE      WHEN 'C' THEN 'INSERT'      WHEN 'U' THEN 'UPDATE'      WHEN 'D' THEN 'DELETE'      ELSE 'UNKNOWN'    END AS change_type  FROM sap_odp_read_delta('BW', 'VBAK$F', 'NIGHTLY_ETL'))SELECT  change_type,  COUNT(*) AS record_count,  MIN(ERDAT) AS earliest_date,  MAX(ERDAT) AS latest_dateFROM delta_dataGROUP BY change_typeORDER BY change_type;
```

### BICS Lineage Analysis

```
-- Aggregate edges related to one BEx query by source/target kindWITH query_edges AS (  SELECT src_kind, tgt_kind  FROM sap_bics_lineage_edges()  WHERE tgt_name = '0D_FC_NW_C01_Q0008' OR src_name = '0D_FC_NW_C01_Q0008'),lineage_analysis AS (  SELECT src_kind, tgt_kind, COUNT(*) AS flow_count  FROM query_edges  GROUP BY src_kind, tgt_kind)SELECT  src_kind,  tgt_kind,  flow_count,  ROUND(flow_count * 100.0 / SUM(flow_count) OVER(), 2) AS percentageFROM lineage_analysisORDER BY flow_count DESC;
```

### RFC Function Discovery

```
-- Discover BAPI functions (columns: FUNCNAME, GROUPNAME, APPL, HOST, STEXT)SELECT FUNCNAME, GROUPNAME, STEXTFROM sap_rfc_show_function(FUNCNAME => 'BAPI*')ORDER BY FUNCNAME;
```

### ODP Subscription Management

```
-- Reconcile ERPL-owned subscriptions against their cursorsWITH subs AS (  SELECT queue_name, subscriber_name, subscriber_proc  FROM sap_odp_show_subscriptions()),cursors AS (  SELECT subscriber_proc, pointer, is_closed, is_delta_extension, request_date  FROM sap_odp_show_cursors(subscriber_name => 'ERPL'))SELECT  s.queue_name,  s.subscriber_proc,  c.is_delta_extension,  c.is_closed,  c.request_date,  CURRENT_DATE - CAST(c.request_date AS DATE) AS days_since_last_use,  CASE    WHEN c.is_closed IS NULL                                       THEN 'NO_CURSOR'    WHEN c.is_closed                                                THEN 'CLOSED'    WHEN CURRENT_DATE - CAST(c.request_date AS DATE) > 7            THEN 'STALE'    WHEN CURRENT_DATE - CAST(c.request_date AS DATE) > 1            THEN 'RECENT'    ELSE                                                                  'CURRENT'  END AS usage_statusFROM subs sLEFT JOIN cursors c USING (subscriber_proc)ORDER BY days_since_last_use DESC NULLS LAST;
```

# ERPL-Web Examples

Real-world examples and patterns using ERPL-Web for API integration, OData services, SAP Datasphere, and delta replication.

---

## HTTP Examples

### Simple API Call

```
-- Get your public IP addressSELECT content::JSON->>'ip' AS my_ipFROM http_get('https://api.ipify.org?format=json');
```

### API with Authentication

```
-- Create secret for GitHub APICREATE SECRET github_api (  TYPE http_bearer,  token 'ghp_your_token_here');-- Get repository informationSELECT   content::JSON->>'name' AS repo_name,  content::JSON->>'stargazers_count' AS stars,  content::JSON->>'forks_count' AS forksFROM http_get('https://api.github.com/repos/duckdb/duckdb');
```

### POST JSON Data

```
-- Send webhook notificationSELECT status, contentFROM http_post(  'https://webhook.site/your-unique-url',  json_object(    'event', 'data_loaded',    'timestamp', current_timestamp,    'records', 12345  )::VARCHAR,  content_type := 'application/json');
```

### API Pagination

```
-- Fetch paginated API resultsWITH page1 AS (  SELECT content::JSON AS data  FROM http_get('https://api.example.com/users?page=1&limit=100')),page2 AS (  SELECT content::JSON AS data  FROM http_get('https://api.example.com/users?page=2&limit=100'))SELECT * FROM page1UNION ALLSELECT * FROM page2;
```

### ETL from API to Parquet

```
-- Extract weather data and save to ParquetCOPY (  SELECT     content::JSON->'location'->>'name' AS city,    content::JSON->'current'->>'temp_c' AS temperature_celsius,    content::JSON->'current'->>'condition'->>'text' AS condition,    current_timestamp AS extracted_at  FROM http_get('https://api.weatherapi.com/v1/current.json?q=London&key=YOUR_KEY')) TO 'weather_data.parquet' (FORMAT PARQUET);
```

---

## OData Examples

### Attach and Query OData Service

```
-- Attach TripPin V4 serviceATTACH 'https://services.odata.org/TripPinRESTierService'   AS trippin (TYPE odata);-- List available tablesSHOW TABLES;-- Query peopleSELECT UserName, FirstName, LastName, GenderFROM trippin.PeopleWHERE Gender = 'Female'LIMIT 10;-- Query with filtersSELECT * FROM trippin.AirlinesWHERE Name LIKE '%American%';
```

### Northwind V2 Service

```
-- Attach Northwind V2 serviceATTACH 'https://services.odata.org/V2/Northwind/Northwind.svc'   AS northwind (TYPE odata);-- Query customers by countrySELECT CustomerID, CompanyName, ContactName, CountryFROM northwind.CustomersWHERE Country IN ('Germany', 'France', 'UK')ORDER BY Country, CompanyName;-- Join customers and ordersSELECT   c.CompanyName,  COUNT(o.OrderID) as order_count,  SUM(o.Freight) as total_freightFROM northwind.Customers cLEFT JOIN northwind.Orders o ON c.CustomerID = o.CustomerIDGROUP BY c.CompanyNameORDER BY order_count DESCLIMIT 10;
```

### Direct OData Read

```
-- Read specific entity set without ATTACHSELECT   UserName,  FirstName,  LastName,  FavoriteFeatureFROM odata_read('https://services.odata.org/TripPinRESTierService/People')WHERE FavoriteFeature IS NOT NULLLIMIT 5;
```

### Export OData to CSV

```
-- Extract OData data to CSVCOPY (  SELECT * FROM northwind.Products  WHERE UnitsInStock < 20) TO 'low_stock_products.csv' (HEADER, DELIMITER ',');
```

---

## SAP Datasphere Examples

### Setup and Discovery

```
-- Create OAuth2 secretCREATE SECRET datasphere (  TYPE datasphere,  PROVIDER oauth2,  tenant_name 'acme-corp',  data_center 'eu10');-- List all spacesSELECT * FROM datasphere_show_spaces();-- List assets in SALES spaceSELECT name, object_type, technical_nameFROM datasphere_show_assets('SALES')ORDER BY name;
```

### Query Relational Data

```
-- Read customer master dataSELECT   customer_id,  customer_name,  country,  revenueFROM datasphere_read_relational('SALES', 'CUSTOMER_MASTER_V')WHERE country = 'United States'ORDER BY revenue DESCLIMIT 100;
```

### Query Analytical Data

```
-- Read sales analytics with specific metrics and dimensionsSELECT   region,  product_category,  SUM(total_revenue) as revenue,  SUM(total_cost) as cost,  SUM(total_revenue - total_cost) as profitFROM datasphere_read_analytical(  'SALES',  'REVENUE_ANALYTICS_V',  metrics := ['TotalRevenue', 'TotalCost'],  dimensions := ['Region', 'ProductCategory'])GROUP BY region, product_categoryORDER BY profit DESC;
```

### Parameterized Views

```
-- Query with input parametersSELECT *FROM datasphere_read_relational(  'SALES',  'MONTHLY_SALES_V',  params := {    'P_YEAR': '2024',    'P_MONTH': '03',    'P_REGION': 'EMEA'  });
```

### Multi-Tenant Access

```
-- Create secrets for multiple tenantsCREATE SECRET datasphere_prod (  TYPE datasphere,  PROVIDER oauth2,  tenant_name 'acme-prod',  data_center 'eu10');CREATE SECRET datasphere_dev (  TYPE datasphere,  PROVIDER oauth2,  tenant_name 'acme-dev',  data_center 'eu10');-- Query production tenantSELECT * FROM datasphere_show_assets(secret := 'datasphere_prod');-- Query development tenantSELECT * FROM datasphere_show_assets(secret := 'datasphere_dev');
```

---

## ODP via OData Examples

### Complete Delta Replication Workflow

```
-- 1. Setup authenticationCREATE SECRET sap_system (  TYPE http_basic,  username 'SAP_USER',  password 'SAP_PASSWORD');-- 2. Discover available ODP servicesSELECT   service_name,  entity_set_name,  descriptionFROM odp_odata_show('https://sap-server:8000', secret='sap_system')WHERE entity_set_name LIKE '%SALES%';-- 3. Create target tableCREATE TABLE sales_facts (  sales_order VARCHAR,  customer_id VARCHAR,  product_id VARCHAR,  quantity INTEGER,  revenue DECIMAL(15,2),  order_date DATE,  RECORD_MODE VARCHAR);-- 4. Initial load (creates subscription)INSERT INTO sales_factsSELECT * FROM odp_odata_read(  'https://sap-server:8000/sap/opu/odata/sap/Z_ODP_BW_1_SRV/FactsOf0D_NW_C01',  secret='sap_system');-- 5. Check subscription statusSELECT * FROM odp_odata_list_subscriptions();-- 6. Delta update (run periodically)WITH delta AS (  SELECT * FROM odp_odata_read(    'https://sap-server:8000/sap/opu/odata/sap/Z_ODP_BW_1_SRV/FactsOf0D_NW_C01',    secret='sap_system'  ))-- Process insertsINSERT INTO sales_factsSELECT * FROM delta WHERE RECORD_MODE = 'N';-- Update existing recordsUPDATE sales_facts tSET   quantity = d.quantity,  revenue = d.revenueFROM delta dWHERE t.sales_order = d.sales_order AND d.RECORD_MODE = '';-- Delete removed recordsDELETE FROM sales_factsWHERE sales_order IN (  SELECT sales_order FROM delta WHERE RECORD_MODE = 'D');
```

### Monitor Extraction Performance

```
-- View extraction statisticsSELECT   subscription_id,  entity_set_name,  request_timestamp,  request_type,  records_received,  execution_time_ms / 1000.0 AS execution_time_secondsFROM erpl_web.odp_subscription_auditORDER BY request_timestamp DESCLIMIT 20;-- Average performance by subscriptionSELECT   subscription_id,  entity_set_name,  COUNT(*) as extraction_count,  AVG(records_received) as avg_records,  AVG(execution_time_ms / 1000.0) as avg_time_secondsFROM erpl_web.odp_subscription_auditWHERE request_type = 'DELTA'GROUP BY subscription_id, entity_set_name;
```

### Export to Data Lake

```
-- Extract SAP data to Parquet with partitioningCOPY (  SELECT     *,    DATE_TRUNC('day', order_date) as partition_date  FROM odp_odata_read(    'https://sap/odata/SALES_DATA',    secret='sap_system'  )) TO 'datalake/sales_data.parquet' (  FORMAT PARQUET,  PARTITION_BY (partition_date),  COMPRESSION 'ZSTD');
```

---

## Integration Patterns

### Federated Analytics

```
-- Combine Datasphere, OData, and HTTP sourcesWITH   datasphere_sales AS (    SELECT customer_id, SUM(revenue) as ds_revenue    FROM datasphere_read_relational('SALES', 'TRANSACTIONS')    GROUP BY customer_id  ),  crm_data AS (    SELECT       content::JSON->>'id' AS customer_id,      content::JSON->>'score' AS crm_score    FROM http_get('https://crm-api.example.com/customers')  ),  odata_customers AS (    SELECT CustomerID, CompanyName    FROM northwind.Customers  )SELECT   oc.CompanyName,  ds.ds_revenue,  cd.crm_scoreFROM odata_customers ocLEFT JOIN datasphere_sales ds ON oc.CustomerID = ds.customer_idLEFT JOIN crm_data cd ON oc.CustomerID = cd.customer_idORDER BY ds.ds_revenue DESCLIMIT 50;
```

### Incremental Data Sync

```
-- Sync API data incrementallyCREATE TABLE api_cache (  id VARCHAR PRIMARY KEY,  data JSON,  last_updated TIMESTAMP);-- Initial loadINSERT INTO api_cacheSELECT   content::JSON->>'id' AS id,  content::JSON AS data,  CURRENT_TIMESTAMPFROM http_get('https://api.example.com/data');-- Incremental update (only new records)INSERT INTO api_cacheSELECT   content::JSON->>'id' AS id,  content::JSON AS data,  CURRENT_TIMESTAMPFROM http_get('https://api.example.com/data?since=' || (  SELECT MAX(last_updated) FROM api_cache)::VARCHAR)ON CONFLICT (id) DO UPDATE SET  data = EXCLUDED.data,  last_updated = EXCLUDED.last_updated;
```

### API Gateway Pattern

```
-- Create views that wrap external APIsCREATE VIEW github_trending ASSELECT   content::JSON->'items'[1]->>'name' AS repo_name,  content::JSON->'items'[1]->>'stargazers_count' AS stars,  content::JSON->'items'[1]->>'html_url' AS urlFROM http_get('https://api.github.com/search/repositories?q=stars:>10000&sort=stars');-- Query the view like a tableSELECT * FROM github_trending;
```

---

## Troubleshooting Examples

### Debug with Tracing

```
-- Enable detailed tracingSET erpl_trace_enabled = TRUE;SET erpl_trace_level = 'DEBUG';SET erpl_trace_output = 'both';SET erpl_trace_file_path = './debug.log';-- Run problematic querySELECT * FROM datasphere_show_spaces();-- Check trace file for details
```

### Test Authentication

```
-- Test HTTP authSELECT status, contentFROM http_get(  'https://api.example.com/test',  auth := 'user:pass',  auth_type := 'BASIC');-- Expected: status = 200-- If 401: Wrong credentials-- If 403: Insufficient permissions
```

### Verify OData Metadata

```
-- Attach serviceATTACH 'https://your-odata-service.com/odata' AS test (TYPE odata);-- List tables to verify metadata loadedSHOW TABLES;-- Query first few rowsSELECT * FROM test.EntitySet LIMIT 5;
```

---

## Next Steps

*   Read detailed documentation:
    
    *   [HTTP Functions](/docs/erpl-web/http-functions.md)
    *   [OData Guide](/docs/erpl-web/odata.md)
    *   [Datasphere Guide](/docs/erpl-web/datasphere.md)
    *   [ODP via OData](/docs/erpl-web/odp-web.md)
*   Check [Functions Reference](/docs/reference/erpl-web-functions.md) for complete API
    
*   See [Real-World Use Cases](/docs/examples/real-world-use-cases.md) for more patterns

# real-world-use-cases

# Connect

## Connection Setup

This page explains how to connect to SAP systems using the ERPL extension.

\[Continue reading...\]

# Installation

## Introduction

**Work in progress:**

This section is still work in progress. We are working hard to provide you with the best possible experience. If you have any questions, please do not hesitate to [contact](/contact) us.

The ERPL is essentially an extension for the [DuckDB](https://duckdb.org) in-process SQL OLAP database management system. This means that you need to install DuckDB first before you can use the ERPL. You can find links to the binary packages and installation instructions for DuckDB in the [original documentation](https://duckdb.org/docs/installation/).

DuckDB provides an [extension system](https://duckdb.org/docs/extensions/overview) which enables to extend the functionality of the database in various directions. The extension are distributed as packages for the platforms Windows, Linux, and macOS. We try to mimic the installation procedure of DuckDB as good as possible.

**Note:**

At this point of time, **ERPL** supports version **0.10.3** and **1.0.0** of DuckDB.

## ➜ Obtaining the ERPL Extension

### Introduction

Building extensions for DuckDB can be challenging due to the varying C++ compiler and library ecosystem. This variability often leads to incompatibilities between locally built extensions and the centrally distributed DuckDB binary, primarily due to differences in the Application Binary Interface (ABI).

### Recommended Approach: Install the Extension directly in DuckDB

The easiest way of installing our SAP DuckDB extension is using the `httpfs` extension.

```
SET custom_extension_repository = 'http://get.erpl.io';FORCE INSTALL erpl;
```

**Note:**

Currently, it is necessary to delete the extension folder when there is a new release of our extension if `FORCE` is not used.

To ensure compatibility and ease of use, we follow a build process similar to that of the DuckDB team. **Our advice is to start with the pre-compiled binaries available in our [GitHub releases](https://github.com/DataZooDE/erpl/releases).** For those interested in building the extension themselves, our [development instructions](https://github.com/DataZooDE/erpl) provide detailed guidance.

### Binary Selection

The assets in each release follow this naming convention:

```
erpl-${DUCKDB_VERSION}-extension-{OS}-{ARCH}.tar.gz
```

Choose the binary that matches your usage scenario. The table below summarizes the available binaries for various platforms and use cases:

| DuckDB Version | Operating System | Architecture | Download Link |
| --- | --- | --- | --- |
| 0.10.3 | Linux | amd64 | [Download](https://github.com/DataZooDE/erpl/actions/runs/9422757678/artifacts/1580606856) |
| 0.10.3 | Linux | amd64 (GCC4) | [Download](https://github.com/DataZooDE/erpl/actions/runs/9422757678/artifacts/1580606859) |
| 0.10.3 | Windows | amd64 | [Download](https://github.com/DataZooDE/erpl/actions/runs/9422757678/artifacts/1580606864) |
| 1.0.0 | Linux | amd64 | [Download](https://github.com/DataZooDE/erpl/actions/runs/9422757678/artifacts/1580606865) |
| 1.0.0 | Linux | amd64 (GCC4) | [Download](https://github.com/DataZooDE/erpl/actions/runs/9422757678/artifacts/1580606867) |
| 1.0.0 | Windows | amd64 | [Download](https://github.com/DataZooDE/erpl/actions/runs/9534891222/artifacts/1605916851) |
| 1.0.0 | Mac OS | arm | [Download](https://github.com/DataZooDE/erpl/actions/runs/9534891222/artifacts/1605916852) |
| 1.0.0 | Mac OS | amd64 | [Download](https://github.com/DataZooDE/erpl/actions/runs/9422757678/artifacts/1580606868) |

### Note on OSX Support

Currently, we support OSX with Mac OS arm and amd64 architectures. Check the download table above for the available binaries for your system.

This revised section aims to provide a clearer, more user-friendly explanation of how to acquire and choose the appropriate ERPL extension, along with a straightforward guide for those interested in building the extension themselves.

## 💻 Installing the ERPL Binaries

### Introduction

Installation of the ERPL extension is straightforward. Please note that this extension is independent of the [DuckDB Foundation](https://duckdb.org/foundation/) and [DuckDB Labs](https://duckdblabs.com/), meaning the binaries are unsigned. Consequently, DuckDB must be initiated with the `-unsigned` flag. Detailed instructions on this process can be found in the [DuckDB documentation](https://duckdb.org/docs/archive/0.9.2/extensions/overview#ensuring-the-integrity-of-extensions).

### Installation Steps

1.  **Enable Unsigned Extensions in DuckDB**: Set the `-unsigned` flag as described in the DuckDB documentation.
    
2.  **Install and Load the ERPL Extension**:
    
    ```
    INSTALL 'path/to/erpl.duckdb_extension';LOAD 'erpl';
    ```
    

### Confirmation of Successful Installation

Upon successful installation and loading, the extension will output the following message:

```
-- Loading ERPL Trampoline Extension. --(Saves ERPL SAP dependencies to '/home/jr/.duckdb/extensions/v0.9.2/linux_amd64' and loads them)ERPL extension saved and loaded from /home/jr/.duckdb/extensions/v0.9.2/linux_amd64/erpl_impl.duckdb_extension.For usage instructions, visit https://erpl.io
```

### Understanding the Extension Loading Process

The ERPL extension is composed of two parts: 1. **Trampoline Extension**: Extracts SAP Netweaver RFC SDK and SAP Business Warehouse BICS libraries from the binary. 2. **Implementation Extension**: The actual functional part of the extension.

The `erpl_init` function in the trampoline extension bundles and extracts dependencies into the DuckDB extension folder. Post-installation, the directory `~/.duckdb/extensions/v0.10.1/linux_amd64` should contain the following files:

```
-rw-r--r-- 1 jr jr 110M 26. Nov 10:23 erpl.duckdb_extension-rw-r--r-- 1 jr jr  34M 26. Nov 10:35 erpl_impl.duckdb_extension-rw-r--r-- 1 jr jr  20M 26. Nov 10:35 libicudata.so.50-rw-r--r-- 1 jr jr  12M 26. Nov 10:35 libicui18n.so.50-rw-r--r-- 1 jr jr 8,4M 26. Nov 10:35 libicuuc.so.50-rw-r--r-- 1 jr jr 9,5M 26. Nov 10:35 libsapnwrfc.so-rw-r--r-- 1 jr jr 1,1M 26. Nov 10:35 libsapucum.so
```

This revised section aims for a clearer, more structured presentation of the installation process, ensuring users can easily understand and follow the steps.

## Adding a license key for BICS and ODP

**Work in progress:**

This section is still work in progress. We are working hard to provide you with the best possible experience. If you have any questions, please do not hesitate to [contact](/contact) us.

## Manual Installation: Total Control

For those who like to roll up their sleeves, we also offer comprehensive guides for manual installation. This is perfect for users who need more control over their installation, prefer specific versions of DuckDB, or are working in environments with unique constraints. We'll walk you through the entire process, from downloading the source code to the final setup, ensuring that you're comfortable at every step.

# Quick Start Guide

Welcome to ERPL! This guide will help you get started with connecting DuckDB to your SAP ecosystem in just a few minutes.

## Prerequisites

Before you begin, ensure you have:

*   **DuckDB 0.10.1** installed on your system
*   Access to an **SAP ERP system** with RFC enabled
*   A **valid SAP user account** with appropriate authorizations
*   **Network connectivity** to your SAP system

## Installation

### Step 1: Install DuckDB

If you haven't already, install DuckDB following the [official installation guide](https://duckdb.org/docs/installation/).

### Step 2: Install ERPL Extension

Install the ERPL extension using DuckDB's extension system:

```
SET custom_extension_repository = 'http://get.erpl.io';INSTALL 'erpl';LOAD 'erpl';
```

### Step 3: Verify Installation

Confirm that ERPL is properly installed:

```
SELECT * FROM duckdb_extensions() WHERE extension_name = 'erpl';
```

You should see the ERPL extension listed.

## Your First Connection

### Connect to SAP ERP

ERPL stores SAP credentials as a DuckDB secret. The connection is established lazily on the first call.

```
CREATE SECRET my_sap (  TYPE sap_rfc,  ASHOST 'your-sap-server.com',  SYSNR '00',  CLIENT '100',  USER 'your-username',  PASSWD 'your-password',  LANG 'EN');-- Validate the connectionPRAGMA sap_rfc_ping;
```

### List Available Tables

Explore what's available in your SAP system:

```
SELECT * FROM sap_show_tables()WHERE table_name LIKE '%KNA1%';
```

### Query Your First Table

Load customer master data from the KNA1 table:

```
SELECT     KUNNR as customer_number,    NAME1 as customer_name,    LAND1 as country,    ORT01 as cityFROM sap_read_table('KNA1')LIMIT 10;
```

## Next Steps

Congratulations! You've successfully connected DuckDB to SAP using ERPL. Here's what you can do next:

1.  **Explore More Tables**: Use `sap_show_tables()` to discover available data
2.  **Read Documentation**: Check out our [Key Tasks](/docs/key_tasks.md) for common use cases
3.  **Join the Community**: Connect with other users on [GitHub](https://github.com/datazoode/erpl)

## Troubleshooting

### Common Issues

**Connection Failed**

*   Verify your SAP system details (host, system number, client)
*   Check network connectivity
*   Ensure your SAP user has RFC authorizations

**Permission Denied**

*   Contact your SAP administrator to grant RFC permissions
*   Verify your user account is active

**Extension Not Found**

*   Ensure you're using DuckDB 0.10.1
*   Check your internet connection for extension download

### Getting Help

If you encounter issues:

1.  Check our [troubleshooting guide](/docs/get_started/troubleshooting.md)
2.  Search existing [GitHub issues](https://github.com/datazoode/erpl/issues)
3.  [Contact our support team](/contact)

## What's Next?

Now that you have ERPL running, explore these advanced topics:

*   [Installation Guide](/docs/get_started/install.md) - Detailed installation instructions
*   [Connection Configuration](/docs/get_started/connect.md) - Advanced connection options
*   [Key Tasks](/docs/key_tasks.md) - Common SAP integration scenarios
*   [SQL Reference](/docs/reference.md) - Complete function reference

# ERPL Quick Start (On-Premise)

In this 5-minute tutorial, you'll connect to an on-premise SAP system and read your first table. By the end, you'll be querying SAP data directly from DuckDB.

**Prerequisites:**

*   DuckDB installed (version 0.10.0+)
*   Access to an on-premise SAP system
*   SAP connection credentials (username/password)
*   Network access to SAP system (port 33xx)

## Step 1: Install ERPL

```
-- Install ERPL extensionINSTALL 'erpl' FROM 'http://get.erpl.io';-- Load the extensionLOAD 'erpl';
```

**Success Check:**

If installation worked, you should see no errors. The extension is now loaded and ready to use.

## Step 2: Connect to SAP

```
-- Store SAP credentials in a DuckDB secret of type sap_rfcCREATE SECRET sap_system (  TYPE   sap_rfc,  ASHOST 'your-sap-host',  SYSNR  '00',  CLIENT '100',  USER   'your_username',  PASSWD 'your_password',  LANG   'EN');
```

**Connection Details:**

Replace the connection details:

*   `ASHOST` - Your SAP application server hostname or IP
*   `SYSNR` - Your SAP system (instance) number, e.g. `00`
*   `CLIENT` - Your SAP client (mandant), e.g. `100`
*   `USER` / `PASSWD` - Your SAP username and password

The functions below pick up a single secret automatically; with several secrets, pass `secret => 'sap_system'` to any function.

### What's Happening?

## Step 3: Read Your First Table

```
-- Read customer master dataSELECT * FROM sap_read_table('KNA1', MAX_ROWS => 10);
```

### Understanding the Result

The `KNA1` table contains customer master data. You should see columns like:

*   `KUNNR` - Customer number
*   `NAME1` - Customer name
*   `LAND1` - Country
*   `REGIO` - Region

## Step 4: Filter and Transform Data

```
-- Get German customers onlySELECT   KUNNR AS customer_id,  NAME1 AS customer_name,  LAND1 AS country,  REGIO AS regionFROM sap_read_table('KNA1')WHERE LAND1 = 'DE'LIMIT 100;
```

## Step 5: Explore More Tables

Here are some common SAP tables you can explore:

```
-- Sales document headerSELECT * FROM sap_read_table('VBAK', MAX_ROWS => 5);-- Material master dataSELECT * FROM sap_read_table('MARA', MAX_ROWS => 5);-- Sales document itemsSELECT * FROM sap_read_table('VBAP', MAX_ROWS => 5);
```

## Common SAP Tables Reference

| Table | Description | Key Fields |
| --- | --- | --- |
| `KNA1` | Customer Master Data | KUNNR, NAME1, LAND1 |
| `VBAK` | Sales Document Header | VBELN, ERDAT, KUNNR |
| `VBAP` | Sales Document Items | VBELN, POSNR, MATNR |
| `MARA` | Material Master Data | MATNR, MTART, MEINS |
| `LFA1` | Vendor Master Data | LIFNR, NAME1, LAND1 |
| `BKPF` | Accounting Document Header | BUKRS, BELNR, GJAHR |

## Troubleshooting

### Connection Issues

**Error: "Connection refused"**

```
-- Check that ASHOST/SYSNR are correct, then ping the systemPRAGMA sap_rfc_ping;
```

**Error: "Authentication failed"**

```
-- Re-create the secret with the correct credentialsCREATE OR REPLACE SECRET sap_system (  TYPE   sap_rfc,  ASHOST 'your-host',  SYSNR  '00',  CLIENT '100',  USER   'correct_username',  PASSWD 'correct_password',  LANG   'EN');
```

### Table Access Issues

**Error: "Table not found"**

```
-- Check table name (case-sensitive)-- Verify table exists in your SAP systemSELECT * FROM sap_read_table('KNA1', MAX_ROWS => 1);
```

## Next Steps

### 🚀 Ready for More?

*   [Read SAP Tables Guide](/docs/guides/simple/read-sap-table.md) - Detailed table reading
*   [RFC Deep Dive](/docs/erpl/rfc.md) - Advanced RFC features
*   [Run BW Queries](/docs/guides/simple/run-bw-query.md) - Execute SAP BW queries

### 🔧 Advanced Topics

*   [RFC Metadata](/docs/guides/advanced/rfc-metadata.md) - For SAP experts
*   [Performance Tuning](/docs/guides/advanced/performance-tuning.md) - Optimize queries
*   [Function Reference](/docs/reference/erpl-functions.md) - Complete API docs

### 💡 Examples

*   [ERPL Examples](/docs/examples/erpl-examples.md) - More real-world examples
*   [Integration with Python](/docs/guides/integration/python-pandas.md) - Use with Pandas

## What You've Learned

✅ **Installed ERPL extension**  
✅ **Connected to SAP system**  
✅ **Read SAP tables**  
✅ **Filtered and transformed data**  
✅ **Explored common SAP tables**

You're now ready to use ERPL for your SAP data analysis needs!

---

**Need help?** Check our [troubleshooting guide](/docs/reference/troubleshooting.md) or browse [more examples](/docs/examples/erpl-examples.md).

# ERPL-ADT Quick Start (CLI + MCP)

In this 5-minute tutorial you'll install erpl-adt, save your SAP credentials, search ABAP objects, read source with syntax highlighting, and wire the MCP server into Claude Code so your AI agent can do all of the above.

**Prerequisites:**

*   A SAP ABAP system reachable by HTTPS (ABAP Cloud Developer Trial, S/4HANA, BW/4HANA, or a partner sandbox — anything that exposes the ADT REST API)
*   Login credentials for that system
*   For Step 5: an MCP-capable client like Claude Code, Cursor, or Gemini CLI

## Step 1: Install ERPL-ADT

The fastest path — no install:

```
uvx erpl-adt --help
```

Or install permanently:

```
pip install erpl-adt
```

Or download a static binary from the [latest release](https://github.com/datazooDE/erpl-adt/releases/latest) (Linux x86\_64, macOS arm64/x86\_64, Windows x64).

**Success Check:**

`erpl-adt --version` prints a date-based version like `2026.05.16`. The binary is statically linked — no JVM, no SAP NW RFC SDK, no Eclipse.

## Step 2: Save Your Connection

Run `login` once. It writes `~/.adt.creds` (chmod 600) so you don't have to repeat the connection flags every time.

```
erpl-adt login \  --host sap.example.com \  --port 44300 \  --https \  --user DEVELOPER# (prompts for the password)
```

For CI or one-off calls, skip `login` and pass the password via env:

```
SAP_PASSWORD='…' erpl-adt search 'ZCL_*' --host sap.example.com --https
```

Want object descriptions in another logon language? Add the global `--language` flag (2-letter ISO, default `EN`) to any command — it's sent as the SAP logon language, so descriptions come back translated:

```
erpl-adt search 'ZCL_*' --language DE
```

**Success Check:**

`erpl-adt discover services` returns a list of ADT services. If you see an authentication error, verify the password; if you see a TLS warning, add `--insecure` for self-signed certificates (dev only).

## Step 3: Search ABAP Objects

The smallest useful command: pattern + type + max.

```
erpl-adt search 'CL_DEMO_*' --type CLAS --max 8
```

You'll see a table like this:

![erpl-adt search demo](/assets/images/search-a45efb2d7888bb582738d7b73ea93f1c.gif)

Add `--json` if you want to pipe into `jq`:

```
erpl-adt --json search 'CL_DEMO_*' --type CLAS --max 8 \  | jq '.[] | {name, package}'
```

**Success Check:**

The Name, Type, Package, and Description columns line up and the bold header is rendered correctly. If columns look squashed, your terminal is narrow — try `--json` instead.

## Step 4: Read ABAP Source with Syntax Highlighting

```
erpl-adt source read CL_DEMO_OUTPUT --color
```

Cyan keywords, green string literals, dimmed comments — the same defaults the Eclipse editor uses.

For data-dictionary inspection:

```
erpl-adt ddic table SFLIGHT
```

The output resolves the `AbapType` and `CheckTable` columns automatically (one extra lookup per data element). Add `--no-resolve-types` for a fast, offline listing if you only care about field names and types.

For a package walkthrough:

```
erpl-adt package tree SABAP_DEMOS_OUTPUT_STREAM --max-depth 2
```

**Success Check:**

Source output is colorized when stdout is a TTY. Piping to a file disables color automatically; force-enable with `--color` or pass `--editor` to open in `$VISUAL`/`$EDITOR`.

## Step 5: Plug ERPL-ADT into Claude Code (MCP)

Same binary, different entry point. Drop this into `~/.claude/mcp.json` (or your project's `.claude/mcp.json`):

```
{  "mcpServers": {    "sap": {      "command": "erpl-adt",      "args": ["mcp", "--host", "sap.example.com", "--port", "44300", "--https"],      "env": { "SAP_PASSWORD": "…" }    }  }}
```

Restart Claude Code. The `sap` tools appear in the tool list — `adt_search`, `adt_read_source`, `adt_run_tests`, `bw_lineage_graph`, etc. Ask the agent:

> What flight-related classes exist in this system, and which ones have failing unit tests?

The agent picks the right MCP tools, calls them in order, and reports back.

For the full MCP setup (Cursor, Gemini, tool catalogue), see the [MCP guide](/docs/erpl-adt/mcp.md).

**Success Check:**

In Claude Code's status bar the `sap` MCP server shows a green dot. Type `/mcp` to see the tool list. If the dot is red, run `erpl-adt mcp --host …` manually in a terminal — startup errors surface immediately.

## Where to Next

*   [Overview](/docs/erpl-adt.md) — the full feature surface and motivation
*   [Metadata Catalog](/docs/erpl-adt/catalog.md) — build a cross-domain DuckDB catalog with search, lineage, and a web explorer
*   [BW/4HANA Modeling](/docs/erpl-adt/bw.md) — dataflow export, ADSO inspection, lineage graphs
*   [MCP Server](/docs/erpl-adt/mcp.md) — Claude Code / Cursor / Gemini integration deep-dive
*   [Command Reference](/docs/erpl-adt/reference.md) — every command, every flag, every exit code
*   [Announcement post](/blog/introducing-erpl-adt) — narrative intro with a recorded agent session

# ERPL-Web Quick Start (Cloud/Web)

In this 5-minute tutorial, you'll connect to a public OData service and query SAP Datasphere from DuckDB. By the end, you'll be reading SAP data via HTTP APIs straight into SQL.

**Prerequisites:**

*   DuckDB installed (version 0.10.0+)
*   Network access from your machine to the OData endpoint or SAP Datasphere tenant
*   For protected services: a Datasphere OAuth2 app, an OData bearer token, or HTTP basic credentials

## Step 1: Install ERPL-Web

```
INSTALL 'erpl_web' FROM 'http://get.erpl.io';LOAD 'erpl_web';
```

**Success Check:**

If installation worked, you should see no errors. The extension is now loaded and ready to use.

## Step 2: Query Your First OData Service

The public `TripPin` reference service is a good starting point — no auth, no setup. `odata_read()` takes the entity-set URL and returns a regular table.

```
-- Read the People entity set directlySELECT *FROM odata_read('https://services.odata.org/TripPinRESTierService/People')LIMIT 5;
```

OData's `top` / `skip` / `expand` / `count` query options are exposed as named parameters:

```
-- Top-5 with $expand for related TripsSELECT *FROM odata_read(  'https://services.odata.org/TripPinRESTierService/People',  top => 5,  expand => 'Trips');
```

### What's happening?

ERPL-Web fetches the OData `$metadata` once, builds a typed schema, then issues paginated requests for the entity set.

## Step 3: Filter and Sort with Plain SQL

You don't need named parameters for filtering or ordering — regular SQL `WHERE` and `ORDER BY` are pushed down to OData `$filter` and `$orderby`:

```
-- Filter by date range — pushed down to OData $filterSELECT FirstName, LastName, UserNameFROM odata_read('https://services.odata.org/TripPinRESTierService/People')WHERE FirstName = 'Russell';
```

```
-- Sort + paginateSELECT *FROM odata_read(  'https://services.odata.org/TripPinRESTierService/People',  top => 10,  skip => 20)ORDER BY LastName ASC;
```

## Step 4: Attach a Service as a Database

If you'll query many entity sets from the same service, attach it once. Each entity set then looks like a table:

```
ATTACH 'https://services.odata.org/TripPinRESTierService'  AS trippin (TYPE odata);SELECT FirstName, LastName FROM trippin.People LIMIT 5;SELECT * FROM trippin.Airports LIMIT 5;
```

The catalog discovers entity sets from `$metadata`; predicate and column pushdown apply to every table.

## Step 5: Connect to SAP Datasphere

Datasphere uses OAuth2 client credentials. Create a secret once, then call the catalog functions — no per-call auth needed.

```
-- One-time: create the OAuth2 secretCREATE SECRET datasphere (  TYPE datasphere,  PROVIDER oauth2,  tenant_name 'mytenant',  data_center 'eu10',  client_id 'your-client-id',  client_secret 'your-client-secret');
```

Discover what's available:

```
-- List spaces you have access toSELECT * FROM datasphere_show_spaces();-- List assets in a spaceSELECT * FROM datasphere_show_assets('SALES_SPACE');
```

Then read an asset by name:

```
-- Read a relational view (the default consumption mode)SELECT *FROM datasphere_read_relational('SALES_SPACE', 'V_DEMAND_FORECAST')WHERE month = '2026-05';
```

For multi-dimensional analytical models, use `datasphere_read_analytical()` instead.

## Common OData Patterns

```
-- Sales orders entity setSELECT *FROM odata_read('https://your-server/odata/SalesOrders', top => 5);-- Customer master with $expandSELECT *FROM odata_read(  'https://your-server/odata/Customers',  expand => 'Addresses,Contacts',  top => 20);-- Count-only querySELECT *FROM odata_read(  'https://your-server/odata/Orders',  count => true,  top => 0);
```

### Filtering — let DuckDB push it down

```
-- Date range — translated to OData $filterSELECT *FROM odata_read('https://your-server/odata/Orders')WHERE OrderDate BETWEEN DATE '2024-01-01' AND DATE '2024-12-31';
```

```
-- Compound predicate — also pushed downSELECT *FROM odata_read('https://your-server/odata/Products')WHERE Category = 'Electronics' AND Price > 100;
```

### Sorting and pagination

```
-- Sort by date desc, server-paginatedSELECT *FROM odata_read(  'https://your-server/odata/Orders',  top => 20)ORDER BY OrderDate DESC;-- Skip first 100SELECT *FROM odata_read(  'https://your-server/odata/Customers',  skip => 100,  top => 50);
```

### Selecting specific fields

```
-- SELECT-list is pushed down as OData $selectSELECT OrderID, CustomerID, TotalAmountFROM odata_read('https://your-server/odata/Orders', top => 10);
```

## Troubleshooting

### Auth failures

Most "authentication failed" errors come from a missing or mis-scoped secret. ERPL-Web looks up the secret whose `SCOPE` best matches the request URL.

```
-- Create a bearer-token secret scoped to the hostCREATE SECRET sap_api (  TYPE http_bearer,  token 'your-api-token',  SCOPE 'https://api.example.com');-- Then any matching call uses it automaticallySELECT * FROM odata_read('https://api.example.com/Orders', top => 5);
```

For Datasphere OAuth2 see Step 5 above.

### Connection issues

*   Verify the URL works in `curl` first.
*   Confirm the service exposes a reachable `$metadata` endpoint — ERPL-Web fetches it before issuing queries.
*   For self-signed TLS, configure your DuckDB extension config accordingly (see [Tracing & Diagnostics](/docs/erpl-web/tracing.md)).

### Type or field errors

OData fields are case-sensitive and depend on the service's `$metadata`. To see the exact schema, run with `top => 1` first and inspect the result columns.

```
SELECT * FROM odata_read('https://your-server/odata/Orders', top => 1);
```

## Next Steps

### 🚀 Ready for more?

*   [OData Deep Dive](/docs/erpl-web/odata.md) — predicate pushdown, ATTACH, $expand details
*   [SAP Datasphere](/docs/erpl-web/datasphere.md) — spaces, assets, relational vs analytical reads
*   [Microsoft 365](/docs/erpl-web/m365.md) — SharePoint, Excel, Teams, Outlook, Planner, Entra ID
*   [Microsoft Dynamics 365](/docs/erpl-web/dyn365.md) — Business Central + Dataverse

### 🔧 Advanced topics

*   [Secrets Management](/docs/erpl-web/secrets.md) — every secret type and provider
*   [Tracing & Diagnostics](/docs/erpl-web/tracing.md) — debugging HTTP and OData calls
*   [Function Reference](/docs/reference/erpl-functions.md) — canonical signatures

### 💡 Examples

*   [ERPL-Web Examples](/docs/examples/erpl-web-examples.md) — real-world scenarios
*   [Integration with Python](/docs/guides/integration/python-pandas.md) — Pandas with ERPL-Web

## What You've Learned

✅ **Installed ERPL-Web extension** ✅ **Queried a public OData service with `odata_read`** ✅ **Used `WHERE` and `ORDER BY` (with pushdown) instead of fake named params** ✅ **Attached an OData service as a DuckDB catalog** ✅ **Connected to SAP Datasphere via OAuth2 secrets**

You're ready to use ERPL-Web for cloud SAP integration.

---

**Need help?** Check the [troubleshooting guide](/docs/get_started/troubleshooting.md) or browse [more examples](/docs/examples/erpl-web-examples.md).

# Troubleshooting Guide

This guide helps you resolve common issues when using ERPL with DuckDB and SAP systems.

## Installation Issues

### Extension Installation Fails

**Problem**: `INSTALL 'erpl'` command fails with an error.

**Solutions**:

1.  **Check DuckDB Version**: Ensure you're using DuckDB 0.10.1
    
    ```
    SELECT version();
    ```
    
2.  **Verify Internet Connection**: The extension is downloaded from the internet
    
    ```
    curl -I http://get.erpl.io
    ```
    
3.  **Try Manual Installation**: Download and install manually
    
    ```
    INSTALL '/path/to/erpl.duckdb_extension';LOAD 'erpl';
    ```
    

### Extension Not Found After Installation

**Problem**: `LOAD 'erpl'` returns "Extension not found".

**Solutions**:

1.  **Check Installation**: Verify the extension was installed
    
    ```
    SELECT * FROM duckdb_extensions() WHERE extension_name = 'erpl';
    ```
    
2.  **Restart DuckDB**: Close and reopen your DuckDB session
    
3.  **Check File Permissions**: Ensure DuckDB can access the extension directory
    

## Connection Issues

### SAP Connection Fails

**Problem**: ERPL fails to reach your SAP system.

Credentials live in a DuckDB secret and the connection is established lazily on the first RFC call. Verify your secret and ping the system.

**Common Causes and Solutions**:

1.  **Invalid Host or System Number** — re-create the secret with correct values
    
    ```
    CREATE OR REPLACE SECRET my_sap (  TYPE sap_rfc,  ASHOST 'correct-sap-server.com',  -- Hostname or IP  SYSNR '00',                       -- System number  CLIENT '100',                     -- SAP client  USER 'your-username',  PASSWD 'your-password',  LANG 'EN');-- Validate the connectionPRAGMA sap_rfc_ping;
    ```
    
2.  **Network Connectivity Issues**
    
    ```
    # Test network connectivityping your-sap-server.comtelnet your-sap-server.com 3200  # Default SAP port
    ```
    
3.  **SAP User Account Issues**
    
    *   Verify username and password
    *   Check if account is locked
    *   Ensure account has RFC authorizations

### Permission Denied Errors

**Problem**: `sap_read_table()` returns permission errors.

**Solutions**:

1.  **Check RFC Authorizations**: Contact your SAP administrator
2.  **Verify Table Access**: Ensure your user can access the specific table
3.  **Test with Different Table**: Try a table you know you have access to

## Performance Issues

### Slow Query Performance

**Problem**: SAP queries are running slowly.

**Optimization Tips**:

1.  **Use Filters**: Limit data with WHERE clauses
    
    ```
    SELECT * FROM sap_read_table('KNA1') WHERE LAND1 = 'DE'  -- Filter by countryLIMIT 1000;
    ```
    
2.  **Select Specific Columns**: Only retrieve needed fields
    
    ```
    SELECT KUNNR, NAME1 FROM sap_read_table('KNA1');
    ```
    
3.  **Use Pagination**: Process data in chunks
    
    ```
    SELECT * FROM sap_read_table('KNA1') LIMIT 1000 OFFSET 0;
    ```
    

### Memory Issues

**Problem**: DuckDB runs out of memory with large SAP tables.

**Solutions**:

1.  **Increase Memory Limit**: Set higher memory limit for DuckDB
2.  **Process in Batches**: Use LIMIT and OFFSET for large tables
3.  **Optimize Queries**: Use more selective WHERE clauses

## SAP-Specific Issues

### Table Not Found

**Problem**: `sap_read_table('TABLENAME')` returns "Table not found".

**Solutions**:

1.  **Check Table Name**: Verify the exact table name (case-sensitive)
2.  **List Available Tables**: Use `sap_show_tables()` to see what's available
3.  **Check Client**: Ensure you're connected to the correct SAP client

### Function Module Errors

**Problem**: Calling SAP function modules fails.

**Common Issues**:

1.  **Function Not Remote-Enabled**: Verify the function module is RFC-enabled
2.  **Parameter Mismatch**: Check function module interface
3.  **Authorization Issues**: Ensure you have permission to call the function

## Debugging Tips

### Enable Verbose Logging

Enable detailed logging to diagnose issues:

```
SET erpl_trace_enabled = TRUE;SET erpl_trace_level   = 'DEBUG';   -- TRACE | DEBUG | INFO | WARN | ERRORSET erpl_trace_output  = 'console'; -- console | file | both
```

### Test Basic Connectivity

Start with simple tests:

```
-- Ping the SAP system (raises an error if the connection is broken)PRAGMA sap_rfc_ping;-- List available tablesSELECT * FROM sap_show_tables() LIMIT 5;-- Try a simple table readSELECT * FROM sap_read_table('T000', MAX_ROWS => 1);
```

### Check SAP System Status

Verify your SAP system is accessible:

```
# Check if SAP system is runningping your-sap-server.com# Test RFC port (usually 3200 + system number)telnet your-sap-server.com 3200
```

## Getting Additional Help

### Self-Service Resources

1.  **Documentation**: Check our [complete documentation](/docs/start_here.md)
2.  **Examples**: Review [key tasks](/docs/key_tasks.md) for common scenarios
3.  **GitHub Issues**: Search existing [issues](https://github.com/datazoode/erpl/issues)

### Community Support

1.  **GitHub Discussions**: Ask questions in [GitHub Discussions](https://github.com/datazoode/erpl/discussions)
2.  **Stack Overflow**: Tag questions with `erpl` and `duckdb`
3.  **LinkedIn**: Connect with other users on [LinkedIn](https://www.linkedin.com/company/datazoo/)

### Professional Support

For enterprise support and consulting:

*   **Email**: [contact@data-zoo.de](mailto:contact@data-zoo.de)
*   **Contact Form**: [Contact us](/contact)
*   **Consulting**: [Schedule a call](https://calendly.com/datazoo)

## Error Code Reference

### Common Error Codes

| Error Code | Description | Solution |
| --- | --- | --- |
| `RFC_ERROR` | General RFC communication error | Check network connectivity and SAP system status |
| `AUTH_FAILED` | Authentication failed | Verify username and password |
| `PERMISSION_DENIED` | Insufficient permissions | Contact SAP administrator for RFC authorizations |
| `TABLE_NOT_FOUND` | SAP table doesn't exist | Verify table name and client |
| `FUNCTION_NOT_FOUND` | Function module not found | Check function name and RFC enablement |

### Log Analysis

When reporting issues, include:

1.  **DuckDB Version**: `SELECT version();`
2.  **ERPL Version**: `SELECT * FROM duckdb_extensions() WHERE extension_name = 'erpl';`
3.  **Error Message**: Complete error text
4.  **SAP System Info**: Version, client, system number
5.  **Query**: The exact SQL query that failed

This information helps us provide faster and more accurate support.

# BICS Lineage Tracking

This guide covers the three BICS lineage functions that ERPL exposes and shows how to use them for impact analysis, governance, and documentation in SAP BW landscapes.

**For BI Administrators and Data Governance Teams:**

The lineage functions read directly from `RSTRAN`, `RSTRANFIELD`, and related BW system tables. They expose the same data Eclipse-based tools see, but as DuckDB tables you can join, filter, and aggregate freely.

## The three lineage functions

All three are zero-positional. Filter and shape results with normal SQL.

| Function | Returns | Best for |
| --- | --- | --- |
| ([`sap_bics_lineage_edges()`](#flat-edge-list)) | Flat edge list (one row per source→target hop) | Catalog-wide lineage, joins, group-bys |
| ([`sap_bics_lineage_trace()`](#forward-trace-from-a-source)) | Walk forward from one source, with hop count and path | "What is downstream of this table/field?" |
| ([`sap_bics_lineage_graph_json()`](#json-graph)) | Full lineage as a single JSON document | Visualization, export to graph tools |

## Flat edge list

`sap_bics_lineage_edges()` returns one row per edge in the BW data flow graph.

**Columns** (all VARCHAR):

| Column | Description |
| --- | --- |
| `edge_type` | Edge category (e.g. transformation step, query element) |
| `src_kind` | Source object kind (DATASOURCE, INFOPROVIDER, QUERY, …) |
| `src_name` | Source object name |
| `src_field` | Source field (empty for object-level edges) |
| `tgt_kind` | Target object kind |
| `tgt_name` | Target object name |
| `tgt_field` | Target field |

**Named parameters**: `scope` (limits the underlying RFC reads to objects matching the name; useful on very large landscapes), `secret` (DuckDB secret name).

### Get everything

```
SELECT * FROM sap_bics_lineage_edges();
```

### All edges related to one BEx query

```
SELECT *FROM sap_bics_lineage_edges()WHERE tgt_name = '0D_FC_NW_C01_Q0008'   OR src_name = '0D_FC_NW_C01_Q0008'ORDER BY src_kind, tgt_kind;
```

### Count distinct source tables feeding each InfoProvider

```
SELECT  tgt_name AS infoprovider,  COUNT(DISTINCT src_name) AS source_table_countFROM sap_bics_lineage_edges()WHERE tgt_kind = 'INFOPROVIDER' AND src_kind = 'DATASOURCE'GROUP BY tgt_nameORDER BY source_table_count DESC;
```

### Field-level edges only

```
SELECT src_name, src_field, tgt_name, tgt_fieldFROM sap_bics_lineage_edges()WHERE src_field <> '' AND tgt_field <> '';
```

### Scope to a specific object

For large landscapes, pass `scope` so the underlying RFC reads filter server-side:

```
SELECT * FROM sap_bics_lineage_edges(scope => '0D_FC_NW_C01_Q0008');
```

## Forward trace from a source

`sap_bics_lineage_trace()` answers "what is downstream of this object?". It walks the edge graph forward and returns one row per hop, including the full chain in the `path` column.

**Columns**:

| Column | Type | Description |
| --- | --- | --- |
| `hop` | INTEGER | Distance from the source (1 = direct downstream) |
| `source_object` | VARCHAR | Source object at this hop |
| `source_field` | VARCHAR | Source field (empty for object-level edges) |
| `target_object` | VARCHAR | Target object at this hop |
| `target_field` | VARCHAR | Target field |
| `edge_type` | VARCHAR | Edge category |
| `path` | VARCHAR | The chain of objects walked from the original source to this row |

**Named parameters**: `source_object` (required for a meaningful trace), `source_field` (optional, for field-level tracing), `secret`.

### Impact analysis: what depends on `VBAK`?

```
SELECT hop, target_object, edge_type, pathFROM sap_bics_lineage_trace(source_object => 'VBAK')ORDER BY hop, target_object;
```

### Field-level trace

If `NETWR` in `VBAK` is renamed, what BW objects need updating?

```
SELECT hop, target_object, target_field, pathFROM sap_bics_lineage_trace(  source_object => 'VBAK',  source_field => 'NETWR')ORDER BY hop;
```

### Count downstream queries

```
SELECT COUNT(DISTINCT target_object) AS downstream_queriesFROM sap_bics_lineage_trace(source_object => 'VBAK')WHERE edge_type LIKE '%QUERY%';
```

## JSON graph

`sap_bics_lineage_graph_json()` returns the entire lineage as one JSON document. Useful for visualization libraries and external graph tools.

```
-- One row with the full JSON graphSELECT * FROM sap_bics_lineage_graph_json();-- Export to a fileCOPY (SELECT * FROM sap_bics_lineage_graph_json())TO 'lineage.json' (FORMAT 'json');
```

The JSON shape is suitable for direct ingestion by D3.js, Cytoscape.js, or the JSON exporter of most graph databases.

## Common patterns

### Find unused DataSources

DataSources with no downstream consumers are candidates for cleanup:

```
WITH all_sources AS (  SELECT DISTINCT src_name AS name  FROM sap_bics_lineage_edges()  WHERE src_kind = 'DATASOURCE'),used_sources AS (  SELECT DISTINCT src_name AS name  FROM sap_bics_lineage_edges()  WHERE src_kind = 'DATASOURCE' AND tgt_kind = 'INFOPROVIDER')SELECT name FROM all_sourcesEXCEPTSELECT name FROM used_sources;
```

### Pre-change impact assessment

Before changing a source table, dump the affected query list:

```
COPY (  SELECT DISTINCT target_object AS affected_query  FROM sap_bics_lineage_trace(source_object => 'VBAK')  WHERE edge_type LIKE '%QUERY%')TO 'vbak_affected_queries.csv' (HEADER, DELIMITER ',');
```

### Persist a lineage snapshot

Capture lineage as it was on a specific date — useful for audit/compliance:

```
CREATE TABLE lineage_snapshot_2026_05_15 ASSELECT * FROM sap_bics_lineage_edges();
```

## Performance notes

*   All three functions read from BW system tables via RFC (`RSTRAN`, `RSTRANFIELD`, etc.). On large landscapes this can be slow on the first call.
*   Use the `scope` parameter on `sap_bics_lineage_edges()` to limit the RFC scan when you only care about one object.
*   For repeated analysis, persist a snapshot to DuckDB (`CREATE TABLE ... AS SELECT * FROM ...`) and query that.

## Combining with metadata functions

The `sap_bics_meta_*` family exposes detailed metadata on individual object types (providers, queries, transformations, fields). Join them against `sap_bics_lineage_edges()` to enrich edges with descriptions:

```
SELECT  e.edge_type,  e.src_kind, e.src_name,  q.description AS query_description,  e.tgt_kind, e.tgt_nameFROM sap_bics_lineage_edges() eLEFT JOIN sap_bics_meta_queries() q ON q.query_name = e.tgt_nameWHERE e.tgt_kind = 'QUERY';
```

(Exact metadata column names vary by function — call each `sap_bics_meta_*()` once to inspect its schema.)

## Next Steps

*   [BICS Protocol Guide](/docs/erpl/bics.md) — full BICS function reference
*   [Function Reference](/docs/reference/erpl-functions.md#lineage-functions) — canonical signatures for the lineage functions

# OData Metadata Deep Dive

Advanced guide to OData metadata handling for SAP experts.

## OData Metadata

OData services expose metadata that describes the data model.

## Metadata Endpoints

```
-- Get service metadataSELECT * FROM odata_metadata('https://api.sap.com/service');
```

## Advanced Topics

*   Metadata parsing
*   Schema introspection
*   Type mapping

# odp-delta-replication

# ODP Subscription Management

This guide covers operational management of ODP subscriptions and cursors from ERPL: how to inspect them, recover from stuck states, coordinate across teams, and keep the SAP side clean.

**For SAP Administrators and Data Engineers:**

You'll get the most out of this guide if you've already worked through [ODP Protocol Deep Dive](/docs/erpl/odp.md). It covers the `sap_odp_read_full` vs. `sap_odp_read_delta` distinction that the rest of this page assumes.

## The Two ODP Cursor Lifecycles

`sap_odp_read_full` cursors auto-close at end of scan — there is no lifecycle to manage. Everything below applies to **delta** cursors registered by `sap_odp_read_delta`.

## Inspecting What's There

### Listing Subscriptions

```
-- ERPL-owned subscriptions (default ERPL_ONLY => TRUE)SELECT queue_name, subscriber_type, subscriber_name, subscriber_procFROM sap_odp_show_subscriptions();-- All subscribers on the system (cross-team visibility)SELECT * FROM sap_odp_show_subscriptions(ERPL_ONLY => FALSE);
```

`sap_odp_show_subscriptions` returns only `queue_name`, `subscriber_type`, `subscriber_name`, `subscriber_proc`. There is no `status`, `created_date`, or `last_used_date` — pair the result with `sap_odp_show_cursors` (below) for activity timestamps.

### Per-Source Subscriber Visibility

`sap_odp_get_subscriptions(odp_context, odp_name [, …])` exposes `RODPS_REPL_ODP_GET_SUBSCR`, which lists every subscription registered against a single source — handy when you want to know who else is reading from a CDS view or BW DataSource.

```
-- All subscribers on a source, with model and queue contextSELECT * FROM sap_odp_get_subscriptions('ABAP_CDS', 'SEPM_IBUPA$P');-- Filter to my pipeline's subscriberSELECT * FROM sap_odp_get_subscriptions(    'BW', '0D_FC_C01$F',    subscriber_name => 'ERPL');
```

Returns `subscriber_type`, `subscriber_name`, `subscriber_process`, `model_name`, `queue_name`, `subscription_id` (DECIMAL(23,9)).

### Inspecting Cursors

```
-- All ERPL cursors with their current stateSELECT subscriber_proc, pointer, is_closed, is_delta_extension, request_dateFROM sap_odp_show_cursors(subscriber_name => 'ERPL');-- Narrow to delta cursors on a specific contextSELECT * FROM sap_odp_show_cursors(    subscriber_name  => 'ERPL',    context          => 'ABAP_CDS',    replication_mode => 'DELTA');
```

Returned columns: `queue_name`, `subscriber_proc`, `subscriber_id`, `pointer`, `is_closed`, `is_delta_extension`, `request_date`. `request_date` is the activity timestamp — use it for staleness checks.

### Probing Source Change Time

`sap_odp_get_last_modified` returns when the source itself was last updated server-side — independent of any subscription:

```
-- Returns (odp_name VARCHAR, last_modified DECIMAL(21,7))-- Format YYYYMMDDhhmmss.fffffff (UTC). Returns 0.0 for unknown ODP names.SELECT * FROM sap_odp_get_last_modified('ABAP_CDS', 'SEPM_IBUPA$P');
```

Combine with run-log state to skip pipelines when nothing has changed.

## Operational Patterns

### Reconcile subscriptions to cursors

`sap_odp_show_subscriptions` lists what is registered; `sap_odp_show_cursors` lists what currently has activity. Joining them surfaces stale or orphaned state:

```
WITH subs AS (  SELECT queue_name, subscriber_proc  FROM sap_odp_show_subscriptions()),cursors AS (  SELECT subscriber_proc, pointer, is_closed, is_delta_extension, request_date  FROM sap_odp_show_cursors(subscriber_name => 'ERPL'))SELECT  s.queue_name,  s.subscriber_proc,  c.is_delta_extension,  c.is_closed,  c.request_date,  CURRENT_DATE - CAST(c.request_date AS DATE) AS days_idle,  CASE    WHEN c.subscriber_proc IS NULL                              THEN 'NO_CURSOR'    WHEN c.is_closed                                            THEN 'CLOSED'    WHEN CURRENT_DATE - CAST(c.request_date AS DATE) > 7        THEN 'STALE'    ELSE                                                              'ACTIVE'  END AS stateFROM subs sLEFT JOIN cursors c USING (subscriber_proc)ORDER BY days_idle DESC NULLS LAST;
```

### Graceful close vs. hard reset

ERPL exposes two cleanup pragmas with different semantics:

| Pragma | Effect | When to use |
| --- | --- | --- |
| `PRAGMA sap_odp_close_delta_cursor` | Closes the cursor's pointer on SAP. Subscription remains registered and resumable. Idempotent — returns `'CLOSED'` or `'NOT_FOUND'`. | End of every healthy pipeline run. The default. |
| `PRAGMA sap_odp_drop` | Invokes `RODPS_REPL_ODP_RESET` — wipes the subscription entirely. Next `sap_odp_read_delta` call performs DELTAINIT. | Cursor is stuck and the close pragma can't reach it, or you intentionally want a fresh snapshot. |

```
-- Standard end-of-pipeline cleanupPRAGMA sap_odp_close_delta_cursor('BW', 'NIGHTLY_ETL', '0D_FC_C01$F');-- Hard reset (rare)PRAGMA sap_odp_drop('BW', 'ERPL', 'NIGHTLY_ETL', '0D_FC_C01$F');
```

### Recovering a delta pipeline

```
-- 1. Look at the cursor — closed? open? when did it last move?SELECT * FROM sap_odp_show_cursors(subscriber_name => 'ERPL')WHERE subscriber_proc = 'NIGHTLY_ETL';-- 2. Try replaying the last unconfirmed packet (no pointer advance)SELECT * FROM sap_odp_read_delta(    'BW', '0D_FC_C01$F', 'NIGHTLY_ETL',    recover => true);-- 3. If still wedged, hard-reset and let the next call DELTAINITPRAGMA sap_odp_drop('BW', 'ERPL', 'NIGHTLY_ETL', '0D_FC_C01$F');SELECT * FROM sap_odp_read_delta('BW', '0D_FC_C01$F', 'NIGHTLY_ETL');
```

### Multiple subscribers on one source

Each `subscriber_process` is independent — different pipelines can register their own delta pointers against the same source without coordinating:

```
-- Three independent pipelines on VBAK$FSELECT * FROM sap_odp_read_delta('BW', 'VBAK$F', 'ANALYTICS_DAILY');SELECT * FROM sap_odp_read_delta('BW', 'VBAK$F', 'REPORTING_HOURLY');SELECT * FROM sap_odp_read_delta('BW', 'VBAK$F', 'CDC_REALTIME');-- See them all (ERPL-owned)SELECT * FROM sap_odp_show_subscriptions()WHERE queue_name = 'VBAK$F';-- Or, including subscribers other tools created on the same sourceSELECT * FROM sap_odp_get_subscriptions('BW', 'VBAK$F');
```

Pick descriptive, stable names per pipeline (`<environment>_<purpose>_<cadence>`); they appear verbatim in SAP's queue UI.

## Cleanup Hygiene

Leaked open cursors accumulate server-side and consume queue slots. Two rules:

1.  **Always close a pipeline's cursor before exiting** with `PRAGMA sap_odp_close_delta_cursor`. The pragma is idempotent — call it even if you aren't sure whether the cursor is open.
2.  **Run a periodic stale-cursor sweep** against the reconciliation query above. Anything with `state = 'STALE'` and a `days_idle` past your retention policy is a candidate for `PRAGMA sap_odp_drop`.

```
-- Generate drop commands for ERPL cursors idle for more than 30 daysSELECT  'PRAGMA sap_odp_drop(''' ||    -- context: derive from queue_name or join sap_odp_show / sap_odp_get_subscriptions    'BW' || ''', ''ERPL'', ''' ||    subscriber_proc || ''', ''' ||    queue_name || ''');' AS drop_sqlFROM sap_odp_show_cursors(subscriber_name => 'ERPL')WHERE CURRENT_DATE - CAST(request_date AS DATE) > 30;
```

Inspect the generated SQL before executing — `sap_odp_drop` is destructive.

## Troubleshooting

**The next call returned a fresh full snapshot — I expected just deltas**

The subscription was dropped (explicit `sap_odp_drop`, an admin reset on the SAP side, or the queue was purged). Auto-DELTAINIT runs whenever a `subscriber_process` is new.

**`sap_odp_close_delta_cursor` returned `'NOT_FOUND'`**

No cursor of that name exists. Either it was already closed, the `subscriber_process` argument doesn't match what `sap_odp_read_delta` was called with, or the subscription was dropped. Confirm with `sap_odp_show_cursors(subscriber_name => 'ERPL')`.

**`sap_odp_read_delta` errors with "delta extraction not supported"**

The source doesn't have a delta capability. Check `sap_odp_describe(context, name)` → `supports_delta` and `delta_modes`. CDS views need `@Analytics.dataExtraction.delta.byElement`; BW fact tables (`*$F`) are normally delta-capable.

**Cursor is wedged — `recover => true` doesn't help**

Hard-reset with `PRAGMA sap_odp_drop`; the next call re-DELTAINITs. If even drop fails, the queue may be locked at the SAP layer — escalate to Basis.

## Next Steps

### Ready for More?

*   [ODP Protocol Deep Dive](/docs/erpl/odp.md) — full context, examples, and the FULL vs. DELTA model
*   [ERPL Function Reference](/docs/reference/erpl-functions.md#odp-functions) — every ODP function and pragma with signatures

### Examples

*   [ERPL Examples](/docs/examples/erpl-examples.md) — runnable ODP snippets
*   [Delta Replicate Stock Data from SAP ERP to Parquet](/docs/key_tasks/replicate_odp.md) — end-to-end pipeline walkthrough

---

**Need help?** Check our [troubleshooting guide](/docs/reference/troubleshooting.md) or browse [more examples](/docs/examples/erpl-examples.md).

# performance-tuning

# RFC Metadata Extraction

This advanced guide covers RFC metadata extraction and analysis using ERPL's RFC functions. Learn how to discover SAP functions, analyze table structures, and build dynamic integration patterns — using the **real** output columns of each function.

**For SAP Developers and Integration Specialists:**

This guide is designed for SAP developers, integration specialists, and data architects who need to understand and analyze SAP system metadata.

## The metadata functions and their columns

| Function | Output columns |
| --- | --- |
| `sap_rfc_show_function([FUNCNAME, GROUPNAME])` | `FUNCNAME`, `GROUPNAME`, `APPL`, `HOST`, `STEXT` |
| `sap_rfc_show_groups([GROUPNAME])` | `name`, `text` |
| `sap_show_tables([TABLENAME, TEXT])` | `table_name`, `text`, `class` |
| `sap_describe_fields(table)` | `pos`, `is_key`, `field`, `text`, `sap_type`, `length`, `decimals`, `check_table`, `ref_table`, `ref_field`, `language` |
| `sap_rfc_describe_function(name)` | `name`, `text`, `function_group`, `remote_callable`, `import`, `export`, `changing`, `tables`, `source` |

For `sap_rfc_describe_function`, the `import`/`export`/`changing`/`tables` columns are **lists of structs**, each struct having `name`, `text`, `abap_type`, `duckdb_type`, `direction`, `length`, `decimals`, `default_value`, `optional`.

## Function Module Discovery

### Listing Available Functions

```
-- List RFC-enabled function modules (filter server-side with FUNCNAME)SELECT * FROM sap_rfc_show_function(FUNCNAME => 'BAPI_FLIGHT*');-- Search by pattern in SQLSELECT FUNCNAME, GROUPNAME, STEXTFROM sap_rfc_show_function()WHERE FUNCNAME LIKE '%BAPI%';-- Analyze function distribution by name prefixSELECT  SUBSTRING(FUNCNAME, 1, 3) AS function_prefix,  COUNT(*) AS function_countFROM sap_rfc_show_function(FUNCNAME => 'BAPI*')GROUP BY SUBSTRING(FUNCNAME, 1, 3)ORDER BY function_count DESC;
```

### Function Group Analysis

```
-- List all function groupsSELECT * FROM sap_rfc_show_groups();-- Functions grouped by their function group (GROUPNAME is already on each row)SELECT  GROUPNAME,  COUNT(*) AS function_count,  STRING_AGG(FUNCNAME, ', ') AS functionsFROM sap_rfc_show_function(FUNCNAME => 'BAPI_FLIGHT*')GROUP BY GROUPNAMEORDER BY function_count DESC;
```

### Function Module Details

```
-- Get the full interface of a function module (one row)SELECT name, function_group, remote_callableFROM sap_rfc_describe_function('BAPI_FLIGHT_GETLIST');-- Count parameters per direction using len() on the list columnsSELECT  name,  len(import)   AS import_params,  len(export)   AS export_params,  len(changing) AS changing_params,  len(tables)   AS table_paramsFROM sap_rfc_describe_function('BAPI_FLIGHT_GETLIST');
```

### Listing individual parameters

The list columns are unnested to one row per parameter:

```
-- Import parameters of a function moduleWITH f AS (SELECT * FROM sap_rfc_describe_function('BAPI_FLIGHT_GETLIST'))SELECT p.name, p.abap_type, p.duckdb_type, p.length, p.decimals, p.optionalFROM f, UNNEST(f.import) AS t(p);-- All parameters with their direction (import + export + tables)WITH f AS (SELECT * FROM sap_rfc_describe_function('BAPI_FLIGHT_GETLIST'))SELECT 'IMPORT' AS direction, p.name, p.abap_type, p.duckdb_typeFROM f, UNNEST(f.import) AS t(p)UNION ALLSELECT 'EXPORT', p.name, p.abap_type, p.duckdb_typeFROM f, UNNEST(f.export) AS t(p)UNION ALLSELECT 'TABLES', p.name, p.abap_type, p.duckdb_typeFROM f, UNNEST(f.tables) AS t(p);
```

## Table Structure Analysis

### Table Discovery

```
-- List SAP tables (filter server-side with TABLENAME)SELECT * FROM sap_show_tables(TABLENAME => '*FLIGHT*');-- Search for specific table patterns in SQLSELECT table_name, text, classFROM sap_show_tables()WHERE table_name LIKE 'KNA%';-- Analyze table naming patternsSELECT  SUBSTRING(table_name, 1, 2) AS table_prefix,  COUNT(*) AS table_countFROM sap_show_tables(TABLENAME => 'KN*')GROUP BY SUBSTRING(table_name, 1, 2)ORDER BY table_count DESC;
```

### Table Field Analysis

```
-- Get table structureSELECT * FROM sap_describe_fields('KNA1');-- Analyze field characteristicsSELECT  field,  sap_type,  length,  decimals,  CASE    WHEN length > 100 THEN 'LONG_FIELD'    WHEN length > 50  THEN 'MEDIUM_FIELD'    ELSE 'SHORT_FIELD'  END AS field_size_categoryFROM sap_describe_fields('KNA1')ORDER BY length DESC;
```

### Data Type Analysis

```
-- Analyze data type distributionSELECT  sap_type,  COUNT(*) AS field_count,  AVG(length) AS avg_length,  MAX(length) AS max_lengthFROM sap_describe_fields('KNA1')GROUP BY sap_typeORDER BY field_count DESC;
```

### Cross-Reference Analysis

The `check_table`, `ref_table`, and `ref_field` columns of `sap_describe_fields` expose how a table's fields reference other tables (foreign-key check tables and currency/quantity reference fields):

```
-- Which tables does KNA1 reference via its fields?SELECT field, check_table, ref_table, ref_fieldFROM sap_describe_fields('KNA1')WHERE check_table <> '' OR ref_table <> '';-- Most-referenced check tables across a table's fieldsSELECT  check_table AS referenced_table,  COUNT(*) AS reference_count,  STRING_AGG(field, ', ') AS referencing_fieldsFROM sap_describe_fields('KNA1')WHERE check_table <> ''GROUP BY check_tableORDER BY reference_count DESC;
```

## Dynamic Integration Patterns

### Function discovery for integration

First discover candidate functions, then describe a specific one to gauge its interface size. `sap_rfc_describe_function` takes a constant function name (one call per function):

```
-- 1. Find candidate BAPIsSELECT FUNCNAME, STEXT FROM sap_rfc_show_function(FUNCNAME => 'BAPI_FLIGHT*');-- 2. Classify a chosen function by its interface sizeSELECT  name,  len(import) AS import_params,  len(export) AS export_params,  len(tables) AS table_params,  CASE    WHEN len(tables) > 0  THEN 'REQUIRES_TABLE_HANDLING'    WHEN len(import) > 10 THEN 'COMPLEX_INPUT'    ELSE 'SIMPLE_INTEGRATION'  END AS integration_complexityFROM sap_rfc_describe_function('BAPI_FLIGHT_GETLIST');
```

### Table schema generation

```
-- Generate a DuckDB CREATE TABLE skeleton from SAP table metadataWITH cols AS (  SELECT    field,    CASE      WHEN sap_type IN ('CHAR', 'NUMC') THEN 'VARCHAR(' || length || ')'      WHEN sap_type = 'DEC'  THEN 'DECIMAL(' || length || ',' || decimals || ')'      WHEN sap_type = 'INT4' THEN 'INTEGER'      WHEN sap_type = 'DATS' THEN 'DATE'      WHEN sap_type = 'TIMS' THEN 'TIME'      ELSE 'VARCHAR(' || length || ')'    END AS duckdb_type  FROM sap_describe_fields('KNA1'))SELECT  'CREATE TABLE KNA1 (' AS ddl_start,  STRING_AGG('  ' || field || ' ' || duckdb_type, ',' || chr(10)) AS columns,  ');' AS ddl_endFROM cols;
```

## Performance Optimization

### Metadata caching

```
-- Materialize metadata locally for repeated analysisCREATE TABLE rfc_function_cache ASSELECT * FROM sap_rfc_show_function(FUNCNAME => 'BAPI*');CREATE TABLE rfc_table_cache ASSELECT * FROM sap_show_tables(TABLENAME => 'KN*');-- Query the cache instead of re-hitting SAPSELECT * FROM rfc_function_cache WHERE FUNCNAME LIKE '%FLIGHT%';
```

### Efficient metadata queries

```
-- Push the filter to SAP with the FUNCNAME parameter, not a SQL WHERE on everythingSELECT FUNCNAME, STEXTFROM sap_rfc_show_function(FUNCNAME => 'BAPI_FLIGHT*')LIMIT 100;
```

## Troubleshooting

**Function Not Found**

```
-- Check whether a function exists / find similar onesSELECT * FROM sap_rfc_show_function(FUNCNAME => 'BAPI_FLIGHT_GETLIST');SELECT * FROM sap_rfc_show_function(FUNCNAME => '*FLIGHT*');
```

**Table Not Found**

```
SELECT * FROM sap_show_tables(TABLENAME => 'KNA1');SELECT * FROM sap_show_tables(TABLENAME => 'KNA*');
```

**Metadata Access Denied**

```
-- Confirm connectivity and basic metadata accessPRAGMA sap_rfc_ping;SELECT * FROM sap_rfc_describe_function('RFC_SYSTEM_INFO');
```

### Debugging Tips

```
-- Enable ERPL tracingSET erpl_trace_enabled = TRUE;SET erpl_trace_level   = 'DEBUG';-- Test metadata functionsSELECT * FROM sap_rfc_show_function(FUNCNAME => 'BAPI*') LIMIT 5;SELECT * FROM sap_show_tables(TABLENAME => 'KN*') LIMIT 5;SELECT * FROM sap_rfc_describe_function('RFC_SYSTEM_INFO');
```

## See also

*   [RFC Protocol Deep Dive](/docs/erpl/rfc.md) — full function reference and parameters
*   [`sap_rfc_authorizations()`](/docs/erpl/rfc.md#sap_rfc_authorizations) — which RFC modules each ERPL function calls

# sap-connection-pooling

# jupyter

# powerbi

# python-pandas

# r-integration

# connect-datasphere

# query-odata

# How to Read an SAP Table

In this guide, you'll learn how to extract data from SAP ERP tables. By the end, you'll be able to query any SAP table directly from DuckDB.

**What You'll Need:**

*   ERPL extension installed
*   SAP connection credentials
*   5 minutes

## Step 1: Connect to SAP

```
-- Install and load ERPLINSTALL 'erpl' FROM 'http://get.erpl.io';LOAD 'erpl';-- Store your SAP credentials in a secretCREATE SECRET sap_system (  TYPE   sap_rfc,  ASHOST 'hostname',  SYSNR  '00',  CLIENT '100',  USER   'your_username',  PASSWD 'your_password',  LANG   'EN');
```

## Step 2: Read Your First Table

```
-- Read customer master dataSELECT * FROM sap_read_table('KNA1', MAX_ROWS => 100);
```

## Common SAP Tables

| Table | Description |
| --- | --- |
| KNA1 | Customer Master Data |
| VBAK | Sales Document Header |
| MARA | Material Master Data |

## Next Steps

*   [Run a BW Query](/docs/guides/simple/run-bw-query.md)
*   [Advanced: RFC Metadata](/docs/guides/advanced/rfc-metadata.md)

# run-bw-query

# Understand and Execute a BW Query

This guide demonstrates how to work with SAP BW (Business Warehouse) queries using ERPL's BICS (Business Intelligence Consumer Services) interface. You'll learn how to discover query metadata, build OLAP cross-tabs, and execute queries to retrieve analytical data.

For the full function reference and protocol details, see the [BICS Protocol Deep Dive](/docs/erpl/bics.md).

## What is SAP BW?

SAP BW is SAP's data warehouse solution that provides:

*   **Multidimensional Data Model**: Cubes, dimensions, and key figures
*   **OLAP Functionality**: Online Analytical Processing capabilities
*   **Data Integration**: ETL processes from various SAP and non-SAP sources
*   **Reporting**: Pre-built reports and ad-hoc analysis

## Prerequisites

Before working with BW queries, ensure you have:

*   ERPL BICS extension installed (subscription required)
*   Access to SAP BW or BW/4HANA system
*   BW user account with query execution permissions
*   A DuckDB `sap_rfc` secret for the system (see [Connecting](/docs/erpl/bics.md#connecting))
*   Basic understanding of BW concepts (InfoProviders, queries, variables)

**BICS runs over RFC:**

BICS uses the same `sap_rfc` secret as the RFC extension — there is no separate BW endpoint. Create a secret once per session and every `sap_bics_*` function picks it up.

## Understanding BW Query Structure

### Key Components

1.  **InfoProvider**: The data source (InfoCube, DSO, CompositeProvider)
2.  **Characteristics**: Dimensions for analysis (e.g., Customer, Product, Time)
3.  **Key Figures**: Measures to analyze (e.g., Sales Amount, Quantity)
4.  **Variables**: Dynamic parameters for queries
5.  **Filters**: Static restrictions on data

### Query Metadata

Every BW query contains metadata that describes its structure:

```
-- Get query metadata (characteristics, key figures, variables)SELECT * FROM sap_bics_describe('0D_NW_C01', '0D_FC_NW_C01_Q0011');
```

This returns the `technical_name`, `text`, the `characteristics` and `keyfigures` structs, and the query's `variables`.

## Discovering Available Queries

### List All Queries

```
-- List all available queriesSELECT * FROM sap_bics_show(obj_type => 'QUERY');-- Or use the dedicated helper with searchSELECT * FROM sap_bics_show_queries(search => 'Q0011');
```

### Get Query Details

```
-- Get detailed information about a specific querySELECT * FROM sap_bics_describe('0D_NW_C01', '0D_FC_NW_C01_Q0011');
```

## The query execution model

BICS queries are **stateful**. You open a query state and give it an `id`, mutate that state with separate calls (placing characteristics on the rows/columns axes and filtering members), then read the result set by the same `id`. The functions are chained by passing the `id` string — they are **not** nested inside one another.

```
-- 1. Open a state on the cubeSELECT state_id FROM sap_bics_begin('0D_NW_C01', id => 'q1');-- 2. Place characteristics on the rows axisSELECT state_id FROM sap_bics_rows('q1', '0D_NW_PROD', op => 'SET');-- 3. Restrict members with a filter (Division = 7)SELECT state_id FROM sap_bics_filter('q1', '0D_NW_DIV', '7', op => 'SET');-- 4. Read the result setSELECT * FROM sap_bics_result('q1');
```

The result columns are the cube's own characteristic and key-figure technical names — for `0D_NW_C01` that includes `"0D_NW_PROD"`, `"0D_NW_NETV"` (net value), and `"0D_NW_QUANT"` (quantity). Quote them with double quotes because they start with a digit.

## Working with Query Variables

Variables in BW queries (e.g. "Enter fiscal year", "Select date range") are prompts a user fills before execution in SAP Analysis for Office.

**Variable metadata is read-only in ERPL:**

ERPL can **read** a query's variable definitions (via `sap_bics_describe`), but **filling** variable values is not yet supported. Queries that have variables execute with their default/empty values. To restrict the result, place the characteristic on an axis and apply a `sap_bics_filter` instead.

```
-- Inspect the variables a query definesSELECT technical_name, variablesFROM sap_bics_describe('0D_NW_C01', '0D_FC_NW_C01_Q0011');-- Restrict the result with a filter rather than a variableSELECT state_id FROM sap_bics_begin('0D_NW_C01', id => 'v1');SELECT state_id FROM sap_bics_rows('v1', '0D_NW_PROD', op => 'SET');SELECT state_id FROM sap_bics_filter('v1', '0CALMONTH', '202401', op => 'SET');SELECT * FROM sap_bics_result('v1');
```

## Practical Examples

### Example 1: Net value by product and division

```
SELECT state_id FROM sap_bics_begin('0D_NW_C01', id => 'sales');SELECT state_id FROM sap_bics_rows('sales', '0D_NW_PROD', op => 'SET');SELECT state_id FROM sap_bics_filter('sales', '0D_NW_DIV', '7', op => 'SET');SELECT    "0D_NW_PROD"  AS product,    "0D_NW_NETV"  AS net_value,    "0D_NW_QUANT" AS quantityFROM sap_bics_result('sales')ORDER BY net_value DESC;
```

### Example 2: Country breakdown

```
SELECT state_id FROM sap_bics_begin('0D_NW_C01', id => 'geo');SELECT state_id FROM sap_bics_rows('geo', '0D_NW_CNTRY', op => 'SET');SELECT    "0D_NW_CNTRY" AS country,    "0D_NW_NETV"  AS net_valueFROM sap_bics_result('geo')ORDER BY net_value DESC;
```

### Example 3: Product × month cross-tab

```
-- Products on rows, calendar month on columnsSELECT state_id FROM sap_bics_begin('0D_NW_C01', id => 'xtab');SELECT state_id FROM sap_bics_rows('xtab', '0D_NW_PROD', op => 'SET');SELECT state_id FROM sap_bics_columns('xtab', '0CALMONTH', op => 'SET');SELECT * FROM sap_bics_result('xtab');
```

## Advanced Query Operations

### Drill-Down Analysis

Add a characteristic to the rows axis to drill deeper. `op => 'ADD'` extends the current axis instead of replacing it:

```
-- High-level: net value by countrySELECT state_id FROM sap_bics_begin('0D_NW_C01', id => 'drill');SELECT state_id FROM sap_bics_rows('drill', '0D_NW_CNTRY', op => 'SET');SELECT "0D_NW_CNTRY" AS country, "0D_NW_NETV" AS net_valueFROM sap_bics_result('drill')ORDER BY net_value DESC;-- Drill down: add product under countrySELECT state_id FROM sap_bics_rows('drill', '0D_NW_PROD', op => 'ADD');SELECT "0D_NW_CNTRY" AS country, "0D_NW_PROD" AS product, "0D_NW_NETV" AS net_valueFROM sap_bics_result('drill')ORDER BY country, net_value DESC;
```

### Time Series Analysis

```
-- Monthly net valueSELECT state_id FROM sap_bics_begin('0D_NW_C01', id => 'time');SELECT state_id FROM sap_bics_rows('time', '0CALMONTH', op => 'SET');SELECT "0CALMONTH" AS calmonth, "0D_NW_NETV" AS net_valueFROM sap_bics_result('time')ORDER BY calmonth;
```

### Comparative Analysis

Build two states — one per slice — and join their result sets in SQL:

```
-- State A: division 7SELECT state_id FROM sap_bics_begin('0D_NW_C01', id => 'div7');SELECT state_id FROM sap_bics_rows('div7', '0D_NW_PROD', op => 'SET');SELECT state_id FROM sap_bics_filter('div7', '0D_NW_DIV', '7', op => 'SET');-- State B: division 15SELECT state_id FROM sap_bics_begin('0D_NW_C01', id => 'div15');SELECT state_id FROM sap_bics_rows('div15', '0D_NW_PROD', op => 'SET');SELECT state_id FROM sap_bics_filter('div15', '0D_NW_DIV', '15', op => 'SET');WITH a AS (SELECT "0D_NW_PROD" AS product, "0D_NW_NETV" AS netv_7  FROM sap_bics_result('div7')),     b AS (SELECT "0D_NW_PROD" AS product, "0D_NW_NETV" AS netv_15 FROM sap_bics_result('div15'))SELECT a.product, a.netv_7, b.netv_15, (a.netv_7 - b.netv_15) AS differenceFROM a JOIN b ON a.product = b.product;
```

## Performance Optimization

### Query Optimization Tips

1.  **Filter early**: restrict members with `sap_bics_filter` to shrink the result set
2.  **Limit the drilldown**: only place the characteristics you need on the rows/columns axes
3.  **Reuse the state**: keep mutating the same `id` instead of re-opening for each variation
4.  **Post-process in DuckDB**: aggregate, sort, and join the result set with ordinary SQL

### Example: Filtered execution

```
SELECT state_id FROM sap_bics_begin('0D_NW_C01', id => 'opt');SELECT state_id FROM sap_bics_rows('opt', '0D_NW_PROD', op => 'SET');SELECT state_id FROM sap_bics_filter('opt', '0CALMONTH', '202401', op => 'SET');SELECT state_id FROM sap_bics_filter('opt', '0D_NW_DIV', '7', op => 'SET');SELECT "0D_NW_PROD" AS product, "0D_NW_NETV" AS net_valueFROM sap_bics_result('opt')WHERE "0D_NW_NETV" > 1000        -- additional client-side filtering in DuckDBORDER BY net_value DESCLIMIT 100;
```

## Error Handling

BICS surfaces server-side problems (unknown query, missing authorization, invalid characteristic) as DuckDB **errors** — the statement fails rather than returning a status column. Wrap risky steps in your pipeline and check connectivity up front:

```
-- Confirm the system is reachable before running a queryPRAGMA sap_rfc_ping;-- Verify the query exists (case-sensitive) before executing itSELECT * FROM sap_bics_show_queries(search => '0D_FC_NW_C01_Q0011');
```

### Common BW Query Errors

1.  **Query/cube not found**: verify the technical name (case-sensitive) via `sap_bics_show`
2.  **Authorization issues**: ensure the user has BW analysis authorizations (RSEC)
3.  **Empty result**: check filter member _keys_ (not display texts) with `sap_bics_describe_infoobject`
4.  **Variable queries**: remember variable filling is unsupported — restrict via filters

## Integration with Data Science

### Export to Python/R

```
-- Build the state, then export its result set to CSVSELECT state_id FROM sap_bics_begin('0D_NW_C01', id => 'exp');SELECT state_id FROM sap_bics_rows('exp', '0D_NW_PROD', op => 'SET');SELECT state_id FROM sap_bics_filter('exp', '0CALMONTH', '202401', op => 'SET');COPY (SELECT * FROM sap_bics_result('exp')) TO 'sales_analysis.csv' WITH (HEADER);
```

### Machine Learning Integration

```
-- Prepare a labelled dataset from the result setSELECT state_id FROM sap_bics_begin('0D_NW_C01', id => 'ml');SELECT state_id FROM sap_bics_rows('ml', '0D_NW_PROD', op => 'SET');SELECT    "0D_NW_PROD"  AS product,    "0D_NW_NETV"  AS net_value,    "0D_NW_QUANT" AS quantity,    CASE WHEN "0D_NW_NETV" > 10000 THEN 'HIGH_VALUE' ELSE 'STANDARD' END AS segmentFROM sap_bics_result('ml');
```

## Best Practices

### 1\. Query Design

*   Use the cube's real technical names for characteristics and key figures
*   Inspect structure with `sap_bics_describe` before building a cross-tab
*   Test queries with different filter combinations

### 2\. Filtering

*   Filter on member **keys**, not display texts
*   Use `op => 'SET' | 'ADD' | 'REMOVE'` to manage selections precisely
*   Apply restrictions on the SAP side with `sap_bics_filter`; refine further in DuckDB

### 3\. Data Quality

*   Verify data completeness for the slice you filtered
*   Handle missing or initial values explicitly
*   Cross-check totals against a known report

### 4\. Security

*   Use a `sap_rfc` secret (SNC for encrypted connections)
*   Rely on BW analysis authorizations to scope access
*   Audit RFC logons via the SAP security audit log

## Troubleshooting

### Common Issues

1.  **Slow performance**: filter and limit the drilldown before reading the result
2.  **Memory**: aggregate in DuckDB rather than pulling every cell
3.  **Connection problems**: check connectivity with `PRAGMA sap_rfc_ping`
4.  **Authorization errors**: verify BW user permissions

### Debugging Tips

```
-- Enable ERPL tracing to see the underlying RFC callsSET erpl_trace_enabled = TRUE;SET erpl_trace_level   = 'DEBUG';   -- TRACE | DEBUG | INFO | WARN | ERROR-- Test with no filtersSELECT state_id FROM sap_bics_begin('0D_NW_C01', id => 'dbg');SELECT state_id FROM sap_bics_rows('dbg', '0D_NW_PROD', op => 'SET');SELECT * FROM sap_bics_result('dbg');-- Check query metadataSELECT * FROM sap_bics_describe('0D_NW_C01', '0D_FC_NW_C01_Q0011');
```

# Call SAP BAPI Functions

This guide shows you how to call SAP BAPI functions using ERPL. BAPIs are standardized programming interfaces that provide stable access to SAP business objects.

## What are BAPIs?

BAPIs (Business Application Programming Interfaces) are standardized interfaces that enable external applications to access SAP business objects. They provide:

*   **Stable Interface**: BAPIs maintain backward compatibility
*   **Business Logic**: Encapsulate complex business processes
*   **Data Validation**: Built-in validation and error handling
*   **Transaction Support**: Support for SAP transactions

## Prerequisites

Before calling BAPI functions, ensure you have:

*   ERPL extension installed and loaded
*   Active connection to SAP ERP system
*   SAP user with BAPI execution authorizations
*   Knowledge of the specific BAPI you want to call

## Basic BAPI Call Syntax

The basic syntax for calling BAPI functions is:

```
SELECT * FROM sap_rfc_invoke(    'BAPI_FUNCTION_NAME',    {'PARAMETER1': 'value1', 'PARAMETER2': 'value2'});
```

## Example: Customer Master Data BAPI

Let's call the `BAPI_CUSTOMER_GETDETAIL2` BAPI to retrieve customer details:

```
-- Call BAPI to get customer detailsSELECT * FROM sap_rfc_invoke(    'BAPI_CUSTOMER_GETDETAIL2',    {'CUSTOMERNO': '0000001000'});
```

### Understanding BAPI Parameters

BAPI functions typically have different parameter types:

1.  **Import Parameters**: Input values
2.  **Export Parameters**: Output values
3.  **Table Parameters**: Input/output tables
4.  **Changing Parameters**: Input/output values

## Advanced BAPI Examples

### Example 1: Material Master Data

Retrieve material information using `BAPI_MATERIAL_GET_DETAIL`:

```
SELECT * FROM sap_rfc_invoke(    'BAPI_MATERIAL_GET_DETAIL',    {'MATERIAL': 'MAT-001', 'PLANT': '1000'});
```

### Example 2: Sales Order Creation

Create a sales order using `BAPI_SALESORDER_CREATEFROMDAT2`:

```
-- Prepare order header dataSELECT * FROM sap_rfc_invoke(    'BAPI_SALESORDER_CREATEFROMDAT2',    {        'ORDER_HEADER_IN': {            'DOC_TYPE': 'OR',            'SALES_ORG': '1000',            'DISTR_CHAN': '10',            'DIVISION': '00'        },        'ORDER_ITEMS_IN': [            {                'ITM_NUMBER': '000010',                'MATERIAL': 'MAT-001',                'REQ_QTY': '10'            }        ]    });
```

### Example 3: Financial Document Posting

Post a financial document using `BAPI_ACC_DOCUMENT_POST`:

```
SELECT * FROM sap_rfc_invoke(    'BAPI_ACC_DOCUMENT_POST',    {        'DOCUMENTHEADER': {            'COMP_CODE': '1000',            'DOC_DATE': '2024-01-15',            'PSTNG_DATE': '2024-01-15',            'DOC_TYPE': 'DR'        },        'ACCOUNTGL': [            {                'ITEMNO_ACC': '0000000001',                'GL_ACCOUNT': '0000400000',                'DEBIT_CREDIT': 'S',                'AMT_DOCCUR': '1000.00'            }        ]    });
```

## Working with BAPI Return Values

BAPI functions return structured data including:

*   **Return Messages**: Success/error messages
*   **Export Parameters**: Output values
*   **Table Parameters**: Result tables

### Handling Return Messages

```
-- Call BAPI and check return messagesWITH bapi_result AS (    SELECT * FROM sap_rfc_invoke(        'BAPI_CUSTOMER_GETDETAIL2',        {'CUSTOMERNO': '0000001000'}    ))SELECT     message_type,    message_id,    message_number,    message_textFROM bapi_resultWHERE parameter_name = 'RETURN';
```

### Processing Export Parameters

```
-- Extract specific export parametersWITH bapi_result AS (    SELECT * FROM sap_rfc_invoke(        'BAPI_CUSTOMER_GETDETAIL2',        {'CUSTOMERNO': '0000001000'}    ))SELECT     parameter_valueFROM bapi_resultWHERE parameter_name = 'CUSTOMERADDRESS';
```

## Error Handling

### Common BAPI Errors

1.  **Authorization Errors**: Insufficient permissions
2.  **Parameter Errors**: Invalid or missing parameters
3.  **Business Logic Errors**: Validation failures
4.  **System Errors**: Technical issues

### Error Handling Example

```
-- Call BAPI with error handlingWITH bapi_result AS (    SELECT * FROM sap_rfc_invoke(        'BAPI_CUSTOMER_GETDETAIL2',        {'CUSTOMERNO': 'INVALID_CUSTOMER'}    )),error_check AS (    SELECT         CASE             WHEN message_type = 'E' THEN 'ERROR'            WHEN message_type = 'W' THEN 'WARNING'            ELSE 'SUCCESS'        END as status,        message_text    FROM bapi_result    WHERE parameter_name = 'RETURN')SELECT * FROM error_check;
```

## Best Practices

### 1\. Parameter Validation

Always validate parameters before calling BAPIs:

```
-- Validate customer number existsSELECT COUNT(*) as customer_existsFROM sap_read_table('KNA1')WHERE KUNNR = '0000001000';
```

### 2\. Transaction Handling

For BAPIs that modify data, follow the SAP-side transactional pattern: call the operation BAPI, then call `BAPI_TRANSACTION_COMMIT` (or `BAPI_TRANSACTION_ROLLBACK`) over the **same RFC session**. ERPL preserves the session for the duration of a single `sap_rfc_invoke` chain when you compose them in one statement:

```
-- Example: create a customer, then commitWITH create_result AS (  SELECT * FROM sap_rfc_invoke(    'BAPI_CUSTOMER_CREATEFROMDATA1',    {'PI_PERSONALDATA': {'FIRSTNAME': 'Alice'}}  ))SELECT * FROM sap_rfc_invoke('BAPI_TRANSACTION_COMMIT');
```

If the create returns errors in its `RETURN` table, invoke `BAPI_TRANSACTION_ROLLBACK` instead. Commit and rollback are standard SAP BAPIs called via `sap_rfc_invoke`.

### 3\. Performance Optimization

*   Use specific BAPIs for your use case
*   Limit data retrieval with filters
*   Cache frequently used data
*   Use batch processing for multiple calls

### 4\. Security Considerations

*   Use secure connections (SAP Router)
*   Implement proper authorization checks
*   Log BAPI calls for audit purposes
*   Validate all input parameters

## BAPI Discovery

### Finding Available BAPIs

1.  **SAP Help**: Check SAP documentation
2.  **SE37 Transaction**: Use SAP GUI to explore function modules
3.  **BAPI Explorer**: Use SAP's BAPI Explorer tool
4.  **SAP Community**: Search for BAPI examples

### Testing BAPIs

Before using BAPIs in production:

1.  **Test in Development**: Always test in development system first
2.  **Validate Parameters**: Ensure all required parameters are provided
3.  **Check Authorizations**: Verify user has necessary permissions
4.  **Monitor Performance**: Test with realistic data volumes

## Troubleshooting

### Common Issues

1.  **BAPI Not Found**: Verify BAPI name and availability
2.  **Parameter Errors**: Check parameter names and types
3.  **Authorization Issues**: Contact SAP administrator
4.  **Performance Problems**: Optimize queries and use filters

### Debugging Tips

```
-- Enable ERPL tracing to see the underlying RFC callsSET erpl_trace_enabled = TRUE;SET erpl_trace_level   = 'DEBUG';-- Test with minimal parametersSELECT * FROM sap_rfc_invoke('BAPI_NAME', {});-- Check the BAPI's RETURN table for messages (TYPE 'E' = error, 'A' = abort)SELECT * FROM sap_rfc_invoke('BAPI_NAME', {}, path => '/RETURN')WHERE TYPE IN ('E', 'A');
```

## Next Steps

Now that you understand BAPI calls, explore:

*   [ERP Table Access](/docs/key_tasks/erp_table.md) - Reading SAP tables directly
*   [BW Query Execution](/docs/key_tasks/bw_query.md) - Working with SAP BW
*   [ODP Replication](/docs/key_tasks/replicate_odp.md) - Data replication
*   [SQL Reference](/docs/reference.md) - Complete function reference

## Additional Resources

*   [SAP BAPI Documentation](https://help.sap.com/viewer/)
*   [ERPL GitHub Repository](https://github.com/datazoode/erpl)
*   [SAP Community](https://community.sap.com/)
*   [Contact Support](/contact) - Get help with specific BAPI issues

# Load an ERP Table

## Introduction

SAP ERP essentially keeps all of its data in a relational schema. This means that all the data is stored in tables and the relationships between the tables are defined by foreign keys. This is a very common approach in the database world and is also used by many other ERP systems. The main difference between SAP ERP and other ERP systems is that SAP ERP is highly customizable. This means that the tables and their relationships can be changed by the customer. In this document we show you

*   How to show you **available ERP tables**, their columns and data types
*   How to get **texts and other metadata** for the columns
*   How to **query ERP tables** directly from DuckDB

## Get yourself prepared

**Required authority objects (RFC):**

To query a table, far reaching access has to be granted to the user issuing the query. For more information, refer our documentation which designated authority objects (RFC) must be created.

## List all available ERP tables

A typical SAP ERP system contains thousands of tables. To find the table you are looking for, you can use the `sap_show_tables` function. This function returns a list of all available tables in the SAP ERP system. The function has the following signature:

```
SELECT * FROM sap_show_tables()
```

This will return a (long) list of tables. To find a specific table, you can supply a search string to the function. The search string is matched against the table name and the table description. For example, to find all tables starting with `FLIGHT` you can use the following query:

```
SELECT * FROM sap_show_tables()WHERE table_name LIKE '%SPFL%'
```

The star operator `*` is a wildcard and matches any number of characters. Supplying no search string is equvivalent to `*`. The result of the query will look like this:

\[Continue reading the full documentation...\]

# Delta Replicate Stock Data from SAP ERP to Parquet

This guide demonstrates how to use SAP ODP (Operational Data Provisioning) to replicate stock data from SAP ERP to DuckDB and export it to Parquet files. ODP provides a standardized way to extract and replicate data from SAP systems.

## What is SAP ODP?

SAP ODP (Operational Data Provisioning) is a framework that provides:

*   **Standardized Data Extraction**: Consistent interface for data replication
*   **Delta Capabilities**: Incremental data updates
*   **Real-time Processing**: Near real-time data availability
*   **Multiple Formats**: Support for various data formats
*   **Error Handling**: Built-in error recovery and monitoring

## Prerequisites

Before starting ODP replication, ensure you have:

*   ERPL ODP extension installed (subscription required)
*   Access to SAP ERP system with ODP enabled
*   ODP extractors configured for your data sources
*   User account with ODP authorizations
*   Sufficient storage space for Parquet files

## Understanding ODP Extractors

### Types of ODP Extractors

1.  **CDS Views**: Core Data Services views as ODP sources
2.  **BW Extractors**: Traditional BW extractors
3.  **Function Modules**: Custom function modules
4.  **Tables**: Direct table access

### Stock Data Extractors

Common ODP extractors for stock data:

*   **MCHB**: Material stock by storage location
*   **MSKA**: Material stock by sales area
*   **MSLB**: Material stock by storage location and batch
*   **MCH1**: Material stock by plant

## Setting Up ODP Replication

### Step 1: Discover Available Extractors

```
-- List all available ODP contextsSELECT * FROM sap_odp_show_contexts();-- List extractors in BW contextSELECT * FROM sap_odp_show('BW');-- Search for stock-related extractorsSELECT * FROM sap_odp_show('BW')WHERE data_source LIKE '%STOCK%';
```

### Step 2: Get Extractor Metadata

```
-- Get detailed information about a specific extractorSELECT * FROM sap_odp_describe('BW', 'MCHB$F');
```

This returns information about:

*   Available fields
*   Key fields
*   Data types
*   Extraction methods

### Step 3: Configure the SAP Connection

ERPL uses DuckDB secrets to authenticate. Create one once per system:

```
CREATE SECRET erp_prod (    TYPE sap_rfc,    ASHOST 'your-sap-server.com',    SYSNR  '00',    CLIENT '100',    USER   'your-username',    PASSWD 'your-password',    LANG   'EN');
```

Subsequent ERPL calls pick up this secret automatically. Pass `secret => 'erp_prod'` if you have several.

## Basic ODP Replication

### Simple Full Replication

`sap_odp_read_full` is a one-shot scan: it opens a FULL cursor on SAP, streams the snapshot, and closes the cursor. No subscription survives.

```
-- Replicate all stock data from MCHB extractorSELECT * FROM sap_odp_read_full('BW', 'MCHB$F');-- Project columns and parallelizeSELECT * FROM sap_odp_read_full(    'BW', 'MCHB$F',    columns => ['MATNR', 'WERKS', 'CLABS'],    threads => 4);
```

### Server-side filters

```
-- Restrict to a single plant via the ODP filter pushdownSELECT * FROM sap_odp_read_full(    'BW', 'MCHB$F',    filters => [{      'FIELDNAME': 'WERKS',      'SIGN':      'I',      'OP':        'EQ',      'LOW':       '1000',      'HIGH':      ''    }]);
```

## Delta Replication

### Understanding Delta Replication

`sap_odp_read_delta` is the delta-capable companion to `sap_odp_read_full`. It takes a third positional argument — `subscriber_process` — that keys a server-side delta pointer:

1.  **First call** with a new `subscriber_process` → SAP performs auto-DELTAINIT, returns the full current snapshot AND registers the pointer.
2.  **Subsequent calls** with the same `subscriber_process` → returns only the changes since the previous call.
3.  **Recovery** → pass `recover => true` to re-stream the last unconfirmed packet without advancing the pointer.

Pick a stable, descriptive `subscriber_process` per pipeline (e.g. `'STOCK_DELTA_DAILY'`). It is the identity that ties calls together; do not change it between runs.

### Initial Delta Setup

```
-- First call: auto-DELTAINIT.  Returns the full snapshot AND registers-- the delta pointer under subscriber_process 'STOCK_DELTA_DAILY'.SELECT * FROM sap_odp_read_delta('BW', 'MCHB$F', 'STOCK_DELTA_DAILY');-- Confirm the cursor existsSELECT * FROM sap_odp_show_cursors(subscriber_name => 'ERPL')WHERE subscriber_proc = 'STOCK_DELTA_DAILY';
```

### Delta Replication Process

```
-- 1. Cheap probe — has anything changed since last run?SELECT last_modified FROM sap_odp_get_last_modified('BW', 'MCHB$F');-- 2. Pull the deltaSELECT * FROM sap_odp_read_delta('BW', 'MCHB$F', 'STOCK_DELTA_DAILY');-- 3. Inspect cursor stateSELECT * FROM sap_odp_show_cursors(subscriber_name => 'ERPL');-- 4. Release the cursor when the pipeline shuts down (subscription stays;--    next run resumes from the same pointer).PRAGMA sap_odp_close_delta_cursor('BW', 'STOCK_DELTA_DAILY', 'MCHB$F');
```

## Complete Stock Data Replication Example

### Step 1: Setup Replication Environment

```
-- Create the target table by introspecting the source schemaCREATE TABLE stock_data ASSELECT * FROM sap_odp_read_full('BW', 'MCHB$F') WHERE 1=0;-- Optional: a run log driven by ERPL's own stateCREATE TABLE replication_run_log (    subscriber_proc VARCHAR,    odp_name        VARCHAR,    run_started     TIMESTAMP,    rows_inserted   BIGINT);
```

### Step 2: Initial Delta Load (DELTAINIT)

```
-- First call returns the full snapshot AND registers the delta pointerINSERT INTO stock_dataSELECT * FROM sap_odp_read_delta('BW', 'MCHB$F', 'STOCK_DELTA_DAILY');-- Record the runINSERT INTO replication_run_logSELECT 'STOCK_DELTA_DAILY', 'MCHB$F', now(), COUNT(*) FROM stock_data;
```

### Step 3: Scheduled Delta Pull

DuckDB doesn't have PL/pgSQL — schedule the SQL below from your orchestrator (cron, Airflow, dbt, …). Each invocation pulls only the changes since the previous one:

```
-- 1. Probe — skip the run if nothing changedWITH probe AS (  SELECT last_modified FROM sap_odp_get_last_modified('BW', 'MCHB$F'))SELECT 'skipped' AS statusWHERE (SELECT last_modified FROM probe) = (  SELECT COALESCE(MAX(EPOCH(run_started)), 0) FROM replication_run_log  WHERE subscriber_proc = 'STOCK_DELTA_DAILY');-- 2. Pull the deltaINSERT INTO stock_dataSELECT * FROM sap_odp_read_delta('BW', 'MCHB$F', 'STOCK_DELTA_DAILY');-- 3. Log the runINSERT INTO replication_run_logSELECT 'STOCK_DELTA_DAILY', 'MCHB$F', now(),       (SELECT COUNT(*) FROM stock_data);
```

### Step 4: Export to Parquet

```
-- Export stock data to ParquetCOPY (    SELECT         MATNR as material_number,        WERKS as plant,        LGORT as storage_location,        CHARG as batch,        CLABS as unrestricted_stock,        CUMLM as stock_in_transit,        CINSM as stock_in_quality_inspection    FROM stock_data) TO 'stock_data.parquet' WITH HEADER;
```

## Advanced ODP Features

### Error Handling

```
-- Inspect cursors for this pipelineSELECT * FROM sap_odp_show_cursors(subscriber_name => 'ERPL')WHERE subscriber_proc = 'STOCK_DELTA_DAILY';-- Re-stream the last unconfirmed packet (no pointer advance)SELECT * FROM sap_odp_read_delta(    'BW', 'MCHB$F', 'STOCK_DELTA_DAILY',    recover => true);-- If the cursor is wedged, hard-reset and let DELTAINIT re-snapshotPRAGMA sap_odp_drop('BW', 'ERPL', 'STOCK_DELTA_DAILY', 'MCHB$F');SELECT * FROM sap_odp_read_delta('BW', 'MCHB$F', 'STOCK_DELTA_DAILY');
```

### Performance Optimization

```
-- Project columns and parallelize at the ODP fetch levelSELECT * FROM sap_odp_read_delta(    'BW', 'MCHB$F', 'STOCK_DELTA_DAILY',    columns => ['MATNR', 'WERKS', 'CLABS'],    threads => 4);-- Preview data before extraction (no cursor side effects)SELECT * FROM sap_odp_preview('BW', 'MCHB$F');
```

### Data Transformation

```
-- Transform data during extractionSELECT    MATNR as material_number,    WERKS as plant,    LGORT as storage_location,    CLABS as stock_quantity,    CASE        WHEN CLABS > 1000 THEN 'HIGH_STOCK'        WHEN CLABS > 100 THEN 'MEDIUM_STOCK'        ELSE 'LOW_STOCK'    END as stock_levelFROM sap_odp_read_delta('BW', 'MCHB$F', 'STOCK_DELTA_DAILY');
```

## Monitoring and Maintenance

### Replication Monitoring

```
-- Inspect ERPL-owned subscriptionsSELECT queue_name, subscriber_type, subscriber_name, subscriber_procFROM sap_odp_show_subscriptions();-- Cross-team visibility on a source — who else is reading it?SELECT * FROM sap_odp_get_subscriptions('BW', 'MCHB$F');-- Monitor extraction cursors (request_date = last cursor activity)SELECT subscriber_proc, pointer, is_closed, is_delta_extension, request_dateFROM sap_odp_show_cursors(subscriber_name => 'ERPL');
```

### Data Quality Checks

```
-- Validate data completenessSELECT     COUNT(*) as total_records,    COUNT(DISTINCT MATNR) as unique_materials,    COUNT(DISTINCT WERKS) as unique_plantsFROM stock_data;-- Check for data anomaliesSELECT     MATNR,    WERKS,    CLABSFROM stock_dataWHERE CLABS < 0 OR CLABS > 1000000;
```

### Cleanup and Archiving

```
-- Archive old dataCREATE TABLE stock_data_archive ASSELECT * FROM stock_dataWHERE extraction_date < CURRENT_DATE - INTERVAL '1 year';-- Clean up archived dataDELETE FROM stock_dataWHERE extraction_date < CURRENT_DATE - INTERVAL '1 year';
```

## Integration with Cloud Platforms

### AWS S3 Integration

```
-- Export to S3COPY (    SELECT * FROM stock_data) TO 's3://your-bucket/stock-data/stock_data.parquet'WITH HEADER;
```

### Google Cloud Storage

```
-- Export to GCSCOPY (    SELECT * FROM stock_data) TO 'gs://your-bucket/stock-data/stock_data.parquet'WITH HEADER;
```

### Azure Blob Storage

```
-- Export to Azure BlobCOPY (    SELECT * FROM stock_data) TO 'az://your-container/stock-data/stock_data.parquet'WITH HEADER;
```

## Best Practices

### 1\. Delta Replication Strategy

*   Use appropriate delta modes (FULL, DELTA, INIT)
*   Monitor delta tokens regularly
*   Implement error recovery mechanisms
*   Schedule regular delta extractions

### 2\. Performance Optimization

*   Use parallel extraction where possible
*   Implement data compression
*   Optimize filter conditions
*   Monitor extraction performance

### 3\. Data Quality

*   Implement data validation rules
*   Monitor data completeness
*   Handle missing or invalid data
*   Maintain data lineage

### 4\. Security and Compliance

*   Use secure connections
*   Implement proper authorization
*   Log all extractions
*   Protect sensitive data

## Troubleshooting

### Common Issues

1.  **Extraction Failures**: Inspect the cursor in `sap_odp_show_cursors`; try `recover => true` before resetting
2.  **Stuck Cursor**: `PRAGMA sap_odp_drop` wipes the subscription; the next call auto-DELTAINITs
3.  **Performance Problems**: Project `columns`, push `filters` server-side, raise `threads`
4.  **Data Quality Issues**: Add post-extraction validation queries against the staging table

### Debugging Tips

```
-- Enable ERPL traceSET erpl_trace_enabled = TRUE;SET erpl_trace_level = 'DEBUG';-- Inspect cursor state for this pipelineSELECT * FROM sap_odp_show_cursors(subscriber_name => 'ERPL')WHERE subscriber_proc = 'STOCK_DELTA_DAILY';-- See every subscriber on the source (not just ERPL's)SELECT * FROM sap_odp_get_subscriptions('BW', 'MCHB$F');-- Cheap preview with no cursor side effectsSELECT * FROM sap_odp_preview('BW', 'MCHB$F');
```

## Next Steps

Now that you understand ODP replication, explore:

*   [ERP Table Access](/docs/key_tasks/erp_table.md) - Reading SAP ERP tables
*   [BAPI Function Calls](/docs/key_tasks/erp_bapi.md) - Calling SAP BAPIs
*   [BW Query Execution](/docs/key_tasks/bw_query.md) - Working with SAP BW
*   [SQL Reference](/docs/reference.md) - Complete function reference

## Additional Resources

*   [SAP ODP Documentation](https://help.sap.com/viewer/)
*   [ERPL GitHub Repository](https://github.com/datazoode/erpl)
*   [SAP Community](https://community.sap.com/)
*   [Contact Support](/contact) - Get help with ODP replication

# configuration

# ERPL Function Reference

Complete reference for all ERPL functions organized by protocol. This reference covers RFC, BICS, and ODP functions with parameters, return types, and examples.

## RFC Functions

### Core RFC Functions

#### `sap_read_table()`

Read data from SAP tables.

**Signature:**

```
sap_read_table(table_name VARCHAR, [COLUMNS LIST(VARCHAR)], [FILTER VARCHAR], [MAX_ROWS UINTEGER], [THREADS UINTEGER], [READ_TABLE_FUNCTION VARCHAR], [READ_TABLE_DELIMITER VARCHAR], [SECRET VARCHAR])
```

**Parameters:**

| Parameter | Type | Description | Example |
| --- | --- | --- | --- |
| `table_name` | VARCHAR (positional) | SAP table name | `'KNA1'` |
| `COLUMNS` | LIST(VARCHAR) | Restrict to a subset of columns | `COLUMNS => ['KUNNR', 'NAME1']` |
| `FILTER` | VARCHAR | OpenSQL-style WHERE-fragment passed straight to RFC. Use for filters too complex to push down. | `FILTER => 'LAND1 = ''DE'''` |
| `MAX_ROWS` | UINTEGER | Maximum rows to return | `MAX_ROWS => 1000` |
| `THREADS` | UINTEGER | Number of parallel RFC threads to read partitions (default: 5) | `THREADS => 4` |
| `READ_TABLE_FUNCTION` | VARCHAR | Override the underlying RFC function module (e.g. `/SAPDS/RFC_READ_TABLE2` for wide rows) | `READ_TABLE_FUNCTION => '/SAPDS/RFC_READ_TABLE2'` |
| `READ_TABLE_DELIMITER` | VARCHAR | Delimiter used by RFC when packing rows. Override for tables whose values may contain whitespace. | \`READ\_TABLE\_DELIMITER => ' |
| `SECRET` | VARCHAR | Name of the DuckDB secret to authenticate with (if multiple SAP connections are configured) | `SECRET => 'erp_prod'` |

**Predicate and projection pushdown:**

A regular SQL `WHERE` clause is automatically pushed down to RFC — you don't need the `FILTER` named parameter for simple predicates:

```
SELECT KUNNR, NAME1 FROM sap_read_table('KNA1') WHERE LAND1 = 'DE';
```

`COLUMNS` is similarly inferred from the SELECT list. Reach for the named parameters when you need precise control over what RFC receives — for example, complex multi-table filters that DuckDB's optimizer can't safely lower.

**Ordering:**

RFC's underlying `RFC_READ_TABLE` has no server-side sort. Apply `ORDER BY` in DuckDB after the read; it executes client-side.

**Example:**

```
SELECT * FROM sap_read_table('KNA1', MAX_ROWS => 100);
```

#### `sap_rfc_invoke()`

Call SAP function modules and BAPIs.

**Signature:**

```
sap_rfc_invoke(function_name VARCHAR, [parameters...], [path VARCHAR], [secret VARCHAR])
```

Input parameters are passed as **positional struct arguments** after the function name — one struct per import / changing parameter the BAPI expects. DuckDB struct syntax (`{'KEY': value, ...}`) lets you express nested structures and tables naturally.

**Parameters:**

| Parameter | Type | Description | Example |
| --- | --- | --- | --- |
| `function_name` | VARCHAR (positional) | SAP function module / BAPI name | `'BAPI_FLIGHT_GETLIST'` |
| `parameters...` | STRUCT (positional varargs) | One struct per RFC import/changing parameter | `{'AIRLINE': 'LH', 'DESTINATION_FROM': {'AIRPORTID': 'FRA'}}` |
| `path` | VARCHAR (named) | Optional path into the response structure — picks a sub-table of the result | `path => '/FLIGHT_LIST'` |
| `secret` | VARCHAR (named) | Name of the DuckDB secret to authenticate with | `secret => 'erp_prod'` |

**Example:**

```
-- Simple call with named import parametersSELECT trim(ECHOTEXT)FROM sap_rfc_invoke('STFC_CONNECTION', {'REQUTEXT': 'Hello'});-- BAPI call selecting a nested result table via pathSELECT *FROM sap_rfc_invoke(  'BAPI_FLIGHT_GETLIST',  {'AIRLINE': 'LH', 'DESTINATION_FROM': {'AIRPORTID': 'FRA'}},  path => '/FLIGHT_LIST');
```

### RFC Metadata Functions

#### `sap_rfc_describe_function()`

Get detailed information about RFC function parameters.

**Signature:**

```
sap_rfc_describe_function(function_name VARCHAR)
```

**Example:**

```
SELECT * FROM sap_rfc_describe_function('BAPI_FLIGHT_GETLIST');
```

#### `sap_describe_fields()`

Get table structure and field information.

**Signature:**

```
sap_describe_fields(table_name VARCHAR)
```

**Example:**

```
SELECT * FROM sap_describe_fields('KNA1');
```

#### `sap_rfc_authorizations()`

List which SAP RFC function module each ERPL function invokes — useful when requesting RFC authorizations from SAP Basis. Returns `extension`, `duckdb_function`, `rfc_function_module`, `invocation`, and `purpose`. Takes no arguments and needs no SAP connection.

**Signature:**

```
sap_rfc_authorizations()
```

**Example:**

```
SELECT DISTINCT rfc_function_moduleFROM sap_rfc_authorizations()WHERE extension = 'erpl_rfc';
```

#### `sap_rfc_show_function()`

List all available RFC functions.

**Signature:**

```
sap_rfc_show_function()
```

**Example:**

```
SELECT * FROM sap_rfc_show_function();
```

#### `sap_rfc_show_groups()`

List all RFC function groups.

**Signature:**

```
sap_rfc_show_groups()
```

**Example:**

```
SELECT * FROM sap_rfc_show_groups();
```

#### `sap_show_tables()`

List all available SAP tables.

**Signature:**

```
sap_show_tables()
```

**Example:**

```
SELECT * FROM sap_show_tables();
```

## BICS Functions

**BICS query composition:**

BICS queries are built **compositionally** by chaining functions: `begin` → `columns` / `rows` / `filter` → `result`. Each step returns a state identifier that the next step consumes. Set `return => 'DESCRIBE'` (default) to inspect the in-progress query plan, or `return => 'RESULT'` to short-circuit and execute.

### Basic Query Functions

#### `sap_bics_show()`

List InfoProviders (default), cubes, queries, or InfoAreas registered on the BW system.

**Signature:**

```
sap_bics_show()
```

**Parameters** (all named):

| Parameter | Type | Description | Example |
| --- | --- | --- | --- |
| `obj_type` | ENUM | One of `'INFOPROVIDER'` (default), `'QUERY'`, `'CUBE'`, `'INFOAREA'` | `obj_type => 'CUBE'` |
| `search` | VARCHAR | Glob pattern to filter results | `search => '0D_NW*'` |
| `search_in_key` | BOOLEAN | Match against object key (default: `true`) | `search_in_key => false` |
| `search_in_text` | BOOLEAN | Match against display text (default: `true`) | `search_in_text => true` |
| `fetch_levels` | UINTEGER | Hierarchy depth to traverse (default: `0`) | `fetch_levels => 2` |
| `secret` | VARCHAR | DuckDB secret name | `secret => 'erp_prod'` |

**Example:**

```
SELECT * FROM sap_bics_show();SELECT * FROM sap_bics_show(obj_type => 'CUBE', search => '0D_NW*');
```

#### `sap_bics_show_cubes()`

Convenience wrapper — list cubes only. Equivalent to `sap_bics_show(obj_type => 'CUBE')` with a slightly different result shape tuned for cubes.

**Signature:**

```
sap_bics_show_cubes()
```

**Parameters** (all named): `search`, `search_in_key`, `search_in_text`, `secret` (same semantics as `sap_bics_show`).

**Example:**

```
SELECT * FROM sap_bics_show_cubes(search => '0D_NW*');
```

#### `sap_bics_show_queries()`

Convenience wrapper — list BEx queries only.

**Signature:**

```
sap_bics_show_queries()
```

**Parameters** (all named): `search`, `search_in_key`, `search_in_text`, `secret`.

**Example:**

```
SELECT * FROM sap_bics_show_queries(search => 'ZFI*');
```

#### `sap_bics_describe()`

Describe cube structure or query.

**Signature:**

```
sap_bics_describe([cube_name VARCHAR], [query_name VARCHAR])
```

**Example:**

```
SELECT * FROM sap_bics_describe('0D_NW_C01');
```

#### `sap_bics_begin()`

Open a BICS query state against an InfoProvider or query. Pass an `id` to name the state; the other query functions reference that **string id** — they are called in sequence, **not** nested inside one another.

**Signature:**

```
sap_bics_begin(cube_or_query_name VARCHAR, [id => VARCHAR, return => 'DESCRIBE'|'RESULT', rows => VARCHAR[], columns => VARCHAR[], filters => VARCHAR[], secret => VARCHAR])
```

**Example:**

```
SELECT state_id FROM sap_bics_begin('0D_NW_C01', id => 'q1');
```

#### `sap_bics_columns()`

Manipulate the column axis: add, set, or remove characteristics and key figures.

**Signature:**

```
sap_bics_columns(state_id VARCHAR, characteristic VARCHAR, [characteristic2 VARCHAR, ...])
```

**Parameters:**

| Parameter | Type | Description | Example |
| --- | --- | --- | --- |
| `state_id` | VARCHAR (positional) | State produced by `sap_bics_begin` or a prior chain step | (chained) |
| `characteristic...` | VARCHAR (positional varargs) | One or more characteristics / key figures to operate on | `'0CALMONTH', '0SALES_AMOUNT'` |
| `id` | VARCHAR (named) | Override the state id passed in (rarely needed) | `id => 'my_state'` |
| `op` | ENUM (named) | `'SET'` (replace, default), `'ADD'` (append), `'REMOVE'` (drop) | `op => 'ADD'` |
| `return` | ENUM (named) | `'DESCRIBE'` (return plan, default) or `'RESULT'` (execute now) | `return => 'RESULT'` |
| `secret` | VARCHAR (named) | DuckDB secret name | `secret => 'erp_prod'` |

**Example:**

```
-- Add a characteristic to the column axis of state 'q1'SELECT state_id FROM sap_bics_columns('q1', '0CALMONTH', op => 'ADD');
```

#### `sap_bics_rows()`

Manipulate the row axis. Same parameter shape as `sap_bics_columns`.

**Signature:**

```
sap_bics_rows(state_id VARCHAR, characteristic VARCHAR, [characteristic2 VARCHAR, ...])
```

**Parameters** (named): `id`, `op` (`'SET'` / `'ADD'` / `'REMOVE'`), `return` (`'DESCRIBE'` / `'RESULT'`), `secret`.

**Example:**

```
SELECT state_id FROM sap_bics_rows('q1', '0D_NW_PROD', op => 'ADD');
```

#### `sap_bics_filter()`

Apply a filter on a specific characteristic.

**Signature:**

```
sap_bics_filter(state_id VARCHAR, characteristic VARCHAR, value VARCHAR, [value2 VARCHAR, ...])
```

**Parameters:**

| Parameter | Type | Description | Example |
| --- | --- | --- | --- |
| `state_id` | VARCHAR (positional) | State produced by `sap_bics_begin` or a prior chain step | (chained) |
| `characteristic` | VARCHAR (positional) | Characteristic to filter | `'0CALMONTH'` |
| `value...` | VARCHAR (positional varargs) | One or more filter values | `'202401', '202402'` |
| `op` | ENUM (named) | `'SET'` / `'ADD'` / `'REMOVE'` | `op => 'ADD'` |
| `return` | ENUM (named) | `'DESCRIBE'` (default) / `'RESULT'` | `return => 'RESULT'` |
| `secret` | VARCHAR (named) | DuckDB secret name | `secret => 'erp_prod'` |

**Example:**

```
SELECT state_id FROM sap_bics_filter('q1', '0CALMONTH', '202401', op => 'SET');
```

#### `sap_bics_result()`

Execute the chained query and return the result rows.

**Signature:**

```
sap_bics_result(state_id VARCHAR)
```

**Example:**

```
-- Full sequence: open a state, shape it, then read the result by idSELECT state_id FROM sap_bics_begin('0D_NW_C01', id => 'q1');SELECT state_id FROM sap_bics_rows('q1', '0D_NW_PROD', op => 'SET');SELECT state_id FROM sap_bics_filter('q1', '0CALMONTH', '202401', op => 'SET');SELECT * FROM sap_bics_result('q1');
```

### Intermediate Query Functions

#### `sap_bics_show_hierarchies()`

List available hierarchies.

**Signature:**

```
sap_bics_show_hierarchies()
```

**Example:**

```
SELECT * FROM sap_bics_show_hierarchies();
```

#### `sap_bics_hierarchy()`

Query hierarchy data — flat list (default) or as a recursive tree.

**Signature:**

```
sap_bics_hierarchy(hierarchy_name VARCHAR)
```

**Parameters** (all named):

| Parameter | Type | Description | Example |
| --- | --- | --- | --- |
| `version` | VARCHAR | Hierarchy version | `version => '01'` |
| `date_to` | VARCHAR | Valid-to date (time-dependent hierarchies) | `date_to => '20261231'` |
| `as_tree` | BOOLEAN | Return as nested tree instead of flat list (default: `false`) | `as_tree => true` |
| `secret` | VARCHAR | DuckDB secret name | `secret => 'erp_prod'` |

**Example:**

```
SELECT * FROM sap_bics_hierarchy('0MATERIAL_HIER', version => '01');
```

#### `sap_bics_describe_infoobject()`

Get InfoObject details.

**Signature:**

```
sap_bics_describe_infoobject(infoobject_name VARCHAR)
```

**Example:**

```
SELECT * FROM sap_bics_describe_infoobject('0MATERIAL');
```

### Lineage Functions

End-to-end lineage from ERP source tables through DataSources, transformations, InfoProviders, and BEx queries. All three functions return system-wide data; filter by query, provider, or source in SQL.

#### `sap_bics_lineage_edges()`

Lineage as a flat edge list. Each row is one source-to-target edge across the BW data flow graph.

**Signature:**

```
sap_bics_lineage_edges()
```

**Parameters** (all named):

| Parameter | Type | Description | Example |
| --- | --- | --- | --- |
| `scope` | VARCHAR | Optional scoping filter (object pattern) to limit the returned edges | `scope => '0D_FC_NW*'` |
| `secret` | VARCHAR | DuckDB secret name | `secret => 'erp_prod'` |

**Example:**

```
-- All lineage edgesSELECT * FROM sap_bics_lineage_edges();-- Filter edges related to a specific querySELECT *FROM sap_bics_lineage_edges()WHERE tgt_name = '0D_FC_NW_C01_Q0008' OR src_name = '0D_FC_NW_C01_Q0008';
```

#### `sap_bics_lineage_trace()`

Trace lineage forward from a specific source object/field. Useful for impact analysis ("if this table changes, what downstream BEx queries are affected?").

**Signature:**

```
sap_bics_lineage_trace()
```

**Parameters** (all named):

| Parameter | Type | Description | Example |
| --- | --- | --- | --- |
| `source_object` | VARCHAR | Source object name (e.g. ERP table) | `source_object => 'VBAK'` |
| `source_field` | VARCHAR | Optional source field for field-level trace | `source_field => 'NETWR'` |
| `secret` | VARCHAR | DuckDB secret name | `secret => 'erp_prod'` |

**Example:**

```
SELECT * FROM sap_bics_lineage_trace(source_object => 'VBAK');SELECT * FROM sap_bics_lineage_trace(source_object => 'VBAK', source_field => 'NETWR');
```

#### `sap_bics_lineage_graph_json()`

Returns the full lineage graph as a JSON document. Use for visualization or export to graph tooling.

**Signature:**

```
sap_bics_lineage_graph_json()
```

**Parameters** (named): `secret`.

**Example:**

```
SELECT * FROM sap_bics_lineage_graph_json();
```

### Metadata Functions

Each returns BW system metadata as a DuckDB table. Most are zero-positional; six take one VARCHAR positional. All accept an optional `secret` named parameter.

#### `sap_bics_meta_providers()`

InfoProvider metadata.

**Signature:**

```
sap_bics_meta_providers()
```

**Named parameters:**

| Parameter | Type | Description |
| --- | --- | --- |
| `type` | VARCHAR | Filter by provider type (e.g. `'CUBE'`, `'ODSO'`, `'MPRO'`) |
| `secret` | VARCHAR | DuckDB secret name |

**Example:**

```
SELECT * FROM sap_bics_meta_providers();SELECT * FROM sap_bics_meta_providers(type => 'CUBE');
```

#### `sap_bics_meta_datasources()`

DataSource metadata.

**Signature:**

```
sap_bics_meta_datasources()
```

**Named parameters:**

| Parameter | Type | Description |
| --- | --- | --- |
| `appcomp` | VARCHAR | Filter by application component |
| `secret` | VARCHAR | DuckDB secret name |

**Example:**

```
SELECT * FROM sap_bics_meta_datasources(appcomp => 'FI');
```

#### `sap_bics_meta_transformations()`

Transformation metadata.

**Signature:**

```
sap_bics_meta_transformations()
```

**Named parameters:**

| Parameter | Type | Description |
| --- | --- | --- |
| `active_only` | BOOLEAN | If true, only active transformations |
| `secret` | VARCHAR | DuckDB secret name |

**Example:**

```
SELECT * FROM sap_bics_meta_transformations(active_only => true);
```

#### `sap_bics_meta_queries()`

Query metadata.

**Signature:**

```
sap_bics_meta_queries()
```

**Named parameters:** `secret` (DuckDB secret name).

**Example:**

```
SELECT * FROM sap_bics_meta_queries();
```

#### `sap_bics_meta_query_elements()`

Query-element metadata for a specific query.

**Signature:**

```
sap_bics_meta_query_elements(query_name VARCHAR)
```

**Named parameters:** `secret`.

**Example:**

```
SELECT * FROM sap_bics_meta_query_elements('0D_FC_NW_C01_Q0008');
```

#### `sap_bics_meta_query_stats()`

Query runtime statistics.

**Signature:**

```
sap_bics_meta_query_stats()
```

**Named parameters:**

| Parameter | Type | Description |
| --- | --- | --- |
| `query_name` | VARCHAR | Filter to one query |
| `from_date` | VARCHAR | Start of the time window (YYYYMMDD) |
| `to_date` | VARCHAR | End of the time window (YYYYMMDD) |
| `secret` | VARCHAR | DuckDB secret name |

**Example:**

```
SELECT * FROM sap_bics_meta_query_stats(  query_name => '0D_FC_NW_C01_Q0008',  from_date => '20260101',  to_date => '20260501');
```

#### `sap_bics_meta_query_usage()`

Query usage metadata for a specific query.

**Signature:**

```
sap_bics_meta_query_usage(query_name VARCHAR)
```

**Named parameters:** `secret`.

**Example:**

```
SELECT * FROM sap_bics_meta_query_usage('0D_FC_NW_C01_Q0008');
```

#### `sap_bics_meta_provider_fields()`

Field metadata for one InfoProvider.

**Signature:**

```
sap_bics_meta_provider_fields(provider_name VARCHAR)
```

**Named parameters:**

| Parameter | Type | Description |
| --- | --- | --- |
| `provider_type` | VARCHAR | Override the inferred provider type if needed |
| `secret` | VARCHAR | DuckDB secret name |

**Example:**

```
SELECT * FROM sap_bics_meta_provider_fields('0D_NW_C01', provider_type => 'CUBE');
```

#### `sap_bics_meta_datasource_fields()`

Field metadata for one DataSource.

**Signature:**

```
sap_bics_meta_datasource_fields(datasource_name VARCHAR)
```

**Named parameters:** `secret`.

**Example:**

```
SELECT * FROM sap_bics_meta_datasource_fields('0FI_GL_4');
```

#### `sap_bics_meta_transform_fields()`

Field-level metadata for one transformation.

**Signature:**

```
sap_bics_meta_transform_fields(transformation_id VARCHAR)
```

**Named parameters:** `secret`.

**Example:**

```
SELECT * FROM sap_bics_meta_transform_fields('TRAN_ID_HERE');
```

#### `sap_bics_meta_hcpr_components()`

Composite provider component metadata.

**Signature:**

```
sap_bics_meta_hcpr_components(composite_provider VARCHAR)
```

**Named parameters:** `secret`.

**Example:**

```
SELECT * FROM sap_bics_meta_hcpr_components('0COPC1');
```

#### `sap_bics_meta_hcpr_mapping()`

Composite provider field mapping.

**Signature:**

```
sap_bics_meta_hcpr_mapping()
```

**Named parameters:**

| Parameter | Type | Description |
| --- | --- | --- |
| `composite_provider` | VARCHAR | Filter to one composite provider |
| `secret` | VARCHAR | DuckDB secret name |

**Example:**

```
SELECT * FROM sap_bics_meta_hcpr_mapping(composite_provider => '0COPC1');
```

#### `sap_bics_meta_infoobjects()`

InfoObject metadata.

**Signature:**

```
sap_bics_meta_infoobjects()
```

**Named parameters:**

| Parameter | Type | Description |
| --- | --- | --- |
| `iobjnm` | VARCHAR | Filter by InfoObject name |
| `iobjtp` | VARCHAR | Filter by InfoObject type (`'CHA'` characteristic, `'KYF'` key figure, etc.) |
| `secret` | VARCHAR | DuckDB secret name |

**Example:**

```
SELECT * FROM sap_bics_meta_infoobjects(iobjtp => 'KYF');
```

#### `sap_bics_meta_objxref()`

Object cross-reference metadata.

**Signature:**

```
sap_bics_meta_objxref()
```

**Named parameters:**

| Parameter | Type | Description |
| --- | --- | --- |
| `tlogo` | VARCHAR | Source object type code |
| `objnm` | VARCHAR | Source object name |
| `tlogo_dep` | VARCHAR | Dependent object type code |
| `secret` | VARCHAR | DuckDB secret name |

**Example:**

```
SELECT * FROM sap_bics_meta_objxref(tlogo => 'CUBE', objnm => '0D_NW_C01');
```

## ODP Functions

### Basic ODP Functions

#### `sap_odp_show_contexts()`

List all available ODP contexts.

**Signature:**

```
sap_odp_show_contexts()
```

**Example:**

```
SELECT * FROM sap_odp_show_contexts();
```

**Full vs. delta extraction in ERPL:**

ERPL exposes two distinct ODP extraction functions:

*   ([`sap_odp_read_full`](#sap_odp_read_full)) is one-shot — it opens a FULL cursor on the SAP side, streams the snapshot, then auto-closes the cursor when the scan completes. No state survives.
*   ([`sap_odp_read_delta`](#sap_odp_read_delta)) takes a `subscriber_process` identifier. The **first** call with a new identifier performs SAP's auto-DELTAINIT (returns the full current snapshot AND registers a server-side delta pointer); **subsequent** calls with the same identifier resume from that pointer and return only the changes since the last call.

Close delta cursors when your pipeline finishes with ([`PRAGMA sap_odp_close_delta_cursor`](#pragma-sap_odp_close_delta_cursor)) (graceful — keeps the subscription resumable) or ([`PRAGMA sap_odp_drop`](#pragma-sap_odp_drop)) (hard reset — next call re-snapshots). Inspect active cursors with ([`sap_odp_show_cursors`](#sap_odp_show_cursors)).

#### `sap_odp_show()`

List data sources in a specific context.

**Signature:**

```
sap_odp_show(context_name VARCHAR, [search VARCHAR], [secret VARCHAR])
```

**Parameters:**

| Parameter | Type | Description | Example |
| --- | --- | --- | --- |
| `context_name` | VARCHAR (positional) | ODP context | `'BW'` |
| `search` | VARCHAR | Glob pattern to filter data source names | `search => '*COST*'` |
| `secret` | VARCHAR | DuckDB secret name | `secret => 'erp_prod'` |

**Example:**

```
SELECT * FROM sap_odp_show('BW', search => '*COST*');
```

#### `sap_odp_describe()`

Describe data source structure.

**Signature:**

```
sap_odp_describe(context_name VARCHAR, data_source VARCHAR, [secret VARCHAR])
```

**Example:**

```
SELECT * FROM sap_odp_describe('BW', 'VBAK$F');
```

#### `sap_odp_preview()`

Preview data without opening a delta cursor. Useful for one-off inspection.

**Signature:**

```
sap_odp_preview(context_name VARCHAR, data_source VARCHAR, [max_rows UINTEGER], [secret VARCHAR])
```

**Example:**

```
SELECT * FROM sap_odp_preview('BW', 'VBAK$F', max_rows => 100);
```

#### `sap_odp_read_full()`

Full one-shot extraction of an ODP source. Opens a FULL cursor on the SAP side, streams the snapshot, and **auto-closes** the cursor when the scan ends — no server-side state survives. For incremental extraction use ([`sap_odp_read_delta`](#sap_odp_read_delta)).

**Signature:**

```
sap_odp_read_full(context_name VARCHAR, data_source VARCHAR, [threads UINTEGER], [columns LIST(VARCHAR)], [filters LIST(STRUCT)], [secret VARCHAR])
```

**Parameters:**

| Parameter | Type | Description | Example |
| --- | --- | --- | --- |
| `context_name` | VARCHAR (positional) | ODP context name | `'BW'` |
| `data_source` | VARCHAR (positional) | Data source name | `'VBAK$F'` |
| `threads` | UINTEGER | Number of parallel RFC fetch threads (default: 5) | `threads => 4` |
| `columns` | LIST(VARCHAR) | Restrict to a subset of columns | `columns => ['BUSINESSPARTNER']` |
| `filters` | LIST(STRUCT) | Server-side select predicates (see `ODP_SELECT_SIGN` / `ODP_SELECT_OP`) | `filters => [{...}]` |
| `secret` | VARCHAR | DuckDB secret name | `secret => 'erp_prod'` |

**Example:**

```
SELECT * FROM sap_odp_read_full('ABAP_CDS', 'Z_MY_CDS_VIEW');-- Tune parallelism and project columnsSELECT * FROM sap_odp_read_full('BW', 'MY_DATASOURCE', threads => 4);SELECT * FROM sap_odp_read_full('ABAP_CDS', 'SEPM_IBUPA$P', columns => ['BUSINESSPARTNER']);
```

#### `sap_odp_read_delta()`

Incremental delta extraction. The first call with a given `subscriber_process` performs SAP's **auto-DELTAINIT** — returns the full current snapshot AND registers a server-side delta pointer keyed by the subscriber tuple. Subsequent calls with the same `subscriber_process` resume from the previous pointer and return only the changes since then.

The delta cursor persists across calls. Close it explicitly with ([`PRAGMA sap_odp_close_delta_cursor`](#pragma-sap_odp_close_delta_cursor)) when your pipeline is done.

**Signature:**

```
sap_odp_read_delta(context_name VARCHAR, data_source VARCHAR, subscriber_process VARCHAR,                   [threads UINTEGER], [columns LIST(VARCHAR)], [filters LIST(STRUCT)],                   [recover BOOLEAN], [secret VARCHAR])
```

**Parameters:**

| Parameter | Type | Description | Example |
| --- | --- | --- | --- |
| `context_name` | VARCHAR (positional) | ODP context name | `'BW'` |
| `data_source` | VARCHAR (positional) | Data source name | `'0D_FC_C01$F'` |
| `subscriber_process` | VARCHAR (positional) | **Stable** identifier that keys the server-side delta pointer across calls. Pick a deterministic name per pipeline. | `'NIGHTLY_ETL'` |
| `threads` | UINTEGER | Number of parallel RFC fetch threads (default: 5) | `threads => 4` |
| `columns` | LIST(VARCHAR) | Restrict to a subset of columns | `columns => ['BUSINESSPARTNER']` |
| `filters` | LIST(STRUCT) | Server-side select predicates (same shape as `sap_odp_read_full`) | `filters => [{...}]` |
| `recover` | BOOLEAN | Re-stream the last unconfirmed packet (`I_EXTRACTION_MODE='R'`) without advancing the pointer. Useful after an interrupted fetch. | `recover => true` |
| `secret` | VARCHAR | DuckDB secret name | `secret => 'erp_prod'` |

**Concurrency:**

Do not run two `sap_odp_read_delta` calls with the same `subscriber_process` in parallel — they will race the server-side pointer.

**Example:**

```
-- First call: auto-DELTAINIT — full snapshot + delta-pointer registered under 'NIGHTLY_ETL'.SELECT * FROM sap_odp_read_delta('BW', '0D_FC_C01$F', 'NIGHTLY_ETL');-- Run the same call later: only the rows that changed since the previous call.SELECT * FROM sap_odp_read_delta('BW', '0D_FC_C01$F', 'NIGHTLY_ETL');-- Recover the last packet if a previous call was interrupted:SELECT * FROM sap_odp_read_delta('BW', '0D_FC_C01$F', 'NIGHTLY_ETL', recover => true);-- Release the cursor when the pipeline is done:PRAGMA sap_odp_close_delta_cursor('BW', 'NIGHTLY_ETL', '0D_FC_C01$F');
```

Delta-capable sources are identified by `supports_delta=true` in `sap_odp_describe`. Not every CDS view is delta-capable — the underlying DDL must carry the `@Analytics.dataExtraction.delta.byElement` annotation; BW fact tables (`*$F`) are typically delta-capable.

#### `sap_odp_get_last_modified()`

Cheap probe that returns the most-recent modification timestamp of an ODP source without opening a cursor or fetching rows. Use it before a heavy `sap_odp_read_delta` call to skip pipelines when nothing has changed since the last run. Backed by `RODPS_REPL_ODP_GET_LAST_MODIF`.

**Signature:**

```
sap_odp_get_last_modified(context_name VARCHAR, data_source VARCHAR, [secret VARCHAR])
```

**Returns:** `odp_name VARCHAR`, `last_modified DECIMAL(21,7)` (UTC, format `YYYYMMDDhhmmss.fffffff`). Returns `0.0` for unknown ODP names — the SAP RFM does not surface "not found" as an error.

**Example:**

```
SELECT * FROM sap_odp_get_last_modified('ABAP_CDS', 'SEPM_IBUPA$P');-- Bail out early if nothing changedWITH probe AS (  SELECT last_modified FROM sap_odp_get_last_modified('BW', 'MY_SRC'))SELECT *FROM sap_odp_read_delta('BW', 'MY_SRC', 'MY_PIPELINE')WHERE (SELECT last_modified FROM probe) > 20260516000000.0;
```

### Advanced ODP Functions

#### `sap_odp_show_subscriptions()`

View active subscriptions (cursors that hold ODP state).

**Signature:**

```
sap_odp_show_subscriptions([ERPL_ONLY BOOLEAN], [secret VARCHAR])
```

**Parameters:**

| Parameter | Type | Description | Example |
| --- | --- | --- | --- |
| `ERPL_ONLY` | BOOLEAN | When true (default), show only subscriptions created by ERPL. When false, show all ODP subscribers. | `ERPL_ONLY => FALSE` |
| `secret` | VARCHAR | DuckDB secret name | `secret => 'erp_prod'` |

**Example:**

```
SELECT * FROM sap_odp_show_subscriptions();SELECT * FROM sap_odp_show_subscriptions(ERPL_ONLY => FALSE);
```

#### `sap_odp_get_subscriptions()`

List every subscription registered against a given ODP source — across all subscribers, not just ERPL. Useful for cross-team visibility ("who else is reading from this CDS view?") and for stuck-subscription forensics before a `PRAGMA sap_odp_drop`. Backed by `RODPS_REPL_ODP_GET_SUBSCR`.

**Signature:**

```
sap_odp_get_subscriptions(context_name VARCHAR, data_source VARCHAR,                          [subscriber_name VARCHAR], [subscriber_process VARCHAR], [secret VARCHAR])
```

**Parameters:**

| Parameter | Type | Description | Example |
| --- | --- | --- | --- |
| `context_name` | VARCHAR (positional) | ODP context (`'ABAP_CDS'`, `'BW'`, `'SAPI'`) | `'ABAP_CDS'` |
| `data_source` | VARCHAR (positional) | ODP object name | `'SEPM_IBUPA$P'` |
| `subscriber_name` | VARCHAR | Filter by subscriber name | `subscriber_name => 'ERPL'` |
| `subscriber_process` | VARCHAR | Filter by subscriber process | `subscriber_process => 'NIGHTLY_ETL'` |
| `secret` | VARCHAR | DuckDB secret name | `secret => 'erp_prod'` |

**Returns:** `subscriber_type`, `subscriber_name`, `subscriber_process`, `model_name`, `queue_name`, `subscription_id`.

**Example:**

```
-- All subscribers on a source (across teams)SELECT * FROM sap_odp_get_subscriptions('ABAP_CDS', 'SEPM_IBUPA$P');-- Just my pipelineSELECT * FROM sap_odp_get_subscriptions('BW', '0D_FC_C01$F', subscriber_name => 'ERPL');
```

#### `sap_odp_show_cursors()`

Inspect ODP cursors — the server-side state behind `sap_odp_read_delta`. Delta cursors created by `sap_odp_read_delta` appear here with `is_delta_extension=true`; their `subscriber_proc` column matches the `subscriber_process` argument you passed.

**Returned columns:** `queue_name`, `subscriber_proc`, `subscriber_id`, `pointer`, `is_closed`, `is_delta_extension`, `request_date`.

**Signature:**

```
sap_odp_show_cursors([erpl_only BOOLEAN], [subscriber_name VARCHAR], [context VARCHAR], [replication_mode ODP_REPLICATION_MODE], [secret VARCHAR])
```

**Parameters:**

| Parameter | Type | Description | Example |
| --- | --- | --- | --- |
| `erpl_only` | BOOLEAN | Filter to cursors owned by the ERPL subscriber | `erpl_only => TRUE` |
| `subscriber_name` | VARCHAR | Filter by subscriber name | `subscriber_name => 'ERPL'` |
| `context` | VARCHAR | Filter by ODP context | `context => 'ABAP_CDS'` |
| `replication_mode` | ODP\_REPLICATION\_MODE | Filter by cursor mode | `replication_mode => 'FULL'` |
| `secret` | VARCHAR | DuckDB secret name | `secret => 'erp_prod'` |

**Example:**

```
SELECT * FROM sap_odp_show_cursors();SELECT * FROM sap_odp_show_cursors(erpl_only => TRUE);SELECT * FROM sap_odp_show_cursors(context => 'ABAP_CDS', replication_mode => 'FULL');
```

### ODP Pragma Functions

#### `PRAGMA sap_odp_close_delta_cursor()`

Graceful counterpart to `sap_odp_drop`. Looks up the delta cursor for the given subscriber tuple and calls `RODPS_REPL_ODP_CLOSE` on its pointer. **Idempotent**: returns `'CLOSED'` if a cursor existed (open or already closed) and `'NOT_FOUND'` if no cursor of that name was found.

Prefer this over `sap_odp_drop` at the end of a delta pipeline — close leaves the subscription registered and resumable from its last pointer; drop wipes the subscription so the next call performs DELTAINIT again.

**Signature:**

```
PRAGMA sap_odp_close_delta_cursor(odp_context, subscriber_process, odp_name [, secret => ...]);
```

**Parameters:**

| Parameter | Description | Example |
| --- | --- | --- |
| `odp_context` | ODP context | `'BW'` |
| `subscriber_process` | Subscriber process identifier (the one passed to `sap_odp_read_delta`) | `'NIGHTLY_ETL'` |
| `odp_name` | Data source name | `'0D_FC_C01$F'` |

Optionally takes a named `secret` parameter to select the DuckDB secret.

**Example:**

```
PRAGMA sap_odp_close_delta_cursor('BW', 'NIGHTLY_ETL', '0D_FC_C01$F');
```

If a pipeline crashes mid-fetch the cursor may not be closeable via this pragma — fall back to `sap_odp_drop` to fully reset (accepting that the next call will re-snapshot via DELTAINIT).

#### `PRAGMA sap_odp_drop()`

Hard reset of an ODP subscription server-side (invokes `RODPS_REPL_ODP_RESET`). The next call to `sap_odp_read_delta` for the same subscriber tuple re-creates the subscription via auto-DELTAINIT and returns a fresh full snapshot.

**Signature:**

```
PRAGMA sap_odp_drop(odp_context, subscriber_name, subscriber_process, odp_name);
```

**Parameters:**

| Parameter | Description | Example |
| --- | --- | --- |
| `odp_context` | ODP context | `'ABAP_CDS'` |
| `subscriber_name` | Subscriber name as stored on SAP (ERPL default: `'ERPL'`) | `'ERPL'` |
| `subscriber_process` | Subscriber process identifier | `'erpl_subs_proc'` |
| `odp_name` | Data source name | `'Z_MY_CDS_VIEW'` |

Optionally takes a named `secret` parameter to select the DuckDB secret.

**Example:**

```
PRAGMA sap_odp_drop('ABAP_CDS', 'ERPL', 'erpl_subs_proc', 'Z_MY_CDS_VIEW');
```

## Custom Types

### BICS Types

#### `BICS_RETURN`

Enum type for BICS return codes.

**Values:**

*   `SUCCESS`
*   `ERROR`
*   `WARNING`

#### `BICS_OPERATION`

Enum type for BICS operations.

**Values:**

*   `BEGIN`
*   `FILTER`
*   `COLUMNS`
*   `RESULT`

### ODP Types

#### `ODP_REPLICATION_MODE`

Enum type for ODP replication modes.

**Values:**

*   `FULL` - Complete data load
*   `DELTA` - Only changed records
*   `RECOVER` - Recovery mode

#### `ODP_SELECT_SIGN`

Enum type for ODP selection signs.

**Values:**

*   `I` - Include
*   `E` - Exclude

#### `ODP_SELECT_OP`

Enum type for ODP selection operations.

**Values:**

*   `EQ` - Equal
*   `NE` - Not equal
*   `GT` - Greater than
*   `LT` - Less than
*   `GE` - Greater than or equal
*   `LE` - Less than or equal

## Function Categories

### By Complexity Level

**Beginner Functions:**

*   `sap_read_table()`
*   `sap_rfc_invoke()`
*   `sap_bics_show()`
*   `sap_bics_begin()`
*   `sap_bics_result()`
*   `sap_odp_show_contexts()`
*   `sap_odp_read_full()`

**Intermediate Functions:**

*   `sap_rfc_describe_function()`
*   `sap_describe_fields()`
*   `sap_bics_describe()`
*   `sap_bics_filter()`
*   `sap_bics_hierarchy()`
*   `sap_odp_describe()`
*   `sap_odp_preview()`

**Advanced Functions:**

*   `sap_rfc_authorizations()`
*   `sap_bics_lineage_edges()`, `sap_bics_lineage_trace()`, `sap_bics_lineage_graph_json()`
*   `sap_bics_meta_*()` (all metadata functions)
*   `sap_odp_read_delta()`
*   `sap_odp_get_last_modified()`
*   `sap_odp_show_subscriptions()`, `sap_odp_get_subscriptions()`
*   `sap_odp_show_cursors()`
*   `PRAGMA sap_odp_close_delta_cursor()`, `PRAGMA sap_odp_drop()`

### By Protocol

**RFC Functions (8):**

*   `sap_read_table()`
*   `sap_rfc_invoke()`
*   `sap_rfc_describe_function()`
*   `sap_describe_fields()`
*   `sap_rfc_authorizations()`
*   `sap_rfc_show_function()`
*   `sap_rfc_show_groups()`
*   `sap_show_tables()`

**BICS Functions (25+):**

*   Basic query functions (7)
*   Metadata functions (15)
*   Lineage functions (3)

**ODP Functions (12):**

*   `sap_odp_show_contexts()`
*   `sap_odp_show()`
*   `sap_odp_describe()`
*   `sap_odp_preview()`
*   `sap_odp_read_full()`
*   `sap_odp_read_delta()`
*   `sap_odp_get_last_modified()`
*   `sap_odp_show_subscriptions()`
*   `sap_odp_get_subscriptions()`
*   `sap_odp_show_cursors()`
*   `PRAGMA sap_odp_close_delta_cursor()`
*   `PRAGMA sap_odp_drop()`

## Common Patterns

### Basic Query Pattern (BICS)

```
-- Standard BICS query pattern: open a state by id, shape it, read by the same idSELECT state_id FROM sap_bics_begin('CUBE_NAME', id => 'q1');SELECT state_id FROM sap_bics_rows('q1', 'ROW_CHARACTERISTIC', op => 'SET');SELECT state_id FROM sap_bics_filter('q1', 'FILTER_FIELD', 'FILTER_VALUE', op => 'SET');SELECT * FROM sap_bics_result('q1');
```

### Delta Replication Pattern (ODP)

```
-- Standard ODP delta pattern: one function, stable subscriber_process.-- 1. First call: auto-DELTAINIT — full snapshot + delta-pointer registered.SELECT * FROM sap_odp_read_delta('BW', 'VBAK$F', 'NIGHTLY_ETL');-- 2. Subsequent calls return only changes since the last call.SELECT * FROM sap_odp_read_delta('BW', 'VBAK$F', 'NIGHTLY_ETL');-- 3. Release the cursor when the pipeline is done.PRAGMA sap_odp_close_delta_cursor('BW', 'NIGHTLY_ETL', 'VBAK$F');
```

### Function Discovery Pattern (RFC)

```
-- Discover and analyze functionsSELECT * FROM sap_rfc_show_function(FUNCNAME => 'BAPI*');SELECT * FROM sap_rfc_describe_function('FUNCTION_NAME');
```

## Error Handling

### Common Error Patterns

**Function Not Found:**

```
-- Check if function existsSELECT * FROM sap_rfc_show_function(FUNCNAME => 'FUNCTION_NAME');
```

**Table Not Found:**

```
-- Check if table existsSELECT * FROM sap_show_tables()WHERE table_name = 'TABLE_NAME';
```

**Subscription Errors:**

```
-- List the registered subscriptions (cols: queue_name, subscriber_type, subscriber_name, subscriber_proc)SELECT * FROM sap_odp_show_subscriptions();
```

## Performance Tips

### Optimization Strategies

**Use LIMIT for large datasets:**

```
SELECT * FROM sap_read_table('KNA1', MAX_ROWS => 1000);
```

**Filter early in BICS queries:**

```
SELECT state_id FROM sap_bics_begin('CUBE', id => 'q1');SELECT state_id FROM sap_bics_filter('q1', 'FILTER_FIELD', 'VALUE', op => 'SET');SELECT * FROM sap_bics_result('q1');
```

**Use DELTA mode for ODP:**

```
SELECT * FROM sap_odp_read_full(  'BW', 'VBAK$F');
```

## Next Steps

### 🚀 Ready for More?

*   [RFC Protocol Guide](/docs/erpl/rfc.md) - Complete RFC documentation
*   [BICS Protocol Guide](/docs/erpl/bics.md) - Complete BICS documentation
*   [ODP Protocol Guide](/docs/erpl/odp.md) - Complete ODP documentation

### 🔧 Advanced Topics

*   [BICS Lineage Tracking](/docs/guides/advanced/bics-lineage-tracking.md) - Advanced lineage analysis
*   [ODP Subscription Management](/docs/guides/advanced/odp-subscription-management.md) - Advanced subscription patterns
*   [RFC Metadata Extraction](/docs/guides/advanced/rfc-metadata.md) - Advanced metadata analysis

### 💡 Examples

*   [ERPL Examples](/docs/examples/erpl-examples.md) - Real-world examples
*   [Real-World Use Cases](/docs/examples/real-world-use-cases.md) - Complete scenarios

---

**Need help?** Check our [troubleshooting guide](/docs/reference/troubleshooting.md) or browse [more examples](/docs/examples/erpl-examples.md).

# ERPL-Web Functions Reference

Complete reference for all ERPL-Web functions with signatures, parameters, and return values.

---

## HTTP Functions

### http\_get

Make an HTTP GET request.

**Signature:**

```
http_get(  url VARCHAR,  headers MAP(VARCHAR, VARCHAR) := NULL,  accept VARCHAR := NULL,  auth VARCHAR := NULL,  auth_type VARCHAR := NULL,  timeout BIGINT := NULL) → TABLE
```

**Parameters:**

*   `url`: Request URL
*   `headers`: Custom HTTP headers
*   `accept`: Accept header value
*   `auth`: Authentication credential (username:password for BASIC, token for BEARER)
*   `auth_type`: `'BASIC'` or `'BEARER'`
*   `timeout`: Timeout in milliseconds

**Returns:** Table with columns:

*   `method` (VARCHAR)
*   `status` (INTEGER)
*   `url` (VARCHAR)
*   `headers` (MAP(VARCHAR, VARCHAR))
*   `content_type` (VARCHAR)
*   `content` (VARCHAR)

---

### http\_post

Make an HTTP POST request.

**Signature:**

```
http_post(  url VARCHAR,  body VARCHAR,  content_type VARCHAR := 'application/json',  headers MAP(VARCHAR, VARCHAR) := NULL,  accept VARCHAR := NULL,  auth VARCHAR := NULL,  auth_type VARCHAR := NULL,  timeout BIGINT := NULL) → TABLE
```

**Parameters:**

*   `url`: Request URL
*   `body`: Request body content
*   `content_type`: Content-Type header
*   `headers`: Custom HTTP headers
*   `accept`: Accept header value
*   `auth`: Authentication credential
*   `auth_type`: `'BASIC'` or `'BEARER'`
*   `timeout`: Timeout in milliseconds

**Returns:** Same as `http_get`

---

### http\_put

Make an HTTP PUT request.

**Signature:**

```
http_put(  url VARCHAR,  body VARCHAR,  content_type VARCHAR := 'application/json',  headers MAP(VARCHAR, VARCHAR) := NULL,  accept VARCHAR := NULL,  auth VARCHAR := NULL,  auth_type VARCHAR := NULL,  timeout BIGINT := NULL) → TABLE
```

**Parameters:** Same as `http_post`

**Returns:** Same as `http_get`

---

### http\_patch

Make an HTTP PATCH request.

**Signature:**

```
http_patch(  url VARCHAR,  body VARCHAR,  content_type VARCHAR := 'application/json',  headers MAP(VARCHAR, VARCHAR) := NULL,  accept VARCHAR := NULL,  auth VARCHAR := NULL,  auth_type VARCHAR := NULL,  timeout BIGINT := NULL) → TABLE
```

**Parameters:** Same as `http_post`

**Returns:** Same as `http_get`

---

### http\_delete

Make an HTTP DELETE request.

**Signature:**

```
http_delete(  url VARCHAR,  headers MAP(VARCHAR, VARCHAR) := NULL,  accept VARCHAR := NULL,  auth VARCHAR := NULL,  auth_type VARCHAR := NULL,  timeout BIGINT := NULL) → TABLE
```

**Parameters:** Same as `http_get` (no body)

**Returns:** Same as `http_get`

---

## OData Functions

### ATTACH (OData)

Attach an OData service as a database.

**Signature:**

```
ATTACH 'service_url' AS database_name (TYPE odata);
```

**Parameters:**

*   `service_url`: OData service root URL
*   `database_name`: Alias for the attached database

**Example:**

```
ATTACH 'https://services.odata.org/TripPinRESTierService'   AS trippin (TYPE odata);
```

---

### odata\_read

Read from an OData entity set.

**Signature:**

```
odata_read(  entity_set_url VARCHAR,  secret VARCHAR := NULL,  expand VARCHAR := NULL) → TABLE
```

**Parameters:**

*   `entity_set_url`: Full URL to OData entity set
*   `secret`: DuckDB secret name for authentication
*   `expand`: OData $expand parameter for navigation properties

**Returns:** Table with columns matching the OData entity schema

**Example:**

```
SELECT * FROM odata_read(  'https://services.odata.org/TripPinRESTierService/People',  expand := 'Trips');
```

---

## Datasphere Functions

### datasphere\_show\_spaces

List all accessible Datasphere spaces.

**Signature:**

```
datasphere_show_spaces(  secret VARCHAR := NULL) → TABLE
```

**Parameters:**

*   `secret`: DuckDB secret name (optional, uses default if not specified)

**Returns:** Table with column:

*   `name` (VARCHAR): Space ID

---

### datasphere\_show\_assets

List assets in a space or across all spaces.

**Signatures:**

```
-- Assets in specific spacedatasphere_show_assets(  space_id VARCHAR,  secret VARCHAR := NULL) → TABLE-- All accessible assetsdatasphere_show_assets(  secret VARCHAR := NULL) → TABLE
```

**Parameters:**

*   `space_id`: Space identifier (optional)
*   `secret`: DuckDB secret name

**Returns:** Table with columns:

*   `name` (VARCHAR): Asset label
*   `object_type` (VARCHAR): Type (View, Table, etc.)
*   `technical_name` (VARCHAR): Technical identifier
*   `space_name` (VARCHAR): Space ID (only when querying all spaces)

---

### datasphere\_describe\_space

Get detailed metadata for a space.

**Signature:**

```
datasphere_describe_space(  space_id VARCHAR,  secret VARCHAR := NULL) → TABLE
```

**Parameters:**

*   `space_id`: Space identifier
*   `secret`: DuckDB secret name

**Returns:** Table with columns:

*   `name` (VARCHAR): Space ID
*   `label` (VARCHAR): Display label

---

### datasphere\_describe\_asset

Get comprehensive metadata for an asset.

**Signature:**

```
datasphere_describe_asset(  space_id VARCHAR,  asset_id VARCHAR,  secret VARCHAR := NULL) → TABLE
```

**Parameters:**

*   `space_id`: Space identifier
*   `asset_id`: Asset technical name
*   `secret`: DuckDB secret name

**Returns:** Table with 15 columns including:

*   `name` (VARCHAR)
*   `space_name` (VARCHAR)
*   `label` (VARCHAR)
*   `asset_type` (VARCHAR)
*   `asset_relational_metadata_url` (VARCHAR)
*   `asset_relational_data_url` (VARCHAR)
*   `asset_analytical_metadata_url` (VARCHAR)
*   `asset_analytical_data_url` (VARCHAR)
*   `supports_analytical_queries` (BOOLEAN)
*   `has_relational_access` (BOOLEAN)
*   `has_analytical_access` (BOOLEAN)
*   `relational_schema` (STRUCT)
*   `analytical_schema` (STRUCT)
*   `odata_context` (VARCHAR)
*   `odata_metadata_etag` (VARCHAR)

---

### datasphere\_read\_relational

Query relational data from a Datasphere asset.

**Signature:**

```
datasphere_read_relational(  space_id VARCHAR,  asset_id VARCHAR,  secret VARCHAR := NULL,  top BIGINT := NULL,  skip BIGINT := NULL,  params MAP(VARCHAR, VARCHAR) := NULL) → TABLE
```

**Parameters:**

*   `space_id`: Space identifier
*   `asset_id`: Asset technical name
*   `secret`: DuckDB secret name
*   `top`: Limit number of rows (OData $top)
*   `skip`: Skip rows (OData $skip)
*   `params`: Input parameters for parameterized views

**Returns:** Table with columns matching the asset schema

---

### datasphere\_read\_analytical

Query analytical data from a Datasphere asset.

**Signature:**

```
datasphere_read_analytical(  space_id VARCHAR,  asset_id VARCHAR,  secret VARCHAR := NULL,  top BIGINT := NULL,  skip BIGINT := NULL,  params MAP(VARCHAR, VARCHAR) := NULL,  metrics LIST(VARCHAR) := NULL,  dimensions LIST(VARCHAR) := NULL) → TABLE
```

**Parameters:**

*   `space_id`: Space identifier
*   `asset_id`: Asset technical name
*   `secret`: DuckDB secret name
*   `top`: Limit rows
*   `skip`: Skip rows
*   `params`: Input parameters
*   `metrics`: List of measures to retrieve
*   `dimensions`: List of dimensions to retrieve

**Returns:** Table with columns matching requested metrics and dimensions

---

## ODP Functions

### odp\_odata\_show

Discover available ODP OData services.

**Signature:**

```
odp_odata_show(  base_url VARCHAR,  secret VARCHAR := NULL) → TABLE
```

**Parameters:**

*   `base_url`: SAP server base URL (e.g., `https://sap-server:port`)
*   `secret`: DuckDB secret name for authentication

**Returns:** Table with columns:

*   `service_name` (VARCHAR)
*   `entity_set_name` (VARCHAR)
*   `full_entity_set_url` (VARCHAR)
*   `description` (VARCHAR)

---

### odp\_odata\_read

Extract data from ODP OData with automatic delta replication.

**Signature:**

```
odp_odata_read(  entity_set_url VARCHAR,  secret VARCHAR := NULL) → TABLE
```

**Parameters:**

*   `entity_set_url`: Full URL to ODP OData entity set
*   `secret`: DuckDB secret name for authentication

**Returns:** Table with columns matching entity schema plus:

*   `RECORD_MODE` (VARCHAR): Change indicator (`''` = update, `'N'` = insert, `'D'` = delete)
*   `ODQ_CHANGEMODE` (VARCHAR)
*   `ODQ_ENTITYCNTR` (VARCHAR)

**Behavior:**

*   First call: Creates subscription and performs initial full load
*   Subsequent calls: Returns only delta changes since last extraction

---

### odp\_odata\_list\_subscriptions

List all active ODP subscriptions.

**Signature:**

```
odp_odata_list_subscriptions() → TABLE
```

**Parameters:** None

**Returns:** Table with columns:

*   `subscription_id` (VARCHAR)
*   `entity_set_name` (VARCHAR)
*   `entity_set_url` (VARCHAR)
*   `subscription_status` (VARCHAR)
*   `last_delta_token` (VARCHAR)
*   `created_at` (TIMESTAMP)
*   `updated_at` (TIMESTAMP)

---

## Pragma Functions

### odp\_odata\_remove\_subscription

Remove an ODP subscription.

**Signature:**

```
PRAGMA odp_odata_remove_subscription(  subscription_id VARCHAR,  delete_on_server BOOLEAN);
```

**Parameters:**

*   `subscription_id`: Subscription identifier
*   `delete_on_server`: If true, also deletes subscription on SAP server

**Example:**

```
-- Remove from local tracking onlyPRAGMA odp_odata_remove_subscription('sub_123', false);-- Remove from local AND SAP serverPRAGMA odp_odata_remove_subscription('sub_123', true);
```

---

## Audit Tables

### erpl\_web.odp\_subscription\_audit

Audit log for all ODP extractions.

**Schema:**

*   `subscription_id` (VARCHAR)
*   `request_timestamp` (TIMESTAMP)
*   `request_type` (VARCHAR): `'INITIAL'` or `'DELTA'`
*   `package_count` (INTEGER)
*   `records_received` (BIGINT)
*   `has_more_data` (BOOLEAN)
*   `new_delta_token` (VARCHAR)
*   `execution_time_ms` (BIGINT)
*   `package_size_bytes` (BIGINT)
*   `response_status` (INTEGER)

**Example:**

```
SELECT * FROM erpl_web.odp_subscription_auditWHERE subscription_id = 'sub_123'ORDER BY request_timestamp DESCLIMIT 10;
```

---

## Configuration Settings

### Tracing Settings

```
-- Enable/disable tracingSET erpl_trace_enabled = TRUE|FALSE;-- Trace levelSET erpl_trace_level = 'TRACE'|'DEBUG'|'INFO'|'WARN'|'ERROR';-- Output destinationSET erpl_trace_output = 'console'|'file'|'both';-- File configurationSET erpl_trace_file_path = '/path/to/trace.log';SET erpl_trace_max_file_size = 10485760;  -- bytesSET erpl_trace_rotation = TRUE|FALSE;
```

### Telemetry Settings

```
-- Enable/disable telemetrySET erpl_telemetry_enabled = TRUE|FALSE;-- Custom telemetry keySET erpl_telemetry_key = 'your-posthog-key';
```

---

## Common Parameter Patterns

### Secret Parameter

Most functions accept an optional `secret` parameter:

```
-- Use default secret (auto-detected)SELECT * FROM http_get('https://api.example.com');-- Use named secretSELECT * FROM http_get('https://api.example.com', secret := 'my_api');
```

### Named Parameters

ERPL-Web functions use DuckDB's named parameter syntax:

```
-- Good: Named parameters (order doesn't matter)SELECT * FROM datasphere_read_relational(  'SALES',  'CUSTOMERS',  top := 100,  skip := 50);-- Also works: Positional parameters (order matters)SELECT * FROM datasphere_read_relational('SALES', 'CUSTOMERS', NULL, 100, 50);
```

### MAP Parameters

```
-- Headers as MAPSELECT * FROM http_get(  'https://api.example.com',  headers := {'Authorization': 'Bearer token', 'X-Custom': 'value'});-- Params as MAPSELECT * FROM datasphere_read_relational(  'SALES',  'VIEW',  params := {'YEAR': '2024', 'REGION': 'EMEA'});
```

### LIST Parameters

```
-- Metrics and dimensions as LISTSELECT * FROM datasphere_read_analytical(  'SALES',  'ANALYTICS',  metrics := ['Revenue', 'Cost'],  dimensions := ['Region', 'Quarter']);
```

---

## Error Codes

Common HTTP status codes returned:

| Code | Meaning | Action |
| --- | --- | --- |
| 200 | Success | Continue |
| 401 | Unauthorized | Check credentials/secret |
| 403 | Forbidden | Verify permissions |
| 404 | Not Found | Check URL/entity name |
| 429 | Too Many Requests | Reduce request rate |
| 500 | Server Error | Check SAP/service logs |
| 503 | Service Unavailable | Retry later |

---

## Next Steps

*   Read detailed guides:
    
    *   [HTTP Functions](/docs/erpl-web/http-functions.md)
    *   [OData](/docs/erpl-web/odata.md)
    *   [Datasphere](/docs/erpl-web/datasphere.md)
    *   [ODP via OData](/docs/erpl-web/odp-web.md)
    *   [Secrets Management](/docs/erpl-web/secrets.md)
    *   [Tracing & Diagnostics](/docs/erpl-web/tracing.md)
*   Check [Examples](/docs/examples/erpl-web-examples.md) for real-world usage
    

---

## Summary

This reference covers all ERPL-Web functions:

*   **5 HTTP functions** for REST APIs
*   **2 OData functions** for OData services
*   **6 Datasphere functions** for SAP Datasphere
*   **3 ODP functions** for delta replication
*   **Configuration settings** for tracing and telemetry

For complete usage examples, see the individual function documentation pages.

# troubleshooting

# Pragma Functions

Pragma functions are special SQL commands for configuring and managing the ERPL extension runtime behavior.

## Available Functions

### Configuration Management

*   **[sap\_rfc\_set\_ini\_path](/docs/references/pragma/sap_rfc_set_ini_path.md)** - Specifies the path to the INI configuration file.
*   **[sap\_rfc\_reload\_ini\_file](/docs/references/pragma/sap_rfc_reload_ini_file.md)** - Reloads the INI configuration file without restarting DuckDB.

### Trace and Debugging

*   **[sap\_rfc\_set\_trace\_level](/docs/references/pragma/sap_rfc_set_trace_level.md)** - Sets the trace level for debugging and analysis purposes.
*   **[sap\_rfc\_set\_trace\_dir](/docs/references/pragma/sap_rfc_set_trace_dir.md)** - Specifies the directory where trace files should be stored.
*   **[sap\_rfc\_set\_maximum\_trace\_file\_size](/docs/references/pragma/sap_rfc_set_maximum_trace_file_size.md)** - Sets the maximum size for trace files.
*   **[sap\_rfc\_set\_maximum\_stored\_trace\_files](/docs/references/pragma/sap_rfc_set_maximum_stored_trace_files.md)** - Defines the maximum number of trace files to keep.

### Connection and Utility

*   **[sap\_rfc\_ping](/docs/references/pragma/sap_rfc_ping.md)** - Checks the connectivity to the SAP system.

To inspect a function module's interface, use the table function [`sap_rfc_describe_function('FUNC_NAME')`](/docs/reference/erpl-functions.md#sap_rfc_describe_function).

## Usage Examples

```
-- Set the trace level to verbosePRAGMA sap_rfc_set_trace_level(2);-- Check SAP connectivityPRAGMA sap_rfc_ping;-- Set the INI configuration file pathPRAGMA sap_rfc_set_ini_path("/path/to/config.ini");
```

For detailed documentation on each function, see the individual function pages linked above.

# Remote Function Call (RFC) Functions

You can find detailed documentation about SAP Remote Function Call (RFC) functions on SAP's official website. This documentation provides comprehensive information on the various types of RFCs, such as Synchronous RFC (sRFC), Asynchronous RFC (aRFC), Transactional RFC (tRFC), Queued RFC (qRFC), Background RFC (bgRFC), and Local Data Queue (LDQ), along with their specific characteristics and uses. It also covers aspects of data transfer and security relevant to RFC communication.

All ERPL functions loaded into DuckDB can be retrieved by the following SQL statement:

```
SELECT * FROM duckdb_functions() WHERE function_name LIKE '%sap%';
```

## Available RFC Functions

### Table Operations

*   **[sap\_show\_tables](/docs/references/rfc/sap_show_tables.md)** - Lists all or specific tables available in the SAP system.
*   **[sap\_describe\_fields](/docs/references/rfc/sap_describe_fields.md)** - Provides detailed information about fields in a specified table.
*   **[sap\_read\_table](/docs/references/rfc/sap_read_table.md)** - Reads data from a specified SAP table.

### Function Discovery and Inspection

*   **[sap\_rfc\_show\_groups](/docs/references/rfc/sap_rfc_show_groups.md)** - Lists SAP function groups that contain RFC-enabled function modules.
*   **[sap\_rfc\_show\_function](/docs/references/rfc/sap_rfc_show_function.md)** - Searches RFC-enabled function modules by name/group pattern.
*   **[sap\_rfc\_describe\_function](/docs/references/rfc/sap_rfc_describe_function.md)** - Returns the full parameter interface of a function module.

### Function Invocation

*   **[sap\_rfc\_invoke](/docs/references/rfc/sap_rfc_invoke.md)** - Executes a specified function module in the SAP system.

## Usage Examples

```
-- List all available tablesSELECT * FROM sap_show_tables()-- Find tables starting with "FLIGHT"SELECT * FROM sap_show_tables(TABLENAME='*SPFL*')-- Get field descriptions for a tableSELECT * FROM SAP_DESCRIBE_FIELDS('SPFLI');-- Search for functionsSELECT * FROM sap_rfc_show_function(FUNCNAME='BAPI_FLIGHT*') ORDER BY 1-- Get function detailsSELECT * FROM sap_rfc_describe_function('BAPI_FLIGHT_GETLIST')-- Invoke a functionSELECT * FROM sap_rfc_invoke('BAPI_FLIGHT_GETLIST', path='/FLIGHT_LIST')-- Read table dataSELECT * FROM sap_read_table("SFLIGHT")
```

For detailed documentation on each function, see the individual function pages linked above.

# sap\_rfc\_describe\_function

## Syntax

Inspect the full RFC interface of a function module.

```
sap_rfc_describe_function(function_name VARCHAR)
```

## Example usage

```
SELECT * FROM sap_rfc_describe_function('BAPI_FLIGHT_GETLIST');
```

See [`sap_rfc_describe_function()` in the function reference](/docs/reference/erpl-functions.md#sap_rfc_describe_function) for full parameter details.

# sap\_rfc\_ping

## Syntax

This functions checks the connection and login credentials to the SAP server. Internally, ERPL tries to establish a connection to the SAP server, performs a ping, and the SQL statement `SELECT 'PONG' as msg`.

The function has the following signature:

```
PRAGMA sap_rfc_ping;
```

## Example usage

```
PRAGMA sap_rfc_ping;
```

# sap\_rfc\_reload\_ini\_file

## Syntax

Reload the SAP RFC INI file.

The function has the following signature:

```
PRAGMA sap_rfc_reload_ini_file;
```

## Example usage

```
PRAGMA sap_rfc_reload_ini_file;
```

# sap\_rfc\_set\_ini\_path

## Syntax

Set the path of the SAP ini file. The function has a positional argument (VARCHAR) with the path to the SAP RFC INI File.

The function has the following signature:

```
PRAGMA sap_rfc_set_ini_path(<VARCHAR>);
```

## Example usage

```
PRAGMA sap_rfc_set_ini_path("path_to_your_ini_file/");
```

# sap\_rfc\_set\_maximum\_stored\_trace\_files

## Syntax

The function has the following signature:

```
PRAGMA sap_rfc_set_maximum_stored_trace_files(<unsigned int>);
```

## Example usage

```
PRAGMA sap_rfc_set_maximum_stored_trace_files(10);
```

# sap\_rfc\_set\_maximum\_trace\_file\_size

## Syntax

Set the maximum size of the trace files. Two positional arguments are possible. The first one is of type `<unsigned int>` and defines the size. The second one defines the unit and is of type `String` with possible values `M` and `G`.

The function has the following signature:

```
PRAGMA sap_rfc_set_maximum_trace_file_size(<unsigned int>, <String>);
```

## Example usage

Set the trace file size to `1024` MB.

```
PRAGMA sap_rfc_set_maximum_trace_file_size(1024, "M");
```

# sap\_rfc\_set\_trace\_dir

## Syntax

This function sets the path to the trace directory. The function has a positional argument (VARCHAR)

The function has the following signature:

```
PRAGMA sap_rfc_set_trace_dir(<VARCHAR>);
```

## Example usage

```
PRAGMA sap_rfc_set_trace_dir("path_to_your_trace_dir/");
```

# sap\_rfc\_set\_trace\_level

## Syntax

Set the trace level. `sap_rfc_set_trace_level` has an unnamed positional parameter that takes unsigned integers. The available trace levels are:

*   `0`: Off
*   `1`: Brief
*   `2`: Verbose
*   `3`: Detailed
*   `4`: Full

The function has the following signature:

```
PRAGMA sap_rfc_set_trace_level(<Unsigned Int>);
```

## Example usage

The following function call set the trace level to `Verbose`:

```
PRAGMA sap_rfc_set_trace_level(2);
```

# sap\_describe\_fields

## Syntax

The `sap_describe_fields` function returns a table that provides metadata about the fields of a given SAP table or structure. Here's a detailed description of each column in the output:

1.  **pos**: Position of the field within the table or structure. This is typically a four-digit number (e.g., 0001, 0002).
    
2.  **is\_key**: Indicates whether the field is a key field ('X') or not (blank). Key fields are crucial for uniquely identifying a record in the table.
    
3.  **field**: The name of the field. This is the technical name used within SAP.
    
4.  **text**: A descriptive text or label for the field. This is a more human-readable description of what the field represents.
    
5.  **decimals**: The number of decimal places for numerical fields. This is typically a six-digit number (e.g., 000000, 000004).
    
6.  **check\_table**: The name of the check table associated with the field. A check table is used to validate the values entered in the field.
    
7.  **ref\_table**: The name of the reference table, if the field references another table.
    
8.  **ref\_field**: The name of the reference field in the reference table, if applicable.
    
9.  **language**: The language code (e.g., 'E' for English) for the descriptive text.
    

The function has the following signature:

```
SELECT * FROM sap_describe_fields([col0 = "<tablename>"])
```

## Example usage

```
SELECT * FROM SAP_DESCRIBE_FIELDS('SPFLI');
```

# sap\_read\_table

## Syntax

To find a table you are looking for, you can use the `sap_read_table` function. This function returns a list of all available tables in the SAP ERP system.

The function has the following signature:

```
SELECT * FROM sap_read_table("<tablename>", [THREADS = NUMBER_OF_THREADS], [MAX_ROWS = NUMBER_OF_MAX_ROWS], [FILTER = '<query filter>'])
```

## Example usage

```
SELECT * FROM sap_read_table("SFLIGHT")
```

# sap\_rfc\_describe\_function

## Syntax

Explore the API of a RFC function.

The function has the following signature:

```
SELECT * FROM sap_rfc_describe_function('$BAPI_NAME')
```

*   `NAME`: Containing the full name of the BAPI, typically this is the same as the argument of the function.
    
*   `IMPORT`: Contains a list with description of all input types. Have a look especially at the required flag. This parameters have to be provided.
    
*   `EXPORT`: Also list with export parameters.
    
*   `CHANGING`: Are so called in/out parameters, which can be input as well as output.
    
*   `TABLES`: This are parameters in form of tables (which are lists of structs in DuckDB). Tables can also have in/out direction.
    

## Example usage

```
SELECT * FROM sap_rfc_describe_function('BAPI_FLIGHT_GETLIST')
```

# sap\_rfc\_invoke

## Syntax

Invoke functions and return values.

The function has the following signature:

```
SELECT * FROM sap_rfc_invoke('function_name', [path=<path name>])
```

## Example usage

```
SELECT * FROM sap_rfc_invoke('BAPI_FLIGHT_GETLIST', path='/FLIGHT_LIST')
```

# sap\_rfc\_show\_function

## Syntax

```
sap_rfc_show_function()
```

### Named parameters

| Parameter | Type | Description |
| --- | --- | --- |
| `FUNCNAME` | VARCHAR | Filter by function name pattern (e.g. `'BAPI_FLIGHT*'`) |
| `GROUPNAME` | VARCHAR | Filter by function group |
| `LANGUAGE` | VARCHAR | Filter by language key (e.g. `'EN'`) |

## Example usage

```
SELECT * FROM sap_rfc_show_function(FUNCNAME => 'BAPI_FLIGHT*') ORDER BY 1;SELECT * FROM sap_rfc_show_function(GROUPNAME => 'SFLIGHT');
```

# sap\_rfc\_show\_groups

## Syntax

```
sap_rfc_show_groups()
```

### Named parameters

| Parameter | Type | Description |
| --- | --- | --- |
| `GROUPNAME` | VARCHAR | Filter by function group name pattern |
| `LANGUAGE` | VARCHAR | Filter by language key (e.g. `'EN'`) |

## Example usage

```
SELECT * FROM sap_rfc_show_groups();SELECT * FROM sap_rfc_show_groups(GROUPNAME => 'SFLIGHT*');
```

# sap\_show\_tables

## Syntax

To find a table you are looking for, you can use the `sap_show_tables` function. This function returns a list of all available tables in the SAP ERP system.

The function has the following signature:

```
SELECT * FROM sap_show_tables([TABLENAME = "<tablename>"], [TEXT = "<search string>"])
```

## Example usage

```
SELECT * FROM sap_show_tables()
```

This will return a (long) list of tables. To find a specific table, you can supply a search string to the function. The search string is matched against the table name and the table description. For example, to find all tables starting with `FLIGHT` you can use the following query:

```
SELECT * FROM sap_show_tables(TABLENAME='*SPFL*')
```

If you are not familiar with the internal naming scheme of SAP tables you can also use the `sap_show_tables` function to find tables by their description text. For example, to find all tables containing the words `Flight Schedule` you can use the following query:

```
SELECT * FROM sap_show_tables(TEXT='*Flight Schedule*')
```



# Welcome to ERPL

ERPL is a DuckDB extension that connects your analytics workflow directly to SAP systems. By the end of this guide, you'll know which extension to use and how to get started.

**What is ERPL?:**

ERPL (Enterprise Resource Planning Loader) is a DuckDB extension that provides native connectivity to SAP systems. It comes in two flavors: **ERPL** for on-premise SAP and **ERPL-Web** for cloud/web-based SAP services.

## Which Extension Do I Need?

### 🏢 ERPL (On-Premise SAP)

**Use when:** Your SAP system is behind a firewall and requires SSH tunneling or direct network access.

**Protocols supported:**

*   **RFC** - Read SAP tables and call function modules
*   **BICS** - Execute SAP BW queries
*   **ODP** - Delta replication from SAP systems

**Example:**

```
-- Read customer master data via RFCSELECT * FROM sap_read_table('KNA1', MAX_ROWS => 100);
```

### ☁️ ERPL-Web (Cloud/Web SAP)

**Use when:** Your SAP system exposes web APIs or you're using SAP cloud services.

**Protocols supported:**

*   **OData** - Query SAP services via HTTP/HTTPS
*   **Datasphere** - Connect to SAP Datasphere
*   **ODP Web** - ODP via OData protocol

**Example:**

```
-- Query OData serviceSELECT * FROM odata_read(  'https://api.sap.com/Orders',  filter => 'year eq 2024');
```

## Quick Start

### For On-Premise SAP (ERPL)

1.  [Install ERPL extension](/docs/get_started/install.md)
2.  [5-minute quickstart guide](/docs/get_started/quickstart-erpl.md)
3.  [Read your first SAP table](/docs/guides/simple/read-sap-table.md)

### For Cloud/Web SAP (ERPL-Web)

1.  [Install ERPL-Web extension](/docs/get_started/install.md)
2.  [5-minute quickstart guide](/docs/get_started/quickstart-erpl-web.md)
3.  [Query your first OData service](/docs/guides/simple/query-odata.md)

## Architecture Overview

## Common Use Cases

### 📊 Data Analysis

*   Extract SAP data for analysis in Python/R
*   Create dashboards in Power BI/Tableau
*   Build machine learning models with SAP data

### 🔄 Data Integration

*   Replicate SAP data to data lakes
*   Real-time data streaming from SAP
*   ETL pipelines with SAP as source

### 📈 Reporting

*   Automated reports from SAP data
*   Cross-system analytics
*   Historical data analysis

## What's Next?

### 🚀 Getting Started

*   [Installation Guide](/docs/get_started/install.md) - Install the right extension
*   [Quick Start Guides](/docs/get_started.md) - Get up and running in 5 minutes

### 📚 Learn by Doing

*   [Simple Guides](/docs/guides/simple/read-sap-table.md) - For data analysts and beginners
*   [Advanced Guides](/docs/guides/advanced/rfc-metadata.md) - For SAP experts
*   [Integration Guides](/docs/guides/integration/python-pandas.md) - Connect with your favorite tools

### 💡 Examples

*   [ERPL Examples](/docs/examples/erpl-examples.md) - On-premise SAP examples
*   [ERPL-Web Examples](/docs/examples/erpl-web-examples.md) - Cloud SAP examples
*   [Real-World Use Cases](/docs/examples/real-world-use-cases.md) - Complete scenarios

### 📖 Reference

*   [Function Reference](/docs/reference.md) - Complete API documentation
*   [Configuration](/docs/reference/configuration.md) - Advanced configuration options
*   [Troubleshooting](/docs/reference/troubleshooting.md) - Common issues and solutions

## Need Help?

**Getting Stuck?:**

*   Check our [troubleshooting guide](/docs/reference/troubleshooting.md)
*   Browse [real-world examples](/docs/examples/real-world-use-cases.md)
*   Join our community discussions

**For SAP Experts:**

Looking for advanced topics? Check out our [advanced guides](/docs/guides/advanced/rfc-metadata.md) covering RFC internals, ODP delta replication, and performance optimization.

---

**Ready to get started?** Choose your path:

*   [🏢 On-premise SAP → Install ERPL](/docs/get_started/install.md)
*   [☁️ Cloud/Web SAP → Install ERPL-Web](/docs/get_started/install.md)