Skip to main content

ERPL-Web Overview

ERPL-Web is a production-grade DuckDB extension for HTTP, OData, and SAP Datasphere integration. Query APIs, web services, and cloud data sources directly from SQL.

Core Capabilities​

HTTP from SQL
Call any REST API with GET, POST, PUT, PATCH, DELETE. Full support for headers, authentication, and custom timeouts.

OData v2/v4
Universal OData reader with ATTACH support, automatic version detection, predicate pushdown ($filter, $select, $top, $skip), and expand navigation for complex queries.

SAP Datasphere
First-class DuckDB integration for SAP Datasphere. OAuth2 authentication, discover spaces and assets, query relational and analytical data with full metadata support.

ODP via OData
Delta replication from SAP using ODP OData protocol. Automatic subscription management, change tracking (Insert/Update/Delete), and comprehensive audit logging.

Secrets Management
Secure credential storage with OAuth2, config, and file providers. Automatic token refresh for cloud services.

Tracing & Diagnostics
Configurable logging for debugging network calls, performance tuning, and troubleshooting.

Installation​

-- Install from ERPL repository
INSTALL 'erpl_web' FROM 'http://get.erpl.io';
LOAD 'erpl_web';

Build from source (developers):

make debug

Quick Start​

HTTP in One Line​

SELECT content::JSON->>'ip' AS ip
FROM http_get('https://api.ipify.org?format=json');

OData as a Database​

ATTACH 'https://services.odata.org/TripPinRESTierService' 
  AS trippin (TYPE odata);

SELECT UserName, FirstName, LastName 
FROM trippin.People 
WHERE Gender = 'Female';

SAP Datasphere​

-- Create OAuth2 secret
CREATE SECRET datasphere (
  TYPE datasphere,
  PROVIDER oauth2,
  TENANT_NAME 'your-tenant',
  DATA_CENTER 'eu10'
);

-- Discover and query
SELECT * FROM datasphere_show_spaces();
SELECT * FROM datasphere_read_relational('SALES', 'CUSTOMERS');

Delta Replication from SAP​

-- Create auth secret
CREATE SECRET sap_system (
  TYPE http_basic,
  username 'SAP_USER',
  password 'SAP_PASSWORD'
);

-- Initial load (creates subscription)
SELECT * FROM odp_odata_read(
  'https://sap/odata/Z_ODP_BW_1_SRV/SalesData',
  secret='sap_system'
);

-- Delta updates (only changed records)
SELECT RECORD_MODE, COUNT(*) 
FROM odp_odata_read(
  'https://sap/odata/Z_ODP_BW_1_SRV/SalesData',
  secret='sap_system'
)
GROUP BY RECORD_MODE;

Documentation​

Getting Started​

Advanced Topics​

Examples​

Features​

HTTP Client​

  • ✅ GET, HEAD, POST, PUT, PATCH, DELETE
  • ✅ Custom headers and authentication
  • ✅ Bearer and Basic auth support
  • ✅ Timeout configuration
  • ✅ Automatic retries

OData Support​

  • ✅ OData v2 and v4 auto-detection
  • ✅ ATTACH as database
  • ✅ Predicate pushdown ($filter, $select, $top, $skip)
  • ✅ Expand navigation properties
  • ✅ EDM type mapping
  • ✅ Automatic pagination

SAP Datasphere​

  • ✅ OAuth2 flows (authorization code, client credentials)
  • ✅ Space and asset discovery
  • ✅ Relational data access
  • ✅ Analytical data access with metrics/dimensions
  • ✅ Parameterized views support
  • ✅ Automatic token refresh

ODP via OData​

  • ✅ Automatic delta replication
  • ✅ Subscription management
  • ✅ Change tracking (Insert/Update/Delete)
  • ✅ Audit logging
  • ✅ SAP-approved extraction method
  • ✅ Incremental data synchronization

Security​

  • ✅ DuckDB secrets integration
  • ✅ OAuth2 provider
  • ✅ Config file provider
  • ✅ File-based secrets
  • ✅ Encrypted credential storage

Observability​

  • ✅ Configurable tracing
  • ✅ File and console output
  • ✅ Log rotation
  • ✅ Performance metrics
  • ✅ Detailed debugging

Use Cases​

API Integration
Query REST APIs, aggregate data from multiple sources, build data pipelines from web services.

SAP Data Extraction
Extract data from SAP systems using approved ODP OData protocol with automatic delta synchronization.

Cloud Analytics
Access SAP Datasphere, combine cloud and on-premise data, build federated analytics.

Data Lake Population
Stream data from APIs and SAP to Parquet files, maintain incremental updates, build data lakes.

ETL & ELT
Transform API data with SQL, sync SAP data to warehouses, orchestrate data flows.

Performance & Resilience​

  • Response caching - HTTP and OData metadata caching
  • Exponential backoff - Automatic retry with intelligent delays
  • Connection reuse - Keep-alive for efficient throughput
  • Predicate pushdown - Server-side filtering for OData
  • Pagination handling - Automatic chunking for large datasets

Next Steps​

  1. Install - INSTALL 'erpl_web' FROM 'http://get.erpl.io'
  2. Learn - Start with HTTP Functions
  3. Explore - Try OData and Datasphere
  4. Build - Check Examples for patterns

Community & Support​

License​

ERPL-Web is licensed under the Business Source License (BSL) 1.1. See LICENSE.

Build API-powered analytics with DuckDB + ERPL-Web. Query the web like it's a table. 🚀