Market Data Ingestion Accelerator

$1490

Generic balancing-energy/market-data ingestion for open APIs (ENTSO-E-style): bronze->silver->gold with XML time-series flattening, revision dedup, and DQ checks. Secret-scoped API token. Ships a dependency-free ingestion library + offline tests. Fully generic Databricks Asset Bundle.

📁 18 files🏷 v1.0.0

Production-readyUnit-testedDatabricks Asset Bundle

✓ Instant download✓ Lifetime updates✓ MIT licensed✓ Secure checkout (Stripe)

⚙ Try the Live Demo interactive

Expand a market-data Period/Point block into per-interval UTC rows — the accelerator's time-series flattening logic, live.

⚡ Open Series Flattener

📋 What's Inside 18 files

✓ README.md
✓ LICENSE
✓ manifest.json
✓ databricks.yml
✓ resources/jobs.yml
✓ src/ingest/api_client.py
✓ src/ingest/realtime_ingest.py
✓ src/transform/bronze_to_silver.py
✓ src/transform/silver_to_gold.py
✓ src/quality/dq_checks.sql
✓ lib/market_api.py
✓ tests/test_market_api.py
✓ conftest.py
✓ guide/01_what-you-get.md
✓ guide/02_getting-started.md
✓ guide/03_architecture.md
✓ guide/04_support.md
✓ guides/market-ingestion-methodology.md

📁 File Structure 18 files

market-data-ingestion-accelerator/
├── README.md
├── LICENSE
├── manifest.json
├── databricks.yml
├── resources/
│   ├── jobs.yml
├── src/
│   ├── ingest/
│   │   ├── api_client.py
│   │   ├── realtime_ingest.py
│   ├── transform/
│   │   ├── bronze_to_silver.py
│   │   ├── silver_to_gold.py
│   ├── quality/
│   │   ├── dq_checks.sql
├── lib/
│   ├── market_api.py
├── tests/
│   ├── test_market_api.py
├── conftest.py
├── guide/
│   ├── 01_what-you-get.md
│   ├── 02_getting-started.md
│   ├── 03_architecture.md
│   ├── 04_support.md
├── guides/
│   ├── market-ingestion-methodology.md

📖 Documentation Preview README excerpt

Market Data Ingestion Accelerator

A generic, deploy-ready ingestion pipeline for balancing-energy / market-data

**open APIs** (ENTSO-E Transparency Platform-style): bronze → silver → gold with

XML time-series flattening, revision de-duplication, and data-quality checks.

Fully generic and environment-agnostic: configure your bronze/silver/gold

catalogs, bidding-zone EIC codes, and API token (via a Databricks **secret

scope** — never hard-coded). No organization-specific dependencies.

What's inside

- **Databricks Asset Bundle** — a 30-minute realtime ingest → transform → DQ job.

- **A thin REST API client** (`src/ingest/api_client.py`) with request building

and secret-scoped auth.

- **Bronze→silver→gold transforms** — XML parse, point flattening, enrich, dedup.

- **Data-quality SQL** — freshness, null, and duplicate assertions per zone.

- **A dependency-free ingestion library** (`lib/market_api.py`): query-param

building, ISO-8601 resolution mapping, Period/Point flattening, exponential

backoff, and dedup — with an offline test suite (5 tests).

Quickstart

```bash

pip install pytest && pytest tests/ -v # validate parsing/dedup offline

databricks bundle deploy -t dev -p <profile>

```

Store your API token in a secret scope, set `bidding_zones` (EIC codes), and the

bronze/silver/gold catalogs.

License

MIT — see `LICENSE`.

... preview truncated, see full README in product download.

📄 Code Sample .py preview

src/ingest/api_client.py# Market Data Ingestion: API client (generic, ENTSO-E-style) # Thin, dependency-light REST client. Token is read from a Databricks secret at # call time by the notebook; nothing organization-specific is embedded. # The request/param + response-flattening logic is unit-tested in lib/market_api.py. import urllib.parse import urllib.request from lib.market_api import build_query_params, rate_limit_delay_ms class MarketApiClient: def __init__(self, base_url, token, rate_limit_ms=1500): self.base_url = base_url.rstrip("/") self.token = token self.rate_limit_ms = rate_limit_ms def build_url(self, bidding_zone, period_start, period_end, document_type="A24"): params = build_query_params(bidding_zone, period_start, period_end, document_type) params["securityToken"] = self.token return f"{self.base_url}?{urllib.parse.urlencode(params)}" def fetch(self, bidding_zone, period_start, period_end, document_type="A24"): url = self.build_url(bidding_zone, period_start, period_end, document_type) req = urllib.request.Request(url, headers={"User-Agent": "market-data-accelerator"}) with urllib.request.urlopen(req, timeout=60) as r: return r.read().decode("utf-8") # XML payload -> parsed downstream

Buy Now — $1490 ⚙ Open Series Flattener Back to Products