A modern financial data estate has hundreds of tables. Three things are almost always true.
Nobody can tell you what a table means
A name like pos_bal_eod or stg_txn_2 carries no semantics a tool can reason about. Is it a ledger, a balance snapshot, a reference table, or transient ETL scratch? The answer lives in a senior engineer's head, and it walks out the door when they do.
Nobody can trace where a number came from
When an auditor asks "how was this consolidated balance derived?", the answer is reconstructed by hand from tribal knowledge and half-remembered joins. Foreign keys describe structure, not flow. The actual lineage is buried in query history nobody parses.
Nobody can prove the numbers reconcile
Sub-ledger rolls up to general ledger rolls up to consolidated. Whether those roll-ups actually balance, to a defined tolerance, with a named audit assertion behind each check, is the entire job of a financial controller and it is done in spreadsheets.
These aren't visualisation problems. A prettier graph doesn't tell you a table is a General Ledger, doesn't parse your query log into column-level lineage, and doesn't assert completeness against a tolerance. They're semantic, lineage, and reconciliation problems. dataLineage solves all three in one pipeline.
An eleven-stage crawl, run entirely against a read-only connection.
A FastAPI + SurrealDB backend with a Next.js graph frontend. Register a read-only database connection; the platform runs an eleven-stage crawl that introspects the schema, classifies every table, builds the lineage graph, and detects schema drift since the last crawl, all asynchronously via Celery.
Every credential is encrypted at rest with AES-256-GCM. Every privileged action is written to an audit trail. Crawls can run on a cron schedule via Celery Beat. The whole stack comes up with a single make up.
The nine-type financial taxonomy.
Every crawled table is classified into one of nine financial-data types. A structural heuristic pass scores each table on column-naming patterns, foreign-key density, and row-count profile; tables that clear a 0.85 confidence threshold are auto-classified, and the rest are sent to an LLM classifier (NVIDIA NIM or Claude, selected via LLM_PROVIDER) with the same nine-type rubric. Classifications are cached, overridable in the UI, and stored with their confidence and source.
| Type | Enum value | What it captures |
|---|---|---|
| Transaction | transaction | Atomic financial events: trades, payments, ledger entries. High row count, append-only, debit/credit columns. |
| Position / Balance | position_balance | Current state of accounts or holdings. Updated in place; balance/quantity columns. |
| Reference / Master | reference_master | Lookup and master data: instruments, counterparties, currencies. Static, many outbound FKs. |
| Subledger | subledger | Subsidiary records that roll up to a GL. GL account refs, period and debit/credit columns. |
| General Ledger | general_ledger | Authoritative accounting record. Chart of accounts, trial balance. High inbound FK density. |
| Transformation | transformation | Derived intermediate data from ETL or business logic. Computed columns, multi-source refs. |
| Reporting / Aggregated | reporting_aggregated | Pre-computed aggregations. Total/summary columns, period groupings, no outbound FKs. |
| Staging / ETL | staging_etl | Landing zones. stg_, tmp_, landing_ naming; load_ts, batch_id columns. |
| Mapping / Enrichment | mapping_enrichment | Cross-reference tables mapping identifiers across systems. Multiple FKs to different masters. |
Three kinds of edge, one graph.
The crawl builds three kinds of edge into a SurrealDB graph and renders them with Cytoscape, colour-coded by classification.
Foreign-key edges
structuralStraight from the crawled constraint metadata.
Inferred edges
heuristicFrom name-overlap and naming-convention matching, e.g. stg_trades → trades.
Query-log edges
observedFrom parsing the database's own statement history (pg_stat_statements, Snowflake QUERY_HISTORY, MySQL events_statements_summary) through sqlglot to extract column-level dependencies.
Upstream/downstream traversal, schema-change detection between crawls (with breaking / warning / info severity), and OpenLineage v2 event export all sit on top of this graph.
The medallion model, and Gold is composition-only.
dataLineage proposes a composable Bronze / Silver / Gold layer set on top of your raw tables, with the discipline a financial estate needs. The generator enforces it, not just documents it.
At the Gold tier the SQL generator rejects aggregate, computed, and type-cast transforms, only carry_forward and rename survive, and emits SELECT + WHERE + ORDER BY only, no GROUP BY, no aggregates. This keeps the Gold layer auditable: it composes trusted Silver output, it doesn't invent new numbers. Every layer carries a full version history (ai_draft → human_edited → approved → archived/stale) with structured diff and rollback, and can be exported as dbt models.
Every checkpoint ties to a named audit assertion.
Reconciliation checkpoints validate that value rolls up correctly through the financial hierarchy.
A checkpoint aggregates a sum_column over group_by dimensions from a source layer, joins it to a target via match_keys, and compares totals within a tolerance (e.g. 0.01 for FX rounding). The result is balanced, variance, or break, with expected/actual totals, variance percentage, and unmatched items surfaced. Checkpoints are auto-proposed from table classifications and run fire-and-forget through Celery. The assertion vocabulary, completeness, existence, accuracy, follows ISA 315 / PCAOB.
Built against named frameworks, not invented ones.
BCBS 239
POST /compliance/bcbs239/assessA fourteen-principle assessment engine scores the estate across governance, data aggregation, and risk-reporting categories, with per-principle evidence.
OpenLineage v2
POST /openlineage/eventsIngest and emit lineage as RunEvents (START / RUNNING / COMPLETE / ABORT / FAIL) so dataLineage interoperates with the open lineage ecosystem.
ISA 315 / PCAOB
Phase 1 implementedAudit assertions back every reconciliation checkpoint. Valuation, rights-obligations, and presentation assertions are planned.
Enterprise foundation
JWT plus API-key dual authentication; RBAC across admin / editor / viewer roles (multi-role per user); AES-256-GCM credential encryption (12-byte nonce, NIST SP 800-38D); per-user Redis-backed rate limiting; a SurrealDB audit trail on every privileged action; scheduled crawls via a custom Celery Beat DynamicScheduler that reloads cron schedules from the database every 60s; in-app and email notifications; and a Prometheus /metrics endpoint.
Eleven stages, end to end, asynchronously.
Frontend, Next.js 16, React 19, Tailwind v4, Cytoscape.js (dagre + fcose layouts), shadcn/ui, Zustand, TanStack Query, Zod, React Hook Form. Auth rides an httpOnly cookie proxied through Next.
Backend, FastAPI (≥0.133), Python 3.12, SurrealDB (graph + document), Celery + Redis, sqlglot for SQL parsing and dialect transpilation, cryptography for AES-256-GCM, Anthropic Claude (or NVIDIA NIM) for classification, Prometheus client, slowapi, loguru.
Supported databases
| Database | Schema crawl | Query-log lineage | SQL generation |
|---|---|---|---|
| PostgreSQL | ✓ pg_catalog | ✓ pg_stat_statements | ✓ postgres |
| Snowflake | ✓ INFORMATION_SCHEMA | ✓ QUERY_HISTORY | ✓ snowflake |
| MySQL | ✓ information_schema | ✓ events_statements_summary | ✓ mysql |
| SQL Server | ✓ sys.* views | ⚠ dm_exec_query_stats (limited) | ✓ tsql |
Connections are read-only. dataLineage never writes to a customer database.
What's production-ready, what's partial, what's planned.
- 9-type classification (heuristic + LLM: NVIDIA NIM or Claude)
- Lineage graph (FK / inferred / query-log)
- Bronze/Silver/Gold layers, versioning, dbt export
- AES-256-GCM credential encryption
- JWT + API-key auth and RBAC
- Scheduled crawls, Prometheus metrics
- Schema-change detection
- BCBS 239 assessment
- OpenLineage v2 ingestion
- All four database connectors
- Reconciliation: checkpoint proposer and runner implemented; some API surface still returns task handles without full async execution
- ISA 315 / PCAOB: Phase 1 assertions only
- AI root-cause analysis for reconciliation breaks
- Cross-entity (Level 4) reconciliation
- Phase 2 audit assertions: valuation, rights-obligations, presentation
- A dedicated auditor role
A single make up brings up the full stack.
| /auth | Register, login, refresh, email verification, password reset, logout |
| /connections | Connection CRUD, crawl trigger + status, table listing |
| /connections/{id}/graph | Lineage graph export |
| /connections/{id}/layers | Propose / generate / export Bronze · Silver · Gold layers |
| /connections/{id}/reconciliation | Checkpoint propose / list / run / results |
| /connections/{id}/compliance/bcbs239 | BCBS 239 assessment |
| /connections/{id}/openlineage | OpenLineage jobs / runs / edges / event ingest |
| /metrics | Prometheus exposition |
Built for teams who have to defend their numbers, not just visualise them.
Proprietary, Single Source Studios (Pty) Ltd. All rights reserved.