Home › Enterprise Intelligence Architecture

Provenance, Lineage & Compliance

The PROV model, column-level lineage from the resolved plan, deterministic replay of a regulatory figure, GDPR erasure across the whole derived-data graph, and the real limits of privacy tech.

Last reviewed: 2026-09-04 · Українською

In this area

'Where did this number come from' is an engineering question, not a paperwork one. If a platform cannot name every transformation, code version and input snapshot behind a specific reporting cell, the number exists only as an assertion. This module treats provenance as a first-class architectural layer: how to capture it from the resolved logical plan rather than from SQL text, how to assemble an evidence package that replays a figure to the cent, and how to run erasure once personal data has already spread into aggregates, embeddings and backups. This is the problem the lineage layer of ontology platforms such as Palantir Foundry is built to solve.

The provenance model and column-level lineage

E6.1

Three node types, and nothing else. The W3C PROV model reduces provenance to three entities: Entity — an artefact (a table version, a file, model weights); Activity — a run that consumed some artefacts and produced others; Agent — the human or service accountable for that run. Everything else is edges: used, wasGeneratedBy, wasAttributedTo, wasDerivedFrom. One discipline is non-negotiable: an entity is immutable. Overwriting a table does not update a node, it creates a new one. Otherwise the graph starts lying retroactively — yesterday's report points at today's content, and no audit will catch it.

Three different questions people keep conflating:

• Where — which physical source cell the value physically came from. This is what you need when investigating 'why does it say 12 and not 14'.
• Why — which input rows caused this output row to exist (the witnesses of the predicate). This is the basis of erasure and impact analysis.
• How — by which sequence of operations it was produced. This is the basis of reproduction.

A catalogue that shows only table-to-table edges answers none of the three, even though it looks like lineage.

Column lineage comes from the plan, not from query text. SQL text is an unreliable source: SELECT * names no columns, nested views hide two levels of renaming, USING collapses two columns into one, and a CTE is reused under different aliases. The resolved logical plan already contains what the text does not: the star is expanded into an attribute list, every attribute carries a stable id, and each output column's expression is a tree over specific input attributes. Lineage extracts mechanically from there, and it distinguishes two edge classes: DIRECT (the value genuinely reached the output) and INDIRECT (the column influenced the result through WHERE, GROUP BY or JOIN, but its values are not present in the output). For compliance that distinction is load-bearing: an indirect edge is not a disclosure, but it is a dependency — and erasure must honour it.

Policy travels along the same edges. If customers.tax_id carries a PII tag, every column whose expression touches it inherits the tag automatically. Removing the tag requires an explicit downgrade recording who did it and why — for a salted hash whose salt lives elsewhere, say. Hand-tagging marts does not scale and drifts out of sync within a week.

Measure coverage, mark the gaps. Python UDFs, external service calls and foreachPartition are opaque to plan analysis. Such a node must be given an explicit opaque status rather than being silently dropped. The working metric is the share of mart columns whose provenance resolves to a root; below 90% the graph is not fit for impact analysis, and you want to know that in advance, not during an incident.

{
  "eventType": "COMPLETE",
  "eventTime": "2026-03-11T02:14:07.318Z",
  "run": {
    "runId": "b1f0c2de-8a41-4f6b-9d55-0c9a7e2f4411",
    "facets": {
      "parent":      { "job": { "namespace": "risk", "name": "daily_exposure" } },
      "codeVersion": { "gitSha": "9f2c1ab", "image": "sha256:4d7e0b31...9c1" }
    }
  },
  "job": { "namespace": "risk", "name": "risk.build_exposure_daily" },

  "inputs": [
    { "namespace": "delta://lake", "name": "core.positions",
      "facets": { "version": { "datasetVersion": "4217" } } },
    { "namespace": "delta://lake", "name": "ref.fx_rates",
      "facets": { "version": { "datasetVersion": "88" } } }
  ],

  "outputs": [
    { "namespace": "delta://lake", "name": "risk.exposure_daily",
      "facets": {
        "columnLineage": {
          "fields": {
            "exposure_eur": {
              "inputFields": [
                { "name": "core.positions", "field": "notional",
                  "transformations": [ { "type": "DIRECT",   "subtype": "TRANSFORMATION", "masking": false } ] },
                { "name": "ref.fx_rates",  "field": "rate",
                  "transformations": [ { "type": "DIRECT",   "subtype": "TRANSFORMATION", "masking": false } ] },
                { "name": "core.positions", "field": "book_id",
                  "transformations": [ { "type": "INDIRECT", "subtype": "GROUP_BY",       "masking": false } ] }
              ]
            }
          }
        }
      } }
  ]
}
// DIRECT = the value reached the output. INDIRECT = it only shaped the grouping.
// Both are dependencies for erasure; only DIRECT is a disclosure.

The evidence package: replaying a figure deterministically

E6.2

What a regulator actually asks for. Not 'show me the log', but 'reproduce cell R3.C7 of the return as of 28 February'. That is retrospective provenance: the graph of a specific run, not the shape of the pipeline. This is exactly where the audit log and provenance part ways. A log says what happened: job X wrote table Y at 03:12 as svc_etl. Provenance lets you repeat it and land on the same number.

Four pins, without which replay is impossible:

• A version for every input. Not 'table positions' but positions@4217 — a Delta version, an Iceberg snapshot-id, an object hash. Reading the 'current' state makes replay non-deterministic by construction.
• A code digest. A git sha is not enough: the same revision on a different base image ships a different numpy and rounds differently. Pin the image sha256.
• Run parameters. Processing date, rates, thresholds — anything the job reads from outside must arrive as an explicit argument.
• Determinism of the logic itself. current_date(), rand(), first() without ORDER BY, collect_set, floating-point reduction order — each of these quietly destroys reproducibility.

The evidence package is a backward closure over the graph. From the output cell you walk back along wasDerivedFrom edges, collecting every intermediate artefact down to the raw extracts from source systems. Every node on the path carries its own four pins. The acceptance rule is simple and checkable in one query: if any node in the closure lacks an input version, lacks an image digest, or is flagged non-deterministic, the figure is not provable. It is a binary outcome, not a 'mostly fine' score.

The shape of the workflow is checkable too. A pipeline as a task graph is a Petri net, and Petri nets have a classical soundness property: termination is reachable from every reachable state, no dangling tokens remain after termination, and there are no dead transitions that never fire. Formal checking catches what no data test catches: a branch that writes the output twice under one flag combination, or a step that never runs and therefore silently leaves a mart two weeks stale.

Pins are captured at run time, not reconstructed later. Trying to rebuild the coordinates from git history and the orchestrator schedule six months on is an assumption, not evidence. The collector must emit the event as the run executes, synchronously with the transaction commit — otherwise the graph fills with runs that have no result and results that have no run.

-- Backward closure: everything a single reporting cell depends on.
-- prov_edge is written by the lineage collector from the resolved logical plan;
-- prov_run holds the four replay pins for the run that produced each edge.
WITH RECURSIVE closure(out_ds, out_col, in_ds, in_col, run_id, depth) AS (

    SELECT e.out_ds, e.out_col, e.in_ds, e.in_col, e.run_id, 1
      FROM prov_edge e
     WHERE e.out_ds        = 'risk.exposure_daily'
       AND e.out_col       = 'exposure_eur'
       AND e.business_date = DATE '2026-02-28'

  UNION ALL

    SELECT e.out_ds, e.out_col, e.in_ds, e.in_col, e.run_id, c.depth + 1
      FROM prov_edge e
      JOIN closure  c ON e.out_ds = c.in_ds AND e.out_col = c.in_col
     WHERE c.depth < 32          -- cycle guard: a sound provenance graph is acyclic
)
SELECT c.depth, c.in_ds, c.in_col,
       r.dataset_version, r.git_sha, r.image_digest, r.params, r.is_deterministic
  FROM closure c
  JOIN prov_run r USING (run_id)
 ORDER BY c.depth, c.in_ds;

-- Acceptance gate. The figure is provable only when this returns zero rows:
--   SELECT * FROM <closure above>
--    WHERE dataset_version IS NULL
--       OR image_digest    IS NULL
--       OR is_deterministic = false;

GDPR erasure and the limits of privacy-preserving analytics

E6.3

Erasure is a forward closure over the graph, not a DELETE. Within six months a personal row has spread into marts, aggregates, a feature store, an embedding index, caches, model weights and backups. GDPR Article 17 covers all copies, so the algorithm starts not with a delete but with walking lineage forward from every dataset where the subject appears.

Four classes of derived artefact, each with its own action:

• Rebuildable from source — delete and re-materialise. Cheap, provided the pipeline is idempotent.
• Aggregate above threshold — if the minimum group size is ≥ k and no quasi-identifier combination is unique, the artefact can be argued non-personal. The argument must be recorded with the numbers, or you will be inventing it under inspection.
• Immutable store — WORM, object lock, tape, 90-day backups. Rewriting is physically impossible, so you use crypto-shredding: the subject's data is encrypted under its own key, and destroying that key makes the ciphertext permanently unreachable.
• Irreversible but personal — embeddings, model weights, feature-store statistics. This is the genuinely hard case: escalate, decide at DPO level, usually retrain on a schedule.

The tombstone is mandatory. Without a suppression-list entry the nightly CRM sync brings the subject back the next morning, and the system will keep reporting a 'completed' erasure every month. The list stores only a hash of the identifier — the identifier itself must never live there.

Semantics matter more than mechanics. An erasure policy must define not 'the row was deleted from table T' but 'no future computation may depend on this data'. Those are different statements, and the auditor tests the second one.

What privacy tech does not buy you. Pseudonymisation is not anonymisation: while a mapping table exists, the data is still personal. k-anonymity breaks under linkage with an external dataset once the quasi-identifiers are narrow enough. Differential privacy gives a real guarantee, but its ε budget depletes and composes: after N queries against one dataset the budget is spent and the next query must be refused — an operational constraint that belongs in the engine, not in a note to analysts. And separately: a query being technically permitted does not make the flow appropriate — the same row that is appropriate in a fraud investigation is inappropriate in marketing segmentation.

# Forward closure from a data subject's rows to every derived artefact, then
# classify each artefact by the only property that decides the action:
# can it be rebuilt, is it aggregated above k, or is it physically immutable?

REBUILD, AGGREGATE_OK, SHRED, ESCALATE = (
    "rebuild", "aggregate_ok", "crypto_shred", "escalate")

def erasure_plan(subject_id, graph, catalog, k_threshold=50):
    plan, roots = [], catalog.datasets_holding(subject_id)

    for ds in graph.forward_closure(roots):        # follows DIRECT *and* INDIRECT edges
        meta = catalog.get(ds)

        if meta.immutable or meta.object_lock:     # WORM, tape, 90-day backups
            action = SHRED                         # destroy the per-subject DEK
        elif meta.rebuildable_from_source:
            action = REBUILD                       # delete + re-materialise
        elif meta.is_aggregate and meta.min_group_size >= k_threshold:
            action = AGGREGATE_OK                  # record the argument WITH the numbers
        else:
            action = ESCALATE                      # embeddings, weights, feature store

        plan.append({"dataset": ds, "action": action,
                     "min_group_size": meta.min_group_size, "sla_days": 30})

    # Without this the subject is re-ingested from the CRM on the next nightly sync
    # and the same erasure gets "completed" again every month.
    plan.append({"dataset": "suppression_list", "action": "insert_tombstone",
                 "key_hash": catalog.pseudonymise(subject_id)})
    return plan

Sources this area derives from

1. A Framework for Policies over Provenance
2. Cryptographically Secure Information Flow Control on Key V
3. A Role and Attribute Based Access Contro
4. A Formal Foundation for Workflow Composi
5. A Petri net based supervisory control im
6. A High Level Distributed Execution Frame
7. A Semantic Hierarchy for Erasure Policies
8. Need to Know Contextual Integrity Grounded Query Rewriting
9. When PETs misbehave A Contextual Integrity analysis
10. A Hybrid Approach to Private Record Link

Continue in this track