"Locks on the Lake" — Security, Privacy and Compliance

Skim Card

One thing to remember: a lake keeps copies on purpose, so privacy means knowing every copy, locking every door to it, and proving deletion in the bytes.

• Classify columns where the table lives (table properties), and fail closed: analysts can’t read an unclassified table.
• Scan values, not just names: session_id carried the clickstream ID, and no naming rule caught it.
• Masks live in one engine: Trino + OPA masked the analyst while PyIceberg read the same files in clear.
• Pseudonymise with a keyed hash whose key analysts can’t read: joins and distinct counts survive, the identity doesn’t.
• An Iceberg delete leaves the row in old files, snapshots, tags and manifest bounds: rewrite, expire, rewrite manifests, remove orphans.
• Deleted objects stay on disk until storage compacts; SeaweedFS waits for 30% garbage per volume.
• Keep a suppression list of keyed hashes: every loader drops or tombstones listed people, so reloads and re-uploads don’t resurrect them.

Use …	When …
Delete the record	Nothing else needs it (profile, clickstream)
Tombstone the identifier	The law makes you keep the record (orders, payments)
Crypto-shredding	Long-lived immutable stores, with legal sign-off

Verified against: Trino 484, Open Policy Agent 1.21.4, Apache Polaris 1.8.0, Apache Iceberg 1.12, PyIceberg 0.12.0, DuckDB 2.0, SeaweedFS 4.51, Apache Kafka 4.4.0, ClickHouse 26.8 · vendor docs and legal texts · 29 October 2026

Jump to: Concepts · Hands-On · Landscape · Design Drill

Story Opening

Farah forwarded the email on Thursday morning. A customer, Freya Jones, had closed their Larkspur account and asked for everything about them to be deleted. “Legal says a month. Can we?”

“One DELETE FROM customers,” Rohan said. “It’s a primary key.”

“It’s a lake,” Ingrid said. “The customer’s ID is in nine tables, and that’s only the tables: snapshots, raw files and a week of Kafka besides. And two Season 1 shortcuts can still read most of it. Find the copies, decide who may see them, delete, then prove it.”

The Quick Map

Concepts, with use cases

Personal data, classified where the table lives

Personal data is anything relating to an identifiable person. Three kinds matter for masking: direct identifiers (name, email, phone, address); pseudonymous identifiers (customer ID, loyalty number, anonymous_id), which name nobody alone but link everything a person left; and quasi-identifiers (postcode, user agent), identifying in combination. The EDPB’s Guidelines 01/2025 on pseudonymisation (consultation version): pseudonymised data that extra information can attribute “is therefore personal”.

Classification labels each column with its kind; a tag stores the label where tools read it. Larkspur keeps reviewed labels in Iceberg table properties (larkspur.pii.email = direct), so they travel with the table.

What breaks without it. Masking rules name columns by hand and miss the next table. Larkspur: a scan of all 88 tables found 27 personal columns in 11, including one no naming rule would catch (Step 1). Elsewhere: a health insurer must list every table carrying a diagnosis code before it can mask them for actuaries.

Who sees what: RBAC, ABAC, row filters and column masks

Role-based access control (RBAC) grants by role (analysts read gold); attribute-based access control (ABAC) decides from attributes of user, data and request (a UK analyst sees UK rows; columns tagged direct come back empty). A row filter is a predicate the engine adds to every query on a table; a column mask is an expression the engine returns instead of the value. Masks remove (NULL), pseudonymise (a keyed hash, identical in every table, so joins and distinct counts work) or generalise (a postcode’s first three characters).

Part 9’s Polaris grants decide which tables a principal may load; Polaris documents no row or column rules, so here Trino asks Open Policy Agent (OPA), a policy engine with rules written in Rego.

What breaks without it. Analysts get the raw table or nothing, so they get it raw. Larkspur: one customer mart, read as a privacy officer and as a UK analyst (Step 3). Elsewhere: a bank’s call-centre agents see the last four digits of a card, fraud investigators all sixteen.

Encryption, secrets and audit

Encryption at rest protects media and backups: Amazon S3 has encrypted every new object (SSE-S3) since 5 January 2023; SSE-KMS adds keys you control. In transit, Trino’s docs: “all authentication requires secure connections using TLS and HTTPS”. Secrets belong in a secret store, short-lived like Part 9’s vended credentials. An audit log records who did what to which data and the answer: OPA’s decision log keeps every question Trino asked; Trino’s event listeners (HTTP, Kafka, MySQL, OpenLineage) keep the SQL.

What breaks without them. A leaked .env is a breach, and “did anyone read the customer’s data after the request?” has no answer. Larkspur: the lab runs plain HTTP with lab secrets in .env.example, and says so; Step 5 reads the decision log. Elsewhere: a hospital audits every opening of a celebrity’s record.

Erasure in an immutable lake

The right to erasure (GDPR Article 17) means deleting a person’s data “without undue delay”, with exceptions such as “compliance with a legal obligation” (17(3)(b)): tax law makes Larkspur keep its sales. So erasure has two actions: delete records with no other reason to exist (a profile, clickstream), and tombstone the identifier in records the law requires (an order keeps its amount, loses its customer ID). An erasure register records what was done where, under a keyed hash; the EDPB’s 2025 erasure report asks controllers to “keep track of erasure requests and comply with them on restored systems”. Crypto-shredding (per-person keys, deleted on request) avoids rewriting history; whether it counts as erasure is for lawyers.

What breaks without it. The row leaves queries and stays on disk for years. Larkspur: Step 6. Elsewhere: a marketplace erases a seller’s personal details and keeps the seller’s invoices for the tax office.

The law in Larkspur’s three countries

For a proposed shortening of India’s 18 months (January 2026), we found no notification by 29 October 2026. Larkspur runs one process everywhere: erasure on request, the strictest deadline.

Retention and residency

Retention is how long each dataset may live, set per purpose: sales lines seven years, clickstream 25 months, CDC topics seven days, snapshots seven days in production. Any copy kept longer than the erasure deadline needs its own erasure step. Elsewhere: a mobile operator keeps call records for the period its regulator sets, then deletes them by partition. Residency is where data may be stored or processed. None of the three imposes general localisation on retail data: DPDP section 16 allows a blacklist of destinations (none notified, per secondary sources), and the RBI’s 2018 directive binds payment system providers such as Larkspur’s PSP.

Under the Hood

How Trino asks OPA

Each check is a JSON input naming a user, an operation and a resource, never the SQL text. The identifier mask, from opa/policy/trino.rego, is Rego v1; Trino’s documented samples use pre-1.0 syntax (future.keywords imports, is_admin { … } without if), which OPA 1.x rejects unless started with --v0-compatible:

batch_masks contains {"index": i, "viewExpression": view} if {
  analyst
  some i, r in input.action.filterResources
  c := r.column
  cls := classified(c).columns[c.columnName]
  view := mask(c, cls)
}

mask(c, cls) := {
  "expression": sprintf(
    "substr(to_hex(hmac_sha256(to_utf8(%s), (SELECT key FROM governed.customer.pseudonym_key))), 1, 16)",
    [quote(c.columnName)],
  ),
  "identity": "masker",
} if cls == "identifier"

The row filter and mask are SQL that Trino folds into the plan, before your WHERE: an analyst filtering on a real customer ID compares it with the pseudonym and gets nothing. The key lives in a one-row table Trino reads as masker, which analysts are refused; inlined, as first written, it showed in an analyst’s EXPLAIN (base64). Now the plan shows hmac_sha256(to_utf8(customer_id), key).

Where an erased row survives

An Iceberg DELETE or UPDATE from Trino writes a delete file and a new snapshot (Part 7). The person is still in, in order of how long they last:

1. The data file, hidden by the delete file: optimize rewrites it without the row.
2. Older snapshots listing the old file: expire_snapshots drops them, except those a tag or branch keeps.
3. The manifests of the current snapshot (lab observation, consistent with the spec). A manifest lists the files its snapshot removed (status DELETED) with their column bounds; string bounds keep 16 characters by default, so a 10-character ID survives whole as a file’s lowest or highest value. optimize_manifests rewrites them; expire again.
4. Unreferenced files (failed writes; expiry leftovers, as in Part 19): remove_orphan_files.
5. The object store’s disk. A deleted object is garbage until the store compacts; SeaweedFS vacuums a volume past 30% garbage (-master.garbageThreshold 0.3, every 840 s).

Hands-On

Companion: the every-record-counts folder of the companion-workspaces repository, tag every-record-counts/part-23. Core + Trino; OPA starts with Trino. The subject is a synthetic UK customer, CGB0000113.

make up-trino
P="uv run python -m scripts.part23_privacy"
$P setup; $P customers; $P discover; $P tag; $P policy-data       # Steps 1-2
$P erase CGB0000113 --request ER-2026-1029-001                    # Step 6 (each step's other commands are shown there)
uv run pytest -q tests/test_part23.py

Step 1: a CRM, and an inventory

customers adds the CRM: crm.customers in PostgreSQL (synthetic names, example.com emails, drama-range phones), copied to raw.customers and published as customer.dim_customer in a new namespace (157,566 customers; 157,565 after this erasure). discover checks every column name against rules, and a random 1,000-row sample of each text column against value patterns:

table                              column             by name     by value (sample of 1,000 rows)
customer.dim_customer              email              direct      email
gold.web_sessions                  session_id         -           anonymous id
part06.clicks                      anonymous_id       identifier  anonymous id
part07.clicks_v2                   anonymous_id       identifier  anonymous id
part07.clicks_v3                   anonymous_id       identifier  anonymous id
raw.pos_uploads                    loyalty_id         identifier  loyalty id
27 columns in 11 of 88 tables

(21 rows trimmed.) session_id is anonymous_id || '#' || n: a derived column carrying an identifier, invisible to name rules. And part06.clicks, part07.clicks_v2 and part07.clicks_v3 (format v3, which Trino 484 reads) are Season 1 teaching copies no governed namespace owns; the two Part 7 tables hold 15 September only, so they never held this part’s subject. No IP addresses and no card numbers (Part 22 guessed “card fragments”): the PSP files carry references only.

Step 2: tags on the tables, read back by the policy

A person reviews the scan into governance/classification.json; tag writes it into table properties (as catalog admin), and policy-data reads the properties back into OPA’s data file, which OPA reloads (--watch):

gold.web_sessions              personal  anonymous_id=identifier, session_id=identifier, user_agent=quasi
raw.orders                     personal  customer_id=identifier
gold.rpt_daily_net_revenue_v2  none
opa/larkspur/classification/data.json: 29 classified tables, 8 with personal columns
readable by analysts but unclassified (OPA refuses them): customer.erasure_register, customer.erasure_suppression, customer.pseudonym_key

(Lines trimmed; last line from the final state.) Every gold table is classified, most as none; analysts are refused unclassified tables, so new tables fail closed.

Step 3: one query, two users

Trino’s new governed catalog reads Polaris as its own principal, gateway (read-only on gold and customer; secret from the environment). OPA does the rest:

$ … as dpo "SELECT customer_id, full_name, email, postcode, city, orders FROM governed.customer.dim_customer
            WHERE country = 'GB' AND orders >= 2 ORDER BY customer_since, city LIMIT 3"
    customer_id  full_name       email                              postcode  city     orders
    CGB0022699   Isla Evans      [email protected]       EC11 8DP  Bristol  2
    CGB0200811   Oliver Johnson  [email protected]  NW5 8AP   Bristol  2
$ … as analyst_gb (same query)
    customer_id       full_name  email  postcode  city     orders
    2857FF1D0CE9CBF3  None       None   EC1       Bristol  2
    C8E842664CE3D363  None       None   NW5       Bristol  2

Conversion by country from gold.web_sessions: the privacy officer gets three rows, analyst_gb only GB’s, with the same numbers (929,032 sessions, 643,961 visitors counted on pseudonyms, 6.19%). Masks apply to analysts only: dpo, dbt_dev and platform users, including Part 9’s Trino user analyst, see clear values, and in the lab anyone can claim any name (Gotcha).

Step 4: who may do what, and Season 1’s shortcuts

Users live in opa/larkspur/users/data.json: platform (every user Parts 1–22 run as, allowed everything, so earlier parts behave as before), privacy (dpo, erasure: also allowed everything, drops included), developers (dbt_dev, now the dev target’s user) and analysts with countries:

analyst_gb  SELECT count(*) FROM lakehouse_pipeline.raw.customers              denied: Cannot access catalog lakehouse_pipeline
dbt_dev     CREATE TABLE lakehouse_dev.dev.p23_probe AS SELECT 1 AS x          ok [1]
dbt_dev     SELECT count(*) FROM lakehouse_pipeline.gold.fct_basket            ok [2506704]
dbt_dev     DROP TABLE lakehouse_pipeline.gold.fct_basket                      denied: Cannot drop table …
dbt         DELETE FROM lakehouse_pipeline.gold.fct_basket WHERE false         ok [0]
intern      SELECT 1                                                           denied: Cannot execute query

(Lines trimmed.) Part 18’s hole is closed: dev reads production for --defer (unmasked, raw.customers included) and writes only lakehouse_dev; Part 18’s slim CI test passes as dbt_dev. An unknown user can’t run a query at all. The remaining debts, settled:

Step 5: the doors around the mask, and the audit

Masks live in Trino. The same analyst through another door:

Trino as analyst_gb, gold.web_sessions       anonymous_id DFD6923C14F03C53, session_id EB3CAC42B5D8E83A
PyIceberg as analyst, gold.web_sessions      anonymous_id a-CGB0148213, session_id a-CGB0148213#1
PyIceberg as analyst, customer.dim_customer  ForbiddenError: … Principal 'analyst' … is not authorized

Part 9’s analyst principal reads gold’s files directly. The customer mart lives where no human principal can load it; gold.web_sessions should move there. A mask holds only if its engine is the only door.

audit summarises OPA’s decision log and lists every refusal:

DENY 2026-10-29T10:24:16Z dbt_dev DropTable lakehouse_pipeline.gold.fct_basket decision fcd35bca
DENY 2026-10-29T10:24:16Z intern ExecuteQuery - decision f11b5bbb

(Counts and eleven refusals trimmed.) The lab keeps decisions on OPA’s console; production ships them to a log store, with Trino’s event listener recording SQL.

Step 6: one erasure, end to end

find copies the CRM row into the case file (the proof needs the email after the row is gone) and counts rows per location:

subject CGB0000113: anonymous_id=a-CGB0000113, loyalty_id=LGB0000113, [email protected], …
table                       rows snapshots  refs   action
raw.customers                  1         1  -      delete
customer.dim_customer          1         3  -      delete
raw.orders                     1         1  -      tombstone
raw.orders_changes             2         1  -      tombstone
raw.pos_uploads                1         1  -      tombstone
raw.pos_events                 1         1  -      tombstone
raw.clickstream_events         9         1  -      delete
gold.web_sessions              1         1  -      delete
part06.clicks                  9         1  -      delete

prove decodes every object under those tables and the matching raw folders (Parquet columns, gzip, Avro manifests) and searches for the customer ID, loyalty ID and email (not the phone: drama-range numbers repeat):

prove before: 892 objects (754 MiB) decoded and searched for 3 identifiers
  raw.clickstream_events           metadata      1 objects
  raw.pos_events                   metadata      1 objects
  raw.pos_uploads                  metadata      1 objects
  objects still holding the subject: 18

(13 data rows trimmed.) Three manifests hold one of the IDs as a file’s lowest or highest value. Kafka’s own dump tool, run on the volume with no broker, finds 2 records name CGB0000113 in the CDC topic. To see what a tag does, we tag raw.orders as finance might at quarter close, then erase:

$ … pin raw.orders q3_close
$ time … erase CGB0000113 --request ER-2026-1029-001
  raw.orders               files and manifests rewritten, expired, orphans removed in  0.6 s; 2 snapshot(s) left; refs: q3_close
postgres crm.customers             delete         1
postgres shop.orders               tombstone      1
raw.clickstream_events             delete         9
real  0m18.664s

(Lines trimmed.) Sources first (CRM row deleted, order tombstoned), then each table (DELETE/UPDATE as user erasure), the register, Under the Hood’s sequence (scoped to the partitions that held the person), and raw files rewritten in place. The tag keeps its snapshot:

prove after erase: 935 objects (751 MiB) decoded and searched for 3 identifiers
  raw.orders                       data          1 objects
  objects still holding the subject: 1
$ … repoint raw.orders q3_close && … clean raw.orders
raw.orders: tag q3_close moved from snapshot 1251060227498834353 to 3878625504778204354
prove after the tag moved: 933 objects (746 MiB) decoded and searched for 3 identifiers
  objects still holding the subject: 0

Re-pointing helps once: the moved tag then pins whoever is erased next, so the lab dropped it (untag). Policy: no long-lived tags on personal tables. Part 19’s and Part 21’s tags sit on a report classified none and stay.

Reloads. Erasure also writes a suppression list (customer.erasure_suppression, a keyed hash per identifier). Every loader that lands personal data (the CRM load, Parts 6, 9 and 17–20, make upload) drops listed people or tombstones their IDs. After re-uploading the raw zone and re-running each loader:

$ make upload
suppression list: raw/clickstream/events (events) delete 9
$ … part20_contracts land
suppression list: 9 rows dropped, 0 identifiers tombstoned
$ … part17_dbt land pos_uploads
suppression list: 0 rows dropped, 1 identifiers tombstoned
$ … prove CGB0000113 --label 'after every loader re-landed its table'
prove after every loader re-landed its table: 987 objects (807 MiB) decoded and searched for 3 identifiers
  objects still holding the subject: 0

(Lines trimmed.) The tests erase a reserved synthetic subject (CZZ0000001, dated 2025) and remove what they seeded.

Not covered in the lab: the CDC topic’s 2 records (Kafka deletes whole segments past retention.ms, so segment.ms decides when; set both below the deadline, or crypto-shred); the tombstoning UPDATE’s before image with the old customer_id (REPLICA IDENTITY FULL) in PostgreSQL’s WAL (the lab’s slot is lost; a live one would stream it); Part 16’s replay, which regenerates the source; data/raw and DuckDB files on the laptop; the case file data/lab/part23/cases/CGB0000113.json, with email and phone (production deletes it when the case closes, keeping hashes); and the erasure job’s SQL, which names the customer ID in Trino’s query history and event-listener logs, so those need their own retention.

The disk. SeaweedFS compacts a volume by itself only past 30% garbage:

bucket         volumes GiB  deleted, on disk GiB  worst volume   after vacuum: volumes, deleted
larkspur-lake        23.89                  2.54          17%           21.35     0.00

(Raw row trimmed: it passed 30% and compacted itself.) No lake volume did, so 2.54 GiB of deleted objects would have stayed on disk.

Published numbers move and must reconcile: because of this erasure, Part 20’s 2,161,516 sessions and Part 6’s Trino copy become 2,161,515 at HEAD (134,223 converted, still 6.21%), and Part 20’s July contract count of events without page goes from 2,675,104 to 2,675,095; the register says why. Tombstoning keeps amounts: UK Q3 net revenue is still £29,790,674.06.

The Landscape

From vendor documentation, 29 October 2026.

Tips & Gotchas

Gotcha — record before you rewrite. Write the register and suppression list before maintenance: a failed rewrite must not lose the record.

Gotcha — a drop-and-create loses the tags. customers drops and recreates raw.customers, and its properties went with it; CREATE OR REPLACE TABLE on dim_customer kept them (Trino 484, observed). Run tag after every load, and let unclassified tables fail closed.

Gotcha — equality delete files contain what you deleted. Writers that delete by key, such as Flink’s Iceberg sink in upsert mode, write equality delete files, which store the deleted rows’ key values (Iceberg spec). Until compaction rewrites them away, the delete file holds the ID you erased.

Tip — keep IDs out of column statistics (not tested in the lab). Iceberg’s write.metadata.metrics.column.<name> (1.12 docs) sets one column’s metrics mode; none on personal columns keeps their values out of manifest bounds, at the cost of file pruning on those columns.

Gotcha — the lab authenticates nobody. OPA decides on the user name, and in the lab that is an HTTP header anyone can set to dbt. Production puts OAuth 2.0, JWT or passwords in front (all need HTTPS), and maps groups from the identity provider into the policy’s data.

Troubleshooting

Design Drill

The question: “A dating app with 30 million members gets 4,000 erasure requests a week. A member’s data is in 40 tables across PostgreSQL, Kafka and an Iceberg lake with two years of snapshots, in Parquet, Avro and JSON, plus a photo store. Design erasure that completes within a month and can be proved.”

Clarify first

• How is the requester’s identity verified, and against which account?
• Which records must be kept: safety reports about the member, payment records for tax, legal holds?
• Is any data special category (sexual orientation, health)?
• How long are backups, topics and snapshots kept today, and who owns each copy?

The shape of a strong answer

• Inventory first: tags on every table, value scans for IDs hidden in other columns, lineage for copies; unclassified tables fail closed.
• One idempotent erasure service per request ID, with a register (keyed hash, actions, counts) and a suppression list every loader and restore applies.
• Per location, delete or tombstone; payments and safety reports keep a tombstoned member ID, or sit under legal hold.
• Batch the weekly requests: one row-level delete per table per week, then rewrite, expire, rewrite manifests and clean orphans, so 4,000 requests cost one rewrite, not 4,000.
• Every retention layer shorter than the deadline (snapshots, topics, object versions, soft delete); backups re-applied on restore.
• Copies outside the warehouse: search indexes, ML feature stores and trained embeddings, CDN caches of photos; and tell recipients and processors (GDPR Article 19) with a tracked acknowledgement.
• Crypto-shredding for long-lived immutable stores (Kafka history, archives) where legal agrees; per-member keys in a KMS.
• Prove it: byte-level scans on sampled requests; register counts that reconcile metrics.

Trade-offs to say out loud

• Weekly batches use most of the month; daily ones cost seven rewrites.
• Crypto-shredding avoids rewrites but adds a key lookup to every read and needs legal sign-off.
• Tombstoning keeps metrics stable; deleting is cleaner but restates history.

Common weak answers

• DELETE FROM the main tables and stop: snapshots, manifests, topics and backups still hold the member.
• No register: the next restore brings the member back.

Further Reading

1. Trino 484, Open Policy Agent access control. https://trino.io/docs/current/security/opa-access-control.html
2. Apache Iceberg, spec (manifests, row-level deletes) and maintenance. https://iceberg.apache.org/spec/ and https://iceberg.apache.org/docs/latest/maintenance/
3. EDPB, Coordinated Enforcement Framework 2025 report on the right to erasure (February 2026). https://www.edpb.europa.eu/system/files/2026-02/edpb_cef-report_2025_right-to-erasure_en.pdf
4. ICO, right to erasure. https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-resources/individual-rights/individual-rights/right-to-erasure/
5. MeitY, Digital Personal Data Protection Rules, 2025 (G.S.R. 846(E)). https://www.meity.gov.in/static/uploads/2025/11/53450e6e5dc0bfa85ebd78686cadad39.pdf

Story Closing

Farah sent the reply on Friday: request ER-2026-1029-001 completed, the register’s line per location, and what Larkspur keeps (an order’s amount, without its customer) and why.

“And the topic?” Ingrid asked.

“Two Kafka records, until the segment ages out,” Rohan said. “The reply says so, and when we check again.”

Of the five promises from Part 2, this part strengthens completeness, read backwards: knowing every copy of a record, and accounting for every record that leaves. Part 24 follows the bill: what each scan, file and copy costs, and three optimisations measured before and after.

This is Part 23 of “Every Record Counts: Data Engineering for Backend Engineers” — Season 2: Pipelines in Production.