The medical study found that same-type surrogate substitution keeps PHI detectable: pooled masked-span recall moved from 75.8% to 74.6% (paper v2; 76.1% → 74.9% in v1), a change that lies within ±2 points.
Here the same protocol is run on three non-medical domains: Finance (Gretel synthetic financial documents), Legal (ECHR court judgments, Text Anonymization Benchmark) and Enterprise (Gretel business documents from 44 industries).
Each domain has 250 English documents, scored by 10 detectors (1 rule-based, 1 fine-tuned, 8 LLMs) under four conditions: the original text, the Custodian Guardian Layer transform, open-source Faker surrogates and redaction (*****).
Recall is measured only on the spans that were masked: can a detector still find the PII-shaped value where the real one used to be? A change is non-inferior if recall drops by less than 2 points (one-sided test) and equivalent if it stays within ±2 points (TOST).
As reported in the paper (arXiv v2)
Paper v2 re-projects gold spans onto word boundaries and uses a document-level cluster bootstrap (90% CI) and the posterior probability that the change lies within ±2 points. The sections below this one use the original per-span analysis (TOST), so their Custodian numbers differ slightly (72.5% → 72.9%).
Masked-span recall by domain (pooled over 10 detectors)
Faker and redaction share one masked set (all gold spans). The Custodian set is the gold spans the transform actually changed, with its own original recall, so compare verdicts across arms rather than raw recall.
By detector family (all three domains)
Per detector × domain
Detection on the original documents (span F1: exact type match / any overlap)
Several LLMs often return no parsable spans on Finance and Legal with the paper's clinical prompt (e.g. Llama-3.1-8B on 204 of 250 Finance documents), so their absolute F1 is partly a prompt-following failure. The paired design above is not affected by this.
Surrogate quality of the Custodian transform
Detection equivalence only says that PII-shaped text is still where the PII used to be. The transform also edits much text that is not gold PII: about 78% of changed characters in Finance, 53% in Legal and 41% in Enterprise fall outside gold spans. These are upper bounds, because the gold annotation is not exhaustive.
- Meaning changes:
Human Rights→Animal Rights,judge→lawyer,Lender→Borrower,USD→GBP. - Spelling errors introduced:
Article→Artcile,Government→Goverment. - Invalid values: dates such as
2/31/2024, and new x-masking (01/03/2023→2xxxxxxx). - Values left unchanged: 12–31% of gold spans still appear verbatim in the output.
Detection equivalence therefore does not mean that the output is a faithful document.
Datasets
| Domain | Source | Sample | Gold spans | Licence |
|---|---|---|---|---|
| Finance | gretelai/synthetic_pii_finance_multilingual (English test split; SWIFT, ISDA, invoices, loan agreements, …) | 250 docs, seeded | 1,283 | Apache-2.0 |
| Legal | Text Anonymization Benchmark (TAB), ECHR judgments, DIRECT + QUASI identifiers | 250 paragraph windows (≤1,500 chars) | 2,307 | MIT |
| Enterprise | gretelai/gretel-pii-masking-en-v1 (test split, healthcare domains removed; 44 industries in the sample) | 250 docs, seeded | 1,056 | Apache-2.0 |
Data subsets (original, Faker and redacted), scoring code and the full write-up: docs/multidomain_results.md in the public repository. Numbers on this page are read from data/domains/summary.json.