Docs
Product

PII redaction

Foam scrubs sensitive data in two places: directly in your code during installation, and in the collector with NER.

In-code scrubbing ships with every installation. Collector-side NER redaction is optional and off by default. Contact perla@foam.ai to enable it for your telemetry.

1 · In your code, at installation (always on)

During installation Foam reads each service and detects the call sites that emit sensitive values: an email in a log line, a patient name in a span attribute, an SSN in console output. The install PRs rewrite those call sites so the value is scrubbed at the source, and the raw value never leaves your app.

services/checkout/handlers.ts
log.info(`checkout by ${user.email}`);
+log.info(`checkout by ${scrub.email(user.email)}`);
services/patients/record.ts
span.setAttribute('patient.name', patient.name);
+span.setAttribute('patient.name', scrub.phi(patient.name));
workers/claims/submit.ts
console.log('claim ssn', claim.ssn);
+console.log('claim ssn', scrub.ssn(claim.ssn));
Merged automatically during installation.

2 · In the collector, with NER (optional)

In-code scrubbing only catches what is visible statically. As a second net, the Foam collector can run every incoming log and span through a NER analyzer and rewrite detected entities to typed markers before anything is stored.

10:42:07 POST /v1/checkout 200 · 96ms user=sarah.chen@acme.com<EMAIL_ADDRESS>
10:42:08 patient.record updated name="James Miller"<PERSON> mrn=84-2210-77<MRN>
10:42:09 support.ticket created phone=(415) 555-0163<PHONE_NUMBER>
10:42:11 auth.login retried ip=73.92.14.208<IP_ADDRESS>
10:42:12 claim.submitted ssn=536-90-4143<US_SSN>
10:42:14 payment.method added card=4242 4242 4242 4242<CREDIT_CARD>
7 entities rewritten before anything reaches Foam DB.

NER data flow

Your appemits traces + logs (OTLP)FOAM COLLECTORNER call: POST /analyzePresidio analyzer 2.2.362spaCy en_core_web_lg NER + pattern recognizersPII found?nostored unchangedyes“call Maria at 555-867-5309”↓“call <PERSON> at <PHONE_NUMBER>”Foam DBfoam_otel_dbFoam featuresfoam.ai/chat · foam.ai/issues · foam.ai/monitors

Redaction is enabled per environment and per service, and fail-closed: if a value can’t be analyzed, it is masked as [REDACTED] rather than stored raw. Nothing is ever dropped.

How NER detection works

The collector runs Microsoft Presidio as a sidecar. Two mechanisms work together: deterministic pattern recognizers with checksums (emails, phone numbers, credit cards, SSNs, IBANs, IPs) and statistical NER (spaCy en_core_web_lg) for names. Domain identifiers like medical record numbers are covered by configurable patterns that only trigger near context words: MRN 4829102 is redacted, order 4829102 is not. Redacted items carry a foam.presidio.redactions marker for auditing.

Benchmarks

Measured on our own telemetry (500 adjudicated documents) and a 14-language test round:

  • Latency: 12 / 81 / 92 ms (p50 / p95 / p99) per document.
  • Pattern entities: IPs 100%, phones 97.4% recall; emails, cards, and IBANs redact across all 14 tested languages, including right-to-left scripts, with zero text corruption.
  • Names: 33.5% recall on production-domain text, since the NER model is English-trained. Its published reference is ~0.85 F1 on OntoNotes 5 English news (see spaCy’s published benchmarks); the gap is domain and language shift, which is why we benchmark on our own data using Presidio’s evaluation framework (presidio-research).