June 2025.
I had just joined Favorited, brought in to think about something most companies still weren’t taking seriously: what happens when your engineers start talking to machines that learn from everything you give them.
A few months earlier, I’d been at a cybersecurity conference in New York. Not the marketing kind—the kind where the people in the room are the ones who break systems for a living.
I asked them about AI. They didn’t hesitate.
“These models will get hacked,” one of them told me. “And when they do, the data will be worth more than the model.”
I went back and did what I knew how to do. I wrote the policy.
No confidential data in AI tools. No source code. Nothing proprietary. And if you absolutely had to use an LLM, make sure the “do not train on my data” box was checked.
It felt responsible. Structured. Controlled.
Until I sat with a simpler question: How would I know if that box actually worked?
I couldn’t inspect the system. I couldn’t audit the training pipeline.
I couldn’t verify the claim.
I grew up in the restaurant industry. My family opened the first Korean restaurant in NYC back in 1984, and I spent my early twenties putting myself through NYU waiting tables at Blue Ribbon Sushi in SoHo before becoming the opening general manager for several of Chef Sam Demarco’s restaurants. In that world, following food safety rules is paramount. Health inspectors exist precisely because a restaurant’s promise of hygiene is insufficient. Without inspection, hygiene claims are marketing, not safety.
Writing that AI policy felt like approving a restaurant without ever seeing the kitchen.
Imagine a security company telling you: “We record everything—but don’t worry, we delete your surveillance footage when you ask.” And then they refuse to let you verify the deletion.
I can’t audit the claim.
That is the unresolved, uncomfortable truth of enterprise AI today. When an LLM vendor says “we won’t train on your data,” there is currently no independent mechanism to verify that statement. We are buying a promise, not a guarantee.
I. What “Trust” Actually Means in Enterprise Software
I spent years testing SOX IT controls for Fortune 500 companies and global banks. The hard truth: mature organizations cannot legally or operationally leverage technology without independent bodies verifying compliance.
After the Enron scandal, the financial world stopped accepting “trust us” as an accounting standard. We demanded audited statements and criminal liability.
Enterprise software today operates on that post-Enron model. Trust is structural. If you use a cloud provider, you can verify access logs and audit encryption standards. But AI today is pre-Enron finance. It operates on disclosure, not verification. SOC 2 covers security controls, not model training behavior. We can audit who accessed a file—but we cannot audit whether a neural network learned from it.
The gap between what is contractually promised and what is technically verifiable is wide, and largely unexamined.
II. Why This Is Harder Than It Looks
Not all vendors are lying. But the current architecture makes lying possible without consequence.
Training pipelines are opaque. “Opt-out” mechanisms are entirely self-reported. It is like reading a nutrition label, knowing there is no FDA randomly testing what is actually inside the box.
We know that even sophisticated companies struggle to maintain these boundaries. In 2023, Zoom faced massive public backlash when ambiguous terms of service updates suggested they were training on user data, forcing a rapid reversal. Samsung engineers famously pasted proprietary code into ChatGPT, exposing it to the model’s context window and forcing the company to ban the tool internally. GitHub Copilot has faced lawsuits alleging it trained on, and reproduced, copyrighted code. Even if unintentional, training data boundaries are clearly not fully controllable or visible.
We have seen this dynamic before. When the pressure to win is high and enforcement is inconsistent, doping doesn’t just occur — it spreads. Not because every athlete is unethical, but because the system rewards those who take the risk. Over time, something more subtle happens: the boundary of what’s considered acceptable quietly shifts. The concern in AI isn’t a single bad actor making a deliberate choice. It’s a system where cutting corners becomes rational — and then normal — before anyone names it as a problem.
And larger institutions are not immune to these pressures. Look at Boeing and the 737 MAX. No one set out to cut corners. But incentives and competition compressed safety margins, and the era of “self-certification” removed independent verification. The system made failure easy.
When trust is unverifiable, the issue isn’t bad actors. It’s a system where good actors are under pressure.
In that world, the problem isn’t that trust is broken. It’s that we stop noticing it was ever required.
III. The Audit Gap — And What Would Actually Close It
We’ve seen how this ends. When systems cannot be independently verified, trust collapses—even without proof of wrongdoing. This is why the US restricted the usage of Chinese hardware like Huawei and DJI; the lack of verifiability regarding data exfiltration made the risk unacceptable, regardless of stated intent.
We solved this before in finance. We mapped financial statements to external auditors using standardized rules (GAAP) and enforced liability for fraud.
In my previous essays, I argued for an “Independent Table”—an oversight body modeled on the International Atomic Energy Agency (IAEA) to govern frontier AI safety. Nations don’t self-report uranium enrichment; inspectors physically verify it. We don’t accept ‘trust us’ when the stakes are existential.
We need that exact same structural independence at the enterprise product layer. Real auditability requires independent technical auditors with model access and standardized, cryptographically secure training data attestations. It won’t be perfect, but it will be structurally independent.
Categories of Trust Risk in AI
Ambiguity — Vague terms of service that confuse users
Scope Drift — Promises that evolve over time
Unintended Leakage — Bugs, logs, or edge cases exposing data
Opaque Training — No independent verifiability of ingestion
Incentive-Driven Corners — Pressure-driven compromises
IV. What Enterprises Should Actually Do Right Now
Until then, AI governance is operating without its audit layer. Enterprise AI deployment is, at some level, an act of faith.
Enterprises cannot wait for regulation. We must adopt a posture of Zero Trust Architecture applied to AI data governance: Assume breach. Verify continuously. Treat AI vendors the way modern security treats networks: trust nothing, verify what you can, and assume the rest is risk.
Practically, this means:
- Tiered Data Classification: Air-gap your most critical intellectual property from all public LLMs, regardless of opt-out promises.
- Contractual Teeth: Demand strict, specific data handling requirements and vendor liability for unauthorized training ingestion in enterprise contracts.
- Probing Questionnaires: Do not accept standard SOC 2 reports as proof of data privacy. Ask specific questions about their training data pipelines and internal access controls.
Enterprise AI adoption is accelerating.
But until we close the audit gap, we are building the future of enterprise technology on an unverifiable promise.
Faith is not a risk management strategy.
