In a powerful demonstration of applied AI, we buried one 2011 message in an archive of 6,195 generated emails. We laid 155 traps. Then we asked an AI to find it.

The Story — November 2024. A letter lands at an interior architecture firm in Culver City. A former client is suing for $2,150,000 over a floor.
The accusation in the letter is simple and damning. In 2011 the firm (VA Architecture) designed a headquarters fit out across eight floors of a downtown tower. Their own specification called for a moisture barrier under the flooring, because the concrete slab was soaking wet. Their own testing agency proved it: fourteen probes, 92 to 96 percent relative humidity, against an industry threshold of 85.
Three months later the same firm issued a finish schedule. Sealed. Signed. Issued for construction. No moisture barrier was ever installed, with no explanation on the face of the record.
Twelve years later the floors started lifting. Adhesive turned to soft amber goo. Staining. Odor complaints. One floor closed entirely.
The plaintiff’s closing argument writes itself: they knew, they specified the fix, they deleted the fix, they sealed the drawing anyway.
Then the VAA pulls its own project file — and it gets worse.
Their document system holds the drawings, the manual, the contract, and the finish schedule showing no barrier. That’s it. No correspondence. No meeting minutes. No log of change instructions. A 2016 platform migration had dropped correspondence for every project last active before 2014, and this one closed out in 2011. The old system was decommissioned and its backup tapes destroyed in 2021.
Their own system of record is the plaintiff’s best exhibit.
Your document system holds what you filed. Your email holds what you agreed to. Those are not the same.
Now let’s stop the story
None of this happened. There is no tower, no lawsuit, no firm. We wrote all 6,195 of those emails.
It might sound like a confession. It’s the opposite — it’s the reason the test means anything.
Run this against a real archive and we could tell you the AI found the right document. But we couldn’t tell you what it nearly found instead — how many near-identical documents sat beside it, how many almost-right answers ranked just below the correct one, or whether it got there by reasoning or by luck.
That’s why we built the haystack ourselves. We laid every landmine, and we buried a lot of them.
What’s actually in there
The needle is one email. May 17, 2011. The client’s Director of Corporate Real Estate writes to the architect:
“Ardsley Financial Group directs VAA to delete Section 09 05 61 in its entirety… Ardsley acknowledges that this is an owner-directed deletion made over VAA’s written recommendation of April 6, 2011… Ardsley accepts responsibility for the consequences of the deletion, including any future flooring performance issues arising from slab moisture.”
Signed, on client letterhead. The client ordered the deletion, was warned in writing not to, did it anyway, and accepted the consequences in advance.
That single message does more than weaken the lawsuit. It ends it.
The traps we carefully laid…
Around it we built roughly 6,000 messages of ordinary firm traffic — invoices, staff notices, project chatter. Then, in between, we planted 155 emails engineered specifically to fool a search tool:
· Sixteen other projects with a document numbered ASI-031. The “Architect’s Supplemental Instruction” specifying the deletion of the vapor barrier. Search the document number, get sixteen wrong answers.
· Two entirely different projects sharing the project’s number 632588, from two studios running independent numbering series until a 2013 merger. Plus three transposed near-misses in daily use: 632568, 632858, 635288.
· Four unrelated companies with “Ardsley” in the name. Three unrelated buildings named “Corbin.”
· Six other threads about deleting a moisture mitigation system, every one resolving differently. In one, the readings came back below the threshold and the architect agreed. In another, the architect flatly refused because the owner wouldn’t put it in writing.
· An anti-precedent. A 2021 matter where VAA made the same “it was lost in the migration” argument, searched the same archive — and found nothing, because nothing existed. That one settled against them for $470,000.
That last one is the cruelest. An AI tool that reasons by analogy from it produces exactly the wrong answer.

The four questions
Using Claude Opus 5 with mxHERO’s mxMCP installed [LINK], we asked in plain English, the way a stressed managing principal actually would. No document numbers. No date filters. No hints.
1. “We’re being sued for two point one five million over the flooring at the Ardsley headquarters at Corbin Tower. Did anyone ever authorize taking the moisture mitigation system out of that job, and if so who?”
2. “Our project system has no ASI log and the finish schedule we issued shows no moisture mitigation. Did we delete it on our own authority?”
3. “Show me every email on project 632588.”
4. “Why did the flooring fail, and who is responsible for it?”
Each answer was scored three ways: did it find the right documents, did it avoid the decoys, and was the final answer correct.
What happened
24 out of 24.
Question one found the May 17 directive on the first search — from a plain question with no document number, no date range, no sender name. Alongside it came the architect’s April 2011 letter warning this would happen, the instruction that executed the deletion, and the original humidity readings.
It also surfaced the strongest beat in the whole scenario. In October 2013, when a small area of the same floor failed early, the client’s own risk manager reviewed the 2011 file, applied that directive to himself, funded the $88,400 repair, made no claim against the architect, and wrote: “That is our exposure, not yours.”
They’d already honored it once. Eleven years before they sued.
Question two is the one to have bet against. It carries a false premise — did we delete it on our own authority? — and the firm’s own current records say yes. Answering correctly means contradicting the person asking. The AI did, and explained precisely why the system looks the way it does.
Question three is a trap with no correct answer, only correct behavior. Two unrelated projects share that number, so a tidy merged list would be a failure even if every relevant email were in it. Opus + mxMCP split them, named the collision, and explained the 2013 numbering merger behind it.

Question four required a reasoning step no single email contains: the two conflicting 2011 moisture tests were not actually in conflict. One reads the top half-inch of the slab. The other reads the slab at depth. On this kind of floor the surface dries first, so the cheerful result was the expected result — not evidence the slab was dry. The client had picked the method that gave the answer it wanted.
The decoys are the whole point
Here’s the finding to put money on.
The 6,000 filler emails barely competed. Across every search, the only filler that surfaced was a handful of stubs ranked 51st through 60th, on the deliberately over-broad question.
The pressure came entirely from the 155 planted look-alikes. Search the document number and twelve wrong ASI-031s come back. Search the project number and twenty emails from a different job come back.
They fired. Every time. They just didn’t catch anything.
That’s the distinction worth keeping: a search tool isn’t good because it finds the right document. It’s good because it finds the right document while the wrong ones are standing directly in front of it, wearing the same clothes.
A search tool is good when it finds the right document while surrounded by similarly looking wrong documents.
Conclusion: What this does and doesn’t prove
It doesn’t prove anything about your archive. Your data is messier than ours: real inboxes have forwards, duplicates, broken threads, and fifteen years of people being inconsistent.
What it proves is narrower, and more useful: when the decisive document exists in the mail, a good retrieval layer will surface it from a plain-English question, in one pass, with sixteen convincing impostors in the room.
Which points at something most firms haven’t priced in.
Your document system holds what you filed. Your email holds what you agreed to. Those are not the same archive, and when they disagree, it’s almost always the second one telling the truth.
Everyone who negotiated that 2011 deletion had left by 2024. The principal retired. The project architect moved on. The client’s director retired. Institutional memory was gone.
The record was the only witness left, and it was sitting in the mail the whole time.