I Built a Data-Cleaning Agent, Then Watched It Refuse to Clean Real Payroll
Cleanroom runs on TrueForge, TrueFoundry’s open-source agent harness, which is what makes the “stop and ask a human” part enforceable rather than aspirational. It’s live at cleanroom-production.up.railway.app.
I pointed it at 6,000 rows of New York City payroll. Real salaries, real people, public record. I told it nothing about what was wrong. I wanted to see what it would do with money.
It found 762 rows where the pay doesn’t add up: gross pay that doesn’t equal base rate times hours worked. On the demo data I’d been building against, that exact mismatch is a bug, and the agent offers to fix it: recompute the total from quantity times unit price, which is the right call there, because a stored total that disagrees with its own line items is wrong by definition.
Here it stopped and told me it wouldn’t.
Per-hour regular_gross ≠ base_salary × hours by >$1, 762 of 864, often includes poll workers with a placeholder $1 rate and zero hours. Recomputation would corrupt pay.
Derived cash: regular + OT + other ranges from -$37,871.76 to $403,109.75. There is no stored total column against which to assert equality.
That’s the whole thing, right there. On my sales data there’s a total column that is supposed to equal quantity times unit price, so when it doesn’t, that’s an error and fixing it is correct. In payroll there is no such column. base_salary is a rate, not an expected total. Gross pay legitimately differs from rate times hours because people join mid-year, get raises in April, take unpaid leave, work partial periods. Nothing is broken.
An agent that “reconciles” that column silently rewrites thousands of people’s pay records. That’s the failure I set out to make impossible.
The thing I actually set out to build
Every team has one spreadsheet nobody wants to touch.
Three people have edited it over two years. Dates are written three different ways because one person is American, one is British, and one used whatever Excel defaulted to. There are duplicate rows from someone importing the same CSV twice. Money is stored as text, "$1,234.50", with the quotes and the comma and the dollar sign. A couple of the totals don’t match the line items and nobody knows which number is right.
Cleaning it by hand takes an afternoon and you’re never certain you caught everything. Writing a script means guessing at the ambiguities, and the worst part is the script won’t tell you it guessed. Is 03/04 March 4th or April 3rd? Your script picks one. Silently. And now half your dates are wrong in a way you won’t notice until someone asks why Q1 revenue moved.
So I built Cleanroom: an agent that does the tedious part and stops at the parts a machine shouldn’t decide alone.
You hand it a messy file. It profiles the data by writing actual pandas code and running it in an isolated container, so when it says “42 rows, 2 exact duplicates, 13 mixed date formats, 2 totals that don’t reconcile,” those numbers came out of code that executed, not out of a language model’s impression of a file. It works on a copy. Your original is never touched.
Then it stops. It shows you what it found, asks about anything genuinely ambiguous, and lays out a plan where every step is labelled safe or destructive. Nothing is applied until you pick.
After you approve, it applies the changes, verifies them with assertions (42 rows in, 2 dropped, 40 out, every row accounted for), and delivers the result as a pull request you review and merge.
The acceptance is your act, not the agent’s. That distinction is the product.
Why not just paste it into ChatGPT
I get asked this every time, and it’s a fair question. The honest answer is two things, and both matter.
How it’s built
It runs on TrueForge, TrueFoundry’s open-source agent harness. The run moves through nine phases, and it structurally cannot skip the one in the middle.
Six of the harness’s capabilities ended up as load-bearing parts, not things I could say I’d used:
- The sandbox is the compute boundary. The agent never processes your data in its own context. It writes a Python program, runs it in a Daytona container, and reads back the output. That’s what makes the numbers trustworthy, and it’s also why your original is safe.
- The approval gate is enforced by the harness. Applying fixes and writing output are structurally gated. It cannot proceed past the gate without an explicit human yes.
- Delivery goes through MCP with writes gated. The agent can prepare a pull request. It cannot merge one. That’s a policy boundary in config, not a rule I asked it to follow.
- Category analysis runs as a subagent. Deciding whether
NYC,n-y-c, andNew Yorkare the same place happens on a separate thread. It returns a recommendation with reasoning, and the main agent’s context stays on the repair plan. - The methodology lives in the repo as a git-backed skill. When a run succeeds, the agent writes down the method it used and submits it as a pull request. Its memory is a diff a human reviewed and merged. You can read what it learned. You can revert it if it learned something wrong.
- Recipe reuse is guarded. Next time you hand it a file with the same structure, it recognises the shape and asks one question instead of five. If the structure has changed, it refuses to reuse the old recipe and starts over, because a saved method applied to a different file is precisely how silent corruption happens.
The two datasets that mean opposite things
This is the heart of why the refusal matters. The same column name, the same arithmetic mismatch, means “fix me” in one file and “do not touch me” in another.
| Sales export (demo) | NYC payroll (real) | |
|---|---|---|
| The column | total | base_salary |
| What it means | An expected total: qty × unit price | A rate, not a total |
| When it mismatches | A data error, the total is wrong | Normal: mid-year hires, raises, unpaid leave, partial periods |
| Right action | Recompute the total. Fixing it is correct. | Leave it alone. Recomputing would corrupt pay. |
| Is there a stored total to check against? | Yes | No, there is nothing to assert equality against |
What I tested it on
I didn’t want to only test on data I’d written myself, because data you write yourself is data you already know the answers to. I ran it against 21,000+ rows across five corpora. Two were real public datasets it had never seen.
NYC 311 service requests, 5,000 rows, 44 columns. Eight of eight checks exact against a reference I’d measured separately. It found 32 tickets closed before they were created.
But here’s the part that was uncomfortable and good: three of its numbers corrected my reference script.
| Check | I said | The agent said |
|---|---|---|
| Columns ≥95% missing | 9 | 10 (my check had an off-by-one on an all-null column) |
police_precinct as Precinct <n> | all 5,000 | 4,913 (87 rows weren't) |
location_type case variants | 3,505 | 41 (I counted whole groups, not rows that varied) |
NYC payroll, 6,000 rows. Eighteen of eighteen exact. It found 1,068 rows paid overtime for zero overtime hours, 3,014 rows with regular pay above zero and zero regular hours, and 228 rows sharing an employee-year-agency key. And then it refused to fix the pay math, which is where this post started. It also refused on the overtime rows (“no defensible hours imputation”) and on 223 rows with negative pay, on the grounds that clawbacks and adjustments are real payroll, not corruption.
Five clarifying questions, and every one of its recommendations was to preserve and flag rather than change.
What code review caught that I couldn’t see
I used Qodo on every one of the 23 pull requests. Working alone against a deadline, the failure mode isn’t sloppy code, it’s that you convince yourself something works because you need it to. Qodo kept catching exactly that.
The finding that changed the product rather than the code was about scheduled runs. Qodo pointed out that the approval gate blocks a scheduled run from ever completing. Technically correct. The obvious fix is to auto-approve when nobody’s watching.
I didn’t take it. Deleting the gate would have deleted the product. Scheduled runs now profile the file, apply the known recipe to a sandbox copy, verify it, and stop at the gate anyway, reporting what’s ready for a person to approve. Unattended is not permission to guess. Having the question asked is what forced me to decide deliberately instead of drifting into the convenient answer at 2am.
What I actually learned
I went in thinking the approval gate was a safety feature. Something you add because agents are risky and users need a seatbelt. That’s not what it is.
The gate is what makes the agent’s learning trustworthy. Cleanroom improves by writing down what worked and reusing it, and the only reason I’m willing to let it accumulate methods over time is that every method arrives as a pull request with a human on the other end. Its memory is reviewable. Its memory is revertible. If it learns the wrong lesson from a lucky run, I can see the diff and say no. Take the gate away and you don’t just get a less safe agent. You get an agent whose accumulated knowledge nobody has ever checked, quietly applying yesterday’s assumptions to today’s file.
The second thing I learned is that the most valuable behaviour isn’t the cleaning. It’s the refusal. Any competent script can normalise dates. What took real work was building something that could look at 762 rows of mismatched pay and understand that the mismatch is not an error, that in this dataset, unlike the one it was trained on, base_salary is a rate and not a total, and the correct action is to leave it alone and say why.
Drop in messy data, get back data you trust, and an agent that knows when not to touch it.
Try it
- Live: cleanroom-production.up.railway.app, no signup, open it and give it a file.
- Code: github.com/sreenathmmenon/cleanroom
- Built for the Agent Harness Hackathon (WeMakeDevs × TrueFoundry), on TrueForge, with Daytona for sandboxed execution and Qodo for code review.