Six errors planted in a fake month of books: custom agents caught five unprompted, missed one sign-off rule, and refused to guess
Speedydooo(Reddit 用户 u/Speedydooo,原帖作者本人,自述对自己的 Agent 配置做了一次破坏性自检)Reddit · 2026-07-16
Read originalSaved text / editorial summary
原帖正文(Reddit r/AIToolsForSMB,作者 u/Speedydooo;本轮经 Reddit 公开 Atom feed 取得,以下英文部分逐字未改写): “Ran an experiment on my own AI agent setup last month that I wasn't planning to post about, but the result stuck with me. I built a fake month of bookkeeping for a business that doesn't exist. Real-looking invoices, vendor payments, a churn curve, the whole thing. Then I quietly planted six specific errors into the data before handing it to the agents: a duplicate invoice, a vendor rate that had drifted without anyone updating it, a threshold that needed sign-off and didn't get one, a customer segment with margin that looked fine on the surface and was actually losing money, a chunk of revenue leakage that only showed up in the churn month, and a vendor payment that was never approved. I wrote an answer key first so I'd know if the agents were actually catching things or just sounding confident. They caught five of six unprompted. The one they missed wasn't a math error, it was a sign-off rule the grading logic itself didn't account for, and two of the agents flagged that gap on their own before I found it. The part that actually built trust wasn't the catches. It was what they refused to do. One agent wouldn't tell me if the business was solvent because it didn't have a number it needed. Another wouldn't estimate customer acquisition cost without real spend data instead of guessing at something reasonable-sounding. A third wouldn't compute a ratio itself when the input was ambiguous, it kicked it back and asked which definition I meant. I also found two actual errors in my own answer key partway through. One was a typo in my math. The other was a rule I'd written that didn't match how the business actually worked. Neither was caught by me, both got flagged because the agents' output didn't match my key and that mismatch forced me to go check who was actually wrong. Whole exercise cost about five and a half dollars and took under half an hour. Everyone talks about agents that get things right. Nobody talks much about agents that know when to say I don't have enough to answer that.” 中文对照:作者先给一家不存在的公司造了一个月假账,写明发票、供应商付款和流失曲线,往里埋六个错;答案键在交给 Agent 之前写好。Agent 无提示抓到五个,漏掉的一条是评分逻辑本身没覆盖的审批规则,两个 Agent 主动指出了这个缺口。他最看重的不是抓错,而是三种拒答:缺关键数字不判断偿付能力、缺真实投放数据不估算获客成本、口径含糊时反问该用哪个定义。因为输出与答案键对不上,他反过来查出自己答案键里的两处错。整次试验他自述约 5.5 美元、不到半小时。
Business model
Reddit user u/Speedydooo wrote in r/AIToolsForSMB about a destructive self-test of his own agent setup. He built a realistic-looking month of bookkeeping for a business that does not exist - invoices, vendor payments, a churn curve - quietly planted six specific errors, and wrote the answer key before handing the data to the agents: a duplicate invoice, a vendor rate that had drifted without being updated, a threshold that needed sign-off and did not get one, a customer segment whose margin looked fine but was actually losing money, revenue leakage visible only in the churn month, and a vendor payment that was never approved. He gave the agents no hints. They caught five of six unprompted. The miss was not arithmetic: it was a sign-off rule his own grading logic did not account for, and two agents flagged that gap on their own. He says the part that built trust was not the catches but three refusals - one would not say whether the business was solvent without a number it needed, one would not estimate customer acquisition cost without real spend data instead of guessing at something reasonable-sounding, and one would not compute a ratio from an ambiguous input, asking which definition he meant instead. Because agent output did not match his answer key, he went back and found two real errors in the key itself: a typo in his math and a rule that did not match how the business actually worked. He says the whole exercise cost about five and a half dollars and took under half an hour - his cost, not a result. The data is fabricated, there are no real clients, no income, no verified savings, no subscription or token breakdown, and no performance claim.
The reusable idea is that refusal is an output, not a failure. Most self-tests only count hits, so a model quietly fills uncertain gaps with a plausible number. This author counted 'I do not have enough to answer that' as a finding - and because the output disagreed with his key, he found two mistakes in the key itself. Write the key first, plant known errors, then allow the key to be overturned. The workflow, prompt and synthetic sample here are AgentFlow proposals and were not run.
Workflow
- Write the answer key before handing the data over, then plant the known errors. (AgentFlow's working order; the post does not state whether the key came before or after the errors.)
- Hand over the material with no hints about what to look for. (The post says the five catches were unprompted.)
- Score each expected finding as caught, missed, or refused - with evidence row numbers. (AgentFlow's formalisation.)
- Record refusals and clarifying questions as their own output, stating what is missing and who must supply it. (The author's own observation, made explicit here.)
- When output disagrees with the answer key, re-check the key before blaming the agents. (The author found two errors in his own key this way.)
- Stop at a synthetic self-test: no real ledger, no external report, no sending. (AgentFlow's boundary; the post does not describe a confirmation step.)
Adaptation plan
The post does not say which agents or models were used, how many agents ran, what the six errors looked like in the raw data, what the two answer-key errors were in detail, how the 5.5 dollars was calculated, or what subscription or token prices applied. It gives no real client data, no income, no savings figure, no accuracy over real books, and no time series. The six-error hit rate cannot be generalised to a real ledger. All execution steps, the prompt and the sample here are synthetic proposals.
Prompt
AgentFlow draft · Untested.
You are my bookkeeping review assistant. You may conclude only from the data I give you. If a field is missing, a definition is ambiguous or evidence is insufficient, write 'cannot determine, missing X' instead of supplying a plausible number. You do not send anything externally and do not modify the source ledger. [INPUT] - Synthetic ledger file: [CSV or table path], with at least date / amount / counterparty / receipt id / category - Period and scope: [start and end dates], [which accounts or lines] - Review rules: [rules to check, e.g. duplicate receipts, vendor rate consistency, approval thresholds, segment margin, cross-period revenue, payment authorisation] - Output format: [required table fields] - Note: my answer key is kept separately and is NOT part of your input. I am giving you no hint about where errors might be. [STEPS] 1. Report how many rows you actually read, whether the date range is fully covered, and which fields are missing. If fields are missing, stop here and say which step is blocked. 2. Check each rule in turn. For every rule state what you checked, which rows matched, and the evidence. Write 'none found' where applicable; never skip a rule. 3. Write 'cannot determine' and list it separately whenever a key number is missing, real evidence is absent, a definition has more than one reading, or a rule requires approval that is not recorded. State what is missing, which conclusion it affects, and who must supply it. 4. List clarifying questions separately: which definition or scope do you need me to confirm before you can compute. 5. Produce one result table: rule / outcome (found, none found, cannot determine) / evidence row ids / missing item / who confirms. 6. End with 'what I could not cover', including periods not checked, fields not read, and any rule you think is itself ill-defined. [OUTPUT] 1) read report with row count and coverage; 2) per-rule findings with evidence row ids; 3) cannot-determine list; 4) questions for me; 5) result table; 6) coverage gaps. [NEEDS MY CONFIRMATION] - Any judgement involving approval, authorisation or a definition: stop and let me confirm before computing. - If my rule list is ambiguous: ask which reading applies, do not pick one. - Connecting this to a real ledger, issuing an external conclusion or sending a report: stop at the synthetic self-test and wait. [IF MATERIAL IS MISSING OR A STEP FAILS] Unreadable file, mismatched fields, missing period, duplicate receipt ids that cannot be resolved: say exactly what is missing and which step is blocked. Never substitute rounding, general knowledge or older data. Prefer an extra 'cannot determine' over a number with no evidence.
Running costs and pitfalls
The post does not say which agents or models were used, how many agents ran, what the six errors looked like in the raw data, what the two answer-key errors were in detail, how the 5.5 dollars was calculated, or what subscription or token prices applied. It gives no real client data, no income, no savings figure, no accuracy over real books, and no time series. The six-error hit rate cannot be generalised to a real ledger. All execution steps, the prompt and the sample here are synthetic proposals. No independent cost benchmark is available. Check subscriptions, permissions and external actions before running a workflow.