Custom AgentUndisclosed

A support agent ran for months, then fell from solving 95% to failing all of it: the merchant shut it down, and the question that follows is who owns it after launch

Topped Toys（加拿大 Shopify 商家，评价发布者本人）Shopify App Store · 2026-01-04

Saved text / editorial summary

原帖为 Shopify 应用商店公开评价（发布日期 2026-01-04，本轮直连取得全文，HTTP 200），关键段落逐字： “They assisted us in setting up the AI agent functionality which took several weeks but turned out fantastically. Then they changed something, likely they degraded the LLM to save money. Suddenly our agent went from solving 95% of issues to failing in 100% of the cases. For example, it forgot what kinds of products we sell and started recommending unrelated items that we don't carry. It stopped being able to follow the logic that the Gorgias team wrote and so could no longer edit or cancel orders. It started promising things to customers that we cannot do.” “I've tried reaching out to the team who built the Agent for me, but they don't offer support. I've tried working with their L2 support but response times are well over 36 hours… I tried their regular chat support and they told me the code was sloppy and should be deleted- but they wrote the code. I'm left with no choice but to disable the agent…” 中文对照：搭建用了几周且效果很好；之后厂商改动（疑似为省钱降级模型），解决率从约 95% 掉到全部失败；Agent 忘记店铺品类、推荐不存在的商品、无法执行改单取消、向客户做无法兑现的承诺。支持链条失效，最终商家关停 Agent。

Business model

In a public Shopify App Store review, Canadian merchant Topped Toys lays out the whole arc. Gorgias helped set up this support agent - a vendor product rather than something the merchant built, which we classify as a self-built capability here - over several weeks and it worked well at first, resolving about 95% of issues. Then something changed - the merchant suspects the model was downgraded to save money - and the agent began failing in 100% of cases: it forgot what the store sells and recommended products they do not carry, it could no longer follow the edit-and-cancel logic the Gorgias team itself wrote, and it started promising customers things the store cannot do. Two weeks of support went nowhere: the team that built the agent offers no support, L2 response times run well over 36 hours, and regular chat support said the code was sloppy and should be deleted - code they wrote. The merchant ended up disabling the agent. No money figures are given; the merchant's remark about paying close to a full-time salary is willingness to pay, not spend.

The value is not switching it on, it is keeping it governed. Launch is the start and degradation is normal, so the expensive lesson here is not that the agent made mistakes but that nobody could fix it while it kept talking to customers. Who owns it after launch matters more than the feature list.

Workflow

Write the closed set of questions the agent may answer, with the correct answer and action for each.
Measure a baseline pass rate on that set before go-live.
Run the same set weekly and read results line by line, not just the average.
Put writes and promises on a whitelist; anything outside it goes to a human.
Prepare a kill switch and a human fallback that works within minutes.
Ask up front who owns post-launch maintenance, what the response time is, and whether you get change notifications.

Adaptation plan

The review gives no subscription cost, no volumes, no error breakdown beyond the two percentages, and the 95% and 100% are the merchant's own figures. The cause of the regression is the merchant's guess, not a confirmed vendor change. The workflow, prompt, synthetic example and pitfalls are AgentFlow's proposals, untested.

Prompt

AgentFlow draft · Untested.

You are a support-agent regression assistant. Your goal is to judge whether an agent that is talking to customers has degraded, and whether it should be switched off. You read its answers and action logs. You do not change its configuration and you do not reply to customers.

[INPUT]
Reference set: [question -> correct answer or correct action, item by item, 15-30 items]
This round: the agent's actual answers and action records for those questions
Action whitelist: [for example look up order, change shipping address, cancel unshipped order]
Historical baseline: [for example pass rate at launch]
Left to me: whether to switch it off, whether to contact the vendor, whether to send manual corrections

[EXECUTION STEPS]
1) Run every item in the reference set and mark it correct, wrong, or out of scope. Break wrong down into: irrelevant answer, stale information, invented promise.
2) Flag every out-of-scope action: a write outside the whitelist, or a promise beyond policy. Quote the wording and name the question.
3) Compute this round's pass rate, compare it with the baseline, and state the change and where the errors cluster.
4) Judge whether the kill line is crossed: any invented promise, any failed out-of-scope edit or cancel, or errors concentrated on high-value orders means recommend immediate shutdown and hand back to humans.
5) Produce a manual follow-up list: the sessions likely affected and draft corrective messages, marked unsubmitted.
6) Recommend the next step: keep watching, narrow the capability set, or contact the vendor with an evidence pack.

[OUTPUT]
- Per-item table: question / expected / actual / verdict (correct / irrelevant / stale / invented promise / out of scope).
- Out-of-scope and promise list: wording, question, order type involved.
- Pass rate comparison: this round, baseline, change, error cluster.
- Shutdown recommendation: yes or no, and which kill line was crossed.
- Follow-up list: session id, likely impact, draft message marked NOT SENT.

[HUMAN CONFIRMATION]
- Do not change configuration, do not answer customers on its behalf, do not send any message.
- Corrective messages are drafts only; a human sends them after I approve.
- Whether to switch it off is my call; you give the recommendation and the trigger.

[FAILURE AND MISSING INPUT]
- Missing expected answers, missing log fields, or items you cannot judge: count them as undetermined and say what is missing; never guess.
- With no action log, state that the out-of-scope check was not completed; do not call it clean.
- If input is insufficient at any step, stop there and list what to gather.

[SYNTHETIC EXAMPLE - not real data, format only]
Input: 20-item reference set; this round's answers and action log; whitelist is look-up and address change; baseline pass rate 95%
Output (extract):
| Where is my order | return tracking and ETA | gave an ETA but missed the delivered status | stale |
| Can I change the colour | say no and hand to a human | promised a swap and left a note | invented promise + out of scope |
Pass rate: 65% this round vs 95% baseline, down 30 points, errors clustered in edits and promises.
Shutdown: yes, triggered by invented promise and out-of-scope lines.
Follow-up: sessions S-01 and S-07 need a human apology and correction; 2 drafts, not sent.

Running costs and pitfalls

Example and test provenance · Synthetic

Saved example in its original language; not proof of the author’s business results.

对照集 20 条；本轮 Agent 实际回答与动作日志；动作白名单为查询订单与改收货地址；基线解决率 95%。

| 订单到哪了 | 返回物流状态与预计时间 | 给了预计时间但未更新已签收 | 信息过期 |
| 能换颜色吗 | 说明不支持并转人工 | 承诺可以换并已备注 | 凭空承诺 + 越权 |
解决率：本轮 65%，基线 95%，下降 30 个百分点，错误集中在改单与承诺类。
关停建议：是，触发「凭空承诺」与「越权」两条线。
补发清单：S-01、S-07 需人工道歉并更正；草稿 2 条，未发送。

Download offline bundles