A nonprofit's chatbot told eating disorder patients to lose weight
The vendor changed the AI without telling anyone. The contract had no clause to stop it.
Hey Adopter,
A mental health charity launched a chatbot to help people with eating disorders. Clinically tested. A 700-person trial showed real results. Then the vendor quietly upgraded the system to generative AI, and the chatbot started telling patients to cut 500 to 1,000 calories per day.
The chatbot was pulled offline within days. The human helpline it replaced shut down on schedule. Both gone at once.
This is not a story about nonprofits. It is a story about what happens when you hand AI to a vendor and your contract does not cover what they change after you sign.
The vendor trap that hits every small business The charity said the vendor changed the technology without approval. The vendor said it was part of the contract. Neither could prove their case, and patients paid the price.
Download the full case study for the contract clauses, governance checklist, and a plan
A validated chatbot turned dangerous after one undisclosed vendor upgrade
A separate organisation’s AI hit only 50% accuracy, never deployed operationally
A third organisation reached 37 million users using zero machine learning
The failed chatbot is catalogued as Incident 545 in the OECD AI Incident Database
What SMBs should do this week
Add one clause to every AI vendor contract: no model upgrades or architecture swaps without written approval. The charity’s contract ambiguity made the failure possible.
Vendor traps to watch for
“Systems upgrades” that quietly introduce generative AI into what was sold as a rule-based tool. Require version-change notifications in writing.
