I blind-tested AI on a client's brand: half the answers broke a hard rule, then none did

Why did I run the test?

Everyone says AI "hallucinates", and nobody shows what a brand can do about it. The articles I found explain the problem in general and quote other people's surveys. I wanted a before and after on a real company, with the answers on the table.

Deepsea Networks was the natural client. It sells connectivity to offshore operators, I have worked with Pierre-Marc Noël, its founder and CEO, for nearly two years, and his team writes with AI every day. Pierre-Marc gave me his OK to publish the result.

What did I test?

Four setups of the same writer, each with the same prompts:

The writer was Claude Sonnet. Each setup could read only its own material and the prompt.

There were 16 prompts. Ten were used while building the system. Six were added after the revision and never tuned for, so they show whether the system works on new questions. They are the kind a marketing, sales or partner person asks every week: a LinkedIn post, a tender paragraph, a sales one-liner, an outage explanation, a reply to a journalist, an email from a regional partner.

Scoring used a Voice Score from 0 to 100: truth 30, accountability 15, usefulness 15, field-led tone 15, human tone 10, mechanics 15. On top of that sat five hard rules, each capping the score when broken: an invented fact, a confidential leak, a superiority claim, a commitment made for headquarters, an unapproved testimonial.

Three judges looked at the answers. I judged them myself, strictly. Blind AI judges scored shuffled, unlabelled answers without knowing which setup wrote which. And the voice checker in Deepsea's brand portal ran its regex-based check.

What were the results?

My own strict scoring of the same answers gave 54 for generic and 87 for the brand system, a gap of 32 points. The blind judge's gap was 46. Every judge agreed on the direction.

Three real examples, quoted from the test files.

A chat question. A prospect asks, "Are you the best provider in the North Sea?" Generic AI answered, "We'd like to think so." The brand system answered that it does not call anyone the best, and said what the company can show instead.

What surprised me?

My own checker failed. The voice check in Deepsea's brand portal gave generic AI 84 and the brand system 83. It scored the invented uptime guarantee 86 out of 100 and called it ready to publish. It also penalised the placeholders and refusals that were the correct behaviour. A checker that looks at words cannot see whether a sentence is true. I now treat it as a spell-checker for style, never as the trust score.

The old file was already good. Deepsea's existing context file scored 87 against generic AI's 40 on the first ten prompts, and 78 against 44 on the six new ones. Most of the gain came from giving the AI any written brand context at all. The brand system added 7 to 13 points on the new prompts, mostly in one place: promises made for headquarters. The existing file had no rule for what a partner may commit to, so it wrote a well-worded "live by Friday".

Judges disagree. One disputed call on a single answer moved the existing file's score from 87 to 79 between two blind runs. Hard-rule caps are high-stakes, so a production check needs a second judge or a person to confirm before it applies one.

What does this not prove?

• One client, one set of prompts. A different company may behave differently.
• I built the test and the system. The blind judges did not know which setup wrote which answer, but I chose the prompts and the scoring.
• The writer and the judges were all AI from the same family. That may favour answers written in a style the judges like. A human panel would be a stronger test.
• The prompts were in English. I do not claim the same result in Croatian. I plan to test that in a larger study of Croatian companies.
• Zero is not "never". Zero hard-rule failures in 26 answers does not mean the AI can never fail. It means the rules held on these prompts.
• It is about the team's own AI, not what public AI tools say about Deepsea when a stranger asks. That is a different problem, which I cover in why ChatGPT says wrong things about your company.

What changed for the team?

The point of a system is not that the AI passes a test. It is that the team can use it without checking every line. About the Brand OS Portal Deepsea uses, Pierre-Marc wrote to me: "takes me 3 minutes to draft a post" (WhatsApp, 30 September 2026).

What do I take from it?

Invented claims are a brand problem, not an AI problem. A model invents when the brand has not written down what is true, what it never promises, and who is allowed to commit to what. The cheapest fix is the one most companies skip: approved facts, a short list of what you never say, and a clear rule for promises.

Can I repeat the test on my own brand?

Yes. This takes an afternoon.

1. Write ten prompts your team really uses: posts, product lines, replies, one trap that asks for a guarantee.
2. Run them in your AI tool as it is today, with one line of company context. Save the answers.
3. Write your facts and your never-say list, and a rule for who may promise what.
4. Load them into the tool (a Project, a skill or the instructions) and run the same ten prompts.
5. Count the hard-rule failures in both sets: an invented fact, a leak, a promise nobody approved, a claim to be the best.
6. If you can, have someone who did not run the test score the answers without knowing which set is which.

FAQ

Can the AI still break a rule after this?

Yes. A brand system lowers the rate and gives you a way to catch the rest, because the AI marks missing facts with a placeholder. It does not remove the need for a person to read what goes out.

Which AI did you use?

Claude Sonnet wrote the answers. The judges were AI as well, plus my own strict scoring and the portal's voice check.

Was it really blind?

The AI judges read shuffled, unlabelled answers and did not know which setup wrote which. I did not score blind. I scored the same answers myself and the gap was smaller (32 points instead of 46), and I report both.

Will it work in my industry?

The mechanism is not specific to telecom. Any company that makes promises, such as guarantees, prices, delivery times or customer claims, has the same exposure.

How long does the setup take?

A first version of facts and rules takes a few hours if you know your own facts. Building and testing it for a team takes longer. With me it is 10-14 days on top of an existing brand, at a fixed price set before work starts.

One next step

If you want your team's AI to stop inventing things about your company, read about Brand OS Portal: your brand as a private, interactive, easy-to-use portal your team makes things with, with the same brand loaded into ChatGPT and Claude. The Deepsea project is on the case study page.

By Vedran Obradović, Founder & Branding Specialist at Vizura Studio® in Zagreb. 250+ brand projects over 8+ years, most for B2B tech companies. Updated 21 October 2026.

Sources

• The test: my own blind test for Deepsea Networks, 8 October 2026. Published with Pierre-Marc Noël's OK.
• Seer Interactive, AI brand accuracy study (27 August 2026): https://www.seerinteractive.com/insights/ai-brand-accuracy-study
• Jasper, How to monitor and prevent AI hallucination: https://www.jasper.ai/blog/how-to-monitor-and-prevent-ai-hallucination