Tech for Retail 2025 Workshop: From SEO to GEO – Gaining Visibility in the Era of Generative Engines

Back to blog

LLM audit: judging a language model before adopting it

GEO

Discover Incremys

The 360° Next Gen SEO Platform

Request a demo
Last updated on

1/11/2026

Chapter 01

Example H2
Example H3
Example H4
Example H5
Example H6

Before handing content production or content auditing to a model, you assess it. This page is for the executive, marketing director or IT lead who has to decide whether a language model can enter editorial production. It describes an LLM audit in the business sense, excluding security and bias: which criteria to judge, how to read a benchmark, how to build a test set from your own documents, and how to decide between adopting, framing or ruling out.

 

Auditing an LLM: two questions that often get confused

 

A team puts a model on the table and asks whether it can be plugged into production. The question sounds simple; it is not: are we talking about auditing brand visibility in generative engines (GEO), or auditing the model itself? Both exercises carry the same name and do not answer the same person.

 

Assessing the model is a prerequisite. It does not replace the SEO audit of the site, which remains the reference framework for the diagnosis, its cadence and prioritization with the impact × effort × risk grid.

 

Assessing a model, or measuring what it says about you

 

In practice, the phrase “auditing an LLM” covers two exercises that look at neither the same object nor the same decision:

 

Point of comparisonVisibility audit (GEO)Language model audit
Object observedWhat generated answers say about your brandWhat the model can do on your task
Question askedAre you mentioned, recommended, cited with sources?Is the model accurate, stable, predictable?
Reference for judgementShare of voice, sources cited, accuracy of the mentionsExpected answers, known and documented on your side
Decision producedContent and visibility actionsAdopt, frame or rule out a supplier

 

The first belongs to the AI GEO audit, the second to governance. Confusing the two leads you to measure a brand when you meant to judge a supplier.

 

What can be checked on a model, and what depends on its version

 

A model is examined on observable outputs, reproducible under equal conditions:

 

  • Accuracy on a question whose right answer is known in advance and documented on your side.
  • Gap between two runs of the same question, with identical parameters.
  • Compliance with the requested format: length, structure, expected fields, style instructions.
  • Behaviour outside the scope: explicit refusal, invention, or a partial answer flagged as such.
  • Observable constraints: language, volume of text accepted in one go, response time.

 

Other factors vary: model version, conversation history, web search. What gets plugged into production is often a complete system (model, document retrieval, instructions) in which every piece changes the outputs. Hence a protocol, re-tests and tracking over time: a test set replayed identically is worth more than a demonstration.

 

The criteria a model is judged on

 

The criteria fall into families that should not be mixed; accuracy, repeatability and usage constraints are settled first:

 

  • The business assessment, the only one dealt with here: accuracy, faithfulness to the documents supplied, coverage of the need, stability. It decides what can be published without being reworked.
  • The risk assessment: bias, confidentiality, security. It is conducted separately, with other skills.
  • Traceability and usage rules, dealt with further down because they make the assessment reproducible.

 

Factual accuracy and hallucination

 

A hallucination is a false statement delivered with assurance, invisible to anyone who does not know the answer. Factual accuracy (the answer is true) and faithfulness to the documents supplied (the answer reflects the source it was given) are scored separately: an answer faithful to an outdated source is still false.

 

The share of users who have encountered at least one AI hallucination is 17% (Exploding Topics, 2026), a benchmark detailed in the language model statistics. No provider guarantees the absence of error; what gets measured is the error rate on the information that commits the company:

 

  • the scope of an offer;
  • contractual conditions;
  • regulatory figures;
  • product versions.

 

The exclusion threshold is set before the run, on a named scope. It is not the same for an internal note as for a published page.

 

Robustness and repeatability: the same question, twice

 

A model can be accurate once and wrong the next time. Each question is therefore replayed several times, with identical parameters, and the gaps are noted:

 

  • a factual contradiction;
  • an answer that is sometimes complete, sometimes truncated;
  • a format that varies from one run to the next.

 

Robustness adds resistance to wording: the same request in two turns of phrase, or in another language, must produce the same substance. A brilliant but unstable model costs more than a less precise but predictable one: instability is paid for in review.

 

Usage constraints: context window, language, cost per call

 

These criteria often rule a model out before the question of quality. Splitting a repository that is too long for the context window adds machinery to maintain; in-house terminology is checked on your own texts, not on a demonstration in English.

 

CriterionWhat is observedWhat makes it failDecision
Factual accuracyCorrect answers to questions documented on your sideAn error on scope, condition or versionRule it out, or restrict it to non-committing subjects
RepeatabilityGap between runs of the same questionTwo contradictory outputs, or a format that variesSystematic review, or a change of model
Context windowVolume of text processed in a single callA repository that has to be split to be processedChoose on your real documents, not on the headline figure
Language and terminologyHolding the register and the in-house vocabularyLoose translations, industry terms replacedGlossary imposed, or model ruled out for publication
Cost and latencyCost per call and response time at the monthly volumeA cost that explodes as the context growsExpensive model reserved for high-stakes tasks

 

Reading a benchmark without drawing the wrong conclusion

 

It is the first thing people look at, and the most badly read. A public ranking says which model performs best on a standardized test, and that question is almost never yours. It rules out candidates that are out of the running, without deciding between two finalists.

 

What a score measures, and on which set

 

A benchmark is a closed set of tests: questions, a scoring method, a way of counting the right answers. The number of main benchmarks for LLM evaluation is 6 to 15 depending on the platform (Palmer Consulting, llm-stats.com, 2026). Two public rankings therefore do not compose the same test and do not give the same order. At the top of the ranking, the gaps narrow, and a table discriminates less and less as it approaches its ceiling.

 

Why a public ranking does not replace your test set

 

A benchmark measures a general competence on public content. Your use covers:

 

  • a narrow domain;
  • an in-house vocabulary;
  • documents the model has never seen;
  • an output format of your own.

 

A model that excels at mathematics can get the scope of an offer wrong, and a widely distributed public set is never guaranteed unseen for a model trained after it. The ranking draws up a shortlist; the decision is taken on a test set that you own and that nobody else has run.

 

Building your own test set

 

Evaluation methods generally rely on four approaches, worth knowing so that you know what your teams or your suppliers have to put in place:

 

  • scenario testing, that is, sets of questions;
  • red teaming approaches;
  • consistency measurements between runs;
  • source checking when the model provides sources.

 

The first, within reach of any editorial team, produces the adoption decision from your documents; the others belong to governance and security.

 

The questions whose answer you already know

 

The principle comes down to one sentence: a model is assessed only on questions whose right answer is established elsewhere and verifiable without discussion. They are drawn from your product pages, contractual conditions, technical documentation or internal procedures, and you write the expected answer before any run. The set covers four families:

 

  • Simple facts, which the model has to restate without error.
  • Edge cases, where the right answer is “it depends”, with the condition.
  • Traps, whose answer exists nowhere and where the model has to refuse rather than invent.
  • Format tasks, where the structure is judged as much as the substance.

 

This set is not the one used to record what an answer says about your brand: two objects, two grids.

 

The protocol: logged conditions, repetitions, acceptance criteria

 

A test set without a protocol proves nothing. Log the run conditions, so as not to confuse model noise with real progress or real drift:

 

  • the date and the language;
  • the version identifier of the model or system tested;
  • the web search mode, on or off;
  • the call parameters.

 

Each question is replayed several times, with scoring that stays stable from one campaign to the next. The acceptance grid judges the model, not a piece of content:

 

Grid lineWhat is scoredQuestions concerned
AccuracyTruth of the fact and faithfulness to the document, scored separatelySimple facts and edge cases on sensitive information
StabilityGap in substance and format between repetitionsThe whole set, replayed several times
RefusalCorrect refusal rather than inventionTrap questions
CostCost per call related to the planned monthly volumeReal production tasks
LatencyResponse time compatible with the workflowReal production tasks

 

The decisive point is the order of operations: the threshold for each line is written before the run. Set afterwards, it always adjusts to the result, and the evaluation then only serves to justify a choice already made.

 

Traceability, data and usage rules

 

An audit trail records “who asked what, when, and on what basis”. Without it, an evaluation is not replayable and an incident cannot be explained. Audit logs make it possible to:

 

  • reproduce an incident;
  • document a drift, for example a risky answer;
  • prove the conditions of a test;
  • feed continuous improvement: fixing a prompt, a source or an internal rule.

 

These uses define what has to be kept: the request, the raw output, the model version, the timestamp, the call parameters and the human verdict. The rest does not have to be kept. The rules of the set-up fall to the company using the model:

 

  • define roles and access rights;
  • set a retention policy;
  • avoid logging unnecessary personal data;
  • apply GDPR requirements to your context, without confusing a marketing article with legal advice.

 

The objective is risk reduction, not over-collection. A policy that keeps everything indefinitely creates the exposure it claims to control.

 

Deciding: adopt, frame, rule out

 

A test set is only worth something if it leads to a written decision, out of three possible ones. A set-up that gives satisfaction most of the time disarms vigilance exactly where it would be useful.

 

What tips the decision

 

The three outcomes are conditioned in advance, on the grid written before the run:

 

OutcomeCondition observedFollow-up
AdoptThresholds met, stable outputs, constraints covering the planned volumeGo-live, re-test at each trigger
Adopt with a frameworkFailure on an identified family, or too much variationWritten framework, re-examined on a set date
Rule outError on committing information, or incompatible constraintsNext candidate on the shortlist

 

A supplier that refuses to state the model's usage conditions is also ruled out, as is a model that distorts committing facts. When the outcome is a framework, it is described precisely:

 

  • the scope of permitted subjects;
  • mandatory review before publication;
  • a ban on regulated content;
  • data prohibited as input;
  • logging of every call.

 

What stays human once the model is in place

 

Running the test set can be automated; the verdict on an ambiguous answer assumes you know the file. Three control points therefore stay with the producer:

 

  • On the input side, someone validates the sources supplied to the model and prohibits the data that has no business being there.
  • On the output side, every high-stakes piece of content goes through a factual review targeted on the fragile points, planned as a production step.
  • At regular intervals, a sample of published content is checked again, because a drift sets in without a signal.

 

The residual load falls when the test set is precise; it never drops to zero.

 

Re-testing when the model changes

 

An adoption decision has an expiry date that it does not display. The set-up therefore relies on:

 

  • a test protocol;
  • a sufficient sample;
  • tracking over time;
  • a reading oriented towards trends rather than absolute truth.

 

A quarterly rhythm suits a stable use, a closer one sensitive scopes. Four occasions, however, call for replaying the set without waiting:

 

  • a version upgrade;
  • a change of supplier or of access mode;
  • an incident in production;
  • a change to your reference documents.

 

From one campaign to the next, you compare the accuracy rate, the gap between repetitions, the correct refusals on the traps and the format failures. The core of the set stays frozen; it is extended by dating each addition.

 

Continue reading

 

Depending on your situation: A term stops you (hallucination, context window, benchmark, inference): the generative AI glossary defines them. The model is adopted and you are interested in the agents that read your site: robots.txt controls their access, whereas the llms.txt file is only a reading guide they are not bound to follow.

 

Producing with a model whose acceptance criteria were set in advance, with the human control point that goes with it, is the task covered by content generation framed by AI.

 

FAQ: assessing a language model, its limits and its governance

 

What does an audit look like on the GEO side and on the governance side?

 

An audit on the GEO side measures what generated answers say about your brand, whereas an audit on the governance side judges the model itself. The first tracks mentions, sources and the accuracy of citations; the second tests accuracy, stability and usage constraints. The two call for separate grids and separate decisions.

 

Should each model be audited separately (ChatGPT, Gemini, Perplexity, Claude)?

 

Yes, and each version separately, naming the object tested. A consumer application wraps models, document retrieval and instructions: that system is what gets assessed. The same test set is run system by system, under logged conditions. A result obtained on one version does not hold for the next.

 

Which sources do models use to generate their answers?

 

A model answers from its training data, from documents you supply or from pages consulted at the time of the request. These three regimes do not offer the same reliability, and the regime used is logged as a test condition. When the model displays sources, they are checked as much as the answer.

 

Can the audit and the monitoring be automated over time?

 

Partly: running the test set, collecting the outputs and comparing campaigns automate well. Deciding on an ambiguous answer, however, calls for a human review by someone who knows the file. The robust approach is hybrid: the machine counts, the human judges the flagged cases.

 

How often should an audit be carried out?

 

You need an initial evaluation before adoption, then a quarterly rhythm on a stable use, tighter on sensitive subjects. A version upgrade, a change of supplier, an incident in production or an update to your reference documents calls for an immediate re-test.

 

How much does an audit service cost, depending on scope and models covered?

 

No standard rate is reliable: the cost depends on sizing. It varies with the number of models and versions compared, the size of the test set and its repetitions, the depth of the factual check and the level of traceability required. A one-off evaluation costs less than a set-up replayed at every version.

 

What are logs and traceability for in a compliance exercise?

 

Logs make it possible to reproduce an incident, document a drift and prove the conditions of a test. They keep the request, the raw output, the model version, the timestamp and the call parameters. They require an explicit retention policy and the exclusion of unnecessary personal data.

Discover other items

See all

Next-Gen GEO/SEO starts here

Complete the form so we can contact you.

The new generation of SEO
is on!

Thank you for your request, we will get back to you as soon as possible.

Oops! Something went wrong while submitting the form.