OpenAI AI agent: what you assemble, evaluate and authorise

This page is for marketing, operations and IT directors whose team has chosen OpenAI and now has to build an OpenAI AI agent that will run with nobody in front of the screen. It helps you decide three things: which blocks you assemble and which stay on your side, how you measure the agent before widening it, and on what condition you grant it a write right.

It builds on the four permission levels and the seven families of agents from our overview of AI agents, which it does not redefine.

Two planes not to confuse: the product and the build layer

At OpenAI, the word “agent” covers two objects that share neither buyer, nor cost, nor lifespan. On one side a product experience: you describe a task in an interface and watch it run. On the other a build layer (at the time of writing, the Responses API, the Agents SDK and callable tools) on which a team assembles an autonomous system.

That second plane is no side road: more than 2 million developers use the API (Chad Wyatt, 2026), a figure recorded in our ChatGPT statistics. The following table sums up what separates the two planes.

If the ecosystem is not settled, comparing families of tools belongs to the choice of an AI agent platform. Here it is assumed to be chosen.

One-off use or a system that runs continuously

Both planes are demonstrated the same way, and that is what misleads: a demo never shows the thirtieth run on incomplete data. If the task comes back once a quarter and a human triggers it, the product is enough; its scope is described under the ChatGPT AI agent mode. If it comes back every week, starts on its own and feeds other systems, you are building.

What the confusion costs on the organizational side

The cost of the confusion is organizational. Product use asks for nobody. An agent built on the API asks for a named owner, who answers for what the system did at three in the morning on a Tuesday, and for skills rarely found in the same person.

• Someone who writes the business rules: what the agent refuses, escalates, never negotiates.
• Someone who holds the integration and the access rights.
• Someone who maintains the evaluation case set.

Maintenance is precisely the item forgotten at scoping. Models change, sources move, rules shift, and each movement can make the agent regress. Budget the replay of evaluations as an on-call duty: periodic, named, independent of the builder. Otherwise the agent turns back into a script nobody dares to modify.

What OpenAI supplies for building, and what stays on your side

A vendor ecosystem is judged on two lists: what it saves you from assembling, and what it leaves you with anyway. The first makes the demos; the second decides your schedule.

The blocks supplied: visual building, development kit, tools

The blocks fall into four families, all resting on the same API, which share neither audience nor exit cost. None of them comes with a ChatGPT subscription. At the time of writing, they go by the following names.

• A visual builder (Agent Builder): you assemble steps, version them and set rules without writing code. Quick to prototype with, and a good medium for review by the business.
• A development kit (Agents SDK): the same logic in code, calling the Responses API, with types and tests. That is where the agents you maintain end up.
• Built-in tools the model can call: sourced web search, search across files, code execution, browser control. These are capabilities, not integrations: they widen what the agent can do, not what it is allowed to do.
• Connectors to third-party applications, with the associated authentication, access scopes and quotas.

A fifth block joins them, invisible because it produces nothing: the evaluation layer (Evals), which replays a case set and grades execution traces, covered further down. A visual sequence reads quickly but exports badly; a coded sequence can be compared and restored. Prototype with the first, ship with the second, and place the switch where the agent gets its first write right.

What the ecosystem does not supply, and what stays with you

The rest belongs to you, and it is what makes the project.

• The business rules: what the agent must refuse, what it must escalate, what is never negotiable.
• The quality of the data consulted: an up-to-date base, with read rights that mirror those of your teams.
• The evaluation case set: nobody can write it for you, because it encodes what “correct” means at your company.
• The connection to your systems: recovery on error, queues, choosing what gets logged.

None of those items is settled by a choice of tool or disappears when you change vendor. That is why a prototype convincing in three days takes three months to reach production: the days consume what the ecosystem supplies, the months what it leaves.

The core of an agent: instructions, output formats, stop criteria

An agent in production is not “a longer prompt”: it is an architecture that connects model, tools, data, memory, evaluation, security and supervision. The core rests on stable instructions, output constraints (schemas, formats) and stop criteria, the moment the agent stops deciding and escalates to a human.

Those elements are written before the first line of code; as long as they are not, you have a prototype about which nobody can say whether it got things wrong. The following table lists what you need to have written.

From the plausible answer to the defensible answer

A model always produces an answer; nothing guarantees it is verifiable. Three decisions move the output from plausible to defensible, and each is paid for in constraint.

• Structured output format (table, schema, plan): more reliable automation (machine reading, tickets, chaining), at the price of an output that is less readable as it stands.
• Validation rules (thresholds, escalation): fewer errors in production, at the price of a high escalation rate in the first weeks.
• Defensibility criteria (sources required): quality control that can be made objective, at the price of occasionally empty answers when the source is missing, which is the intended behaviour.

The third is the one removed first when the agent “does not answer enough”. That is the mistake: an empty answer points to a hole in the base; a plausible answer with no source is a debt discovered at a customer’s.

The retrieval strategy decides more than memory does

An agent’s performance depends less on “magic memory” than on a retrieval strategy: which sources to query, when, at what freshness. In a company, the hierarchy fits in three lines.

• The internal base prevails: internal documents, product reference data, brand rules.
• The Web serves to complete: current events, comparisons, cross-checking.
• The Web may contradict the base: the contradiction is flagged, not settled silently.

Write that rule as a priority: an agent that queries the Web first will hand you answers that are right about the market and wrong about you, with the same confidence. Building the base weighs more on final quality than the choice of model.

Evaluate before widening: case sets, graders, trace grading

The measurement setup is part of what you build. You do not widen an agent’s autonomy because it “looks good”, but because a replayed case set gives a stable result. The loop has three stages.

• Define a set of realistic cases (simple tasks first, then edge cases).
• Measure quality (accuracy, completeness, citations, rule compliance, cost).
• Optimize (prompts, tools, retrieval strategy), then repeat.

Graders are automatic checkers: to each case you attach what can be observed mechanically in the output (a field present, a format respected, a citation, a refusal pattern). They observe without appreciating meaning: an answer can satisfy every criterion and still be wrong. Semantic criteria therefore stay arbitrated by a human, on a sample.

Trace grading applies afterwards to real runs, put through the same criteria. Graders tell you whether the agent holds up on what you had imagined; traces, on what you had not.

Writing a usable case set: how many, what variety, who writes it

A case set is not a statistical sample: it is a list of situations whose right answer you already know. A few dozen are enough to start, half nominal cases, half cases that must fail cleanly. The expected failures cover at least four situations.

• Missing data.
• Contradictory sources.
• An out-of-scope request.
• An attempt to obtain an unauthorized action.

It is written by the business, which knows that an answer omitting a contractual constraint is wrong, however well phrased; the technical team turns it into executable cases and graders. And that set is alive: every incident in production adds a case to it, and that case never leaves. It is the only non-regression you have on a probabilistic system.

What the result authorizes, and what sends it back down

A score is useless unless it is attached to a decision. Settle, before measuring, what each axis authorizes and what it takes away; otherwise autonomy widens out of weariness.

A setup that only knows how to widen is not a control, it is a ratchet. Name who can send the agent back down a notch, without a meeting: that step back must cost less than an incident. The “first write right” corresponds to level 2, controlled writing, of the four permission levels in the overview.

The guardrails specific to an agent that browses

As soon as an agent reads the open Web and accesses your data, the risk changes in kind: the danger is no longer only that it gets things wrong, it is that it obeys someone other than you, with nothing to flag it.

Prompt injection: what it is, and why a better prompt is not enough

A prompt injection is a malicious instruction hidden in content the agent consults: a page, a document, a message. The agent does not structurally distinguish what it is asked from what it reads; a well-placed text can make it disclose content, bypass a rule or trigger an action. Firmer instructions change nothing. Three countermeasures hold.

• Explicit confirmations before any consequential action.
• Switching off connectors not needed by the current task.
• Minimal permissions on the ones that remain.

The logic is constant: you do not stop the agent reading hostile content, you make obeying it useless. Restricting writing reduces the surface without removing it: a read-only agent remains exposed to disclosure and to the exfiltration of data slipped into a tool query or a link. Writing adds the unintended action, carried out in your name.

Minimal permissions, human approval, traceability

Three protections are set at scoping, not after the first incident.

• Minimal permissions: read access by default, writing only on low-risk scopes.
• Human validation: mandatory for emails, purchases, irreversible changes and sensitive content.
• Traceability: log the tools called, the sources consulted, the decisions and the results.

Traceability is the only one of the three that cannot be caught up later: a permission can be narrowed, an approval can be added, but a trace not written at execution will never exist. Decide what gets logged before go-live, and check on a real case that you can answer the question behind every incident: which source, and why that action?

Going to production: connecting, supervising, holding costs

A good deployment favours repeatability and supervision: you start small, you measure, then you widen. The connection rules that cover almost every case number three.

• Expose your data for reading with minimal permissions.
• Break actions down into atomic operations: create a ticket, export a report, propose a draft.
• Add webhooks and approvals as soon as the agent leaves “advice” and enters “execution”.

The third rule carries the only risk threshold worth memorizing: as long as the agent advises, the worst outcome is lost time; the moment it executes, it is a system state to undo. Atomic operations make each action separately reversible. Scaling up then follows its own rules.

• Supervision: dashboards and alerts on consumption drift, weak sources and repeated errors.
• Breakdown: avoid “monolithic” agents that do everything in one go and cannot be diagnosed.
• Autonomy: increase it only after stable evaluations, never after a good week.

Cost, finally, is not ChatGPT’s: a subscription to the product covers neither the API, nor the development kit, nor the tools called, nor the evaluations. Amounts move too fast to be copied out here: read them off the official pricing page. The following table gives the items to budget.

The quota question tells you whether the agent stops cleanly or leaves a job half done. And what clears deployment settles in: your agent will still be there in a year, with its debt.

Continue reading

Depending on where you are stuck, another article takes over.

• You want the method before the tool: scoping, writing the rules, testing and governing is set out in the guide to create an AI agent.
• Your sticking point is the document base: chunking, index and freshness are covered on the RAG AI agent.
• Your single agent has become too heavy: coordination, conflict arbitration and replay belong to AI agent orchestration.
• What remains is connecting the blocks to your systems: connectors, webhooks and recovery on error are the subject of AI agent integration.

And if what you are missing is the internal base the agent must be anchored in, validated and up-to-date content, that is the task framed by content generation with a personalized AI.

FAQ on OpenAI AI agents

How do you use the OpenAI API to create an agent?

You create an agent by combining a model, instructions and callable tools, orchestrated by a versioned sequence. The approach has four stages: scope the use case and the action limits, connect the sources, constrain the output formats, then put the evaluations in place before increasing autonomy.

What is the OpenAI agent platform?

The OpenAI agent platform is the set of building blocks offered around the API. At the time of writing, it brings together a visual builder (Agent Builder), a development kit (Agents SDK) on the Responses API, built-in tools, third-party connectors and an evaluation layer (Evals).

What is the pricing?

There are two separate pricing models, and confusing them throws off any budget. ChatGPT is billed as a per-user subscription and opens no rights on the build layer, which follows usage: requests, length of outputs, tool calls. The amounts are checked on the official pricing page; human supervision comes on top.

What are the capabilities of OpenAI agents?

An agent built on this layer reasons and then acts through tools: sourced web search, reading files, code execution, browser control and connectors to business applications, keeping a context between steps. Those capabilities describe what it can do; what it is allowed to do depends on the permissions granted.

What is the difference between an OpenAI agent, a conversational assistant and a tooled workflow?

The difference is the right to act and what gets logged about it. A conversational assistant answers without guaranteeing execution or traceability. A tooled workflow chains predefined steps and turns rigid off the expected path. An agent chooses its tools according to context, iterates and asks for validation before important actions.

What minimum guardrails limit errors, risky actions and data exfiltration?

Three protections are enough to start: an explicit validation before any irreversible action, minimal permissions with only the necessary connectors, and a trace of tools, sources, decisions and outputs. Prompt injection makes the second point decisive: the fewer rights the agent has, the less a hostile instruction can exploit.

How do you evaluate an agent before deploying at scale?

You evaluate an agent on a case set built from real data, edge cases included, where each case states what a grader must observe. You measure five axes: accuracy, rule compliance, source quality, cost, latency. You replay after every change and only widen the rights after several stable replays.