How AI agents work: models, tools, memory and plans

Ask an AI agent "why did sales drop last week?" and a minute later you get an answer: the marketplace paused your listing on Tuesday over a missing tax ID. In between, it may have run half a dozen database queries, hit an error, fixed its own query, searched your email and checked its conclusion. So how do AI agents work, exactly? This article opens the hood: the model, the tools, the loop, memory, planning, guardrails and cost. It's written for a curious non-engineer, so every term gets explained the first time it shows up.

If you're brand new to the idea, start with what is an AI agent? and come back.

How do AI agents work? The short version

An AI agent is a few simple parts wired together:

Around all of that sits the harness: the ordinary software that runs the loop, calls the model, executes the tools and enforces the rules. The model is the clever part. The harness is the responsible part. Let's take each piece in turn.

1. The model: a very good next-word predictor

At the centre is a large language model (LLM). It's first trained on enormous amounts of text to predict what comes next, then trained further to follow instructions. Do that well enough, at enough scale, and you get something that can summarise a contract, write SQL, or reason through a plan.

Three facts about the model explain most of how agents behave.

It reads and writes in tokens. A token is a chunk of text, often part of a word. OpenAI's rule of thumb for English is that one token is about four characters, or roughly three-quarters of a word. Models are priced and limited in tokens.

It only knows what's in front of it. Everything the model can use for a decision has to be in its context window: the text sent to it on that call, up to a fixed limit of tokens. Your instructions, the conversation, the tool results so far: all of it is re-sent every time. Between calls, the model itself remembers nothing. When an agent seems to "remember", the harness is putting things back in front of it.

It can be confidently wrong. Models sometimes produce false statements that sound right, known as hallucinations. A 2025 research paper argues that part of the cause is how models are trained and tested: a confident guess scores better than admitting uncertainty. That's why good agents check facts with tools instead of relying on what the model "knows".

2. Tools: how a model does things

A model on its own can only produce text. Tool calling (also called function calling) is the trick that lets that text turn into action.

It works in five steps, as OpenAI's documentation lays out:

1. The harness tells the model which tools exist. Each tool has a name, a plain-language description, and the inputs it takes.
2. The model, if it decides a tool would help, replies with a tool call: structured text naming the tool and the inputs.
3. The harness runs the real code.
4. The harness sends the result back to the model.
5. The model answers, or asks for another tool.

Here's what a tool description looks like to the model:

{
  "name": "find_invoices",
  "description": "List unpaid invoices older than a number of days.",
  "parameters": {
    "type": "object",
    "properties": { "older_than_days": { "type": "integer" } },
    "required": ["older_than_days"]
  }
}

And here's the model's reply when it wants to use it:

{ "tool": "find_invoices", "arguments": { "older_than_days": 7 } }

The key point: the model never touches your systems. It writes a request. The harness decides whether to carry it out. That gap between "the model asked" and "the software did" is where every serious safety measure lives: permissions, approvals, budgets and logs.

OpenAI's guide sorts tools into three kinds:

• Data tools fetch information: query a database, read a PDF, search the web.
• Action tools change things: send an email, update a customer record, hand a ticket to a person.
• Orchestration tools are other agents, used as tools by a lead agent. More on that in multi-agent AI explained.

Two more terms you'll hear. MCP (the Model Context Protocol) is an open standard for connecting AI applications to outside tools and data; its own documentation compares it to a USB-C port for AI. Anthropic introduced it and in December 2025 handed it to the Agentic AI Foundation, a fund under the Linux Foundation. And computer use is a tool that gives the model screenshots, a mouse and a keyboard, so it can operate apps that have no API, the way a person would.

3. The loop: act, observe, repeat

One tool call is not an agent. The loop is. The harness keeps calling the model, running the tools it asks for and feeding back the results, until the model says it's done or a limit is hit. OpenAI's guide calls this loop "central to the functioning of an agent."

The pattern got a name from a 2022 research paper, ReAct (reasoning plus acting): the model alternates between thinking out loud and taking an action, and the results of each action shape the next thought. Here's an illustrative trace for the sales question from the start:

Goal:   Why did sales drop last week?
Think:  Compare last week's orders with the week before.
Act:    run_sql("select date, count(*) from orders ... group by date")
See:    Error: column "date" does not exist
Think:  Wrong column. Look at the table first.
Act:    describe_table("orders")
See:    id, customer_id, total, placed_at, channel
Think:  It's placed_at. Rerun, split by channel.
Act:    run_sql("select channel, ... where placed_at >= ...")
See:    Shop orders flat. Marketplace orders fall sharply from Tuesday.
Think:  Something happened on the marketplace on Tuesday. Check email.
Act:    search_email("marketplace", since: "Tuesday")
See:    "Your listing has been paused: missing tax ID"
Answer: Sales fell because the marketplace paused your listing on Tuesday.

Look at what the loop bought you. The agent made a mistake (the wrong column), saw the error, and fixed it. Then it followed a clue somewhere no script would have gone. Anthropic's engineers call this getting "ground truth" from the environment at each step, and it's the difference between an agent and a model guessing.

Every sensible harness also has stopping conditions: a maximum number of steps, a time limit, a spending cap. Without them, a confused agent can keep looping, and spending, indefinitely.

4. Planning: from goal to steps

For anything bigger than a quick lookup, the model first breaks the goal into steps, often as an explicit to-do list the harness keeps for it. Then, as results come in, it re-plans: drops steps that turned out unnecessary, adds ones it didn't foresee.

Two refinements are common in capable agents:

• Reflection. After a failed attempt, the agent writes down in plain words what went wrong and tries again with that note in front of it. A 2023 paper, Reflexion, showed this improves results without retraining the model.
• Delegation. A lead agent hands parts of the plan to sub-agents, each with its own focused context, and combines what they return. It's powerful and costly; see why a crew can beat a single bot.

5. Memory: what an agent knows, and for how long

Because the model forgets everything between calls, memory is something the harness builds. There are two layers.

Short-term memory is the context window: the instructions, the conversation and the tool results of the current job. Researchers call it working memory.

Long-term memory is anything stored outside the model and brought back when it's relevant. A 2023 framework for language agents, CoALA, splits it into three kinds that are easy to remember:

• Episodic: what happened before ("last month the supplier's invoice had the wrong VAT number").
• Semantic: facts about you and the world ("big orders over $500 need a deposit first").
• Procedural: how to do things: instructions, playbooks and code.

How does an agent find the right memory among thousands? Usually by retrieval: before answering, it searches a store of documents or notes and puts the best matches into its context. This is retrieval-augmented generation (RAG), named in a 2020 paper. The search often uses embeddings, lists of numbers that capture what a piece of text means, so a question like "how much is it?" can find a page titled "Pricing" even though they share no words.

Why not just put everything in the context window? Because more isn't better. A 2023 study found models use information best when it sits at the start or end of a long input, and noticeably worse when it's buried in the middle. Anthropic calls the wider effect "context rot": as the context grows, the model's ability to recall what's in it drops. Choosing what goes in front of the model, now often called context engineering, is a large part of building a good agent.

6. Guardrails: the rules that don't depend on the model

You can tell a model "never delete anything" in its instructions, and it will usually listen. Usually is the problem. Instructions are advice to the model. Guardrails are limits enforced by the harness, whatever the model writes.

The ones that matter most:

• Least privilege. The agent gets only the tools and access the job needs: read-only database credentials for a reporting job, drafts but no send for an email job.
• Approval gates. Actions that are risky or hard to undo pause until a person says yes. OpenAI's guide suggests rating every tool low, medium or high risk by whether it writes, whether it can be undone, what permissions it needs and what money it touches.
• Budgets and step limits. A cap on spend and on steps per job.
• Checks on what goes in and out. Filters on inputs (blocking known attacks) and outputs (catching leaked personal data).
• Sandboxes. Running the agent's code in an isolated machine, so a bad command hurts nothing important.
• Logs. A record of every tool call, result and approval.

Why so strict? Because an agent reads text written by other people: web pages, emails, documents. Any of it can contain prompt injection, instructions planted to hijack the agent. OWASP puts it first in its Top 10 risks for LLM applications, and advises enforcing permissions in the systems themselves rather than trusting the model to police itself. We cover the attacks and defences in AI agent security and prompt injection, and the approval side in human in the loop AI.

7. Cost: why agents use more tokens than chat

Models are billed per token, for what's sent in and what comes out. A chat answer is one call. An agent job is many calls, and each one re-sends the growing context.

Here's the arithmetic with simple made-up numbers. Say a job starts with 2,000 tokens of instructions and goal, and each step adds 1,500 tokens of tool results and reasoning. Step ten alone sends 15,500 tokens. All ten steps together send 87,500, because every step re-reads everything before it. Tool descriptions add more: OpenAI notes they count against the context limit and are billed as input tokens.

Real measurements back this up. Anthropic found that agents typically use about four times as many tokens as chat, and multi-agent systems about fifteen times.

Two things keep it reasonable. Prompt caching lets providers bill repeated, unchanged text at a discount (OpenAI says up to 95% off for cached input). And budgets stop a single job from running away.

What this means when you choose an agent

Knowing the parts gives you better questions to ask any agent product:

• Tools: What can it reach, and with what permissions?
• Approvals: What does it do alone, and what does it ask about first?
• Memory: What does it remember about me, and can I see or change it?
• Budget: What's the spending cap per job, and what happens when it's hit?
• Logs: Can I see every step it took, not just its summary?
• Secrets: Does the model ever see my passwords?

Here's how crew bot answers them. You hand a job to the Lead, which plans it and splits it between specialists who work at the same time; a skeptic attacks every finding before it reaches you. Each seat uses a model that fits the job, from several AI labs. The crew remembers how you like things done, so you explain a job once. It asks before sending an email, changing an account or running anything that writes, unless you've told it to go ahead. Every job has a budget, fifty US cents by default, and stops to ask if it needs more. Logins are encrypted and never shown to a model, and each job keeps a record of what it read, clicked, ran and found. Plans and limits are on the pricing page.

For definitions of everything above in one place, see the AI agent glossary.

FAQ

Do AI agents actually think?

They reason in text: the model writes out its reading of the situation and the next step, and that written reasoning shapes what it does. Whether that counts as thinking is a philosophical question. Practically, it means agents can plan, notice errors and change course, and also that they can reason their way confidently to a wrong answer, which is why checks matter.

Do AI agents remember everything?

No. The model itself remembers nothing between calls. An agent remembers what its harness stores and brings back: the current job's context, plus long-term notes such as your preferences or past results. Good products let you see and edit what's remembered.

What's the difference between an LLM and an AI agent?

An LLM is the model: text in, text out. An AI agent wraps that model in a loop with tools, memory and rules, so it can take actions toward a goal and check the results. The LLM is the engine; the agent is the whole car, brakes included.

Why do AI agents make mistakes?

Mostly for three reasons: the model can state wrong things confidently, an early wrong step compounds through later ones, and the agent can lack context a person would have. Tool checks, a skeptical reviewer, approvals before important actions and a step log catch most of it.

How much does it cost to run an AI agent?

It depends on the model, the number of steps and how much text each step carries. Simple jobs cost fractions of a cent; long research or multi-agent jobs cost far more, because each step re-sends the growing context. Always look for a per-job budget so one job can't run away with your money.