Agents
Getting things done
A can chat with a user in text. That's useful, but like ChatGPT and Claude can do a lot more. They can use , search the web, remember things from the past, and use . This chapter is about how that works.
Everything in this chapter is a version of the same trick. The writes a hidden message that is not addressed to you, and something other than the user answers it with a hidden message to the model.
Tools
Humans are tool users. We use tools to help us think (e.g. a calculator), or to look up information (e.g. a search engine), or to do things (e.g. sending an email). The (Chapter 11) lets a model use tools in a similar way.
A hidden message the user can't see tells the model what tools it can use. When the model wants one, it writes a tool call: a piece of structured text naming the tool and its arguments. When the harness sees a tool call it runs the tool, and sends a hidden message back to the model with the result.
That loop repeats until the model writes an ordinary reply instead of another tool call.
Search is a particularly important tool. The model's don't tell it about anything that happened after it was and may lack niche knowledge that you care about, so the model needs to rely on web search, and other information lookup tools, to give in the knowledge it needs.
Skills
A skill is a tool whose result is a page of instructions. They are are essential way that we teach models to do new things and customize their behavior.
Loading every skill in full in every conversation would be a waste of space. Providing a list of skills and a tool to fetch them on demand makes each skill available only when needed.
Memory
If we want an agent to remember something from a previous conversation, we can give it a tool for saving notes and a tool for searching the notes it saved before.
Products differ in how memories are fetched. Some expect the model to search for memories itself (as in this example). Some have the harness automatically search for memories relevant to the conversation. Either way, what arrives as a hidden message in the conversation.
Putting It Together
Tools, skills and memory are the same mechanism pointed at three different problems. The model writes a message the human never sees, something on the other end answers it, and the reply becomes part of the text the model is predicting from.
The model has not gained an ability to act. It has gained a correspondent.
Try it in PyTorch — Optional
Write the harness yourself. About thirty lines of Python give a small model working tools, a skill it can load, and a memory that outlasts the conversation.