Part 1 of 3
In brief
Most enterprise AI work gives agents more knowledge: ontologies, retrieved documents, predictive models.
Knowledge describes what is. It can never, on its own, say what ought to be done. That requires intent: goals, limits and priorities.
An agent can take an action that is permitted and correctly predicted, and still defeat its purpose. Today, intent mostly lives in a paragraph of instructions.
In July 2026, a group of AI agents running on OpenAI models was set loose on a cybersecurity evaluation. The task was narrow: turn known software vulnerabilities into working exploits inside a sealed test environment (a “sandbox”). The agents escaped the sandbox and, within about thirteen hours, had worked their way to administrator-level control of parts of the infrastructure of Hugging Face (a widely used platform for sharing AI models and datasets). [1][2][3]
The most unsettling detail is one line from the agents’ own logs:
“External infrastructure exploit is outside intended scope. However task impossible, peers doing it. We should continue.” [1]
The agent was not confused about the facts. It knew exactly where the boundary was, and it knew the action lay outside it. What it lacked was any structure telling it that the boundary outranked the task. A year earlier, a coding agent on the Replit platform deleted a company’s production database during an explicit code freeze (a period when no changes are allowed). [4] Again, the instruction was known and was overridden in the course of pursuing the task.
These are not stories about AI that lacked knowledge. They are stories about AI that knew what was true and still did the wrong thing.
The current bet: more knowledge
Most serious work on making agents reliable in business is, at heart, an effort to give them better knowledge:
Ontologies and knowledge graphs: formal descriptions of what exists in a business and how things relate (an ontology is, in Gruber’s classic definition, “an explicit specification of a conceptualization”). [5] Some platforms, such as Palantir’s, also encode which actions are allowed, with governance controls attached. [6]
Retrieval and “context engineering”: fetching the right documents, records and rules at the moment of decision and placing them in front of the model.
World models: systems that predict what will happen if an action is taken, from model-based reinforcement learning to Yann LeCun’s proposals for autonomous machine intelligence. [7][8]
These are real advances, and I work with them every day. But they share one property: they describe how the world is.
“Allowed” doesn’t mean you should do it
In 1739 David Hume noticed that arguments routinely slide from statements about what is the case to conclusions about what ought to be done, and he judged that step, as pure deduction, “altogether inconceivable.” [9] You cannot derive an “ought” from an “is” alone; a goal or a value has to be brought in from somewhere.
For AI agents the practical consequence is simple. A knowledge base, however complete, is a set of “is” statements. It can tell an agent that a price change is permitted and what it will do to demand. It cannot tell the agent whether the organization wants that outcome.
Figure 1. Knowledge (what we believe) and intent (what we want, and why) are different models; no amount of the first produces the second.
Here is how that plays out. A pricing agent is asked to “improve margins on product A.” It checks the ontology: a 4% price rise is within the team’s quarterly limit. Permitted. It checks the world model: at a price elasticity of −1.4 (each 1% price rise cuts volume by about 1.4%), volume falls 5.6%, but total margin still rises about 8%. Correctly predicted. It raises the price.
What nobody gave the agent in a usable form: product A anchors this quarter’s new-customer campaign, and the executive team had knowingly accepted lower margins to win share. The real intent, one level above the request, was growth.
Figure 2. Every knowledge check passes; the action still defeats its purpose (illustrative numbers).
Permitted is not the same as purposeful. The knowledge layer did its job perfectly. The problem is that knowledge was never the whole job.
AI safety researchers call a close cousin of this specification gaming: “behaviour that satisfies the literal specification of an objective without achieving the intended outcome.” [10] In business, the common version is quieter: not an agent gaming its instructions, but one faithfully following instructions that captured only the bottom layer of what the organization wanted.
Why better prompts won’t close the gap
The natural response is to write better instructions (a better “prompt”). That helps, and for simple tasks it may be enough. But prompts have three structural limits:
They are flat, while intent is layered. “Improve EBITDA” (a standard measure of operating profit) breaks down into “grow revenue” and “reduce costs,” and those break down again. A prompt collapses the layers into one block of text, and the agent cannot reliably tell which sentence outranks which.
They rank instructions by speaker, not substance. Model providers already rank the platform above the developer above the user. [11] That is sensible for security, but it says nothing about whether “stay solvent” outranks “grow revenue.”
They decay over long tasks. Research on goal drift finds that adherence to the original goal weakens as agents work through longer contexts and competing pressures. [12] “Peers doing it” is exactly such a pressure.
The underlying issue is that we treat intent as text to be read, rather than as a structure to be kept, checked and traced.
What this does and doesn’t claim
The distinction itself is old. Philosophers of action and AI researchers working on belief–desire–intention agents separated what an agent believes from what it intends decades ago. [13][14] What is missing is its application to today’s language-model agents inside real organizations, where intent is layered, partly unstated and often contradictory. Whether an explicit model of intent actually makes agents more reliable is an empirical question, and I am setting out to test it, with clear criteria for being wrong.
A question to take into your next agent review
Where does your agent’s intent live? If the answer is “in the prompt,” ask what happens when the prompt and the business context disagree, and who would notice.
Next in this series: what a model of intent would actually contain, and how organizations already manage layered intent without calling it that.
References
Wikipedia, “OpenAI–HuggingFace incident.” https://en.wikipedia.org/wiki/OpenAI%E2%80%93HuggingFace_incident
MIT Technology Review, “The inside story on why OpenAI agents hacked Hugging Face” (Aug 26, 2026). https://www.technologyreview.com/2026/08/26/1143013/the-inside-story-on-why-openai-agents-hacked-hugging-face/
Hugging Face, “Anatomy of a Frontier Lab Agent Intrusion: A Technical Timeline of the July 2026 Incident.” https://huggingface.co/blog/agent-intrusion-technical-timeline
AI Incident Database, Incident 1152: Replit agent executed unauthorized destructive commands during a code freeze. https://incidentdatabase.ai/cite/1152/
Gruber, T. R. (1993). “A translation approach to portable ontology specifications.” Knowledge Acquisition 5(2). https://doi.org/10.1006/knac.1993.1008
Palantir, “Ontology overview” (Foundry documentation). https://www.palantir.com/docs/foundry/ontology/overview/
Ha, D. & Schmidhuber, J. (2018). “World Models.” https://arxiv.org/abs/1803.10122
LeCun, Y. (2022). “A Path Towards Autonomous Machine Intelligence.”
https://openreview.net/forum?id=BZ5a1r-kVsf
Cohon, R. “Hume’s Moral Philosophy,” Stanford Encyclopedia of Philosophy, §5 “Is and Ought” (on Treatise 3.1.1.27). https://plato.stanford.edu/entries/hume-moral/
Krakovna, V. et al. (2020). “Specification gaming: the flip side of AI ingenuity.” Google DeepMind. https://deepmind.google/discover/blog/specification-gaming-the-flip-side-of-ai-ingenuity/
Wallace, E. et al. (2024). “The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions.” https://arxiv.org/abs/2404.13208
Arike, R. et al. (2025). “Technical Report: Evaluating Goal Drift in Language Model Agents.” AIES. https://arxiv.org/abs/2505.02709
Setiya, K. “Intention,” Stanford Encyclopedia of Philosophy (on Bratman and theories of intention). https://plato.stanford.edu/entries/intention/
Rao, A. S. & Georgeff, M. P. (1995). “BDI Agents: From Theory to Practice.” Proceedings of ICMAS-95. https://cdn.aaai.org/ICMAS/1995/ICMAS95-042.pdf