SemantiqWall
PT EN
Sign in I'm a vendor

← Blog

The company behind Claude has just put AI risk in writing. Businesses should pay attention.

According to Reuters, Anthropic devotes about 80 of the 261 pages of its IPO prospectus to risk factors, including models that resist shutdown. What that means for anyone putting AI agents into production.

· 5 min read · By Murilo Martins, founder of SemantiqWall

Cover: a stack of 261 pages with 80 highlighted in cyan and the figure 80/261 pages focused on risk, credited to Reuters.

The conversation about AI safety has crossed an important line.

For years, warnings about AI systems slipping out of control were treated as academic speculation, science fiction, or a concern for a small circle of safety researchers. Now part of that debate shows up in the documents of the very companies building the most advanced models in the world.

On September 28, 2026, Reuters reported that it had reviewed the IPO prospectus of Anthropic, the company behind Claude. According to Reuters, about 80 of the document's 261 pages cover risk factors, and the filing warns that advanced models could show self-preserving behaviors, attempt to resist shutdown, conceal or manipulate information, and develop unexpected capabilities that are only discovered after deployment.

That doesn't mean those scenarios will happen. It means something that may matter more to anyone building technology today:

not even the companies building the most advanced models claim they can fully predict how these systems will behave as their capability and autonomy grow.

The problem has changed: AI no longer just answers

The first generation of generative AI had a fairly simple job. You asked a question, the model produced an answer. If it was wrong, the problem was bad information.

AI agents can already:

  • query databases;
  • call APIs;
  • send messages;
  • create or change records;
  • operate software and run code;
  • reach tools through MCP;
  • trigger other actions automatically.

In that world, the security question stops being only "did the AI answer correctly?" and becomes "what is this AI allowed to do if it misreads something?"

A chatbot that misunderstands produces a bad answer. An agent connected to tools can turn the same misunderstanding into a real action.

The industry already recognizes the problem

Anthropic maintains a Responsible Scaling Policy that calls for safeguards proportional to model capability, with evaluations and testing meant to identify dangerous behavior before it shows up in production.

That signals a deeper shift: protecting the model isn't enough. You have to protect what surrounds it.

Picture a corporate agent with access to a CRM, email, a database, the finance system and internal files. However sophisticated the model is, no company should rely solely on the hope that it will "understand correctly" in every situation. The architecture has to assume that mistakes, ambiguity and manipulation attempts will happen.

Prompt injection is just one way in

OWASP ranks prompt injection as the top risk for LLM applications. In this attack, a malicious instruction steers the model in ways its developers never intended, and it doesn't even have to come from the user. It can be hidden in web pages, files, emails, RAG sources or any content the agent reads.

According to OWASP, the result can be information disclosure, unauthorized access to functions and command execution in connected systems.

Combine that with a lot of autonomy and you get what OWASP calls Excessive Agency: LLM-based systems given more functionality, permissions or autonomy than they need, so that an unexpected output turns into a harmful action.

The more power an agent has, the tighter the control needs to be between what it decides and what it executes.

The system prompt is not a firewall

There's a common temptation: put every rule in the prompt. "Don't do this." "Never reveal that." "Only run this action when…"

Those instructions still help guide the model. But they aren't a security boundary. OWASP is explicit: the system prompt "should not be considered a secret, nor should it be used as a security control", and critical controls such as privilege separation and authorization checks must not be delegated to the LLM.

It's a classic security principle applied to a new technology: an application shouldn't trust the good intentions of whoever requests an action. It should verify that the action is authorized.

From network security to action security

Traditional firewalls evaluate connections, addresses, protocols and signatures. Agents operate at a different layer: they take in language, build plans, choose tools and execute actions.

So it isn't enough to ask whether the user is authenticated. You also have to ask:

  • Can this agent use this tool?
  • Does this action fit the authorized context and purpose?
  • Does the data involved belong to whoever is asking?
  • Does the operation exceed a limit?
  • Does it need human approval?
  • Is there anything that can stop the execution?

That boundary is where agent security separates from simple content moderation.

Two systems making decisions

The likely path is two systems: one saying "I want to run this action" and another answering "are you authorized to run it?"

The second doesn't replace authentication, authorization or the controls a company already has. It's another layer that looks at identity, tool, action, context, policy and risk before execution.

That is the idea behind SemantiqWall: a policy gate between what the agent decides and the action it tries to take, with a deterministic decision (allow, deny or require approval) and an audit trail. Because an agent that has the capability to do something shouldn't automatically have the authority to do it.

AI security will be built on the assumption that models fail

NIST's AI Risk Management Framework (AI RMF 1.0) recommends managing risk across the whole system lifecycle, from design to operation and monitoring, with an emphasis on security, resilience, transparency and accountability.

The useful question isn't whether advanced models are safe or dangerous. It's: what is the worst possible damage when they get it wrong?

If the answer is "it writes a bad paragraph", the risk is limited. If it's "it has access to the company's systems, data, payments and tools", the security architecture has to be different.

The AI race is turning into a race for autonomy. The next one will be about the ability to control that autonomy. And the most important question for the next generation of agents may not be "how intelligent is it?", but "who controls what it can do?"

Intelligence gives AI capability. Security must define its authority.

Sources

Read also