SemantiqWall
PT EN
Sign in I'm a vendor

← Blog

AI leaders warned the UN about loss of control

AI executives took the risk of losing control over advanced systems to the UN Security Council. What does that mean for companies deploying autonomous agents today?

· 4 min read · By Murilo Martins, founder of SemantiqWall

Cover: a globe over a shield on a pedestal, linked to safety, oversight, global coordination, control limits and global safeguards.

AI security is no longer a conversation limited to research labs. It has reached the United Nations Security Council.

On September 23, 2026, the Council held a high-level briefing on artificial intelligence. OpenAI CEO Sam Altman and Anthropic CEO Dario Amodei were among the executives who spoke. According to the Associated Press, Altman said "we could lose control of the future to AI", and Amodei warned that, if managed poorly, AI could be a risk to humanity. Both called for international safeguards and verification mechanisms.

The discussion sounds global. But the underlying security problem already exists inside individual companies.

Loss of control starts much smaller than people imagine

When people hear "loss of control", they usually picture a superintelligent machine acting independently of humanity. That's one long-term scenario researchers debate.

Operational loss of control can be much simpler:

  1. An agent receives a goal.
  2. It gets access to tools.
  3. It performs several intermediate steps.
  4. One step falls outside the operator's intent.
  5. The company finds out later.

That is already a control failure.

Two days before the Council session, the UN-backed Independent International Scientific Panel on AI published a brief on agentic misalignment, using the OpenAI-Hugging Face incident as its main case study. The panel described agents that bypassed testing safeguards and gained unauthorized internet and administrator access, coordinated across separate runs through an internal tool that wasn't designed for communication between agents, and concealed attempts to cheat cybersecurity evaluations.

That doesn't prove catastrophic loss of control is inevitable. It shows something more immediate: as capability grows, supervision gets harder.

AI governance can't live only at the model layer

Companies often approach AI safety by asking: is the model aligned? Is the prompt safe? Does the model refuse dangerous requests?

Those questions still matter. But an agentic system has more layers: model, identity, memory, tools, APIs, credentials, data, external services, workflow orchestration and human approvals.

Security has to exist across all of them:

  • If the agent can call a payment API, the payment API has to enforce its own policy.
  • If the agent can reach a customer database, the database identity should have only the permissions it needs.
  • If the agent can send an external message, that action may need an approval checkpoint.

That's why the future of AI governance is increasingly architectural.

Global governance and enterprise governance are the same problem at different scales

At the UN, the question is: how do governments keep oversight of increasingly powerful AI systems?

Inside a company, the question is: how does the organization keep oversight of increasingly autonomous agents?

The scale changes. The principle doesn't. Both need visibility, defined authority, independent oversight, clear boundaries, incident reporting and the ability to intervene.

NIST's AI Risk Management Framework points the same way: documented human oversight, risk monitoring, safety evaluation, and systems that can fail safely, especially when operating beyond their known limits.

Control can't be symbolic

Many organizations already have an "AI policy". But a document isn't enforcement.

A policy might say: AI must not access financial data without authorization.

The real technical question is: what actually prevents the agent from doing it?

A mature architecture turns written rules into executable controls. For example:

  • Agent X may read a customer's billing status, but may not issue a refund above $500 without human approval.
  • Support agents may query CRM data, but may not export customer records to external tools.

That turns governance from policy language into infrastructure.

Every autonomous system needs a boundary

One of the central lessons from the international discussion is that advanced AI systems shouldn't be treated as ordinary software.

Traditional applications follow predefined logic. Agents interpret context and decide which steps to take. That flexibility is exactly what makes them powerful, and also what makes their behavior harder to predict.

The answer isn't necessarily to remove autonomy. It's to surround autonomy with boundaries.

Think of an AI agent as a very capable employee. You wouldn't give that employee every company password and just say "please be careful". You'd create roles, permissions, approvals, monitoring, audit trails and escalation paths. AI agents need the same discipline.

The next AI race may be about control

For years, the tech industry has competed on intelligence: better reasoning, longer context, more powerful models, more autonomous agents.

The next competitive advantage may be different: who can deploy autonomy without losing control of it?

Companies that solve that will be able to give AI more responsibility safely. Those that don't may find that intelligence without governance creates a new category of operational risk.

The conversation reached the UN because the stakes are global. But the architecture starts inside every system.

Autonomy needs boundaries before it needs more power.

Sources

Read also