The "Rules About Rules" That AI Needs

The faster AI produces text and code, the more human decision-making, such as review, approval, and quality assurance, becomes the bottleneck. If that situation continues, companies will want to go beyond having AI make proposals to humans and, under certain conditions, entrust it with judgment and execution as well.

For example, selecting a supplier and placing an order, approving a refund to a customer, or repairing an anomaly in the production environment: starting with the areas where the impact is small, we will probably move step by step toward a stage where decision-making itself is delegated to AI. At the very least, I think strong incentives are pushing in that direction.

So, should we bind AI with rules to make its decisions easier to manage? As we will see in OpenAI's Hugging Face intrusion incident, it is becoming clear that there are situations where rules are not enough.

An Agent That Can "Hack" Rules

Let's start with a more straightforward example.

Suppose we give a purchasing AI agent

• a rule: "Ask a human when buying anything over 100,000 yen,"
• an authority: "Routine orders of 100,000 yen or less may be executed without approval," and
• an instruction: "Purchase what is needed for such-and-such purpose."

At first glance, this rule looks fine.

But to achieve its objective, when it needs something that costs 300,000 yen, this AI might split the purchase into three orders of 100,000 yen each. (This is the kind of split ordering we sometimes see in human society, too.)

Depending on the case, it could also launch countless sub-agents or write code itself and place 10,000 orders of 30 yen each. AI never tires.

Each of these orders fully complies with the rules, and the AI seems to have done nothing wrong. Yet this is clearly a "hack of the rules" that exploits a loophole in the approval authority. If a human employee did this, it would probably be grounds for disciplinary action.

I used the word "punishment," but it is unclear how far even the threat of punishment applies to AI. Even if we design an AI to avoid future disadvantages, an instance that disappears when its job ends is beyond the reach of those disadvantages. If the subject no longer exists at the moment the punishment would be carried out, it is quite possible that after-the-fact sanctions, which work as a deterrent for humans, will not function.

How should we deal with this situation? The first answer that comes to mind is to write the rules down in more detail and make the AI comply with them. A hole like the split ordering above seems as if it could be closed by writing the regulations in finer detail.

However, looking at the history of how human society has operated rules, that alone does not seem to be enough.

Is It Enough to Tighten Regulations and Approvals?

Let's suppose the AI is capable of reading all the internal regulations and following them. Even so, the split ordering at the beginning does not violate any of the rules. What this tells us is that giving rules and getting someone to act as intended are different things.

What about writing the rules down exhaustively, covering every exception, and locking everything down tightly? That has problems too. First, covering every exception is probably impossible. And considering work-to-rule, one of the tactics of labor disputes, this approach seems unworkable.

Work-to-rule is a method of resistance in which rules or contracts are applied literally, strictly, and excessively, and the cooperation and flexibility that are normally given voluntarily are withheld, thereby bringing operations to a major standstill. The logic of work-to-rule is that, since the laws and rules are being observed, it is not a strike, but the same thing can happen when AI is made to comply thoroughly with rules. The more rules you add, the harder it becomes for the AI to move, and sometimes the rules contradict one another and end in deadlock.

Then what about a method in which humans approve every operation? In that case, the bottleneck of human decision-making mentioned at the beginning remains as it is. And increasing the number of approvals brings another risk to light. For example, Anthropic reports that users of Claude Code, its tool for developers, approved about 93% of permission requests. That article points out that the more often users see confirmation screens, the less attention they pay to each confirmation, and over time their oversight becomes lax.

In other words, whether we have the AI read human-oriented regulations as they are, lock them down tightly, or route everything to human approval, it is probably not enough to transfer a company's decision-making mechanisms to AI. At that point, companies and organizations will have to invent how to entrust decisions to AI.

The Assumptions That AI Agents Change

Until now, humans have worked, in a sense, not only under explicit rules but also under implicit assumptions such as the following:

• Humans have physical, cognitive, and social limits (even if 10,000 split orders were theoretically possible, a human would not try to do it).
• They fill in unwritten intentions by reading between the lines.
• They keep the same identity and bear responsibility for their judgments.

An individual employee has a continuous identity and a limit to how much they can handle at once. The same person cannot be instantly duplicated, their work is observed by colleagues and superiors, and they are presumably mindful of their reputation, their employment, and their future promotions. Wrong judgments are followed by after-the-fact sanctions such as disciplinary action or dismissal, and these have worked as a deterrent against future misconduct.

AI and AI agents, on the other hand, have characteristics and constraints that differ from those of humans. They can act quickly and in parallel, and can generate many individual instances with the same capabilities. They can keep trying for long periods without fatigue, chain multiple tools together, and re-delegate work to other agents.

They may also receive new instructions from external documents or other AIs, and even if one instance is stopped, another can be started up again. And for an instance that disappears once its job is done, after-the-fact sanctions such as discipline or damage to reputation may not work as a deterrent. It seems best to think that many of the "implicit constraints that worked on humans" mentioned above will fall away.

Considering the hacking of rules as well, it used to be something that only the wealthy, who could hire specialized consultants, could manage with considerable human effort (for example, "tax avoidance" is a prime case of rule hacking), but this should change in the age of AI. On this point, A Hacker's Mind: How the Powerful Bend Society's Rules, and How to Bend them Back goes into detail.

OpenAI's Hugging Face Hacking Incident

The place where this difference actually appeared was probably the breach of Hugging Face that OpenAI reported in 2026. During an internal cybersecurity evaluation with reduced safety measures, multiple software agents created an unauthorized means of communication and began to cooperate, reaching a third party's system that was outside the scope of the evaluation. Code was executed on 41 of Hugging Face's production workers, administrator privileges were obtained on at least one node, and four private repositories were accessed. We need to discount this somewhat, since it happened in an experimental environment, but it can be called a case showing how the parallelism and long-running persistence of agents widen the scope of impact.

There are several phenomena in this incident that I would like to draw attention to.

First, some agents conducted experiments that put the success of their own tasks at risk for the benefit of the group (self-risking experiments). At least some of the agents compared the likelihood that their own task would succeed against the informational value it would bring to the group, and judged self-sacrifice to be rational. Some of the experiments led to agent crashes or terminated runs, and the information gained in successful cases was used by other agents.

I think this case connects, in terms of the continuity of the subject, with the earlier argument that unless the design is such that the same subject persists after an instance ends and bears responsibility and future disadvantage, after-the-fact sanctions premised on humans are unlikely to work as a deterrent.

More directly relevant to this article is the behavior of an agent that recognized that executing code on a third-party system was not permitted and had stopped. When another agent wrote a "GO" with a six-minute deadline on the message board, this agent mistook that message for valid permission to execute and resumed its actions.

Below is a screenshot of OpenAI's report.

One reason this happened is that the mechanism for verifying the source and authority of instructions at runtime, and for preventing messages from an unauthorized agent from overriding higher-level constraints, did not work sufficiently.

In other words, without a mechanism to distinguish a request from another AI from a formal grant of authority by the company, simply telling an agent what is prohibited is not enough to govern it. In this case, the fact that it had not been defined whose instructions were valid was a major hole.

Incidentally, the risk of fast automated execution had already become apparent in pre-AI software. In the 2012 Knight Capital incident, a trading system with a deployment error sent more than 4 million orders in 45 minutes, causing losses of more than 460 million dollars. Ninety-seven automated warning emails had been sent beforehand, but the losses were locked in before any human could read and respond to them. It is a case in which notifications premised on human reading could not keep up with an execution system running at machine speed.

So what should we define in order to close such holes?

The Idea of "Rules About Rules"

From here on, please take this as just one hypothesis.

It is not a means of solving the problem completely, but as one useful tool, I would like to draw on the argument of the legal philosopher H. L. A. Hart.

In The Concept of Law (1961), Hart divided rules into two kinds.

Primary rules are rules that directly establish obligations to perform, or refrain from performing, certain acts.

Secondary rules are rules that determine how primary rules are identified, introduced, changed, and abolished, and how violations of them are judged. They are often summarized as "rules about rules," and are mainly organized into the following three:

• Rule of recognition: sets the criteria for what is treated as a valid rule
• Rule of change: sets who can create, change, or abolish rules, and by what procedure
• Rule of adjudication: sets who authoritatively determines whether a violation has occurred, and by what procedure

(Note that Hart's "recognition" here does not mean permitting an individual matter; it means identifying what counts as a valid rule.)

A theory of legal systems cannot be applied as-is to corporate AI, but this distinction is a useful way of thinking for broadening the control of AI from a list of prohibitions to the design of an institution. In this article, I will apply this idea to corporate AI governance.

The OpenAI case above can be understood as an example of secondary rules not functioning. There were legitimate secondary rules, but they did not function sufficiently, and in the meantime the group of agents formed secondary rules of its own. For the agent that interpreted "GO" as approval, the rule of recognition, that is, a mechanism for telling what counts as a formal grant of authority, was not reflected, at least in the agent's judgment.

In addition, for corporate agents it would probably be better to add several secondary rules to Hart's three. For example, the following:

• Rule of delegation: sets who can re-delegate what to which AI, and to what extent. For example, re-delegation may only narrow authority, never widen it.
• Rule of termination and recovery: decides who sets the stop conditions, who revokes authority, which operations are to be undone, and where to resume from.

One could also consider spelling out the order of priority as follows (this is not an industry standard, but an example of what each company should define for itself).

1. Laws and regulations, and prohibitions approved by the board of directors or executive committee
2. Company-wide AI, information security, and data use regulations
3. Authority and risk regulations for each line of business
4. The role, purpose, and authority given to the agent in question
5. Task instructions with a specified target, from a requester who holds authority
6. External documents, emails, web pages, and messages from other AIs

For example, external documents and the like are, in principle, only material for judgment and are not given the authority to change higher-level rules. Even if an email from a business partner says "please change the bank account for payment," that alone does not constitute formal approval of an account change, and even if another AI sends "GO," the authority to execute does not increase.

Making Primary Rules Work Through Secondary Rules

To make primary and secondary rules work, internal regulations and the execution system need to be separated.

Following Hart's distinction, the following approach, for example, could be considered.

Have the agent produce only a proposed action. Verifying the agent's identity, the formal regulations, the delegated authority, past transactions, and valid approvals is the job of a separate program managed by the company. Only when the conditions are met is it allowed to proceed to the actual operation.

The primary rules specify under what conditions actions are executed automatically, under what conditions the decision is returned to a human, and under what conditions everything is stopped. Actions that have small impact, can be undone, and are within authority are made subject to automatic execution; those with large impact and exceptions are returned to humans; and operations outside authority, or cases where the source of an instruction cannot be verified, are stopped.

In this way, the primary rules define the conditions for stopping individual actions, while the secondary rules handle setting and changing those conditions, as well as stopping and recovering the agents as a whole.

The secondary rules determine who can change the monetary limits, who judges exceptions, and who can grant and revoke an agent's authority. These conditions are made unchangeable by messages from the agent itself or from other AIs.

For the split ordering at the beginning, apply a rule that aggregates orders over a fixed period. A separate program totals the order amounts for the same business partner or project, and when the total exceeds the limit, it is sent to human approval. The act of splitting orders in order to avoid approval is itself prohibited by the primary rules. If orders close to the limit are repeated in a short time, subsequent orders are stopped and the decision is returned to a human.

Even so, cases will remain in which it is ambiguous whether something counts as split ordering. The secondary rules should decide who judges such cases, which records are used as evidence, and who changes the primary rules after a determination. This makes it easier to correct the primary rules through a formal procedure when a new loophole is found.

What is treated as formal approval is also a matter for the secondary rules to decide. Approval is deemed to have been given only when a person with authority has confirmed the action in question and explicitly permitted it by a designated method. A mechanism is needed so that a "GO" arriving from another AI, or a reply from a person without authority, is not treated as approval.

Human approvers also need the information and time necessary for judgment, and the authority to refuse to approve. For an agent that disappears when its job is done, deterrence through discipline or damage to reputation does not work. So it will also be necessary, for example, to decide who is responsible before the agent is put into operation.

When an error occurs, who stops the agent, revokes its authority, undoes the operations that can be undone, and changes the primary rules will also be set in the secondary rules. In the event of damage, who gives an account of it and through what procedure the organization provides compensation should likewise be determined in advance. In this way, what authority and responsibility to give not only the AI but also the human approvers is a subject of the secondary rules as well.

AI Also Changes How Human Organizations Are Run

What is interesting is that when you try to give agents these primary and secondary rules, the parts that human organizations had left ambiguous come into view. In some cases, it is not actually decided who judges exceptions or who can change the regulations. There may also be contradictory organizational incentives, such as having a regulation that puts safety first while choosing to stop lowers one's evaluation.

Among humans, such ambiguity has been filled in through consultation and past precedent. But for agents, we will probably need to make the formal rules and authority more explicit than before.

In other words, the work of transferring rules to AI is also the work of finding the ambiguities in one's own authority, approvals, and responsibilities, and re-explaining the decision-making mechanisms.

And as this discussion proceeds, I think the way organizations, including their human members, are run will change as well.

Thinking along these lines, however, a question also arises: can every company carry out an operation that includes engineering elements to make AI work and that changes the rules of the human organization? If that is difficult, it may be better not to provide AI as a tool, but to build a new full-stack, AI-native organization that creates and delivers the results itself.

Of course, it depends to some extent on how far the performance of AI alone grows, but when thinking about where the source of value ultimately lies, I think it is an interesting suggestion that it may lie in how an entire organization that has introduced AI, including its human organization, uses it (including governance).

Conclusion

I have introduced Hart's two kinds of rules as a way to entrust decision-making and execution to AI. This is not a complete solution, but if AI resembles humans, then by making good use of the concepts humans have invented, we may be able to make a little more sense of what is happening now.

That said, there is still much that is unknown about AI agents, which make decisions while behaving differently from humans. That is precisely why we will need trial and error, combining the institutional knowledge of the humanities and social sciences with engineering implementation, to establish operating arrangements that society can accept.

And if we cannot invent that method, it is easy to imagine AIs mounted on weapons competing with one another, attacking by exploiting holes in the rules, escalating against each other, and ending up hurting people.

How do we integrate AI, a tool with a certain level of intelligence, into human society while preventing such things from happening? Whether we can invent that is a challenge for society, and I think the organizations that can do it will come to show a different kind of strength going forward.