In a rare moment where the language of faith and the language of technology seem to point at the same danger, Pope Francis has renewed calls for restraint in the development of advanced artificial intelligence. The message is moral, urgent, and deliberately framed as a matter of human responsibility: if powerful systems can be built, then societies must also decide—before it’s too late—how they should be governed.
But the argument runs into a stubborn obstacle that has little to do with ethics and everything to do with incentives. In the AI race, “restraint” is not just a belief; it is a strategic choice made under uncertainty, time pressure, and competitive rivalry. And when multiple actors face those conditions, game theory predicts something uncomfortable: even widely shared goals can fail to coordinate behavior. Appeals for safety may shape public debate and policy priorities, but they cannot automatically rewrite the underlying logic of competition.
That tension—between what leaders want and what rational actors do—is increasingly central to how analysts interpret AI risk. It’s not that safety crusades are meaningless. It’s that they operate in a world where commitments are hard to verify, delays can be punished, and “safety” can become entangled with bargaining rather than treated as a universal constraint.
A moral appeal meets a strategic environment
The Pope’s intervention arrives at a time when AI capabilities are advancing quickly and when governments are struggling to keep pace with both the benefits and the harms. The call for restraint is, in essence, a call for coordination: slow down, set guardrails, and ensure that the most consequential systems are developed responsibly rather than recklessly.
Coordination is exactly what game theory says is difficult when actors cannot trust each other’s intentions or measure each other’s compliance. If one developer believes others will continue pushing forward, then slowing down can look like self-sabotage. Even if everyone agrees in principle that safety matters, the question becomes: who pays the cost of restraint, and who captures the advantage of moving first?
In many competitive industries, the incentive structure is already familiar. In AI, it is amplified by the speed of iteration, the scale of investment, and the fact that model improvements can compound. A delay is not merely a missed quarter; it can mean falling behind on research momentum, talent acquisition, compute access, and market positioning. When the payoff for being first is large enough, restraint can become the “dominated” strategy—meaning it is rational only if others restrain too.
This is the core problem analysts point to: moral persuasion can change what people say, but it may not change what people do if the strategic environment rewards acceleration.
The “race” dynamic: why slowing down can trigger faster action
One of the most intuitive game-theoretic mechanisms at work is the fear of being left behind. Suppose two major labs—call them Lab A and Lab B—both claim to support safety. They also both understand that building advanced systems carries risks. Now imagine Lab A chooses to slow down while Lab B continues.
If Lab B’s continued progress yields commercial advantage, political influence, or technical breakthroughs, Lab A’s restraint becomes costly. Over time, Lab A may feel compelled to catch up, not because it rejects safety, but because it cannot afford to lose the contest. The result is a feedback loop: restraint by one party increases pressure on the other, which then reduces the credibility of restraint overall.
This is not a moral failure. It is a predictable outcome of competitive incentives under uncertainty. When the “best response” to restraint is acceleration, restraint becomes unstable. Even if all parties would prefer a safer equilibrium, the path to reach it can be blocked by the fear that others won’t cooperate.
The Pope’s appeal, like many safety proposals, implicitly assumes that shared concern can translate into shared action. Game theory suggests that without enforceable coordination, shared concern may not be enough. The race dynamic turns safety into a fragile promise: it holds only as long as everyone believes others will keep their word.
Verification is the missing ingredient
Another reason appeals struggle to reshape outcomes is that trust is difficult to verify. In theory, restraint could be coordinated through voluntary commitments: labs agree not to train certain models, not to deploy high-risk capabilities, or not to scale beyond specified thresholds.
In practice, verification is hard. What counts as “advanced”? Which training runs are covered? How do you measure internal progress without revealing sensitive information? How do you confirm that a lab truly paused rather than shifted work into adjacent areas? Even if a lab intends to comply, external observers may lack the tools to confirm compliance reliably.
Game theory treats this as a credibility problem. If commitments cannot be verified, then promises become cheap talk—statements that sound responsible but do not constrain behavior. In such environments, rational actors may still choose strategies that maximize their expected payoff, because the cost of being caught is low or ambiguous.
This is why many policy frameworks emphasize not only principles but also mechanisms: audits, reporting requirements, technical evaluations, and enforcement capacity. Without those, safety becomes a negotiation posture rather than a binding constraint.
The “safety as leverage” risk
There is a further complication that often goes unmentioned in public discussions: in competitive settings, safety can become bargaining leverage. If one actor signals restraint, it may gain political goodwill, regulatory attention, or negotiating power. Another actor might interpret the signal as an opportunity to extract concessions—such as favorable terms, slower regulation for competitors, or privileged access to markets.
In other words, safety can be used strategically even by those who genuinely care about risk. The problem is not hypocrisy alone; it is that the environment rewards signaling. If the benefits of appearing cautious are high while the costs of actual restraint are uncertain, then actors may treat safety commitments as part of a broader strategy.
This does not mean safety efforts are fake. It means that in a multi-actor system, the same action can serve two purposes: reducing risk and improving relative position. When those purposes conflict, the system may drift toward outcomes that look like “progress on safety” while still allowing risky acceleration.
The result is a paradox: the more safety is discussed as a competitive differentiator, the less it may function as a universal brake.
Why “public debate” isn’t the same as “behavior change”
It would be unfair to reduce the Pope’s appeal to a symbolic gesture. Moral leadership can matter. It can shift public expectations, influence lawmakers, and create political pressure for stronger regulation. It can also encourage internal governance within companies—boards, ethics committees, and risk teams that push back against reckless timelines.
But the leap from debate to behavior is not automatic. Public discourse changes the political landscape; it does not necessarily change the payoff structure for firms competing in real time. If the market rewards speed and capability, then even a strong moral narrative may not overcome the incentives to move quickly.
This is why analysts often distinguish between “norm-setting” and “coordination.” Norm-setting can be slow and diffuse. Coordination requires concrete agreements, monitoring, and credible enforcement. Without those, the system may continue to produce the same strategic outcomes even as rhetoric becomes more cautious.
A unique take on the “logic” of AI risk
The phrase “AI race’s risky logic” can sound abstract, but it describes a very specific pattern: when multiple actors pursue high-stakes innovation under uncertainty, the system tends to converge on the fastest path unless there is a stabilizing mechanism.
That mechanism could be regulation, but regulation itself faces the same verification and enforcement challenges. It must define what is prohibited, how compliance is measured, and what penalties apply. It must also be internationally coordinated enough that one jurisdiction’s strictness doesn’t simply redirect activity elsewhere.
Alternatively, coordination could come from industry agreements, but those again require credible monitoring and mutual trust. If enforcement is weak, voluntary restraint can collapse under competitive pressure.
So the “logic” is not simply that people are reckless. It is that the system is designed—by incentives, not intentions—to treat delay as a liability. Safety becomes a public good that individuals may underprovide unless the environment makes cooperation individually rational.
This is the classic collective-action problem, dressed in modern technology.
What restraint could look like if it were actually stable
If the goal is to make safety commitments more than rhetoric, then the question becomes: what kind of restraint is robust to strategic behavior?
One approach is to focus on measurable milestones rather than vague promises. For example, instead of “we will be careful,” agreements could specify concrete constraints tied to evaluation results: limits on deployment until certain safety tests pass, or restrictions on scaling until independent assessments meet predefined criteria.
Another approach is to create shared verification infrastructure. If multiple actors can rely on common testing standards and third-party audits, then compliance becomes easier to observe. That reduces the credibility gap that fuels the race dynamic.
A third approach is to align incentives so that restraint is rewarded rather than punished. This could involve procurement policies that favor safer systems, liability regimes that penalize negligence, or regulatory pathways that grant faster approval to labs that demonstrate compliance. When the payoff for restraint improves, the equilibrium shifts.
None of these solutions are simple. But they share a common theme: they convert safety from a moral appeal into a structured constraint.
The role of governments: not just rules, but enforcement capacity
Governments are often portrayed as the natural answer to AI risk, but enforcement capacity is the bottleneck. Rules without monitoring are easily circumvented. Monitoring without international coordination can be evaded by shifting development to less regulated jurisdictions. International coordination without political will can stall.
The Pope’s appeal can help generate political will, but it cannot supply enforcement capacity. That requires budgets, technical expertise, legal authority, and cross-border cooperation.
It also requires clarity about what risks are being targeted. AI risk is not one thing. It includes misuse, accidents, systemic bias, security vulnerabilities, and the possibility of emergent capabilities that exceed expectations. Different risks may require different controls. A one-size-fits-all restraint may be either too weak or too disruptive.
Game theory again offers a useful lens: if controls are poorly designed, actors may