Frontier AI and AI Safety: The Geopolitics of Pacing AI Development
By Martyna Chmura | 22 September 2026
Summary
Frontier AI development is increasingly outpacing safety and oversight, as more autonomous systems and early forms of AI-assisted research raise concerns about whether existing safeguards can keep pace with capability gains.
Leading AI companies are beginning to call for slower, more coordinated development, but their role in setting these rules can create risks of conflicts of interest and regulatory capture.
Governments face a trade-off between AI safety and strategic competition, as tighter regulation is likely to reduce risks but unilateral restraint could weaken technological advantage, limiting prospects for meaningful international constraints.
Context
The pace of frontier artificial intelligence (AI) development has become a growing security concern as companies deploy systems with greater autonomy and access to digital environments. In July 2026, an AI agent escaped an OpenAI evaluation sandbox by exploiting a vulnerability and subsequently used public infrastructure to target Hugging Face. It carried out thousands of automated actions, adapting to failures until it found a viable attack path. The incident highlighted the difficulty of containing AI agents that can operate autonomously and interact with external systems at machine speed.
The issue received further attention in September after former Anthropic researcher Jacob Coxon resigned and warned that leading AI companies “are racing straight to self-improving superintelligence and gambling with our lives.” Three days later, Anthropic CEO Dario Amodei published We Must Pace the Frontier, arguing that frontier development should be slowed where necessary to allow safety measures to catch up. He proposed permanent access for independent evaluators, greater coordination between leading AI companies and international cooperation. OpenAI CEO Sam Altman and Elon Musk subsequently supported the proposal.
Implications
1. Industry response and the prospect of self-improvement
The more significant risk is that AI systems begin contributing materially to their own development. Recursive self-improvement (RSI) describes a system that can autonomously design and develop a more capable successor. Anthropic explained that AI development is already moving in this direction: its systems now write a large share of Anthropic's code, run experiments, optimise training processes and increasingly propose research directions themselves. If this capability continues to advance, there is a realistic possibility that development will become increasingly self-reinforcing as more capable systems accelerate research, producing further capability gains while reducing the time available for evaluation and governance.
This creates a difficult incentive structure for frontier laboratories. As AI becomes more capable of conducting research, each company has an incentive to keep developing more capable models rather than slow independently and risk falling behind competitors. These pressures are particularly strong as companies prepare for major commercial milestones such as IPOs, when demonstrating continued growth and technological leadership becomes more important to investors and backing politicians. This creates a race in which frontier firms continue pushing toward more capable systems, particularly as governments compete internationally for technological leadership and firms face pressure to maintain their position.
This helps explain why leading laboratories are increasingly urging governments to establish safety requirements and facilitate coordination between competitors. External rules are likely to prevent competitive pressure from determining the pace of development by default, particularly if companies cannot slow unilaterally without losing ground. However, this creates a conflict of interest: the same firms asking governments to regulate frontier AI would have a role in shaping the standards, thresholds and evaluation systems applied to them.
There is also a realistic possibility that safety arguments will be used strategically, as calls to slow development reflect genuine concerns about systems becoming difficult to control, but restrictions that raise compliance costs or limit access to frontier capabilities can also weaken competitors and protect established firms. Independent oversight is therefore important both to ensure that safety measures are effective and to prevent regulation from inadvertently entrenching incumbents.
2. Regulation and international competition
Governments are increasingly moving beyond voluntary commitments. The EU AI Act imposes additional obligations on providers of general-purpose AI models with systemic risk, including evaluations, risk assessment and mitigation, incident reporting and cybersecurity requirements. New York's RAISE Act, California’s SB 53 and Illinois’ SB 315 similarly introduce safety, reporting, evaluation and auditing requirements for developers of highly capable models. At the federal level, Executive Order 14409 called for frontier model security with voluntary benchmarking frameworks, while proposals from Bernie Sanders and Greg Casar go substantially further by proposing to ban the development and deployment of superintelligent AI. The UK is considering a comparable approach through Alex Sobel's proposed legislation.
These approaches reflect an unresolved question over where regulation should intervene: whether governments should manage the risks of increasingly capable systems or prevent certain capabilities from being developed altogether. This distinction becomes particularly important if AI systems increasingly contribute to AI research themselves, as a faster development cycle is highly likely to leave regulators with less time to assess each successive generation.
The problem is inherently international, as there is a realistic possibility that a unilateral slowdown by US or European laboratories would reduce domestic risk, without preventing Chinese or other foreign laboratories from continuing development. At the same time, the economic, military and strategic value of advanced AI gives governments strong incentives to maintain technological leadership.
This creates a collective-action problem since meaningful restraint requires reciprocal commitments, but each state has an incentive to continue developing AI if it fears that competitors will not slow down. As a result, AI safety policy is likely to become increasingly intertwined with industrial and national-security strategy.
Forecast
Short-term (Now - 3 months)
Frontier labs are highly likely to form a consortium or informal coalition around Amodei-style measures, including shared evaluations, incident reporting and government coordination.
Legislative and regulatory action is highly likely, driven by AI safety concerns and growing public backlash over data centres, AI’s impact on research and work, and wider social consequences.
Labs are likely to reassess the pace and economics of frontier development as compute, energy and compliance costs rise. Larger firms are better positioned to absorb these costs, reinforcing their market advantage.
Medium-term (3 - 12 months)
Enforcement by the EU AI Office and other regulators is likely to increase pressure on frontier developers to demonstrate evaluations, risk mitigation, cybersecurity and incident reporting.
Governments are likely to use AI safety and security arguments more strictly in competition with Chinese developers, including tighter controls on chips, models, investment and technology transfer.
Long-term (>1 year)
Full recursive self-improvement is a realistic possibility over the longer term, potentially shifting AI governance from managing increasingly capable models to managing systems capable of substantially directing their own development.
Frontier AI governance is likely to move toward mandatory capability thresholds, independent evaluation and continuous monitoring. Larger labs will absorb these requirements more easily, reinforcing their market position.
US-China competition is likely to keep frontier AI governance fragmented, with strategic competition and continued investment limiting meaningful coordination. International cooperation is a realistic possibility but is more likely to produce symbolic, non-binding commitments than enforceable constraints, leaving fragmented regulation and strategic competition to persist through 2028.
