There’s a question I keep coming back to, late at night, scrolling through the latest round of AI safety resignations, open letters, and corporate press releases about “responsible development.” The question isn’t whether AI systems are dangerous — that debate, at this point, feels almost quaint. The question is simpler and far more uncomfortable: why would the people building the most powerful AI systems on Earth have any real incentive to make them safe?
I don’t mean this as provocation. I mean it as structural analysis. Because once you look at the actual incentive architecture — the capital flows, the competitive dynamics, the labor markets, the geopolitical pressures — you start to see something that looks less like a bug in the system and more like the system working exactly as designed.
The incentive trap is not subtle
Let’s start with the obvious. The companies building frontier AI systems — OpenAI, Google DeepMind, Anthropic, Meta AI, xAI — are locked in what the industry itself sometimes calls a “race.” That metaphor isn’t incidental. A race implies a finish line, competitors, and — critically — a cost to slowing down. When you’re in a race, safety isn’t a feature. It’s friction.
This isn’t speculation. It’s visible in the organizational decisions these companies make. OpenAI dissolved its Superalignment team — the team explicitly tasked with ensuring that superintelligent systems remain aligned with human values — after key researchers, including co-lead Jan Leike, resigned and publicly stated that “safety culture and processes have taken a backseat to shiny products.” That was May 2024. The company’s valuation at the time was somewhere north of $80 billion. By early 2025, it was reportedly approaching $300 billion.
Think about what that means structurally. The market did not punish OpenAI for dismantling its safety infrastructure. It rewarded it. Or more precisely — it rewarded the speed and product aggressiveness that dismantling safety enabled. The signal to every other lab was unmistakable: the capital markets do not price safety. They price capability and speed to deployment.
Google DeepMind, for its part, published a framework for approaching AGI safety in late 2024 — a document that is genuinely thoughtful and worth reading. But the framework exists in tension with Google’s own competitive posture. Gemini models are being rushed into products across the Google ecosystem. The safety framework is aspirational; the product timeline is contractual. When aspiration meets obligation, obligation wins. Every time.
Anthropic — the company most explicitly founded on safety-first principles, by researchers who left OpenAI over precisely these concerns — has raised over $7 billion in funding. Its investors include Google and Amazon. I respect what Anthropic is trying to do. But I also understand what $7 billion in venture capital expects. It expects returns. And returns in AI come from deployment, from market share, from speed. Not from caution.
What the capital structure actually tells us
I spend a lot of time thinking about what capital structures reveal about real priorities — as opposed to stated priorities. And the capital structure of frontier AI development tells a very clear story.
The labs need enormous amounts of compute. Compute requires chips. Chips require fabrication facilities. Fabrication requires rare earth minerals and geopolitically sensitive supply chains. This is why trade policy and AI development are far more entangled than most people realize — the recent intensification of rare earth export controls and stock market volatility around semiconductor companies isn’t separate from the AI safety conversation. It is the AI safety conversation, refracted through the lens of geopolitics and industrial policy.
When you need hundreds of millions — sometimes billions — of dollars just to train a single model, your dependency on capital providers isn’t a side note. It’s the main text. And capital providers have a very specific set of preferences: they want moats, they want speed, they want competitive advantage. Safety, in this framework, is either a marketing strategy or a cost center. It is almost never the product itself.
This creates what I’d call an incentive inversion. The people with the deepest understanding of the risks — the researchers inside the labs — have the least structural power to act on that understanding. The people with the most structural power — the investors, the board members, the executives setting product timelines — have the least visceral understanding of the risks. And the people who would bear the consequences of things going wrong — the rest of us — have essentially no seat at the table at all.
I’m not writing from a position of purity here. I run a media company. I use these tools. My team uses these tools. I’m embedded in the same ecosystem I’m analyzing. But that’s precisely why this matters to me. The question isn’t whether to participate. The question is whether we understand the structure we’re participating in.
The labor market as a safety valve — that doesn’t work
One theory I used to find compelling was the idea that the AI safety community could serve as an internal check on the labs. That talented researchers, motivated by genuine concern about existential risk, would choose to work at companies that prioritized safety — and that this talent pressure would, over time, discipline corporate behavior.
The evidence suggests this theory has failed. Not because the researchers aren’t sincere — they are, often profoundly so — but because the labor market for AI talent has become so intensely competitive that it overrides safety preferences. A New York Times investigation revealed that AI researchers at top labs are being offered compensation packages in the tens of millions of dollars. At those numbers, the labor market doesn’t function as a safety valve. It functions as a capture mechanism.
And there’s a subtler dynamic at work too. Researchers who care about safety often convince themselves that they can do more good inside the lab than outside it. This is sometimes true. But it also means that the most safety-conscious people are frequently the ones most willing to stay quiet, to accept compromises, to defer to organizational timelines — because they believe their presence matters more than their voice. The result is a form of self-silencing that looks, from the outside, like consensus.
When someone does speak up — when they resign publicly, when they write the open letter, when they break the NDA — they’re often treated as an outlier. A disgruntled former employee. A pessimist who couldn’t hack the pace. The structural incentives that created their dissent are rarely examined. The person is interrogated; the system is not.
What this reveals about the real structure of tech
Here’s what I think the AI safety crisis actually reveals, and it’s something that extends far beyond AI: the tech industry, as currently structured, is incapable of self-regulation on any issue where self-regulation conflicts with competitive advantage.
This isn’t a new observation. We saw it with social media and mental health. We saw it with data privacy. We saw it with gig economy labor practices. In every case, the industry told a story about responsible innovation, about self-governance, about moving fast but not breaking things — or at least not breaking important things. In every case, the structural incentives won.
But with AI, the stakes are categorically different. We’re not talking about teenagers spending too much time on Instagram — though that’s serious enough. We’re talking about systems that, within a few years, may be capable of autonomous action in domains that include scientific research, military strategy, financial markets, and critical infrastructure. The gap between what these systems can do and what our governance structures can manage is not closing. It’s widening. Rapidly.
What the AI safety crisis tells us about the “real structure” of tech is that the industry’s governing logic — move fast, capture markets, return capital — is not a strategy that can be reformed from within. The people who understand this best are, paradoxically, the ones most constrained by it. The CEOs who publicly call for regulation are often the same ones lobbying against specific regulations that would actually slow them down. The researchers who publish safety papers are often doing so with one hand while shipping capabilities research with the other. It’s not hypocrisy, exactly. It’s structural contradiction.
The space between rhetoric and reality
I’ve been struck recently by how much of the AI discourse operates in a strange rhetorical space — a space where everyone agrees that safety matters, where everyone agrees that governance is important, and where almost nothing actually changes. It reminds me of how the climate conversation functioned for about two decades: universal acknowledgment of the problem, near-universal paralysis on solutions.
The reason, in both cases, is the same. The costs of the problem are distributed broadly and temporally — they fall on everyone, but in the future. The benefits of inaction are concentrated and immediate — they accrue to specific companies, specific investors, specific national strategies, right now. This is the oldest problem in political economy, and we have not solved it. We’ve just dressed it up in new language about “alignment” and “responsible scaling policies.”
There’s also something uncomfortable about the way the safety conversation is geographically distributed. I’m writing this from Singapore — a city-state that has very deliberately positioned itself at the intersection of AI development, trade, and geopolitical hedging. The view from here is different than the view from San Francisco or London. From here, you can see very clearly that AI safety is not just a technical problem or a corporate governance problem. It’s a geopolitical problem. And geopolitical problems don’t get solved by voluntary commitments. They get solved — to the extent they get solved at all — by binding agreements backed by credible enforcement. Which, in the current international climate, feels approximately as likely as OpenAI voluntarily slowing down.
The space industry offers an instructive parallel here. Space exploration — another domain of enormous technological capability, high stakes, and concentrated power among a few actors — was eventually subjected to international governance frameworks. Imperfect ones, yes. But frameworks that at least established norms, created transparency requirements, and imposed costs on reckless behavior. We have nothing comparable for AI. Not really. Not yet.
What I think actually matters now
I don’t have a clean solution to offer. Anyone who does is either lying or selling something — usually both. But I do think there are a few things that matter more than the current discourse suggests.
First, we need to stop treating AI safety as a sub-discipline within AI research and start treating it as a governance problem. The technical work matters enormously. But the reason safety keeps losing is not that we lack good alignment research. It’s that we lack institutional structures that can translate safety research into binding constraints on corporate behavior. That’s a political problem, not a technical one.
Second, we need to be far more honest about what the financial structure of the AI industry actually incentivizes. Every time a lab raises another multi-billion-dollar round, the clock speeds up. Every time a new model is released, the competitive pressure on every other lab increases. The current structure is an accelerant, not a brake. And treating it as anything else is a form of denial.
Third — and this is the one I keep coming back to — we need more people who understand these systems to speak clearly about what they see. Not in the careful, hedged language of corporate communications. Not in the abstract formalism of academic papers. But in plain language, with specific claims, backed by structural analysis. The most valuable thing any of us can do right now is refuse to be comfortable with the gap between what is said and what is done.
I started this piece with a question: why would the people building the most powerful AI systems on Earth have any real incentive to make them safe? The answer, I think, is that they don’t — not under the current structure. And that tells us something important. It tells us that the structure itself is the problem. Not the individuals within it. Not their intentions, which are often genuinely good. But the architecture of incentives, capital, competition, and geopolitical pressure that makes safety structurally subordinate to speed.
Understanding that architecture — clearly, honestly, without illusion — strikes me as the first step toward changing it. Maybe the only step that matters.
