The Inside Baseball of Existential Risk
In the high-rise offices of San Francisco and the research labs of London, a peculiar debate dominates the water cooler. It is a debate about whether a future superintelligence might decide to wipe out the human race. Some AI researchers treat this 'existential risk' with a shuddering gravity, while others—the engineers writing the code—treat it as a punchline. To many developers, the idea of an AI with "the dog" to hunt humanity is a comical fantasy. They argue that code doesn't possess biological drives, malice, or the instinct for self-preservation. They are technically correct. A Large Language Model (LLM) doesn't want to kill you; it doesn't want anything at all.
This dismissal of doomsday scenarios is often framed as a victory for realism. By mocking the "Terminator" trope, these workers claim to be grounding the industry in reality. However, this focus on whether a machine might develop a soul or a sense of vengeance serves a secondary, more cynical purpose. It shifts the conversation away from what the machines are actually doing right now. While the elite debate the hypothetical intentions of a god-like entity, the practical consequences of current-gen AI are already being felt by people who don't have the luxury of theorizing about the end of the world.
The Intent Trap
The technical defense of AI—that it lacks intent, therefore it cannot be dangerous—is a convenient sleight of hand. It conflates harm with malice. A bridge doesn't need to hate you to collapse and kill you; it just needs to be poorly designed or improperly maintained. Similarly, AI doesn't need to "want" to ruin lives to achieve that end. It achieves it through implementation. When an algorithm is deployed to screen resumes, it doesn't need to be biased; it only needs to be trained on historical data that reflects decades of systemic racism and gender discrimination.
When we focus on the "intent" of the machine, we grant the humans behind the machine a pass. If a predictive policing tool disproportionately targets low-income neighborhoods, we often hear discussions about "algorithmic bias" as if it were a minor bug to be patched in the next update. This language treats a civil rights violation as a technical glitch. By treating AI harm as a problem of math rather than a problem of power, developers avoid the messy, difficult work of addressing the social inequities their products amplify.
Who Defines Safety?
The tech industry has become very good at co-opting the language of ethics. They speak of "AI Safety" and "Alignment" as if these are purely mathematical hurdles to be solved by more compute and better datasets. In this framework,
