AI RISK

Machines are resolving long-standing mathematical conjectures, generating novel scientific research, and carrying out cyberattacks on their own initiative. The leading AI laboratories say that their goal is to build AI systems that surpass humans at every cognitive task. Drawing on his book with Eliezer Yudkowsky, If Anyone Builds It, Everyone Dies, Nate Soares (President, Machine Intelligence Research Institute) argues that the current AI trajectory would predictably end in human extinction, not because the AI will hate us but because it simply won’t care, and will pursue other strange drives. But all is not lost: the dangerous sort of AI does not exist yet, and there is ample time to change the trajectory and avert catastrophe.

AI safety has a concentration problem of its own. The field is geographically clustered in a few cities, institutionally clustered inside a handful of frontier labs, and intellectually clustered around a few agendas. Distributed approaches to AI safety, including AI for privacy-preserving, secure, and cooperative AI, are systematically underfunded relative to their importance, especially under short AGI timelines which favor centralized approaches. Foresight’s AI Nodes are a structural response: funding a d/acc portfolio to AI safety directly, while diversifying how safety research gets done, geographically, institutionally, and across regulatory jurisdictions. Join this talk to learn about some problems in the space, promising projects and opportunities for funding, collaboration and exchange with Foresight’s AI Nodes.

Artificial intelligence (A.I.) will likely be the most important technology we have ever developed. Technologies such as electricity, semiconductors, and the internet have been transformative, reshaping economic activity and dramatically increasing living standards throughout the world. In some sense, artificial intelligence is simply the latest of these general purpose technologies and at a minimum should continue the economic transformation that has been ongoing for the past century. However, the case can certainly be made that this time is different. Automating intelligence itself arguably has broader effects than electricity or semiconductors. What if machines—A.I. for cognitive tasks and A.I. plus advanced robots for physical tasks—can perform every task a human can do but more cheaply? What does economics have to say about this possibility, and what might our economic future look like?

How AI will affect society is driven by the dynamics underlying AI development—the constraints, incentives, and oversight under which AI systems operate. A powerful lever to shape these dynamics is scalable measurement: technology that creates high-quality, systematic public understanding of important properties of AI systems. In this talk, we will discuss three forms of scalable measurement: scalable oversight of AI agent transcripts, scalable discovery of long-tail behaviors, and scalable explanation of language model internal states. In each case, an important component is using AI to understand AI: building AI-driven tools to investigate other AI systems, thus allowing us to turn compute into understanding.

China looms large in US conversations about AI policy. But often the stories that spread about Chinese AI are oversimplified—"China is poised to overtake the US in the AI race," "DeepSeek proved that chip export controls have failed," "AI is one area where the US and China should obviously cooperate." Since reality is usually more complicated, this talk, featuring Helen Toner (Georgetown CSET) in conversation with Rachel Stern (Berkeley Law), will run through some common narratives about AI in China, compare them against the available evidence, and draw conclusions for the US AI ecosystem.

As transformative AI approaches, the power of regular people over government and society may dramatically decrease. Where does this power come from, how does AI shift it, and what can we do about it? This talk will argue that labor disruption will be the driving force behind AI-driven political action, and that there could be a window where significant displacement arrives before a sudden takeoff in AI capabilities. This window is the best opportunity to shape AI development, but it won’t stay open forever.

This article explores the connection between our sociotechnical imaginaries of AI and our emerging and potential regulatory futures. Historical institutional scholarship brings temporality to descriptions of institutional stability and change; new sociological institutionalism studies the adoption of norms and cultural practices within organizations and how such structures are adopted into law. We bring a STS perspective and a materialist approach to this analysis through the analytic idiom of sociotechnical imaginaries to connect AI assemblages–associated practices, discourses, and performances–to public visions of the future, which direct, stabilize, and shape anticipated regulation. We describe three competing imaginaries of AI–responsible AI, frontier AI, and pragmatic AI–and their reciprocal governance futures.

Drawing on empirical interpretative document analysis and qualitative interviews with “responsible AI” practitioners as well as “AI safety” researchers, we describe evolving AI sociotechnical imaginaries through professional technical practices, policy documents, public representations, and discourses. We provide examples of how corporate and technical practitioners endogenously shape standard setting processes, discourses, and research to preconfigure understandings of “good” governance and regulators’ choices. These configurations of material as well as symbolic structures offer an example of how the logics and practices of computer science are being institutionalized in quasi-regulatory spaces among key organizations. This is influencing debates about the content and meaning of law in public legal institutions, setting the pre-conditions for managerialization of law. We conclude with a discussion on regulatory possibilities through the shaping of sociotechnical imaginaries. Proposed regulatory responses should be attentive to sociotechnical imaginaries motivating alternative regulatory visions.

Joint work with Victor Z. Wang.

PauseAI co-founder and Dr. of Evolutionary Biology Holly Elmore lays out her deep worldview on genetic conflict, human society, and social change and explains how that is cashed out in PauseAI’s theory of change. PauseAI is for anyone who wants to pause AI development (and pledges nonviolence!). But PauseAI was organized according to the idea that the real AI danger isn’t simply a technical issue awaiting a bugfix. The problem is that we are developing machines of arbitrary intelligence. Arbitrary intelligence is arbitrary power, and the existence of such power will disrupt many societal equilibria, some we anticipate and who knows how many that we may not even be aware of. It may take many years to bring human society to the point where we can coexist comfortably with advanced AI, or human society may be fundamentally incompatible with superhuman AI. Whatever we do about advancing AI as a society must take all of these possibilities into account.

AI Safety is now a common term, used in a variety of regulatory, education and funding contexts. However, most formulations of 'AI safety' fail to capture a coherent picture of the socio-technical nature of AI systems, simultaneously under-estimating and over-estimating certain risks and impacts. In this talk, I’ll share the perspective of those hoping to adopt this socio-technical view, and the implications for AI development, adoption and policymaking.

The AI governance and risk management landscape has evolved with, for example, the release of The General-Purpose AI Code of Practice and at least a dozen frontier AI safety frameworks from leading AI companies. I will compare these developments to the shifting AI risk landscape and highlight key priorities to help developers, policymakers, and researchers stay a step ahead in order to realize AI’s benefits — and prevent its greatest harms.

In 1951, Alan Turing predicted the eventual loss of human control over machines that exceed human capabilities. I will argue that Turing was right to express concern but wrong to think that doom is inevitable. There are technical and regulatory directions that ensure human flourishing, if we choose to pursue them.

This paper examines changes in the labor market for occupations exposed to generative artificial intelligence using high-frequency administrative data from the largest payroll software provider in the United States. We present six facts that characterize these shifts. We find that since the widespread adoption of generative AI, early-career workers (ages 22-25) in the most AI-exposed occupations have experienced a 13 percent relative decline in employment even after controlling for firm-level shocks. In contrast, employment for workers in less exposed fields and more experienced workers in the same occupations has remained stable or continued to grow. We also find that adjustments occur primarily through employment rather than compensation. Furthermore, employment declines are concentrated in occupations where AI is more likely to automate, rather than augment, human labor. Our results are robust to alternative explanations, such as excluding technology-related firms and excluding occupations amenable to remote work. These six facts provide early, large-scale evidence consistent with the hypothesis that the AI revolution is beginning to have a significant and disproportionate impact on entry-level workers in the American labor market.

LLMs are quickly learning capabilities needed to increase catastrophic risks. When a model pushes the frontier of capabilities, how can we know whether it’s safe or not? In this talk, I will go into METR’s recent work on evaluating frontier AI models, using METR’s evaluation of GPT-5 as a case study. I will talk about measuring AI capabilities over time, creating safety cases for frontier AI models, and what AI evaluations and safety assessments might look like in the future as we approach AGI.

I’m going to give a whirlwind tour of AI Safety as a set of concerns, concepts, and communities. I’ll talk about the history of AI Safety and my personal history in the field, and how I and others in AI Safety see the state of play. I’ll touch on some of my past work, including on Gradual Disempowerment — a risk that’s been historically neglected within AI Safety. I’ll also talk about how I think academics across disciplines can (and should) contribute to shaping the future of AI. And I’ll describe my current work on raising public awareness of AI risk.