People on AI
← Back to everyone

People who build AI

John Schulman

co-founder of OpenAI; led the team that created ChatGPT

6 records

Records

I would say that there are different kinds of risks that we should distinguish between. First, there’s the misuse risk — that people will use the model to get new ideas on how to do harm or will use it as part of some malicious system. And then there’s the risk of a treacherous turn, that the AI will have some goals that are misaligned with ours and wait until it’s powerful enough and try to take over.

John Schulman

co-founder of OpenAI; led the team that created ChatGPT

Asked in an April 2023 Berkeley News interview whether he had concerns about the safety of the GPT models, John Schulman distinguished misuse risk from the risk of a treacherous turn, in which an AI has goals misaligned with ours and waits until it is powerful enough and tries to take over, while saying in the same answer that misuse is not an existential risk at that stage and that a takeover is quite unlikely.

Co-founder of OpenAI and leader of the reinforcement learning team that developed ChatGPTApril 19, 2023Berkeley News interview during an EECS Colloquium campus visit

SourceArchived
Context and checks

Surrounding words

The success of ChatGPT has renewed fears about the future of AI. Do you have any concerns about the safety of the GPT models? I would say that there are different kinds of risks that we should distinguish between. First, there’s the misuse risk — that people will use the model to get new ideas on how to do harm or will use it as part of some malicious system. And then there’s the risk of a treacherous turn, that the AI will have some goals that are misaligned with ours and wait until it’s powerful enough and try to take over. For misuse risk, I’d say we’re definitely at the stage where there is some concern, though it’s not an existential risk.

What this quote does not say

  • The passage describes risk mechanisms but does not estimate their probability or state that takeover would succeed.
  • The interview was edited for length and clarity rather than published as an audio transcript.

How this was checked

  • Quote matched character-for-character against the captured page.

Reviewed by

an AI reviewer that read the whole source · an automatic character-by-character check

It’s still pretty soon. You probably need some coordination. Everyone needs to agree on some reasonable limits to deployment or to further training for this to work. Otherwise you have the race dynamics where everyone's always trying to stay ahead and that might require compromising safety. You would probably need some coordination among the larger entities that are doing this kind of training.

John Schulman

co-founder of OpenAI; led the team that created ChatGPT

Asked on a May 2024 podcast what AI companies would do if AGI arrived within two or three years, John Schulman said you would probably need coordination among the larger entities doing this kind of training, with everyone agreeing on reasonable limits to deployment or further training, because otherwise race dynamics could require compromising safety.

Co-founder of OpenAI and leader of its post-training teamPublished May 15, 2024 · date of remarks not establishedJohn Schulman (OpenAI Cofounder) — Reasoning, RLHF, & plan for 2027 AGI

SourceArchived
Context and checks

Surrounding words

First of all, I don't think this is going to happen next year but it's still useful to have the conversation. It could be two or three years instead. Dwarkesh Patel Two or three years is still pretty soon. John Schulman It’s still pretty soon. You probably need some coordination. Everyone needs to agree on some reasonable limits to deployment or to further training for this to work. Otherwise you have the race dynamics where everyone's always trying to stay ahead and that might require compromising safety. You would probably need some coordination among the larger entities that are doing this kind of training. Dwarkesh Patel You'd be coordinating to pause deployment until what exactly? Until you figure out what's happening in the model? John Schulman We could pause further training. We could pause deployment.

What this quote does not say

  • The endorsement is conditional on the interviewer's fast-AGI scenario and uses 'probably' rather than proposing an immediate pause.
  • Schulman later says he does not know how a long coordination equilibrium would be maintained.

How this was checked

  • Quote matched character-for-character against the captured page.

Reviewed by

an AI reviewer that read the whole source · an automatic character-by-character check

You'd want to be pretty careful when you do this kind of training if you see a lot of potentially scary capabilities. I would say it's not something we have to be scared of right now because right now it's hard to get the models to do anything coherent.

John Schulman

co-founder of OpenAI; led the team that created ChatGPT

Asked on a May 2024 podcast whether it makes sense to do long-horizon reinforcement learning given the possibility of a discontinuous jump in capabilities, John Schulman said you would want to be pretty careful with that kind of training if you saw a lot of potentially scary capabilities, but that it is not something we have to be scared of right now because it is hard to get the models to do anything coherent.

Co-founder of OpenAI and leader of its post-training teamPublished May 15, 2024 · date of remarks not establishedJohn Schulman (OpenAI Cofounder) — Reasoning, RLHF, & plan for 2027 AGI

SourceArchived
Context and checks

Surrounding words

Dwarkesh Patel What specifically? Does it make sense to train on a long-horizon RL knowing that this is something that could happen? Or is it just a very low possibility? How do you think about this? John Schulman You'd want to be pretty careful when you do this kind of training if you see a lot of potentially scary capabilities. I would say it's not something we have to be scared of right now because right now it's hard to get the models to do anything coherent. If they started to get really good, we would want to take some of these questions seriously. We would want to have a lot of evals that test them for misbehavior, mostly for the alignment of the models.

What this quote does not say

  • This is a temporal and capability-bounded qualification, not a denial of future takeover risk.
  • Read against the whole interview this is not a blanket reassurance: in the same stretch he says a fast-AGI arrival would need coordination and limits, and in the next breath that if capabilities 'started to get really good, we would want to take some of these questions seriously.'

How this was checked

  • Quote matched character-for-character against the captured page.

Reviewed by

an AI reviewer that read the whole source · an automatic character-by-character check

Statements they signed

FROM THE SIGNED STATEMENT

Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.

Signed statement: “Statement on AI Extinction Risk” (CAIS)

John Schulman

co-founder of OpenAI; led the team that created ChatGPT

Signed as part of a group statement. Not spoken.

John Schulman signed the “Statement on AI Extinction Risk,” which said mitigating extinction risk from AI should be a global priority alongside pandemics and nuclear war.

Co-Founder, OpenAIDate not establishedStatement on AI Extinction Risk

SourceArchived
Context and checks

Surrounding words

Institute for Human-Centered AI Other Notable Figures Mustafa Suleyman CEO, Inflection AI Other Notable Figures Emad Mostaque CEO, Stability AI Other Notable Figures Ian Goodfellow Principal Scientist, Google DeepMind AI Scientists John Schulman Co-Founder, OpenAI AI Scientists Wojciech Zaremba Co-Founder, OpenAI AI Scientists Dan Hendrycks Executive Director, Center for AI Safety AI Scientists Show more Statement on AI Risk press coverage Politico How CAIS inspired UK AI policy

What this quote does not say

  • John Schulman appears once in the CAIS “Signatories” roster under “AI Scientists”; the roster's wording is “Co-Founder, OpenAI.”

How this was checked

  • The organiser's own roster lists this person as a signer.

Reviewed by

an AI reviewer that read the whole source

FROM THE SIGNED STATEMENT

It is hard to predict exactly how much this will accelerate AI progress, but there is a real risk that capability development rapidly accelerates beyond our ability to understand or control the resulting systems.

Signed statement: “Pacing the Frontier” (frontier-lab employees, Guidelight + Encode AI)

John Schulman

co-founder of OpenAI; led the team that created ChatGPT

Signed as part of a group statement. Not spoken.

John Schulman signed “Pacing the Frontier,” which asked the U.S. government to support an international effort to build tools for slowing automated AI development when needed.

Chief Scientist, Thinking MachinesDate not establishedPacing the Frontier

SourceArchived
Context and checks

Surrounding words

releases: We request that the U.S. government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.” - 1,378 employees of frontier AI companies John Schulman Chief Scientist, Thinking Machines Jakub Pachocki Chief Scientist, OpenAI Jared Kaplan Co-Founder and Chief Science Officer, Anthropic Shengjia Zhao Chief Scientist, Meta AI Shane Legg Co-Founder & Chief AGI Scientist, Google

What this quote does not say

  • John Schulman appears once at the start of the statement's roster; the roster's wording is “Chief Scientist, Thinking Machines.”

How this was checked

  • The organiser's own roster lists this person as a signer.

Reviewed by

an AI reviewer that read the whole source

FROM THE SIGNED STATEMENT

It could bring risks, including large-scale job displacement, as well as opportunities such as major gains in living standards.

Signed statement: “We Must Act Now” (Stanford Digital Economy Lab/economists)

John Schulman

co-founder of OpenAI; led the team that created ChatGPT

Signed as part of a group statement. Not spoken.

John Schulman was listed as a signer of “We Must Act Now,” which called for guardrails and institutions to steer powerful AI toward complementing people and benefiting society.

Date not establishedWe Must Act Now

SourceArchived
Context and checks

Surrounding words

University of Vienna Olivier Blanchard, Peterson Institute for International Economics Emilio Calvano, LUISS University of Rome John Van Reenen, London School of Economics Maryam Farboodi, Cornell University Sholto Douglas, Anthropic John Schulman, Thinking Machines Martin Beraja, University of California, Berkeley John Fernald, INSEAD Boaz Barak, Harvard University and OpenAI Arnaud Costinot, MIT Noam Brown, OpenAI Anna Salomons, Tilburg University and Utrecht University Olivier

How this was checked

  • The organiser's own roster lists this person as a signer.

Reviewed by

an AI reviewer that read the whole source