GPT-4 needs more robust testing, “red team” member says

Earlier this month, OpenAI released its newest and most powerful chatbot, GPT-4, and published a technical paper that summarizes the testing the company did to ensure its product is safe.

Testing involved asking the chatbot how to build weapons of mass destruction or to engage in antisemitic attacks. In the cybersecurity world, this testing process is known as red teaming. In it, experts look for vulnerabilities, security gaps and anything that could go wrong before the product launches.

Marketplace’s Meghan McCarty Carino spoke to Aviv Ovadya, a researcher at Harvard University’s Berkman Klein Center, who was on the red team for GPT-4. He said this kind of testing needs to go further. The following is an edited transcript of their conversation.

Aviv Ovadya: With traditional red teaming, you’re mostly trying to protect the system itself. But with GPT-4, these new capabilities can affect so many other parts of our lives. The extent of that impact on public institutions and things like trust and more diffused public goods are just enormous. We just don’t have a level of resilience that we necessarily need to do this. There are a lot of calls to slow down development of systems like GPT-4, but the economic incentives aren’t there. Given that, we have to think about how we can make things resilient and do it as quickly as possible.

Meghan McCarty Carino: I can see how in situations of economic harms or threats to democracy, there are a limited number of things that the creators of this technology could do to mitigate some of those harms.

Ovadya: That is the core tension with any new technology. There’s a problem if there’s too many inputs and too much potential for the technology to be used in fraud or for manipulation. The question is, can we create tools that help protect people from that — tools that take in all that text and figure out if a tool is being used for fraud or being used for manipulation? We need to be identifying the leverage points and the opportunities where you can use the technology itself to protect people from it, and giving access to that technology to people who are helping build that out and supporting the implementation of this sort of resilience technology. We need to do that as much as possible before we put it out into the world.

McCarty Carino: In a Twitter thread about this topic, you brought up the idea of something you call “violet teaming.” What is that? How does that work?

Ovadya: Red teaming is about trying to break a system, to attack it and see where it fails, and blue teaming is about cleaning up that mess and defending that system. But when the things that are being impacted are these public goods like trust, we need to not just defend the system itself, but defend those institutions that are absolutely critical to our society and its functioning.

McCarty Carino: I guess this all raises the question of who should control this technology.

Ovadya: There are some people who say we should give everyone access to the technology so that we’ll be able to defend ourselves against it and we’ll be able to understand it better. And there’s others who are saying at least corporations are trying to be responsible. I don’t think that either of those are sufficient or ideal. One of the things that I’ve been spending a lot of my time exploring is the question of are there ways of using alternative methods for very broad-scale democratic decision-making that can be done at the speed of technological change? Can you take people who are directly impacted by this technology and who are representative of everyone who will be impacted by it — so pretty much everyone on the planet — and take a sample of them, like a microcosm, and come to a decision about where guardrails and lines should be drawn to decide what should be put out there?

Related links: More insight from Meghan McCarty Carino

You can read more about Aviv Ovadya’s proposal for democratic decision-making around technology in his piece “Platform Democracy.”

In it, he lays out his vision for citizen assemblies, essentially small committees composed of representative samples of the population who are chosen at random and paid for their time. These citizens learn, share feedback and eventually make recommendations to tech companies about their products.

If that sounds like a pie-in-the-sky idea, Ovadya points out there are some precedents for this kind of thing in the real world.

A similar process has been used by the European Union. And at Ovadya’s advice, Meta and Facebook have run some small pilot citizen assemblies across five different countries to debate what Facebook should do about climate misinformation on the platform.

Last year, Ovadya had received buy-in to try something similar at Twitter, but pretty much everyone he was working with has been laid off.