There’s no driving test for self-driving cars — only metrics

There’s no widespread standard for measuring how well autonomous cars are performing. For the most part, we have to rely on what self-driving car companies tell us or news reports about accidents or fatalities.

California’s Department of Motor Vehicles has something called a disengagement report, where companies have to tell the state how many times a human had to take over driving from a fully autonomous vehicle. That report came out last week, but it only records data for cars in California. The companies themselves say the metrics are inaccurate and misleading.

I dig into this topic in “Quality Assurance,” the segment where I take a deeper look at a big tech story. I spoke with Jack Stewart, who covers transportation for Marketplace. I asked him what the problem is with the California’s metrics. The following is an edited version of our conversation.

Jack Stewart: Say a company rolls out a brand-new build of their software, and they’re testing that in California because it’s near their headquarters. That software could be extra buggy at the beginning, and you could see a bunch of disengagements, but that same company could be running a commercial service somewhere like Arizona, where they don’t have to collect these reports. That service could be running super smoothly. You don’t really get a picture of a company’s overall performance just by looking at this one really tight little metric. It was a nice idea of California some years ago to start collecting some information, but it’s not really doing what it was originally intended to do nowadays.

Molly Wood: Fair, but doesn’t it seem like we should at least be able to say, “You have to have achieved a certain technological competency in order to get permission to operate your vehicle completely autonomously”?

Stewart: Yeah. A driving test, the same thing that we have to give humans, would be one way of doing that. Put these robots through a test. If I have to prove that I’m capable of not hitting the car in front and obeying speed signs, then maybe we could come up with something similar for robots. It doesn’t necessarily have to be on the roads. There’s ways to do these things through simulation as well that could be really effective. You could run through a whole bunch of scenarios really quickly in a simulation and prove effectiveness before these vehicles are put on what are ultimately public streets interacting with members of the public.

Wood: This seems like a super good idea that we have come up with here. Is anyone working towards something like this?

Stewart: There are companies that are working toward a test and a way of providing a baseline, but it’s not something that, really, anybody is pushing really hard at the moment.

Wood: Would it be helpful to have federal regulations or a federal set of standards and metrics around autonomous car performance?

Stewart: I think if you ask the companies, they would say that federal oversight would be more useful than state oversight, because they definitely don’t want a patchwork of regulations. At the same time, they’re hardly clamoring for extra oversight at the moment, so I don’t think you would find anybody saying, “Yes, we want the feds to come in and start limiting what we’re doing.”

Wood: I want to ask you about a different topic. Waymo has raised $2.25 billion, [which is] a lot of money and the first time that it’s gone outside of the parent company Alphabet to raise money. Is this a sign that this is a company that basically costs so much that it needs more than one parent? It needed to go dual income on this?

Stewart: It is a different topic, but it’s also related, because I think it’s a sign of who is perhaps a leader in this race. Waymo is often called the leader in this race. You’ve got to imagine that these investors, these corporate investors, have done a lot of due diligence into the progress in this field and researched all these companies very comprehensively before putting up such a lot of money. It is a sign that maybe Waymo is doing what some people perceive as good work. It is also a sign of just how much money it costs to do this development. We still haven’t seen any commercial service that makes money. Waymo is finally running a service with passengers, paying passengers, in Phoenix, Arizona, area, but it’s a very, very limited trial. It’s not the thing that’s making profits yet, so it’s still the case that everybody’s investing because this is seen as one of the futures for the auto industry, but it’s going to take a lot of money before it starts making money.

Wood: This brings me back to metrics. What do investors look at when they’re trying to determine the health of one of these companies or the prowess?

Stewart: I imagine they get as much information as they possibly can about the performance of the software, which is what it all comes down to at the end of the day. It’s not really about the vehicle. These vehicles are quite often just standard vehicles — minivans and [such]. It’s not really about the sensors. There are things that can be relatively bought off the shelf and, more so these days, there are just a few companies that you can go to and pick up the hardware that you need. It’s the software, and it’s how many miles that software has driven both in simulation and on the real roads without any incident. You can look at metrics, like the number of miles driven between disengagements, and it might show you the progress of an individual company. It might not be a good way to compare across companies if they have different policies, but perhaps a company could present it and say, “Look, we’re doing so much better. The robot only needs a human to take over once every 1,000, 2,000, 10,000 miles,” and then you’re starting to get a system that’s pretty robust.

Related links: More insight from Molly Wood

Just this week, Uber published a blog post outlining some of the tech it uses for its self-driving cars that’s pretty detailed. It talks about the various components of its technology, how it trains the artificial intelligence that drives the cars and how it uses modeling and simulation to determine whether the cars are road-ready. I still wouldn’t mind having a five-star rating system established by a government body with some oversight, because I really don’t want to understand how Uber is “accelerating machine learning research with continuous integration and delivery” to be part of my new consumer duties.

Also this week, Waymo, which got that big infusion of cash, put out a fancy ad showing all its future delivery, trucking and ride-hailing services, which The Verge notes are all quite far in the future still.

Also, in self-driving talent gossip, remember that guy Anthony Levandowski who allegedly stole a bunch of Google’s self-driving car tech and then started a company that immediately got bought by Uber? Well, Uber fired him, Google sued him and he lost that lawsuit and then appealed. On Wednesday, a judge said he had to pay $179 million in damages. Levandowski promptly filed for bankruptcy. 

Also watching

Under pressure not to spread political misinformation through ads, Facebook has taken down campaign ads paid for by President Donald Trump’s reelection committee. The social giant said the ads referred to the 2020 census, and it has a policy against interfering with the census. I mean, it’s something.

In another story about “you should have done something sooner,” Twitter CEO Jack Dorsey — also the current CEO of Square — said he will reconsider his idea to spend part of this year in Africa. The decision is not because it’s tough to effectively run two American companies while you also live in Africa half-time but, Dorsey said, it’s partly because of coronavirus. He also said it was because an activist investor group bought 4% of Twitter with the goal of reshaping its board and kicking Dorsey out. Except Dorsey didn’t mention that specifically and just called it “everything going on.”

Maybe the idea we mentioned yesterday — the disappearing tweets thing — is an attempt to show that Jack Dorsey is super checked in to innovating Twitter. But bolting on an idea from Snapchat, which Instagram already stole successfully, may not make the strongest case.