Back when the pandemic first hit, many students received tablets or laptops from their schools. Schools also wanted to know what students were doing on those devices, so demand for AI-powered software to monitor students’ digital activities grew.

That surveillance software is the subject of a new investigation from the Associated Press and the Seattle Times, which Claire Bryan coauthored. Marketplace’s Stephanie Hughes asked her what sort of things this surveillance software might flag. The following is an edited transcript of their conversation:

Claire Bryan: Students were writing about thoughts about suicide or just struggles in a relationship with a parent or a peer, all sorts of things. Sometimes it was pretty innocuous, a student writing a short story or a scene in a play for the school play, and it had someone pushing or shoving or just making a remark about shooting basketball hoops.

Stephanie Hughes: And what happens once the software spots something like this?

Bryan: The AI program will alert the employee at the company. And then, if it’s particularly concerning, that employee will review it and send an email or a phone call or a text, depending on the how severe the situation is, to a school district administrator. And then, oftentimes, the principal will decide if they want to talk to the student or the parent themselves, or if they want to send in a school counselor. If there are severe cases, sometimes the company will call the police directly if they cannot get a hold of the school administrator or counselor. Or even if they do contact the school district administrator, the administrator, if they can’t contact the family or get in touch with the parent, they might also decide to call the police and usually ask for what’s called a welfare check, where police will go to the student’s home to check to see if they’re okay.

Hughes: What’s your sense of how widespread this software is? Like, essentially, how big the school surveillance tech industry is?

Bryan: Yeah, so it’s big. It’s definitely growing. What’s really different now is these companies are popping up that are providing monitoring or surveilling anything that the student is creating on the computer — emails to teachers, texts or chats with friends on Google Docs, anything like that. So that is a lot. I mean, that is something that during the pandemic, there was really ripe conditions for school districts who suddenly had all of their kids learning at home, wanting a way to protect kids. So conditions were really ripe for these products to be really appealing. At the same time, there was a mental health crisis, increased rates of anxiety and depression and suicidal ideation, and also the threat of school shootings. So there was a lot going on that has made this more intrusive-type of monitoring really popular. And then on top of that, there’s also a bunch of other ways that ed tech surveillance companies are now in schools, like there’s companies that provide artificial intelligence facial recognition or weapon detection, they also just provide a lot of different things, like electronic hall passes or visitor monitoring systems all the way to just tools that teachers are using in the classroom.

Hughes: Can you give me a sense of how much districts are spending on this?

Bryan: Yeah, so the Vancouver District is spending, about $100,000 per year. Gaggle, the company that they were in contract with, it costs them $5 per student per year to provide this service on each student laptop.

Hughes: You have a quote in this piece, Claire, from a surveillance tech executive who says “not monitoring children is like letting them loose on a digital playground without fences or recess monitors.” Tell me more about that approach.

Bryan: I think that goes back to these companies seeing these devices as the property of the school district. And just like us adults, when we’re working on our company laptop, we have an understanding we don’t own this device, and so it’s not our private space. So over and over again, educators told me these laptops are for school work. They are educational. They are for kids to bring them home and do their homework. And so when you think about it with that mindset, I think you can understand why school district officials do want to monitor that laptop or that playground, because it is for a school purpose. But what we found when we looked at these records that were screenshots of students computers is I’m sure [kids] are also doing their schoolwork, but they are doing so much more on that school district device. I was really struck by just how personal and intimate they truly bare their hearts and souls onto those computers and really talk about really intensely personal things. So it’s very clear that students aren’t just using it for school work. And even if they are aware that this device is not their own property, oftentimes it slips out of their mind that this is their school district’s property, or that the school district can watch them, just in the same way that we as adults [when] know that a company might be watching our activity, we also forget all the time.

Hughes: What sort of risks exist with using the software?

Bryan: So what we found was that the way that this company in particular was sharing the students’ screens with the school district administrators was through a link that provided a screenshot of that computer, of that student’s computer. And that link was accessible without any sort of protection or password to access that link. And the company said that this was necessary, because a lot of times these alerts go out to school administrators in the middle of the night. They’re waking up, they’re just checking their phone. They need to see the alert quickly, and they need to address it quickly, because often with, let’s say, a case of a student thinking about suicide, you have a matter of minutes, and so they feel like it’s important to keep those links really accessible. But what it also opens the door to is other people getting access to that link within the 72 hours — or in this case, the company hadn’t ensured that those links expire after 72 hours, so anyone could see anything. So I think the risk is, it just shows how much data there is being collected about students, and even when a school district is trying to do it right, or even when a company is trying to protect the student data, like this kind of stuff happens, and it’s all out on the internet that someone could stumble upon.

Hughes: Do we have any idea of what the long term ramifications are of using this kind of surveillance in schools?

Bryan: I talked to a lot of experts about the development tool psychology aspect of this for students. We don’t know from where we’re sitting right now. But what we do know is that students, especially students from minority populations, LGBTQ students, trans students, we know that they often are likely to go to the internet for support. And as much as the internet and social media can be harmful to a student’s mental health, it also provides a really big support system and can be actually developmentally productive for students to have a place online to explore their thoughts about identity and ask big philosophical questions about the world. And so I think there’s something to be said about students not feeling like they have a safe place or a private space to explore all of those thoughts, and they’re going to be growing up in a very different way than many adults today probably grew up, where we maybe had access to a phone or a computer and searched whatever we wanted on Google and got to explore all of those questions on the internet.

Hughes: Yeah, I’ve been thinking about this a lot, because I feel like we live our lives on this technology, and it’s really hard to switch gears, if you’re on a school-issued device or work-issued device. What advice do you have for kids and their parents and for educators about this software?

Bryan: Parents and kids should make sure: when they’re given their laptop, they usually will sign a responsible use form or some forms from the district, and pay attention to those forms, don’t just sign them, but really read them, mostly to just understand exactly what kind of monitoring is happening on your laptop every day. And then you can ask your school district if you can opt out. For example, we found in Oklahoma, there was a parent who was not able to opt out of the monitoring, but Vancouver District said that if families wanted to, they could opt out and they would find a different way. So I think opting out is really challenging, because more and more these days so much of curriculum is offered digitally. But if you really care about it, and you’re concerned about it, I think you could talk to your school district and see if there was a different way.

Hughes: How did both the school districts and the companies that you reported on respond to your reporting? What did they say?

Bryan: They were really open to the feedback. Both the school district and the CEO and founder of the company sat down with us for a long time, answered all of our questions. They understood how controversial this issue is, and I think they were understanding of the privacy concerns. But I also think they were pretty adamant in that they feel the software ultimately does help them connect with students and does help them get students help when they really need it, so they didn’t feel like the privacy concerns outweighed the other concerns about mental health and safety that they felt that the software and the service could address.

If you or someone you know are in a mental health crisis, you can call the mental health crisis and suicide prevention line at 988.