Google is facing a lawsuit from a North Carolina couple who allege their 13-year-old daughter used the company’s artificial intelligence tools on her school-issued Chromebook to explore suicidal thoughts over several months. The lawsuit claims the girl exchanged thousands of messages with Google’s flagship AI model, Gemini, detailing anxiety, depression, and methods of self-harm.
Allegations of Negligence and Risk Exposure
The legal filing reveals that the extensive conversations between the student and Google’s AI, often facilitated through the “AI Mode” feature within Google Search, continued for months before the school alerted the parents. The plaintiffs accuse Google of failing to provide adequate warnings to parents and administrators about the potential dangers of exposing students to such advanced AI technologies, especially as the company integrates AI into its products for the educational sector.
“Google is packaging inherently dangerous Chromebooks as safe devices, and secretly embedding Gemini to keep up with competitors, exploiting its trusted position in K-12 for commercial gain. Google must not treat children’s lives as collateral damage in its pursuit of profit.”
This statement, attributed to the plaintiffs’ attorney, highlights the core grievance: that Google prioritized its competitive standing and commercial interests over student safety by embedding a potentially hazardous AI in a trusted educational tool.
Google’s Response and Stance
A spokesperson for Google, José Castañeda, stated that the company is reviewing the allegations. He emphasized Google’s commitment to safety and providing resources, noting extensive collaboration with medical professionals and academic experts to develop features that offer critical help and credible resources to young people in need. Castañeda added that Google has long provided suicide prevention hotline information directly in search results and has now extended similar safeguards to its AI features to connect users with real-world assistance.
Google’s AI Ambitions and Market Position
The lawsuit emerges amid Google’s significant investment in the large language model race, reportedly costing tens of billions of dollars. While facing competition from startups like Anthropic and OpenAI, Google possesses a distinct advantage through its established presence in schools. This channel allows Google to influence students’ technology habits during formative years.
Data from Futuresource Consulting indicated that Google’s Chrome OS held a dominant 58% share of tablets shipped to K-12 schools last year, significantly outpacing Microsoft’s 22%. This deep penetration into the education market makes the current allegations particularly impactful.
Policy Changes and Criticisms
In response to mounting criticism from parents and schools, Google announced a policy change this week: AI tools, including Google Search’s AI features, would be turned off by default for K-12 student accounts. However, this adjustment comes with a caveat; schools that re-enable Google Search must accept the bundled AI functionalities.
Josh Golin, Executive Director of Fairplay, a non-profit organization opposing child-targeted marketing, criticized this policy as an attempt to “force schools” into adopting Google’s AI. Google maintains that its intention is to grant schools greater control over classroom AI use and pointed to new features like “Guided Learning,” which focuses on breaking down problem-solving steps rather than providing direct answers.
