xAI can’t deny Grok makes CSAM anymore. So it’s suing users.

While Musk was posting through it, though, Harwood was allegedly ignoring Musk’s warning and prompting Grok as many times as it took to get the chatbot to generate likely nonconsensual explicit images. Now Musk seemingly can’t deny that Grok makes CSAM. xAI’s lawsuit claimed that Harwood used at least two accounts with convoluted usernames—“ceae2cb4-a9f6-4885-8ae9-6e2096d084f4” and “befccb94-4029-454d-9f1f-0d4945e8fa7c”—to generate illegal content from December 8 to February 18. Sometimes Grok safeguards did prevent harmful outputs, “refusing to follow” some of the prompts “on the basis that such material violated Grok’s content moderation guardrails.” The lawsuit includes an example of an especially creepy prompt that Grok rejected, using phrases like “white slime” to mask intent to generate sexualized images. That particular prompt may have been rejected for an obvious reason, though. Harwood explicitly asked the chatbot to “remove all her clothing,” which directly violates xAI user terms against requests to undress real people. In the proposed class action, it’s alleged that xAI overlooks a lot of bad requests, only reporting to NCMEC one prompt to depict “gang rape” out of 7,000 harmful outputs in the most recent victim’s case. Likely to avoid other bad actors circumventing safeguards, xAI did not include examples of Harwood’s successful prompts or describe methods used to bypass filters. xAI only alleged that Harwood modified prompts to get around safeguards “in clear violation of the xAI Terms of Service and of US law,” including “some” requests for “obscene” images involving the “likeness of minor children.” A spokesperson for the South Carolina attorney general’s office told Ars that Harwood’s case is still pending. Specifically, he has been charged with “distributing, transporting, exhibiting, receiving, selling, purchasing, exchanging, or soliciting CSAM that was ‘through the use of an artificial intelligence platform.’” The spokesperson was not authorized to verify if Grok was the platform used. However, xAI’s complaint alleged that “upon information and belief, at least some of the images at-issue in the Harwood Criminal Action were generated or altered through Defendant’s violative use of Grok.” xAI did not immediately respond to Ars’ request to comment. It’s further noted that any CSAM uncovered by xAI is reported to NCMEC. In its complaint, xAI claimed that Harwood alone is responsible for his outputs because he “flagrantly violated” xAI’s rules and “went to great lengths to circumvent” Grok’s “technological safeguards.” Harwood allegedly did this by relying on “misleading prompts,” xAI said. And Harwood also failed to police himself once he saw that he could generate illegal content, xAI argued. In the complaint, xAI alleged that Harwood should have known that he was banned from using Grok after the first time he relied on the chatbot to make illegal content. Glaringly, though, xAI does not indicate that Harwood received any warnings that his account risked penalties. Instead, Harwood allegedly “continued to use Grok during the Relevant Period after violating the xAI Terms of Service,” xAI argued. “The xAI Terms of Service to which he agreed prohibited his use after his prior violations.” xAI is hoping the US district court will rule that Harwood violated xAI’s terms and breached his contract with xAI. But perhaps more importantly, Musk wants the court to recognize an indemnity clause that holds that only users—and not xAI—are liable for Grok-generated CSAM and NCII. According to xAI, when people use Grok, they are responsible for all of their content, which xAI insisted includes both inputs and outputs. Whether the court will agree that users are responsible for AI outputs has yet to be seen. Perhaps notably, the Copyright Office does not view AI outputs as human-created. That could throw a wrench in xAI’s offense, if the court struggles to see how child sex images generated by an AI tool could be created by the user if any other image could not be legally credited that way. If xAI wins this fight, Harwood could owe substantial damages, including damages for “any real harm to third parties,” xAI’s “exposure to potential third-party claims and lawsuits,” and “any xAI reputational harm,” the complaint said.