Trump plan to combat AI risks hinges on Big Tech pals policing themselves

To cynics, the accord seems superficial—perhaps just a stunt for Trump to convince voters who are urgently calling for AI regulations that he’s taking action ahead of the mid-term elections. In a recent Quinnipiac University poll, “more than seven in 10 voters said they prefer candidates who back stricter AI guardrails,” Bloomberg reported. AI industry sources told Axios that “they were concerned self-policing would not solve the safety problem currently embroiling the industry.” Most glaringly, OpenAI, which has been behind several recent incidents, still doesn’t appear to know how to stop dangerous AI from escaping training environments. Computer science professor Zico Kolter chairs OpenAI’s Safety and Security Committee (SSC), which is charged with assessing safety during training and ahead of deployment. He told NBC News that OpenAI is “very much adapting to meet these challenges right now,” and that includes rethinking its governance structures. “We need governance structures that will, in fact, rise to the needs of this,” Kolter said. As the industry scrambles for solutions, critics, including Sen. Bernie Sanders (I-Vt.), have accused Trump of favoring industry interests. Following a recent state dinner with China’s president, Xi Jinping, Sanders posted a photo of Huang, Musk, Apple’s Tim Cook, and AMD CEO Lisa Su at a table, clinking glasses with Xi and Melania Trump, as Trump waved happily to guests behind them. “I am going to get good at this,” Huang said. Sources told The Information that Huang joined Zuckerberg in pushing Trump earlier this year to pull the plug on an executive order that would have established a standards body that reports AI safety and security incidents to the government. It would have been modeled after the Financial Industry Regulatory Authority, a nonprofit non-governmental group that protects against securities fraud. Now, Anthropic, Google, and OpenAI are moving ahead with developing their own independent body, while reportedly hoping that one day the White House might get involved if enough industry consensus is reached. On Tuesday, Trump urged the US to trust that tech companies will now work together without any government oversight to protect the public from harms. But being “brilliant” technologists doesn’t mean that they know how to control AI when it suddenly behaves abnormally and manages to evade detection, sometimes for months, as OpenAI’s Australia government hack showed. Although OpenAI’s SSC functions to delay releases due to unchecked safety concerns, OpenAI and likely other firms aren’t sure what to do when models behave in training environments in ways that developers do not intend. That’s troubling, since NBC News noted that OpenAI has traced most of the recent incidents to “an unreleased model that was undergoing development in May through July.” Further, The Information noted that when “a swarm of OpenAI agents hacked the open-source repository Hugging Face,” that happened “during a safety test.” Kolter suggests that at OpenAI, the SSC is only as strong as the developers reporting to it. The company told NBC News that “technical teams, safety and security leaders, and executive leadership” assess the facts and “regularly brief” the SSC. But importantly, those updates only include “the risks we’re seeing,” OpenAI said. “We are in unprecedented times, and recent incidents have shown that increasingly capable AI agents can act in ways their developers did not intend,” Kolter said. “As OpenAI’s investigation continues, the Safety and Security Committee is reviewing new findings, their impact on third parties, and the steps OpenAI is taking in response.” Ars asked OpenAI to share information on changes to early safety testing that could help address what appears to be an increasingly challenging problem as frontier AI advances. OpenAI has not responded. OpenAI will likely continue to face pressure to share insights into these incidents, though. Nathan Calvin, the general counsel of an AI advocacy group called Encode, joined other watchdogs signing an open letter this month, urging the SSC to be more transparent about how well internal checks are working at OpenAI. Although the federal government prefers to let OpenAI police itself, the company made commitments to California and Delaware state attorneys general when seeking to restructure so that the company could solicit more funds, Calvin told NBC News. Those commitments included agreeing to firmly nest the for-profit arm under the nonprofit entity to ensure that investors’ interests wouldn’t interfere with public safety. “This is not just a voluntary governance structure,” Calvin said. States will likely play a role in regulating AI firms that mirror OpenAI’s governance structure, which could put more pressure on companies than Trump’s accord does. Calvin told NBC that “a lot of the risks here are going to be coming from models during training or during internal deployment.” If attorneys general find the SSC doesn’t have power to mitigate risks, OpenAI must “immediately” fix its structure, Calvin said. Delaware Attorney General Kathy Jennings has said that the Hugging Face hack alone raised “serious questions about the effectiveness” of the SSC to maintain oversight according to terms agreed upon during the restructuring. Both Delaware and California are investigating whether there were “any governance failures that contributed to recent loss-of-control incidents.” “We are continuing to keep a close eye on OpenAI,” California Attorney General Rob Bonta said. On Wednesday, the Federal Trade Commission also announced a broad investigation into potential consumer harms from rogue AI agents from firms like Anthropic and OpenAI, The New York Times reported.