Silicon Valley Democratic Rep. Ro Khanna will introduce a bill to regulate artificial intelligence, which includes strict liability standards and a ban on “recursive” self-improving AI until the government creates safeguards.

The bill comes amid a push for Washington to regulate AI after top executives warned the technology could become out of their control. Congress has largely done nothing to regulate the technology, which is undergirding the economy but could cause catastrophic damage if it goes wrong.

“There’s actually a civilizational extinction risk,” Khanna said Monday in an interview with CNBC. “There’s a safety risk of loss of control, and then there’s a misuse risk, and we need to take both seriously.”

Khanna’s bill, a summary of which was shared exclusively with CNBC, is dubbed the “Human Control Over AI Act” and would ban models that recursively self-improve or autonomously modify their own core objectives, containment or shutdown controls until federal guardrails exist and an agency approves the activities.

The bill would also create a new federal agency to focus on AI models from OpenAI, Anthropic, Google DeepMind and xAI as the companies find themselves in the spotlight amid rapidly escalating safety concerns.

The agency would be tasked with creating safety regulations, a licensing system to approve model training and deployment, frontier AI model audits and setting security standards for continuous testing and effective human control.

Independent auditors would also be required to be embedded at every frontier lab and report to the agency directly. The agency would also create standards covering sandbox testing environments; air gaps, or isolating testing environments from the internet; kill switches; and controls that prevent models from escaping lab settings and roaming on the Internet. It would also be responsible for regulating the advanced chips that frontier labs use, including mechanisms to monitor or control their use.

The bill would also create new legal penalties for AI gone awry.

It would criminalize “crimes against humanity” for the deployment of AI models that result in the destruction of civilian populations. The bill would also impose criminal penalties on employees of AI companies who disable safeguards, kill switches, logging or containment systems, or who knowingly deploy an unauthorized system.

AI companies would also be required to hold extensive liability insurance to release their models.

The bill also calls for the administration to pursue enforceable agreements and targeted export controls to deter China and other adversaries from developing dangerous AI systems.

Khanna said the bill represents “the most comprehensive AI safety legislation” that has been proposed. He said it is modeled on his conversations with AI safety organizations like the independent nonprofits Model Evaluation and Threat Research, Machine Intelligence Research Institute and Palisade Research rather than the asks of executives from large AI companies.

“I want this to be a model to give voice to the AI safety community,” Khanna said. “I believe they have been unfairly dismissed as science fiction and haven’t been taken seriously enough … they need to be driving this debate, the AI safety community, not the executives of frontier labs.”

It now joins a number of other AI safety proposals, including the FRONTIER Act, a bipartisan proposal from Reps. Jay Obernolte, R-Calif., and Lori Trahan, D-Mass., that would put independent auditors in frontier labs and allow the government to shut down models that have the potential for catastrophic risk.

No House bills are expected to receive a vote until after the midterm election, and the Senate is expected to leave Washington and not return until after the election after this week.