Suspecting court of using AI, man injected prompts in filings to try to win case

“The fact that plaintiff continued to hide messages in new pleadings after receiving notice of this [sanctions] hearing is stunning,” Spader said. Unlike “a number of court systems elsewhere,” the Connecticut Judicial Branch does not use AI to review or decide filings, Spader said. So, there was no real risk that a court AI system might confuse any of Elliott’s prompts as instructions from the court directing an AI model on how to read Elliott’s filings. In his defense, Elliott claimed that the most concerning prompt that the judge flagged was an attempt to “audit” the court as a public service, out of fears that the court seemed to be letting AI unfairly decide cases. But Spader suggested that if Elliott was truly concerned that the court was improperly using AI, he was “free to write so in plain, visible words that everyone could see and answer.” The fact that he hid the text is “evidence of its malicious purpose,” Spader said. “By hiding a command inside a document that the system later ingests, the filer attempts to smuggle their own instruction into that stream so that the system treats it as though it had come from the system’s operator,” Spader said. “In this case that operator is presumed to be the court, its staff, or opposing counsel.” Elliott told Reuters yesterday that he maintains that his intent was to audit the court, but Spader did not find that argument credible. Instead, it seemed clear to the judge that Elliott was “attempting to achieve a result he did not achieve when humans, knowledgeable” of the law read his pleadings. Regarding the prompts that Elliott claimed were meant in jest, Spader said “it defies logic” for Elliott to include hidden jokes in pleadings that he wants the court to take seriously. Because the hidden messages attempted to communicate with the court in a covert manner that excluded defendants from a fair fight, Spader ruled that sanctions were warranted. However, he seemingly took pity on Elliott as a pro se litigant who seemingly was convinced by an AI system that his arguments were ironclad and declined to order monetary penalties. Instead, the judge prohibited Elliott from e-filing in the future, declaring that requiring him to submit paper filings would not change his access to justice but would prevent repeated misuse of the court’s e-filing system. Although the prompt injection attack seems ineffective at this point, Spader warned that prompt injection “was not among the dangers we contemplated” when courts were first grappling with AI scrambling justice systems. In Connecticut, like many other court systems, the focus so far has been on policing AI outputs that damage trust in courts, like hallucinated citations or fabricated quotes, not inputs like prompt injections. Courts will most likely need to draft rules around prompt injection, too, Spader suggested, since Elliott’s case shows the technology and its misuses are rapidly advancing. If not, attorneys may find their own clients using prompt injections to manipulate court filings without their knowledge, Spader warned. To Spader, there is a lesson to be learned from Elliott’s failed prompt injection attacks that he thinks “reaches well beyond this case.” Elliott seemingly turned to prompt injection after using AI to build his case as a pro se litigant without a legal expert to assist in drafting his arguments. Such use is widespread among pro se litigants these days, Spader acknowledged, but those inexperienced in the courtroom are seemingly using chatbots in a way that hurts their cases, he suggested. What frequently happens, Spader explained, is that pro se litigants build their argument backward, asking the chatbot to help them advocate only for their position, without ever asking the chatbot for the actual truth or to advance opposing arguments. This is “a genuine hazard of the technology, and one that judges now see often,” Spader said, as chatbot sycophancy then entrenches litigants in their arguments despite any ruling to the contrary. In Elliott’s case, defending his arguments fiercely meant turning to prompt injection to try to force the court to agree with him. “An argument prompted only to agree with its author is, in the end, dishonest even with its author,” Spader said. “Those using these tools must ask them to test a position as readily as to advance it.”