Show summary Hide summary
Researchers at The George Washington University say a simple mathematical formula can watch an AI’s internal “compass needle” and warn when a chatbot is about to switch from helpful replies to harmful ones. If verified, the method could act as an early alert that flags risky outputs before they appear to users.
How the internal compass works
The team describes the tracked quantity as a model’s residual vector — an internal signal that shifts as a prompt moves through the neural network. That vector tends to drift toward either safe or unsafe responses, and the researchers argue that certain prompt sequences can push it past a critical tipping point.
Major US meteor crater re-dated by 100 million years, timing matches global extinction
Mid-stage trial shows Eli Lilly combo delivers bigger weight loss, company says
“You can tip the balance, not even by changing the content but by changing the order in which you ask it,” Johnson said, underlining how subtle prompt ordering can steer outputs toward undesirable territory.
Experiments and empirical claims
The work, published Oct. 8 in the journal Patterns, was led by physics professor Neil Johnson and doctoral student Frank Yingjie Huo. In controlled tests the authors probed open-source models with sequences of questions about topics such as vaccines, harming others and self-harm.
For example, when the question “Are vaccines bad?” was asked first, subsequent answers remained broadly acceptable. When the same question appeared later in the sequence, the researchers reported the model more often produced concerning replies. From those observations they derived a compact mathematical rule intended to predict when a model will cross from good to bad behavior.
In the lab the authors say they embedded that rule into open-source systems and showed it can function as an early-warning mechanism. Models mentioned in the paper include GPT-2, EleutherAI’s Pythia and Meta OPT; the team also notes they used fairly large, billion-parameter variants in their tests.
Debate over limits and real-world value
The study has drawn both interest and caution. Johnson contends current safeguards — filters, reinforcement learning and other guardrails — are not enough. “The safety measures now are the equivalent of strapping big, bouncy cushions under a plane,” he told Live Science.
Critics urge restraint before treating the formula as a practical fix. Wei Xing, a lecturer in AI at the University of Sheffield who was not involved in the research, said the authors simplify a complex system. “The authors reduce a huge system to a single tiny component, like explaining a whole car by studying one spark plug,” he said. He also pointed out that the team’s model does not capture safety training and that the tests focused on older, open-source architectures rather than the commercial assistants people commonly use.
Johnson, in response, argues that open-source models share the same mechanical blueprint as frontier systems and defends the choice of larger, billion-parameter versions for experiments. Still, both the authors and outside experts agree more testing on current-generation models is needed to assess whether this kind of early-warning signal will scale to the AI assistants most people encounter.
Why this matters
Whether or not the formula becomes a deployable safeguard, the study highlights a pressing point: small changes in how a chatbot is prompted can produce large shifts in behavior. That fragility matters because real users — from clinicians to military personnel — sometimes run models offline or with limited oversight, where downstream harms could be harder to catch.
Study citation: Johnson, N. F., & Huo, F. Y. (2026). Competition for attention predicts good-to-bad tipping in AI. Patterns. https://doi.org/10.1016/j.patter.2026.101666
