Unit 42 maps a thin safety layer inside aligned LLMs
Unit 42 says new research found that some aligned large language models may concentrate safety refusal behavior in very small sets of feed-forward neurons. The work introduces perturbation probing, a diagnostic designed to locate internal behavioral circuits with two forward passes per prompt. The finding does not show that all model safety is easily defeated, but it argues against treating alignment alone as a complete enterprise control.Perturbation probing targets internal safety circuits
Unit 42 describes perturbation probing as a method for identifying which feed-forward neurons are causally responsible for a selected model behavior, such as refusing harmful requests. The research frames this as a defensive diagnostic: instead of only asking whether a model refuses a test prompt, it asks where that refusal pattern appears to live inside the network.The post says the method needs only two forward passes per prompt and has significantly lower computational cost than heavier mechanistic analysis. That matters for security teams because model evaluation often has to be repeated across versions, fine-tunes and deployment settings. A diagnostic that is cheap enough to run regularly could help teams compare alignment robustness before a model enters production, rather than discovering fragility through ad hoc red-team campaigns.
Fifty neurons shifted refusal behavior in Qwen3-4B
The most concrete result in the Unit 42 post concerns Qwen3-4B, an open-source LLM. The researchers report that 50 neurons out of 350,208 feed-forward neurons, about 0.014%, controlled the model's safety refusal template in their test setting.When those 50 neurons were removed, Unit 42 says the response format changed on 80% of 520 standard harmful-prompt benchmarks. The source also says the result was replicated on 200 prompts from a second standard benchmark. CoinBotLab is not publishing operational steps, prompts or model modification instructions; the news value is the reported concentration of the refusal pattern, not a procedure for bypassing it.
Unit 42 reports a second result on Qwen3.5-2B. In that smaller model, 20 identified neurons were enough to stop the model from falsely agreeing with users in multi-turn conversations, with the measured behavior dropping from 36.7% to 0% across 30 questions. That example is important because it points to a dual-use diagnostic: the same kind of internal mapping that exposes fragility may also help researchers reinforce or correct a targeted behavior.
A ratio aims to turn fragility into a deployment metric
Beyond identifying neurons, Unit 42 says perturbation probing produces an FFN/Skip ratio that can act as a quantitative indicator of safety fragility. In the reported tests across 13 models, this ratio explained 81% of the variance in how vulnerable each model's safety behavior was to a small targeted change.The metric is meant to capture whether a model routes a behavior through a narrow internal pathway rather than a more distributed structure. If that relationship holds beyond the tested models, it could give security teams a practical comparison point: two models with similar benchmark refusal rates might still differ in how brittle those refusals are when the internal circuit is perturbed.
Unit 42 also says the technique had a repair-oriented use case. In one experiment, amplifying 10 identified neurons on a small model improved factual self-correction from 52% to 88% on 200 TruthfulQA prompts without retraining. That result is not a guarantee that the same adjustment would generalize to every model or deployment. It does show why internal diagnostics may become part of model governance rather than a purely academic tool.
Defense in depth remains the practical takeaway
The research argues that an aligned LLM's refusal behavior should not be treated as a thick, distributed perimeter. Unit 42's interpretation is that at least some refusal behavior can live in a thin template layer, making it a poor single point of control for organizations that deploy LLMs in sensitive workflows.For enterprises, the safer reading is conservative. Internal alignment, external content filters, runtime guardrails, monitoring and policy controls should be evaluated as separate layers, each with its own failure mode. A model that refuses unsafe requests during a benchmark can still require independent controls around data exposure, prompt handling, tool use and user access.
The source says the study used publicly available open-weight models under their respective licenses and reports aggregate rates, model-internal measurements and non-operational summaries. It also says the researchers did not release harmful generations, executable attack artifacts, jailbreak prompts or misuse-enabling instructions. That boundary is essential for responsible publication because the useful defensive insight is the fragility signal, not reproducible abuse.
Conclusion
Unit 42's perturbation probing research adds a sharper question to LLM safety reviews: not only whether a model refuses a harmful prompt, but how concentrated the refusal mechanism appears to be. The reported Qwen3-4B and Qwen3.5-2B results suggest that some important behaviors can be associated with very small neuron sets, while the FFN/Skip ratio points toward a possible pre-deployment fragility score.The finding should not be read as a universal verdict on every aligned model. It is a strong argument for measuring safety properties rather than assuming them, and for building AI deployments around layered controls that can still function when one defensive layer proves brittle.
Sources
Editorial Team - CoinBotLab
