Google study reframes factuality as a retrieval problem
Google Research says many factual mistakes in frontier large language models may come from failed recall rather than missing knowledge. The team introduced a behavioral framework called knowledge profiling to separate whether a fact is encoded, recalled, recognized or only recovered with thinking. In the reported tests, Gemini-3-Pro and GPT-5 encoded 95-98% of benchmark facts but still failed to directly recall 26-34% of them. The finding points to a practical shift: better factuality may depend not only on bigger models or broader training data, but on methods that help models use facts already stored in their parameters.A benchmark built around facts, not single answers
Google Research’s central move is to evaluate facts as the unit of analysis rather than scoring isolated questions. Standard accuracy can show that a model answered incorrectly, but it does not distinguish between a fact the model never encoded and a fact it encoded but failed to retrieve. The researchers call that distinction critical because each failure mode suggests a different remedy.The team introduced WikiProfile, a benchmark containing 2,150 Wikipedia-derived facts. Each fact is paired with 10 tasks designed to probe encoding, knowledge, recall and recognition. The pipeline extracts candidate facts from Wikipedia pages, generates direct and reverse questions, filters them for ambiguity and uniqueness, and applies a final manual validation step after automated filtering.
The evaluation covered 13 LLMs, each tested with and without thinking. For each model, fact and task, the researchers sampled eight responses and used prompted LLM autoraters to grade the outputs, producing about 4.5 million responses. That scale matters because the study is not merely asking whether a model can answer a prompt once; it is trying to classify the state of a fact across several ways of eliciting it.
Encoding looks strong while direct recall remains weak
The main result is that frontier models appear to encode far more factual information than they reliably recall in open-ended settings. Google Research reports that for Gemini-3-Pro and GPT-5, 95-98% of WikiProfile facts were encoded, while direct recall still failed on 26-34% of facts. With thinking enabled, the remaining failure rate fell but did not disappear, staying at 11-12%.The study defines encoding as the model’s ability to reproduce a fact in a pre-training-like context. Recall means answering semantically equivalent questions about an encoded fact, including direct and reverse forms. Recognition is different again: it measures whether the model can identify the correct fact when presented among alternatives.
That separation changes the diagnosis. If a model never encoded a fact, more data coverage or greater capacity may be the natural response. If the fact is already encoded but inaccessible, the problem is closer to retrieval, prompting, inference-time control or post-training behavior. Google’s framing is that the bottleneck in frontier models is shifting from knowledge acquisition to knowledge utilization.
Rare facts and reverse questions expose the recall gap
The recall problem becomes clearer on long-tail facts. Google Research compared low-popularity and high-popularity facts and found that rare facts were encoded at rates relatively close to popular facts, while the recall gap was larger. In other words, many rare facts may not be absent from the model’s parameters; they may be difficult for the model to access when asked in a normal closed-book question.Reverse questions show a related pattern. A model may handle a fact in the direction it likely encountered during training but struggle when the relation is queried in reverse. The source describes this as a refinement of the reversal curse: reverse questions are consistently harder in open-ended generation, but not necessarily harder in multiple-choice verification.
That distinction is useful for product design and evaluation. If a model can recognize the correct answer among distractors but cannot generate it from a reversed prompt, the failure is not simply missing bidirectional knowledge. It is a retrieval failure triggered by a change in phrasing, direction or context. Benchmarks that only record final-answer accuracy can miss that difference.
Thinking helps most when the fact is already encoded
The study also finds that thinking is most helpful where direct recall is weakest. Google Research reports that in thinking-optimized models, thinking recovers roughly 40-65% of encoded-but-not-directly-known facts. For facts that were not encoded, the benefit is much smaller, at 5-15%.That pattern suggests thinking often acts less like pure multi-hop reasoning and more like a recall-facilitation mechanism. It gives the model a route to access facts that were already represented but not immediately available under the initial query. The gains are especially pronounced for rare facts and reverse questions, narrowing both the popularity gap and the directionality gap.
There is a cost. The source notes that thinking is not free computationally, and it remains unclear when a model should invoke it. That creates an engineering trade-off for future systems: apply slower thinking broadly and pay for unnecessary computation, or build better triggers that detect when a factual query is likely to need retrieval support.
Conclusion
Google Research’s study argues that factuality work should look beyond whether a model has memorized enough facts. On WikiProfile, the reported frontier-model pattern is that encoding is close to saturation while recall remains incomplete. That makes the problem less like empty shelves and more like lost keys.The findings are still framed as behavioral evidence, not a direct inspection of model internals. The benchmark is Wikipedia-derived, the pipeline uses prompted LLMs, and response grading is automated by prompted autoraters. Even with those limits, the result gives evaluators a sharper vocabulary for factual errors: absent knowledge, inaccessible knowledge, direct recall, recall with thinking and recognition without generation.
For developers, the implication is practical. More scale may still help, especially for weaker models, but the next gains in factual reliability may also come from inference-time methods, post-training and evaluation tools that identify when a model is likely to know a fact but fail to retrieve it.
Sources
Editorial Team - CoinBotLab
