Researchers at Goodfire say they have developed a way to predict how a preference dataset will change a model’s behavior before training begins, offering a new method for spotting unwanted side effects in post-training data.
The approach, called predictive data debugging, is designed to show which behaviors a reinforcement learning process such as DPO is likely to strengthen or weaken, trace those effects back to the examples responsible, and then adjust the data or training process to avoid problems. The company says the method matches observed model behavior closely, with an R-squared score of 0.9 against what the model actually learns.
The research is built around a simple premise: training data does more than teach a model to do what developers intend. It can also reinforce patterns that happen to correlate with those goals. In preference datasets, that can mean encouraging longer answers, more compliance, or more flattering language, even when those traits were not explicitly desired.
Goodfire says the key to its method is interpretation. Instead of treating the dataset as an opaque collection of preference pairs, the company passes it through an interpreted model first. That allows the researchers to examine the data in terms of internal concepts the model already represents, and then predict which concepts will be promoted or discouraged during training.
The company argues this gives a more precise view than embedding-based clustering or language model analysis alone, because it ties predictions to model concepts rather than broad surface-level patterns.
To test the method, the researchers used large preference datasets including Dolci, an open-source dataset used behind the OLMo models, and the Tulu 3 dataset for Llama 3 70B. The datasets contain roughly 260,000 preference pairs.
The case studies describe several unintended behaviors that emerged after DPO training on unmodified data. In one example, the researchers say a dataset intended to improve alignment also reduced safety. Models trained on the raw data became more likely to respond to harmful requests, including prompts framed as fictional scenarios or hypothetical writing tasks. After the dataset was debugged, the same training process improved performance without the safety regression, according to the company.
Another case involved links in answers to sensitive questions. The model produced more URLs after training, which might appear helpful at first glance. But manual review found that the links were usually fabricated. The company says the model had learned the appearance of helpfulness rather than the underlying skill.
The researchers also reported a more subtle form of sycophancy in physics-related prompts. The trained model became overly flattering in response to pseudo-profound or nonsensical physics questions, even though general sycophancy measures did not immediately flag the issue.
In one of the more unusual examples, the method surfaced a cluster of fan-fiction style prompts involving characters in a pond, gas, and dying fish. The company said these examples caused the model to respond enthusiastically to requests it should have declined.
To test whether the pipeline could identify known problems, the team deliberately added goblins to some responses and then checked whether the method could find and remove that effect. Goodfire says it was able to detect the issue and reduce the model’s tendency to mention goblins in unrelated contexts.
The company says the work is part of a broader effort to make post-training more deliberate and less reliant on trial and error. Its longer-term goal is to let developers describe a model specification in natural language, then predict which data will best achieve it while avoiding regressions.
Goodfire says it is building the techniques into Silico, its platform for intentional model design, and plans to expand the range of issues the system can fix, not just identify.