In uncertainty quantification, we often ask how sensitive predictions are to perturbations of the model or the input. I am increasingly interested in a different question. For scientific agents, how sensitive are conclusions to perturbations of the organization of knowledge itself?
To think about that question, it helps to separate two notions of stability in agentic systems: output stability and path stability.
Output stability asks whether the agent reaches the same answer. Path stability asks whether it gets there the same way. The distinction is not new, but it becomes increasingly useful once agents begin operating over large search spaces of knowledge, tools, and actions. The two can move independently. An agent may produce highly stable outputs while exhibiting substantial path diversity. The answer is invariant. The trajectory is not.
Knowledge Organization Shapes Search
One possibility is that this diversity reflects differences in the organization of knowledge itself. Every organization defines a different search space. It determines which concepts become neighbors, which connections are preserved, and which paths through the space become available for exploration. Perturb the organization, and the search changes with it. The same conclusion may therefore emerge from very different paths.
This matters because the path may contain information that the final output does not. Multiple paths converging on the same conclusion often feel like evidence of robustness. Sometimes they are. But not always. Some of it may be a waste. When we ask an agent to convert Fahrenheit to Celsius, there is a direct and efficient path to the answer. Path variation there is simply unnecessary computation. Some of it may indeed reflect robustness. Different paths, supported by independent lines of evidence, converge on the same conclusion. Some of it may reflect uncertainty. The available evidence may remain consistent with multiple competing explanations, yet one explanation is consistently easier for the agent to reach than the others.
From the output alone, these regimes can look similar. Path diversity therefore becomes more than an execution artifact. It becomes a window into the organization of knowledge over which the agent reasoned. The trajectory reflects not only the decisions made by the agent, but also the structure of the search space that made those decisions possible.
Execution Paths as Signals
This raises a practical question: how do we evaluate a knowledge organization? Today, we largely do so indirectly through downstream task performance. Modern benchmarks for scientific agents evaluate whether an agent can answer questions, solve scientific tasks, or generate hypotheses. But the feedback they provide is sparse. It requires labeled tasks, conflates the quality of the representation with the capabilities of the agent, and often provides little guidance on how the knowledge representation itself should change.
Execution paths offer a different kind of signal. They are a consequence of the organization of knowledge itself. If different organizations consistently produce different patterns of exploration, those patterns may reveal properties of the representation that the final answer cannot.
This perspective becomes useful when we try to understand why agents fail. LifeSciBench (Liu et al., 2026) highlights an interesting failure mode. Agents often perform plausible local reasoning yet fail on tasks requiring adherence to the exact constraints under which that reasoning remains valid. Consider a student solving a mathematical problem. Every algebraic step is correct. Along the way, however, a special case is treated as general. An assumption is forgotten. Another student may even reach the same answer through a valid derivation. The difference is not in the conclusion. It is in the conditions under which that conclusion remains justified.
In scientific reasoning, every result is conditioned on assumptions, methodology, population, uncertainty, and scope. A knowledge organization must therefore preserve more than facts; it must preserve the boundaries that determine when those facts remain valid. A representation that collapses those boundaries may no longer faithfully represent the space in which valid scientific reasoning can occur.
The interesting question is not simply why the agent failed. It is what property of the underlying knowledge organization that failure reflects. This suggests a different view of synthesis. The challenge is not simply to connect or unify knowledge, but to preserve the distinctions that define its boundaries while enabling new relationships to emerge.
Learning Better Organizations
Viewed this way, execution paths become more than traces of an agent's behavior. They become observations of the search space induced by the knowledge organization itself. Properties such as waste, robustness, and uncertainty therefore reflect not only the agent's reasoning process, but also the representation that made those paths possible.
Perhaps that is the larger opportunity. Instead of evaluating a knowledge organization only through downstream task performance, we may be able to learn from the paths it induces. Much of the focus has been on giving agents better skills and better tools to navigate the search space. But perhaps the goal is not only to build agents that navigate better. It is to build knowledge organizations that continuously reorganize themselves to make better navigation possible.
We've recently seen highly skilled mathematicians use AI to solve one of Paul Erdős's long-standing open problems. The breakthrough did not come from becoming more skilled. It came from discovering paths that were previously hidden.