Chapter 6 — What Does a Result Bear On? Testing, Failure, and Revision

1. From a result to what it bears on

On Arfini and Costa's reading, what was to be explained for Penzias and Wilson changed in the course of the work of excluding possible causes of the noise picked up by the horn reflector (Chapter 4, Section 4). On this reading, results obtained within the work were revising the question. But what a result revises is determined neither by the result itself nor by the availability of candidates. The availability of a candidate does not determine whether believing, accepting, developing, rejecting, or leaving it unresolved is warranted. Nor does conducting an operation called a test determine what its result means. What this chapter mainly takes up is the relation between a result and what it bears on.

An unexpected result does not say which commitment calls for revision. A passed check does not say, by itself, how far its support reaches. This chapter asks how, while an inquiry is still changing, the work of determining what a result bears on is done, what guides it, and where it becomes hard. Chapter 2, Section 5 set out a schema, testing a claim by a method under auxiliary commitments to obtain an outcome, interpreting that outcome as evidence in light of background knowledge and standards, revising an attitude because of that evidence, and asking whether the revised attitude is knowledge, and, reading it line by line, called these four "steps." The same section also said that the schema locates the questions to be assessed and is not an algorithm for settling them. This chapter takes up what were there called the second and third arrows: the move from an outcome to its evidential bearing, and from that bearing to a revised attitude. This chapter, however, reads the order of the schema's lines not as the order in which inquiry proceeds but as a dependence: where a result bears as evidence, and what it revises. The issue here is which of the question, the candidates and their dependencies, and the methods, measurements, concepts, and standards through which results are assessed a result bears on, and how it constrains and revises them.

The central claim is limited:

A result is an outcome of an interdependent arrangement. Moving from that outcome to a change in epistemic attitude or practice requires a further process of interpretation and diagnosis. Neither success nor failure identifies its own cause, target, or appropriate revision.

This claim does not entail unrestricted holism, the equal revisability of every commitment, or the impossibility of rational diagnosis. Its point is that locating what a result bears on is work that inquirers do, not something the result supplies. This book calls that work localization: diagnosing which commitment to doubt, examine, or alter next, rather than dividing a failure among everything that contributed to it; Section 2.4 gives the reasons for this usage. That work draws on background knowledge, error models, independent checks, interventions, practical skill, convictions and expectations, and further inquiry, among other things (Sections 2.3 and 4.1), rather than following mechanically from a scalar outcome. Section 2 examines what guides it and where it becomes hard; how far it can be accomplished for some kinds of arrangement is itself disputed (Section 2.4).

The claim is about evidential bearing, not merely causal complexity. An outcome can have indefinitely many causal conditions. The narrower problem is that a registered outcome becomes a reason to alter some commitment only through a represented or reliably managed relation among the question, the dependency arrangement (defined in Section 2.1), possible outcomes, and candidate revisions. The world constrains the result, but the result does not arrive with its evidential interpretation attached. Evidential bearing is meant here in what Kelly calls the normative sense of evidence, evidence as what makes it more reasonable to believe something, rather than in the sense of a reliable indicator, a relation that can hold before anyone knows of it. Because evidence in this sense bears on belief, this chapter works mainly through the case of knowledge. Where the investigators are artificial systems, it speaks of their commitments and of what a result gives them reason to revise, without assuming that they are themselves the subjects who believe or know (Chapter 2, Sections 2 and 2.1). Where inquiry aims at understanding, results also bear on models and explanations, and their assessment adds the questions of idealization discussed in Chapter 3, Section 3.4. Some philosophers take evidence in the normative sense to be a three-place relation, on which a result is evidence for a hypothesis only relative to a background theory (Kelly 2014, §3; compare the second arrow of the schema in Chapter 2, Section 5, from a result and background standards to evidential bearing, and Longino's account of evidential relevance in Chapter 2, Section 6). In the indicator sense, a result can already point to a faulty element that no one has yet located; the claim of this chapter concerns what a result gives investigators reason to revise. This does not make truth or evidence products of social agreement. Relativity to background knowledge is not relativity to agreement: in Kelly's example, someone who does not know that Koplik spots indicate measles lacks a reason that a better-informed observer has, and would have it on acquiring the relevant medical knowledge (Kelly 2014, §§3–4).

The argument begins with unexpected results and their diagnosis (Section 2), then asks how far successful checks support a claim (Section 3). It follows both judgments across a sequence of results (Section 4), before turning to revision of the standards through which results are assessed (Section 5). Section 6 states what this account leaves to the analysis of artificial actors. This is an order of argument, not a sequence every inquiry follows.

The chapter describes how inquirers distribute doubt and protection in practice and sets out positions on what would warrant doing so.

2. When a result runs against expectation

This section asks which commitment a result that runs against expectation gives reason to revise, and how that comes to be settled. Its five subsections take up, in turn, where a result comes from, and what that suggests for research by machines (Section 2.1); why, although a contrary result bears in principle on the whole arrangement, not everything is equally in doubt (Section 2.2); what inquirers in fact doubt first (Section 2.3); what guides localization, where it becomes hard, and between whom it is contested (Section 2.4); and failures that have not reached the focal claim at all (Section 2.5). This order runs from the structure that in principle makes localization a problem, through the prioritizing that research in practice does, to the practice of localization and the kinds of failure it meets.

2.1 A result comes from an arrangement

A result does not interpret itself. The world can causally constrain what happens without thereby supplying a label that says which commitment the result supports or undermines. An expected observation is derived or anticipated through an arrangement that can include a focal claim, background theory, auxiliary assumptions, initial and boundary conditions, materials and organisms, environmental conditions, instruments and calibration, observers' readings, code and parameters, data selection and transformation, controls, statistical models, inferential rules, and divisions of competence, among other things. The reported result is also produced and made available through an arrangement. This chapter calls such a set of elements, together with the relations among them on which producing and interpreting a given result depend, a dependency arrangement. It is the part of an inquiry configuration (Chapter 5, Section 1.1) on which one result draws, seen from the side of the test. Deriving an expectation through such an arrangement is one way of articulating a candidate's consequences and dependencies, the work that Chapter 5, Section 4 noted serves appraisal as well as pursuit. What inquiry can register as agreement or conflict is therefore a relation among world-constrained but theoretically, materially, and inferentially mediated products—not a direct comparison between one hypothesis and an uninterpreted world. Nor does the point depend on the contested thesis that observation itself is theory-laden: even investigators who agree on how a result is to be described can, on views that relativize the bearing of evidence to background theory, priors, or paradigms, still disagree about whether, and how strongly, it supports a given hypothesis (Kelly 2014, §4).

A version of this point is generally known as the Duhem–Quine thesis, a name given jointly to two claims of different scope (compare Chapter 2, Section 5). Duhem argued, for physics, that an experiment cannot submit an isolated hypothesis to test but only a whole group of hypotheses, and that when a prediction fails the experiment teaches only that at least one proposition in the group is in error, not where the error lies (Duhem 1906, Part II, ch. VI, §2). Duhem also treated the laws of logic and the rules of algebra as rules to be respected in constructing a theory, and held that the test by facts bears on the theory's conclusions (Duhem 1906, Part II, ch. VI, §7). On Duhem's holism, then, what a contrary result can cast doubt on is a group of physical hypotheses, and logic can be read as not among them. Quine extended the point to the whole of knowledge, from geography and history to pure mathematics and logic: statements about the external world face experience "only as a corporate body," any statement can be held true come what may if drastic enough adjustments are made elsewhere in the system, and no statement, not even a law of logic, is immune to revision (Quine 1951, 38–40). Ariew notes that Quine credited Duhem only with the first of these claims and that Duhem did not make the second (Ariew 2026, §2.1). This chapter relies only on the narrower claim the two share: a result that conflicts with an expectation derived from several commitments does not by itself identify which of them is at fault.

The structure Duhem described for physics also appears outside it. Statistical model checking offers an example. Gelman and Shalizi treat a posterior predictive check as a test of a model's assumptions, the prior among them, which has implications for the data only in conjunction with the likelihood; when a model fails such a check, on their account, the analyst has to work out what went wrong and where the problem lies before tinkering with the model or redesigning it altogether (Gelman and Shalizi 2013, 12–13 and n. 12, 25). Chapter 2, Section 5.3 sets out a related argument of Otsuka's, including the claim that the error rates a significance test reports depend on assumptions the test itself does not examine.

A failed prediction does not by itself refute the focal hypothesis. What logic delivers, where the expectation was deduced from the commitments, is that something is false somewhere in their conjunction (an outcome that a statistical model had only made improbable does not establish even that); to reject the hypothesis one must add the claim that the remaining commitments held, a claim that can itself be wrong. To say that a hypothesis has been refuted is to make that claim implicitly. The same holds for success. A prediction that comes true does not by itself settle which commitments it supports, or how much (Sections 2.2 and 3.2), since a false hypothesis can combine with a false auxiliary to yield a true prediction. Nor does a "crucial experiment" settle, in that simple form, which of two hypotheses is true. On Duhem's account, what Foucault's comparison of the speed of light in water and in air decided between was not the emission and wave hypotheses but two whole systems, Newton's optics and Huygens's, each taken as a block; and it had not been excluded that light might be something other than a stream of projectiles or a wave (Duhem 1906, Part II, ch. VI, §3). Reading off from a result where the fault lies is therefore not something the result supplies but work done in addition to it. This is the work that Section 1 called localization.

When a machine conducts research, what enters the arrangement on which a contrary result bears, and where and by whom is the attribution made? First, the contents of the arrangement change. In machine research, the arrangement can include code and data processing that the system has written itself, tools it has called, outputs of other systems, and the means of evaluation, among others. Much of this may have been assembled by the system itself, or may not be surveyable as a whole even by those who assembled it. In such an arrangement, the possibility that something elsewhere in it is wrong need not be smaller than the possibility that the focal hypothesis is. Second, what was just said applies directly to machines' reports of results. When a system reports that a hypothesis has been refuted, the report implicitly contains the claim that the rest of the arrangement held; a report of success likewise contains a claim about how far the support reaches (Section 3). A system that takes the result of an experiment directly as a verdict on the focal hypothesis therefore does not take up the work of localization, and building such a system does not by itself build one that determines for itself what to learn from a result. Third, the scale of trials changes. When many trials are run automatically, verdicts can accumulate without differences that come from the arrangement being distinguished from differences that come from the hypothesis.

Who checks such attributions changes with the combination of A and S. If A is a machine and S human, members of S check the attribution contained in the system's report against the record of the arrangement, that is, what was assembled in what order and what had been checked (Chapter 5, Section 6.3). If the record does not suffice for that, those members cannot tell from the record what a reported refutation or confirmation bears on; other grounds for relying on the report may still be available (Chapter 11, Section 5.2). If both A and S are machines, the one that checks is also an inquirer with an arrangement of its own, and checking inherits the same question. In either case, what the Duhem–Quine point suggests for research by machines is that receiving a result raises not only the question what the result was but also who judged, and how, where in the arrangement it bears. The end of Section 2.4 and Section 4.3 return to this question from actual records.

2.2 Not everything is equally in doubt

In principle, then, a contrary result bears on the whole arrangement, and where to attribute the fault is not settled by the result. Research in practice nonetheless doubts something first, keeps something, and starts examining somewhere, since checking one element relies on others held fixed (compare Neurath's image, taken up in Section 5.2, of sailors who cannot rebuild their ship in dock and must rebuild it at sea), and an inquirer with limited attention cannot examine many at once in any case. That prioritizing can rest on various things, including degrees of confidence in each element (Chapter 5, Section 4.2; Section 2.3), how much else connected with an element would have to be reconsidered if it were revised, the personal and social goals that support a theory's entrenchment (Section 2.3), and the tacit knowledge practitioners have of their own models (Section 2.4). Of these, the personal and social goals come from Chinn and Brewer's review of episodes from the history of science and of studies of students; what, if anything, plays their part when research is done by a machine is a question Section 4.2 returns to. This subsection and the next two ask what such prioritizing depends on and how far it can be grounded. This subsection first brings out that the elements of an arrangement do not all bear on a result in the same way. It then sets out why neither Duhem's point nor Quine's implies that every commitment is equally open to revision, why some philosophers reject the holism often drawn from them, how Duhem described the choice that logic leaves open, and how the same question arises within mathematics, where the commitments a counterexample can bear on might seem to be fully listed.

Not every element of an arrangement bears on a result in the same way, so localizing a fault is not a choice among interchangeable items. In the notebooks of the drug-discovery project Open Source Antibiotics (Chapter 5, Section 2), for example, the material that came off a silica column as a yellow solid was run again on a reversed-phase column, which separates by different properties, and divided into two: a white powder that the chemist labelled "DEBROMINATED CORE," the core structure with its bromine lost, that is, a by-product, and a much smaller amount of the intended product, which the next run scaled up (Tse 2023, EGT 522-1). Readings taken on the yellow solid could have been reliable as measurements and still have been readings of a mixture rather than of the intended compound. Had such readings run against expectation, what called for doubt might have been neither the hypothesis nor the instrument but what the result was a result about. Recognizing that a result comes from an arrangement also does not mean that the arrangement must be revised wholesale, or that an inquiry must represent every causal condition in the universe. What a description has to represent is set by the inferences and corrections that an inquirer or a system claims to make.

Which dependencies are salient also changes with the question asked. A reagent's storage history matters only for some questions; a classification scheme may be background in one episode and the focal object in another. The process that produces a measurement can itself become the object when a different question is put to the same records. In electronic health records from two Boston hospitals, for example, patients whose white blood cell count was normal when measured at 4 a.m. had lower three-year survival than patients whose count was abnormal when measured at 4 p.m., and for 118 of 174 laboratory tests the timing of the orders predicted survival better than the results; the authors attribute much of this to the processes of care that determine when tests are ordered, while allowing that daily biological variation may also contribute, and in their study the process of care, which a test result does not itself record, became the object of analysis (Agniel, Kohane, and Weber 2018).

Duhem stated the narrower claim for physics, contrasting it with the logically simpler testing of physiology, while noting that any science relying on physical instruments also presupposes physical theory; later discussion has extended it to theoretical testing in other fields (Duhem 1906, Part II, ch. VI, §1; Stanford 2023, §2.1). The claim does not imply that every logically available revision is equally reasonable, does not amount to radical conventionalism, the view that which revision to adopt is purely a matter of convention, and does not imply that every component is always equally doubtful. Quine's broader holism does not imply this either. Alongside the claim that any statement could be retained, Quine described a statement's nearness to the periphery of the system as nothing more than a loose association reflecting how likely particular statements are to be chosen for revision, and named among what guides actual revisions "our natural tendency to disturb the total system as little as possible," conservatism, and the quest for simplicity (Quine 1951, 40–41, 43). The stronger thesis that every such retention is rationally defensible is the one Laudan argued had never been adequately supported: on Laudan's account, as Stanford reports it, arguments for underdetermination show only that many responses to contrary evidence are logically possible, and although deductive logic alone may not single out one response, logic together with the ampliative principles of reasoning used in science, principles of inference to conclusions not contained in the premises, may do so (Stanford 2023, §2.2). One reason the tendency not to disturb the system can matter lies in Quine's own account of why a contrary experience spreads: re-evaluating some statements entails re-evaluating others, because of their logical interconnections (Quine 1951, 39). Trying to revise a commitment on which much else has come to rest can therefore mean reconsidering what rests on it, and there are cases in which, unless something consistent can take its place, what is put in its place will not hold either. Doubting something that has withstood many checks can, in such cases, be costly. Duhem's example, discussed in Section 2.3, can be read as one such case: the principle that light travels in straight lines in a uniform medium had served almost as the physical definition of a straight line, and was abandoned when diffraction effects stopped being attributed to some source of error.

Some later philosophers accept the logical point while rejecting the holism often drawn from it. Sober separates a simple logical point from a contentious epistemological one, accepting the first and rejecting the second. Hypotheses rarely make observational predictions without auxiliary assumptions, but it does not follow that observations confirm or disconfirm only the conjunction, because auxiliary assumptions are often tested independently, so that a result can discriminate between rival hypotheses while leaving the shared auxiliaries untouched. A pregnancy test, in Sober's example, provides evidence about whether a woman is pregnant but leaves the woman's confidence in the reliability of the test unchanged; testing that reliability takes a different experiment. Sober takes epistemological holism to be wrong "in the vast majority of cases," while allowing that there may be circumstances in which a conjunction cannot be pulled apart (Sober 1999, 54–56). A later treatment in terms of likelihoods describes the typical situation in the calibration of measuring instruments and the validation of test procedures: their error characteristics "are first ascertained and then the procedure is applied to new individuals," as Galileo gauged the telescope on terrestrial objects before turning it to Jupiter, whose observation then told much about Jupiter's moons and little about the telescope (Sober 2004, 230–231). Whether holism is right about how evidence bears on a conjunction, Sober concludes, "crucially depends on what the alternative conjunctions are"; holism has its place, but in scientific inquiry it is, in Sober's words, "by far, the exception and not the rule" (Sober 2004, 241). This analysis of how far holism holds relies on testing auxiliaries independently. Section 2.4 takes up an argument that this response becomes unavailable in some arrangements.

Elgin's argument, discussed in Chapter 3, Section 7.2, gives a general reason in the same direction: commitments independently supported by other well-confirmed constellations are less likely to be the locus of error. Researchers can have strong reasons to protect some dependencies and investigate others. On such accounts, independent checks, calibration histories, robustness, error models, background theory, mechanism, cost, and later interventions can all help locate what to revise; whether successful mutual adjustment suffices to justify a localization is disputed (Section 2.4). Either way, localization is not contained in the word failed.

Duhem's own answer to what guides the choice was not a rule, nor did it treat the choice as arbitrary. On that answer, when experiment contradicts a theory it does not say what to change, no absolute principle guides the search, and a physicist who saves the fundamental hypotheses by adding corrections and one who alters a fundamental supposition cannot accuse each other of illogic. Duhem added that some opinions consistent with logic are nonetheless unreasonable, and assigned the choice between the two physicists to good sense, whose reasons lack the rigor of logic and do not become clear to everyone at once. Biot and Fresnel provide the example of a disagreement that logic did not settle and that lasted as long as it did for this reason: Biot defended the emission theory of light with corrections and auxiliary hypotheses, and Fresnel kept opposing it with new experiments, until one side gave way; Biot abandoned the theory after Foucault's experiment, although, on Duhem's account, logic alone did not compel Biot to (Duhem 1906, Part II, ch. VI, §10). The case also raises a question of timing, how long such a disagreement can reasonably last, to which Section 4.1 returns.

A dialogue of Lakatos's raises the question of this section within mathematics, where the commitments a contrary result can bear on might seem to be fully listed. In this classroom reconstruction of the history of Euler's conjecture about polyhedra, a history noted at the opening of Chapter 2, counterexamples are, as Musgrave and Pigden sort them, of three kinds: counterexamples to the conclusion of a proof that refute none of its premises, which Lakatos calls global but not local (Lakatos's term, distinct from this chapter's localization); counterexamples to a premise that leave the conclusion standing, local but not global; and counterexamples to both. On this analysis a counterexample to the conjecture is a reason to improve the proof rather than to abandon it (Musgrave and Pigden 2026, §3.1). One response to such a counterexample, as Aberdein recounts the dialogue, is to defend the proved theorem by declaring the counterexample no polyhedron at all, a monster rather than a counterexample, and to propose a further redefinition for each new monster; Lakatos calls this monster-barring and characterizes it as a "sometimes deft but always ad hoc" redefinition (Aberdein 2019, 5, 9). The dialogue can be read as posing for a proof the question this section poses for a test: which part of a structured commitment a contrary result bears on, and when protecting the rest becomes ad hoc. It also shows that even a proof whose steps have been written out may not list every commitment a counterexample bears on: a counterexample to the conjecture that refutes none of the stated lemmas sends the inquirers back to the proof to unearth a "hidden lemma" (Musgrave and Pigden 2026, §3.1).

2.3 What is doubted first

Section 2.2 left open which commitments inquirers in fact doubt when a result runs against expectation. On the second view of Chapter 5, Section 4.2, an inquirer moves toward what seems likely, with degrees of confidence that depend on the standards on which it stands and on what it trusts. The same degrees of confidence bear on where doubt goes when a result runs against expectation. Bayesian treatments of Duhem's problem make this explicit, as an answer to the question what would warrant blaming one conjunct rather than another. On the treatment as Strevens develops it, a refutation considered solely as such, apart from whatever else the evidence says about the hypothesis, concentrates suspicion on the conjuncts that were more doubtful beforehand: conditional only on the conjunction's being false, the hypothesis loses less probability the more probable it was beforehand, and typically more the more probable the auxiliary was (Strevens 2001, §3). Mayo, writing against Dorling's earlier Bayesian reconstruction, argues that it can be run in reverse—someone who assigns the same degrees of belief with the hypothesis and the auxiliary interchanged reaches the opposite verdict—and that blaming an auxiliary is warranted only when the claim that it is at fault has itself passed a severe test (Mayo 1996, 106, 108–109), one that would very probably have failed the claim were it false (Mayo and Spanos 2006, 328–329). Whether degrees of confidence by themselves ground such an attribution is thus disputed. This subsection asks what inquirers in fact do, and it describes the answer loosely.

Which part of an arrangement is doubted first has also been studied empirically. Drawing on episodes from the history of science and on experimental studies in psychology and science education, Chinn and Brewer distinguish seven responses to data that contradict a held theory: ignoring the data, rejecting them (for example as a methodological error, random variation, or fraud), excluding them from the theory's domain, holding them in abeyance, reinterpreting them, making peripheral changes to the theory, and changing the theory. They argue that these exhaust the psychologically plausible combinations of three decisions: whether to accept the data, whether and how to explain them, and whether to change the theory (Chinn and Brewer 1993, 4, 12). They also report conditions that favor the six theory-preserving responses, in a list they do not present as exhaustive. One is the entrenchment of the prior theory, which they characterize by its evidential support, the range of explanations it takes part in, and the personal or social goals it serves. Another is the lack of a plausible alternative theory, often of a plausible mechanism. A third concerns the data: data that lack credibility, are ambiguous, or rest on a single study open to methodological criticism or reinterpretation (Chinn and Brewer 1993, 13–26). Discussing instruction, they add that it is frequently more rational to discredit anomalous data than to change theories (Chinn and Brewer 1993, 27). In experiments on everyday reasoning with conditionals, Walsh and Johnson-Laird found that participants asked to explain an inconsistency mostly revised the general conditional rather than the particular fact, and on most trials made more than a minimal change, as the authors measure minimality, by counting the propositions whose truth values change; they argue that people seek to explain an inconsistency rather than to minimize change (Walsh and Johnson-Laird 2009, 626–630). Neither study observed working researchers: Chinn and Brewer's examples of scientists come from histories of science, and Walsh and Johnson-Laird's participants reasoned about everyday conditionals.

Records kept while research was under way let one trace more directly what was doubted first. When Rosemary Redfield began in 2011 to repeat, in a research blog kept in the open, the claim that the bacterium GFAJ-1 can use arsenic in place of phosphorus, the cells grew on agar plates but not in liquid medium. The first posts suspected the tap water in the medium, the new culture tubes, and the air admitted by loose caps; a post three days later reported that it was not the water, the tubes, or the parafilm; and after a correspondent suggested potassium, two errors came to light, one in Redfield's own phosphate stock and one in the published recipe, which supplied no potassium (Redfield 2011a; Redfield 2011b; Redfield 2011c). When a flask with arsenate and limiting phosphate later showed thick growth, a result that seemed to favor the claim being tested, Redfield's first explanation was that a mistake had been made in setting up the flasks (Redfield 2011d). Doubt went first to the replication's own arrangement, and the claim under test was examined only once that arrangement worked. The OPERA collaboration's report that neutrinos had apparently arrived sooner than light would allow shows a different shape. The collaboration wrote that it would investigate possible still unknown systematic effects and deliberately offered no theoretical interpretation of the anomaly (OPERA Collaboration 2011, 22). Among the two possible effects it identified five months later was the connector of the optical fibre that brought the external GPS signal to its master clock (CERN 2011, update of 23 February 2012), in a timing chain whose fibre delays it had measured by two-way procedures (OPERA Collaboration 2011, 13). The result itself was doubted first, but what came into question was an element that had been checked. A third record shows how what is given up depends on which explanation is at hand. In 2021 a user of the transformers library reported that gradient accumulation gave different training curves from the equivalent batch size; a maintainer judged that the difference could be explained by precision, the user accepted the explanation, and the issue was closed, to be reopened a month later when the user found one source of the difference in how training examples were grouped by length (Hugging Face Transformers 2021). In 2024 a similar report met the same suggestion of precision, which the reporter set aside after a rerun in full precision showed the gap persisting; the gap was then traced to how the library's default loss averaged over accumulated batches (Hugging Face TRL 2024; Hugging Face 2024). An explanation that was always available closed the question once, and an element trusted as a default turned out to be at fault.

Taken together, these studies and the cases in this chapter support only a loose description. The result itself is often among the first candidates for doubt. Beyond that, doubt tends to gather on elements that seem less secure by the standards on which an inquirer stands, but what is given up also depends on what explanation of the conflict is available and on whether an alternative is at hand, that is, on which candidates are under consideration (Chapter 5, Section 4.2). The description states a tendency, not a rule that only the least secure element is revised. Duhem observed both sides of it. Among the theoretical elements entering a test, Duhem wrote, there are always some that the physicists of a period agree to accept without check, and a physicist who has to modify the theory will surely direct the modification elsewhere; yet logic does not compel this, and refusing to invoke causes of error, and reforming propositions held by common agreement to be untouchable, can open a new path for theory. The example given is the principle that light travels in a straight line in a homogeneous medium, which had served almost as the physical definition of a straight line and was given up when inquirers stopped attributing the diffraction effects observed by Grimaldi to some cause of error (Duhem 1906, Part II, ch. VI, §8).

Observations of working laboratories show the branching that this description leaves open. Dunbar followed meetings and other activities in eight molecular-biology and immunology laboratories. In the reported cases, unexpected findings were variously attributed to unknown causes, methodological problems, mistakes, or potentially new mechanisms; follow-up included replication, further analysis, adopting a new protocol, and abandoning the current protocol. The exact frequencies are specific to the sample and coding scheme. Within that bounded sample, the observations support the existence of a branching diagnosis problem: researchers did not invariably convert an unexpected result into revision of the principal hypothesis (Dunbar 2000). Even their replications included minor changes of method, and the senior researchers attended particularly to unexpected findings in control conditions, both of which bear on loci other than the principal hypothesis (Dunbar 2000, 53). Dunbar also reports how the branches were handled: the scientists took an unexpected finding and tried to build a causal model of the events underlying it, deciding on their next step once that model was built, and when they were designing or fixing experiments their analogies were to very similar experiments or the same organism; Dunbar advises attending to unexpected findings in control conditions because they reflect either a conceptual or an experimental problem (Dunbar 2000, 53–54, 57). What these observations show is less an order in which commitments are doubted than that attribution branches, and some of the ways researchers work through the branches.

Where doubt goes can also involve convictions that come from outside the result at hand. Where a result allows either attribution, the choice between them can involve convictions and expectations, and even views about what is essential. Continuing to commit to an idea is one form of this, and is itself an attribution of results, in that it attributes results that do not work not to the idea itself but to not yet having realized it well. Judgments about where to attribute one failure after another may, accumulating, shape which ideas are kept. Hinton's recollections about neural networks, and those of the Transformer authors, can be read as cases of this; they are taken up in Section 4.1, together with the question of how long protecting a commitment remains reasonable.

2.4 What guides localization, where it becomes hard, and between whom it is contested

Granting that doubt is distributed loosely, what can guide the search for what to revise? This subsection sets out positions on that question: what experimenters use to argue that a result reflects the phenomenon rather than the apparatus, whether successful mutual adjustment suffices to justify a result, whether such guidance survives in arrangements of many jointly tuned components, who carries out localization, and what happens when the parts of an arrangement belong to different parties. It then states in what sense this chapter uses the word localization, and considers what comes into view for artificial research systems.

The philosophy of experiment has described some of the resources listed in Section 2.2 as strategies by which experimenters argue that a result reflects the phenomenon rather than the apparatus. Among them are checking that an apparatus reproduces known phenomena, eliminating plausible sources of error and alternative explanations, and using an apparatus based on well-corroborated theory (compare Chapter 5, Section 6.1); when a check of the first kind fails, Franklin and Perović write, there is good reason to question the results obtained with that apparatus. They present these as strategies from which physicists choose rather than as rules, neither exclusive nor exhaustive, and hold that no single one of them, nor any fixed combination of them, guarantees that a result is valid (Franklin and Perović 2023, §§2.1.2, 2.2.4). A more basic doubt has also been raised about whether such strategies can work at all. As Franklin and Perović report, Collins developed an argument and named it the experimenters' regress: a correct result is one obtained with a properly functioning apparatus, and a properly functioning apparatus is one that gives correct results; there are no formal criteria for deciding whether an apparatus is working properly, and the regress is eventually broken by negotiation within the community. Discussing the dispute over the detection of gravity waves, Franklin disagrees: the critics had exchanged data and analysis programs and calibrated their apparatus by injecting pulses of known energy, and the community, on this view, made a reasoned judgment (Franklin and Perović 2023, §2.2.1). The dispute is a reminder that a judgment of localization can itself be a result that does not interpret itself, since the judgment that the fault lies in the apparatus also comes out of the arrangement of checks that led to it. If so, where localization stops may not be settled by the logic of localization alone; the form is like that of the regress, discussed in Chapter 5, Section 6.5, that arises from trying always to decide by inquiry which inquiry to undertake. What stops it—reasoned judgment, negotiation, or both—can be read as what is in dispute here.

Whether stabilization by mutual adjustment suffices to justify a result is itself disputed. As Franklin and Perović set out Pickering's account of Giacomo Morpurgo's searches for free quarks, the observation of continuously distributed charge resisted his expectations. Widening the gap between the capacitor plates and revising the theoretical account of the apparatus brought integral-charge measurements and agreement among the apparatus, its theoretical account, and the theory of the phenomenon. They present this as Pickering's dialectic of resistance and accommodation (Chapter 5, Section 6.1) (Franklin and Perović 2023, §2.2.4). Franklin and Perović read the episode differently. Discussing Pickering's 1987 account of the episode, they argue that charge quantization had been strongly supported since Millikan, so that it was the failure of Morpurgo's apparatus to reproduce that known result, the failure of an important experimental check, that cast doubt on the measurements, whatever theoretical models were on offer; only when the apparatus could reproduce known measurements could it be trusted in the search for fractional charge. Against the 1995 account just outlined, they object that Pickering does not ask whether a given mutual adjustment of theory, apparatus, and phenomenon is justified, and seems to treat any adjustment that achieves stabilization as acceptable (Franklin and Perović 2023, §§2.2.2, 2.2.4). The two readings thus differ over what warrants localization: in Franklin and Perović's report of Pickering, the emphasis falls on the adjustment that achieves agreement; in their criticism, on an independently supported result that can guide the adjustment. Both deny that the result itself identifies the fault.

Humphreys describes guidance available before a test for derived theoretical templates, general mathematical representations obtained from explicit construction assumptions. A correction set supplies suggestions, not always explicitly formulated, for improving idealizations, abstractions, approximations, and other assumptions already known or suspected to be false at construction (Humphreys 2019, §3, advance-version 3 and n. 8). Knuuttila and Loettgers report the corresponding account of computational models: construction assumptions include ontology, idealizations, abstractions, and constraints, while a correction set anticipates avenues of adjustment to fit the data (Knuuttila and Loettgers 2023, §2). They also report Humphreys's more radical possibility of changing the ontology, with his qualification that this approaches constructing a new template (Knuuttila and Loettgers 2020, manuscript 31, n. 17). Ariga reports Humphreys's criticism of the no-ownership perspective, which treats templates as already finished objects rather than attending to their construction and modification (Ariga 2009, 85). The guidance described here concerns models derived from explicit assumptions; Lenhard and Winsberg's argument, taken up next, concerns arrangements of many jointly tuned components.

The response to a Duhemian situation discussed in Section 2.2, testing an auxiliary independently, is precisely what some philosophers of climate modeling argue becomes unavailable as an arrangement grows. Lenhard and Winsberg, as O'Loughlin reconstructs their argument, claim that climate modelers lack what they call analytic understanding of their models, the ability to identify the extent to which each component contributes to a model's successes and failures, and that the complexity of the interactions among components makes it impossible to assess the merits of each independently (O'Loughlin 2023, §§2–2.1). On their account, as O'Loughlin reports it, a model's output is produced by the interaction of its modules during a run, not by combining the modules' separate results; parameterizations are built and tested on the basis of those already in the model; and kludges, made to improve the model as it then exists, become entrenched as other components come to depend on them. Examining a new component and the rest of the model separately then does not show how the component changes the model's behavior, since it cannot be told which of its interactions is responsible (O'Loughlin 2023, §2.1). They take three model intercomparison projects, whose aim of attributing differences in performance to components was repeatedly postponed or scaled back, as evidence for this view, while granting that the sources of some errors were tracked down (O'Loughlin 2023, §2.2).

O'Loughlin replies with diagnoses of disagreement among models from the same and later intercomparisons. In one, an implied ocean heat transport that ran in the wrong direction in some models was traced to their simulation of the radiative effects of clouds, which the investigators suspected first because models were known to disagree about clouds, and the diagnosis was checked by substituting observation-based values. O'Loughlin argues that the difficulty may lie more in the intercomparisons than in the models—dozens of models differing in unsystematic ways, with knowledge of how each was built and tuned dispersed across institutions—and that a standard requiring the extent of each component's contribution, as Lenhard and Winsberg's account of analytic understanding does, could discount any diagnosis that names a component: since a named component produced the error only in conjunction with the rest of the model, one could always ask whether it was wholly or only partly to blame. O'Loughlin suggests that the success of such diagnoses may reflect practitioners' tacit but working knowledge of their models, "a kind of Duhemian 'good sense'" (O'Loughlin 2023, §§1, 2.3, 3.2, 4.1, 5).

Localization need not take the form of a researcher's explicit verbal reflection. A reliable control structure, diagnostic procedure, or coordinated division of labor can localize some errors without any one participant representing the entire dependency arrangement. A notebook kept during a directed-evolution project and later made public provides an example of the last. At 12:28 on 23 November 2007, the researcher running a screen reported that none of six plates, each with more than a thousand colonies, yet appeared to hold variants with the new activity, attributed this to low mutation rates in earlier rounds, and planned DNA shuffling and a further round, while also returning the plates to the incubator to see whether color would develop over a longer time. At 16:15 the investigator leading the project commented that, "from memory," when a colleague had run the screen "it took a while for the colonies to go properly blue" and the screen "needed quite a lot of x-gal," the indicator substrate. At 17:56, after 24 hours of incubation, the researcher reported color just beginning to develop and eight possible variants, and the next day, asked about photographs, added that the color was "very very faint" and might not show on camera (Neylon 2012, post 5305). Read in sequence, the entries suggest that what the negative-looking reading bore on—the mutation rate, the time allowed, or the amount of substrate—was narrowed across the entries of two participants and the recalled experience of a third, and over hours rather than at the moment of reading. In the terms of Chapter 5, Section 6.2, this is localization distributed in the thin sense. The published paper reports, for an experiment other than the run this notebook records, that colonies were picked after 24 hours of incubation (Smith, Hale, and Neylon 2011, 2); given as method, the figure does not show that in at least one run of the screen how long to wait had been in question, and the pair does not show where the figure came from. The trace was chosen for this example and shows nothing about how often localization proceeds in this way. Chapter 5, Section 6.4 described two other forms: in Keeling's case a visiting colleague's glance at the pen traces of three analyzers found the one at odds with the others, and in Pauling's notebook the writer's own reading was treated as one of the things the result depended on.

Where parts of the arrangement belong to different parties, localization can itself be contested. O'Loughlin's point about knowledge dispersed across modeling centers is one case of this. The review of a report on replication attempts in cancer biology that could not be completed, one of which Section 2.5 describes, shows it too. The reviewers asked the authors to address limitations in their own technical expertise as a possible explanation, and the authors replied that a lack of detailed methods and reagent information was more likely, while agreeing that no conclusion about the causes could be drawn (Errington et al. 2021b, decision letter and author response). Collingwood offers a classical analysis of one way such contests arise, although for one restricted sense of "cause." In the practical sciences of nature, such as medicine, Collingwood held, a cause is a handle by which people can produce or prevent what they wish, a sense distinguished from the one used in theoretical science. No one enumerates the conditions of an event: when a result one has been obtaining fails to appear, what one wants to know is not what all its conditions are but which one has changed. And for each party, the cause is the condition it is able to produce or prevent, so that the driver, the county surveyor, and the car manufacturer name different causes of the same skid (Collingwood 1940, 296–301, 303–304). The two contests differ in kind. In Collingwood's example the three parties need not disagree about the conditions of the skid: each names as the cause the condition it can alter, and on that analysis each answer holds for the party that gives it, so that further evidence about the skid would not by itself make their answers converge. The exchange over the replication report concerned instead which condition had probably made the difference, a question on which further checks could bear, although none had settled it. What is in dispute can also be which of the conditions counts as an error. Herndon, Ash, and Pollin, replicating Reinhart and Rogoff's averages of growth by level of public debt, attributed the low growth reported for the highest-debt category to three problems: a spreadsheet error that omitted five countries, the exclusion of some early postwar years, and a weighting that counted each country equally regardless of how many years it contributed (Herndon, Ash, and Pollin 2013, abstract, 7–10). Reinhart and Rogoff's errata accepted the coding error, reporting an effect in line with the one Herndon, Ash, and Pollin had found, and answered the other points by placing them in the context of the paper's longer sample, its medians, and the peculiarity of the years just after the war (Reinhart and Rogoff 2013, 1–2, 4).

The discussion so far brings several things into view. There are resources that guide localization, but they are not rules, and what justifies a successful adjustment is itself disputed. In arrangements of jointly tuned components, testing components separately can lose its purchase. Localization need not happen in one head, and where parts of an arrangement belong to different parties, localization itself can be contested. Collingwood's analysis can be read as describing one shape that localization can take: a search for the dependency that changed, or that some party can alter, rather than a division of the failure among all the dependencies. Records can build such a search into their form: entries in Edison's storage-battery tests that read "Same as 149" and then list what was changed name an earlier test as their baseline and record the changes against it, at times a single one, as in test 152, entered as "Same as 151 Except pkd water moist"; and the note to test 149 appears to say that the test was meant to show whether the fault lay in the grid or in the nickel oxide (Chapter 5, Section 6.3). Sober's analysis of apportioning causal responsibility gives a reason why such a division can have no answer. Where causes act jointly, Sober argues, the question how much each contributed has an answer only if the causes are commensurable in the way they produce the effect, as gravitational and electrical forces acting on a particle are; this condition is offered as a conjecture. Where there is no such common currency, as with genes and environment in development, how much difference each made depends on which alternatives are taken for comparison, and how much each contributed is, in Sober's words, "not a factual matter at all" (Sober 1988, 303–304, 309–312, 318). Herndon, Ash, and Pollin's own table shows this dependence on the comparison. Applying any one of Reinhart and Rogoff's three choices alone to the corrected data lowers the estimate for the highest-debt category from 2.2 to 1.9 percent a year, while applying the exclusion and the weighting together lowers it to 0.3 percent; the table reports every combination (Herndon, Ash, and Pollin 2013, 10, 21). How much each choice contributed thus depends on whether it is compared with the corrected data or with data to which the other choice has already been applied.

These cases and arguments give reason to understand localization less as apportioning a failure among its contributors than as determining where to intervene next—what to doubt, examine, or alter—and that is the sense in which this chapter uses the word (Section 1). That determination can also lead to recasting the question (Chapter 4, Section 4). The cases do not show that localization always takes this shape. What they show is rather how varied its shapes are. Localization can be guided by independent checks and by successful adjustments; it can be one person's judgment or spread across participants and time; it can be built into the form of a record or contested between parties. That variety is itself suggestive. Before treating localization as a capacity that one component or one procedure could implement, there is room to ask which form of localization is called for, in which arrangement, and by whom.

How do matters look for artificial research systems? The climate-model dispute above can be carried over to artificial research systems assembled from many jointly tuned components, with the difference that the localizing may be done either by the system itself or by those who build and maintain it; the practitioners' working knowledge to which O'Loughlin appeals corresponds to the second unless the system has a counterpart of its own. In either case, how far failures can be localized may depend on how the system was built, on how knowledge of it is distributed, and on what is taken to count as localization. On Humphreys's account, construction assumptions can supply advance guidance for correction. Humphreys does not apply this account to artificial systems, but applied to them, a system or members of S receiving a model without those assumptions may lack the corresponding guidance. By analogy, they may face the model as an already finished object, in the no-ownership perspective Ariga describes. The logbook of the training of OPT-175B, a large language model, at Meta illustrates a related loss of diagnostic information. After a run in which, in the logbook's words, "tons of stuff" had been changed at once, the team decided to "bisect recent changes to see what fixes instability" and rolled back one change, but in the run after that changed two things together; which layer was "really the problem" was recorded as an "unsolved mystery," and a run resumed from an earlier checkpoint was noted as not equivalent to the run it continued, because the restart had accidentally reset a state that checkpoints did not save (Meta AI 2022, 86–90). Because changes went in together and a resumed run was not the run it continued, even the team that built the model can be read as having had few means of checking which change had worked.

Collingwood's analysis concerns causes in the practical sciences, in a sense it ties to human control over what people wish to produce or prevent, not localization in research. Carried over to localization in the sense used here, where to intervene next, the analysis suggests that localization can also be relative to what the one localizing is able to alter and to what it seeks to produce or prevent. If so, an inquirer with means of intervention different from a human's, one that can, for example, repeat the same run any number of times or run many comparisons that change a single thing, may localize the same failure in a different place from a human. And whether the locus of a fault can be found afterward can depend less on the moment the result arrives than on how the arrangement was recorded before it (Chapter 5, Section 6.3), since what was changed when cannot be undone to check it unless it was kept.

Which localizing, the system's own or that of those who build and maintain it, is attributed to the system depends on where the boundary of the actor is drawn (Chapter 7, Section 2), and, as Chapter 5, Section 6.5 noted, when the inquirer is a machine the line between inquirer and element is not given either. What evidence could support attributing a more general capacity for diagnosis is taken up in Section 4.2. An inquiry can also render an anomaly harmless without localizing it—by a workaround that removes the symptom while saying nothing about its cause—and move on, much as in the response Chinn and Brewer call holding data in abeyance; a later account may then assign the anomaly a single cause (compare the BLOOM records in Chapter 5, Section 2).

2.5 Failures that have not reached the focal claim

Sections 2.1 to 2.4 have dealt mainly with cases in which the focal test was performed and its prediction failed, and have asked where in the arrangement that failure is to be placed. Not everything recorded as a failure, however, has reached the focal claim. Some failures bear on it only weakly or not at all, and for these, what the failure is has to be asked before where it is to be placed. As Chapter 1, Section 3 noted, failure is not exhausted by one verdict: a project can miss its original target while establishing boundary conditions, ruling out a possibility, or exposing an experimental artifact; the cases here add that which outcome a failure has depends on what it is taken to bear on, and that this can change after the fact.

A replication attempt can fail without the focal test being reached. Study 5 of the Reproducibility Project: Cancer Biology, for example, preregistered an in vivo test of a glioblastoma-tumor claim, but made that test conditional on producing stable cell populations that met a specified GFP-expression threshold (Chroscinski et al. 2015). After replacement materials, repeated optimization, enrichment, and changes suggested by the original authors, the required condition was not achieved for all of the relevant cell populations (Errington et al. 2021a, box 5; Errington et al. 2021b). The replication team stopped the attempt, citing the rising cost of repeated optimization with no feasible path to a solution, and the focal xenograft experiment was never performed (Errington et al. 2021a, box 5; Errington et al. 2021b). The attempt therefore neither replicated nor refuted the target tumor claim. It produced a record of where this implementation failed to reach its planned test, without identifying why it failed or showing that the original result is unreplicable. That a replication attempt failed, that a prerequisite could not be realized, and that the focal test returned a negative result are different things, and what can be said of this replication goes no further than the first two. Between them lie still other cases, in which a focal test is run in part, or under conditions that only partly meet its plan.

An implementation that does not realize the stated method can mislead the diagnosis of failure itself. In one language-model project, for example, a model trained and diagnosed as a 104-billion-parameter configuration turned out, because of a configuration error, to have been "a very lopsided 58B model instead of 104B all along," so that the failing runs had been made with a different model from the one described (BigScience 2021).

Even when the focal test has been performed and has returned a negative result, that result is not one thing. A null result may, for example, constrain a claim, reveal insufficient sensitivity (compare the screen in Section 2.4, read before its color had developed), expose poor operationalization (the step from a concept to a procedure for measuring it), or remain uninformative, and a statistically non-significant or otherwise indeterminate outcome may fail to discriminate among the relevant alternatives; a proof checker may likewise reject a valid idea because its formalization is defective. Which of these a result is depends on the question, measurement, sensitivity, and inferential setting. Collins's case study shows how, in gravitational-wave physics, a non-detection can be transformed into an upper limit, while also emphasizing that the resulting scientific value varies with field and audience (Collins 2003, abstract). What a negative result bears on can also depend on a question asked later. For example, several mutants of Mycobacterium tuberculosis unable to make the sulfated glycolipid SL-1 showed no detectable defect in mouse infection models when their makers reported them (Kumar et al. 2007; Gilmore et al. 2012); a different laboratory later used strains supplied by the same group to show that SL-1 contributes to cough in infected guinea pigs, and took the earlier in vivo findings as evidence that SL-1 is not a classic virulence factor, evidence with which its own results were consistent (Ruhl et al. 2020). What the earlier result was worth depended in part on a question its authors had not asked. In light of these cases, the claim that the target failed, the claim that the procedure failed, and the claim that the research episode failed are different claims, and what a negative result is taken to show changes with which of them is made.

For artificial research, these distinctions imply that a record of failed runs does not by itself say which failures bear on the hypothesis under test. A system that logs an unperformed test, a misdescribed implementation, and an insensitive null result alike as evidence against its hypothesis presupposes, as Section 4.2 discusses, where the fault lies.

The same holds for a system that reads records kept by others. The labels that participants put on their own work can record what is to be carried forward without recording which of the states or claims distinguished above applies to a result. In the public chemistry notebook described in Chapter 5, Section 6.3, for example, three pages of attempted reactions from October 2007 were marked "aborted" because the sample of an aldehyde used in most of the reactions was judged bad ("contaminated" and "providing false negative results," in one page's words), and two of the pages say that the compounds are to be attempted again; on a later page, two reactions were marked "Void" because ten times the intended amount of one acid had been added, so that an apparent precipitate came from insolubility rather than from reaction (UsefulChem 2010, Exp137–139, Exp141). The first label marks a prerequisite that was not met; the second marks a procedure that failed, and what it sets aside is an apparent success rather than a failure. Yet one of the aborted pages enters "aborted", and the later page enters "Void", in the table column where whether a precipitate formed would otherwise be entered, and on two of the pages the label also covers reactions that did not use the suspect aldehyde. A system that learns from or aggregates such records and reads their labels as results presupposes where the fault lies as much as one that logs all its own failures alike as evidence against its hypothesis. Nor are parallel attempts independent with respect to what they share. On the first of these pages, all eight reactions had been made with the same doubtful sample, so that one reagent set aside eight results at once; a count of parallel trials is not a count of independent checks on an element they share (compare Section 3.2).

3. When a check is passed

Section 2 argued that a result that runs against expectation does not by itself identify what it bears on. It also saw that where such a result is attributed can depend not only on the result but on convictions and expectations, on the means of intervention at hand, on records, and on exchanges between parties; that what a failure is can itself change afterward; and that such judgments may shape which paths inquiry goes on to take. When a result runs as expected, that is, when a check is passed, the pass likewise does not by itself say what it has established: which part of the inquiry's question it bears on, how firmly it settles that, whether the success depends on something other than what was meant to be tested, and which of the commitments that produced it the success supports.

What was asked on the side of failure thus has counterparts on the side of success. Just as it was asked of a failure what it is in the first place (Section 2.5), it is asked of a pass what part of the question it bears on and how firmly it settles that (Section 3.1), and whether the success is only apparent (Section 3.2); and just as it was asked where a failure is to be attributed (Sections 2.1–2.4), it is asked which commitments a success supports (Section 3.2). There is also a difference. A result that runs against expectation signals that something is wrong somewhere and prompts a search, whereas a pass can be treated as a reason to stop looking, so that a misreading of a pass can go unnoticed longer than a misreading of a failure (compare the primary mirror of the Hubble Space Telescope in Section 3.2).

When research is done by a machine, how it is arranged who sets a check, where in the inquirer it is placed, and who reads a pass and how, can differ from human research; and a pass may be recorded without the record keeping these differences (compare Chapter 2, Section 5). Section 3.3 returns to these arrangements. This section asks what a passed check establishes. It first fixes how the word verification is used here, then distinguishes the target and the force of a check (Section 3.1), turns to successes that are only apparent, to what a success that is not merely apparent supports, and to the security that agreement among checks provides (Section 3.2), and ends with who sets a check and who reads a pass (Section 3.3).

Chapter 2, Section 5 treated verification as a family of practices whose common function is the assessment of a candidate's epistemic standing under uncertainty, and Chapter 5, Section 5 compared heterogeneous operations by their constraint-bearing role. This chapter uses verification only as a broad comparative umbrella when the common function matters, and at the point of analysis it names the more specific practice and the inference that practice licenses. What is asked here, whatever the stage that did it is called, is what was assessed, by what operation, under which assumptions and standards, and with what bearing on the live alternatives. Sober's account of testability puts the question in a similar way. Defending the notion of testability, Sober argues that testing is contrastive, so that the same proposition can be testable relative to one competing hypothesis and untestable relative to another; what is empirical or nonempirical is in the first instance a problem or question, and propositions are testable only derivatively (Sober 1999, 52–53). The same paper describes the inferential tools by which scientists evaluate competing hypotheses in the light of observations as a toolbox whose contents are neither fixed for all time nor the same in all scientific disciplines (Sober 1999, 68). The function considered here is wider than Sober's, since it includes operations, such as proof, that do not weigh hypotheses against observations. Practitioners have themselves compared practices by their function without erasing the differences. Two mathematicians, Jaffe and Quinn, held that rigorous mathematical proof plays a role functionally analogous to experiment in the natural sciences, checking the reliability of claims and yielding new insight as a byproduct, while declining to call proofs experimental (Jaffe and Quinn 1993, 2). What the practices share, in this limited sense, is that something remains unresolved, a claim that might resolve it is on the table, and an operation is used to determine whether, and how firmly, the claim does so. The practices do not become epistemically equivalent merely because they can occupy a comparable functional position, nor, as Chapter 2, Section 5.1 and Chapter 5, Section 4 argued, need verification follow candidate generation as a separate phase.

3.1 Target and force

Chapter 2, Section 5.1 noted that verification practices differ in what they bear on, the alternatives they discriminate, their defeasibility, and the kind of standing they can provide. This subsection takes two of those differences to compare checks: their target, what part of an inquiry's question an operation bears on, and their force, how firmly it settles that. The pair is chosen because the bare report that a check was passed leaves both unstated; together they do not make a complete description of a check.

A valid proof settles what follows from its premises, but whether that answers an inquiry's question depends on the question (on what counts as an answer to a question, see Chapter 4, Section 2). Where the theorem is itself the question, as is often the case in pure mathematics, a valid proof answers it, and what remains open is chiefly whether a formal statement captures the intended theorem (Chapter 3, Section 5.1). Where a derivation concerns a model used to answer a question about the world, the same proof establishes a property of the model, not that the model represents its target well enough for the purpose. Simulation practitioners, using the word "verification" in a narrow sense, mark this difference as one between verification, whether a computation approximates the solutions of the model's equations, and validation, whether the model is a good-enough representation of the target for the purpose; Winsberg, as Winsberg and Alvarado report, argues that the two cannot always be kept apart, since equations are sometimes chosen to offset the errors of the method that solves them (Winsberg and Alvarado 2026, §2.2; compare Chapter 3, Section 3.4). The difference between the two cases is one of target, not of force: a proof in pure mathematics is not weaker for answering a question that is itself mathematical.

Operations other than proof differ in target in the same way; in each of the following cases, what passing settles can be something other than what the focal question asks. Passing a calibration can support an instrument model without supporting the focal scientific claim: the two-way measurements by which OPERA determined the fibre delays of its timing chain (Section 2.3) supported an assumption about how the timing chain worked, not the claim about the speed of neutrinos. What a successful experiment establishes may be not which of two antecedently formulated hypotheses is true but that a phenomenon can be produced stably, or that a new variable is involved. An interpretation's surviving criticism can show that a reading holds under the contexts and contrasts that were considered without settling whether it holds under others. That an engineered artifact works can establish possibility under one set of conditions while leaving generality, mechanism, and explanation unresolved. Passing a check that tests an artifact against a specification, as an executable checker typically does, supports the artifact's conformity to the specification, while whether the specification expresses what was wanted remains a further question (Angius, Primiero, and Turner 2025, §§2.2–2.3, 6.2). In an experiment that Lange and colleagues report (Chapter 4, Section 8), a loosely worded rubric that did not explicitly forbid backpropagation produced a result with high validation accuracy that did not implement the intended learning rule.

Operations on one target can also differ in force. In mathematics, for example, the same argument was checked by two operations that differed in how far they could settle its correctness. In the spring of 1993 Andrew Wiles taught the central construction of the proof of Fermat's Last Theorem in a course that did not reveal its purpose, and Nicholas Katz, who after a few weeks was the only one still attending, saw no error in it. That summer, reading the manuscript line by line as a referee, Katz sent query after query until one exposed what one of the two later called a fundamental error; the account is a retrospective one edited for television (NOVA 1997). What the episode suggests is that the difference lay less in the target than in how each operation was organized: following a course whose purpose was undisclosed is not designed to settle correctness, whereas refereeing line by line is. Force can, moreover, be reported in two ways, which Chapter 2, Section 5.2 distinguishes as confirmation as relevance and as firmness: how much passing raises the support for what the check bears on, and how well supported that is after passing, which also depends on how well supported it was before; whether that is enough for the use at hand is a further judgment (Chapter 11, Section 4).

What was said above of proofs holds also for proofs of a limitation. A valid proof firmly settles what cannot be done under its premises. What it settles, however, concerns the model its premises describe; how far that model represents actual devices or tasks, and how much the limitation weighs for actual uses, remain outside the proof. The force of such a result can therefore come to be read as the breadth of its target. The history of Minsky and Papert's Perceptrons (1969) can be read as one case of this. One of the things the two proved is that no perceptron of finite order, one whose features each depend on no more than a fixed number of input points however large the input grows, can represent the connectedness predicate. Bottou, in the foreword to the reissue, notes that this says that a finite-order perceptron cannot represent every predicate, and does not necessarily mean that a learning system cannot represent the predicate that tells whether an image shows a cat (Bottou 2017, xi). On multilayer perceptrons the two offered not a proof but an outlook, "our intuitive judgement that the extension is sterile," which they called an important research problem to elucidate or reject (as quoted in Olazaran 1991, 163).

How the book was read is a further matter. Looking back on graduate school at Edinburgh, Hinton recalls that nearly everyone there said that Minsky and Papert had shown neural networks to be nonsense. On Hinton's account, what the two showed was that simple networks with one layer of adaptive weights could not do many things; they did not show that deeper networks could not do them, or that there could be no learning procedure for deeper networks (Hinton 2024, 3–4). Hinton also remarks on the two having expressed this limitation as the inability to compute whether two inputs differ (XOR): had they expressed it as the inability to compute whether two inputs are the same, it might have drawn attention to an operation that compares things for their similarity (Hinton 2024, 3). This is a later, counterfactual conjecture. Olazaran, tracing the controversy from the standpoint of the sociology of knowledge, holds that the book had two parts, one on the limitations of the single-layer perceptron and the other an intuitive judgement about learning in multilayer perceptrons, and argues that the link from the limitations of the single-layer perceptron to the rejection of neural-network research as a whole was not a logical or necessary one (Olazaran 1991, 140, 181). On Olazaran's account, many of the two's arguments had been known since the late 1950s, the book was a final push in a crisis already under way, and the institutionalization of symbolic AI was important in settling how the book was read (Olazaran 1991, 113, 177). Bottou, on the other hand, reports that when research on learning machines revived in the 1980s, the question arose whether the book had interrupted a promising line of research by overplaying the consequences of its theoretical analysis. Papert wrote in 1988 that the question whether the two had tried to kill connectionism called for something more complex than a plea of guilty or innocent, and Bottou's own reading is that the two conceived the book not as an attempt to kill this research but as a first step toward correcting the lack of an adequate basic theory (Bottou 2017, viii–ix). How far the book bore on the subsequent course of research is disputed among these readings, and is not something to be settled here.

A similar shape appears in results that span theory and experiment. Bengio, Simard, and Frasconi argued, theoretically and experimentally, that under the condition that a recurrent network robustly stores information, the gradients needed to learn dependencies between distant times by gradient descent vanish. They then added that the result does not mean that it is impossible to train a recurrent network on a particular task, but that gradient descent becomes increasingly inefficient as the temporal span of the dependencies increases (Bengio, Simard, and Frasconi 1994, §6). LSTM later sought to change the very conditions under which the problem arises, by providing a path along which error flows constantly (Hochreiter and Schmidhuber 1997). LeCun, Bengio, and Hinton write that in the late 1990s it was commonly thought that simple gradient descent would get trapped in poor local minima, and go on to say that poor local minima are rarely a problem with large networks (LeCun, Bengio, and Hinton 2015, 438; compare Hinton's recollection in Section 4.1).

A proof writes out its premises, though not always every lemma it relies on (the "hidden lemma" of Section 2.2). In empirical experiments, and in research that reports having built something, not all the conditions that made a result possible are written out, and some may be unknown even to those who report it (compare the model in Section 2.5 that was meant to have 104 billion parameters and had 58 billion). A result comes from the whole arrangement that made it possible. Yet since readers cannot take on the whole arrangement each time they read a result, a result can be read as a simpler conclusion. And if a pass can be treated as a reason to stop looking, such simplified readings may be shared without being revisited, and may even bear on which ideas a community pursues. Olazaran, writing of the end of the controversy, quotes Harvey to the effect that accepted beliefs quickly cease to be easily comparable with rejected beliefs, because the former become the basis for future practice (Olazaran 1991, 180). The same can be asked when research is done by a machine. A system that learns from published conclusions or from others' reports can receive those conclusions in a form cut off from the arrangement that made them possible, in the same way as a system that reads the labels in others' records as results (Section 2.5). Harvey's point concerns a community, but applied to one system it holds within the system as well: a conclusion the system has come to build on can cease to be compared with what it displaced, unless the system represents which of its commitments play that role or has a procedure that does that work (Section 4.2; compare the framework that is "transparent" while in use, Chapter 4, Section 4). A finite inquirer, moreover, has to stop looking somewhere (Chapter 5, Section 6.5). Whether stopping at a pass settles the question, or only ends a run, is a distinction Chapter 4, Section 5 drew from the side of questions. For a machine inquirer the cost of rerunning a check can differ from a human's, but passes can also pile up in large numbers. If noticing that something is off is what prompts a second look, where such noticing is located in a machine inquirer is a question to be asked.

For artificial research, the differences of target and force appear as a question of what the record of a pass keeps. When A is a machine, a system can record that it passed the checks it ran, but that record need not keep which part of the question a check bore on or how firmly it settled that. A pass on a check that a model's equations are solved correctly and a pass on a check that the model represents its target well, for example, can both be recorded alike as a pass. And since force also depends on how an operation is organized, an inspection of the same output can settle different things by its pass depending on whether it is organized to settle correctness. If S is human, members of S have to read from a recorded pass what it settled and how firmly, and if the record does not suffice for that, those members cannot tell from the record what the pass established. If both A and S are machines, the system that receives the pass also takes over the work of judging how far it reaches. How it is arranged who sets a check, where in the inquirer it is placed, and who reads a pass is taken up in Section 3.3.

3.2 Apparent success, what a success supports, and the security of agreement

A passed check can rest on something other than what it was meant to test. Apparent success may depend on leakage, confounding, an accidental proxy, a permissive threshold, an implementation that does not realize the stated method, and the like. A proof checker can likewise accept a derivation whose premises do not represent the intended mathematical question (compare Section 3.1): the check passed, but on a question other than the one asked (for this case seen from the side of questions, see Chapter 4, Section 4). Apparent success may also depend on a reference standard that is itself wrong but stable. The primary mirror of the Hubble Space Telescope, for example, was polished precisely to match a reflective null corrector that had been assembled incorrectly, and two tests that disagreed with it were discounted at the time as being themselves flawed (Allen et al. 1990, iii–iv). It can be inferred that the corrector was not doubted partly because it occupied the position of the standard against which the other tests were judged. The case also marks a limit of the appeals to independent support discussed in Section 2.2: here the standard relied on to judge other checks was itself what had gone wrong. Section 5 returns to standards that are themselves in question.

A success that is not merely apparent still does not say which of the commitments that produced it it supports. The debate over scientific realism (Chapter 5, Section 3) has posed this question for theories. Against the argument that, since many past theories were successful and yet false, success is no ground for inferring truth, realists such as Kitcher and Psillos, Lyons reports, hold that a theory's successes support only the constituents responsible for them, not its idle parts, a position Lyons calls deployment realism, in the sense that it takes only the constituents actually used (deployed) in reaching a success to be supported by it. Psillos, on Lyons's report, counted a constituent as essential to a successful prediction if the remaining hypotheses and auxiliaries could not yield the prediction without it and no available alternative consistent with them, independently motivated, not ad hoc, and potentially explanatory could replace it in the derivation (Lyons 2006, 538–539). Lyons objects that whether such an alternative was available has no bearing on whether a constituent in fact contributed to the derivation; that the criterion does not say at what time availability is to be assessed, and that, since inquirers seldom seek replacements for constituents that are succeeding, the lack of one can be a historical accident; and that Psillos's own study of ether theories took a constituent's abandonment by later theories as evidence that it had not been essential, although abandonment does not show that the criterion was unmet (Lyons 2006, 540–542 and n. 1). Credit given instead to every constituent that actually led scientists to a prediction, Lyons argues, extends to mystical posits, weak analogies, and mistaken calculations: Kepler's posit that a moving power emanating from the Sun pushes the planets, together with a step of calculation Kepler knew to be inexact, led to the area law, and Adams and Le Verrier predicted where Neptune would be found from values for its distance, eccentricity, and mass far from those now accepted (Lyons 2006, 543, 545–548, 552–553). None of these constituents, the argument continues, yields a prediction except in combination with others (Lyons 2006, 555).

Set against this chapter's argument, the dispute is the counterpart, for a success, of the question of Section 2: a success also comes from an arrangement (Lenhard and Winsberg's argument in Section 2.4 concerned successes as well as failures), and it does not say which of the commitments it drew on it gives reason to retain. The question is not the apportioning of how much each constituent contributed (compare Section 2.4), but which constituents a success supports; on Lyons's cases, having contributed to producing a success does not settle that, in keeping with the distinction drawn in Section 1 between the causal conditions of a result and its evidential bearing (compare Kepler's vicarious hypothesis, which Kepler kept as yielding true equations from a false cause, Chapter 5, Section 4.2). The same question arises when what is sought is a way of bringing something about. What produced the success of a training run, for example, is the arrangement as it was run, including the architecture, the data and their preparation, the hyperparameters, and the code, and the method a paper states describes part of that arrangement. The BLOOM record of Chapter 5, Section 2 shows the gap: a log written during the run recorded that removing the changes one at a time would cost too much to tell which of them had removed the instability, while the later paper reports that one of them proved to solve it. Whether the stated method is what the success supports is then an instance of the question Lyons presses, not something the success itself settles. For artificial research, a system that raises its confidence in every component of a configuration that passed a check presupposes which components the success supports, as one that blames its focal hypothesis for every mismatch presupposes where the fault lies (Section 4.2); and whether a later configuration retained or dropped a component does not by itself show whether that component produced an earlier success. Applied to a machine, Lyons's objection that the lack of an alternative can be a historical accident also takes a definite form: where a system stops searching at a pass (Section 3), whether an alternative to a component was available depends less on the component's role than on the system's own search policy. And where a machine doing research attributes its own successes can bear on which ideas and changes remain afterward. Like the attribution of failure discussed in Section 2.3, the attribution of success can be a judgment that decides which path a search takes (compare Section 4.2).

Agreement among several checks raises a further question: how much security it adds. Within one inquiry, heterogeneous operations can be arranged to check one another, and the first image of the black hole in M87 is an example. In it one can see such an arrangement perform a function and establish features of the image to different degrees, and the independence of the converging analyses was then disputed. For that image, the Event Horizon Telescope Collaboration had four teams reconstruct images without contact with one another for seven weeks, then surveyed tens of thousands of imaging-parameter settings on synthetic data, and reported the ring diameter as insensitive to the choice of imaging method while other features, such as the ring's width, differed between pipelines (Event Horizon Telescope Collaboration 2019, 1, 9, 17, 29). Doboszewski and Elder reconstruct this reasoning as a robustness argument suited to an observation that no second instrument could repeat: variation was introduced into the analysis instead, across teams, algorithms, and parameter settings, and features stable across it were trusted while others were not. On their reading, the tests on synthetic data also showed that the procedure yields different images from differently shaped sources, and the blurring and averaging of the final images gave up resolution for security. They add that the first stage, the four teams' separate reconstructions, guarded mainly against errors due to the choices and expectations of the people doing the imaging, and that its value depends on whether the teams' biases were independent, which overlapping education and careers could undermine (Doboszewski and Elder 2024, 10, 13, 17–21). The teams' independence covered only the first stage of imaging: all four worked from the same early release of data, with the same initial calibration already applied (Event Horizon Telescope Collaboration 2019, 9).

After the data were made public, an outside reanalysis argued that the ring, like the absence of a jet, was an artifact of gaps in the array's sampling combined with a narrow field of view (Miyoshi, Kato, and Makino 2022a); the collaboration replied that ring-like structures are recovered under a broad range of imaging assumptions (Event Horizon Telescope Collaboration 2022), and the authors of the reanalysis answered that three of the four analyses cited in that reply could not count as independent, since one had started from a ring and two had used the collaboration's method or its software (Miyoshi, Kato, and Makino 2022b). Doboszewski and Elder do not discuss this exchange, but it can be read as turning on the condition on which, on their analysis, the value of the first stage depended: the independence of the converging analyses (compare Chapter 11, Section 2, which, on the view that locates the value of criticism in the errors it detects, treats independence as a relation among routes of error). The same question arises in configurations in which machines do research. When analyses by many instances of one base system, or by systems trained on the same data, agree, the security their agreement adds may be limited if they share routes by which they may err (Chapter 11, Section 5.4).

3.3 Who sets a check, and who reads a pass

The answers to the questions of Sections 3.1 and 3.2 do not come from the pass itself. What a pass bears on depends on who set the check and how. How much security a pass or an agreement adds depends both on how sharp the check is, that is, whether it would have failed had the claim been wrong, and on how far the routes by which the check could err differ from those by which the claim could. And whether a pass is taken to settle anything depends on who reads it, against which standards. In human research, many checks come from others, such as standard tests, instruments, and referees; experimenters also design controls of their own; and those who read a pass are a human community. When research is done by a machine, such matters become matters of how the artificial configuration doing the research is arranged.

One of the things that can change is the relation between the checks and the inquirer. As Chapter 5, Section 6.5 noted, when the inquirer is a machine, what is placed inside the inquirer and what is placed outside it as tool or record is not given but is allocated within its configuration. A machine inquirer can write its own checks, choose which to run, and rewrite them, and the checks then become part of its arrangement. What a pass on a check one has built oneself establishes depends also, as with the controls a human experimenter designs, on whether the claim and the check can err by the same routes. If processes of the same make produce both the hypothesis and the check, however, the two can share routes by which they may err. On the other hand, a machine inquirer can have its checks written by a differently made process, or tie them to measurements or a formal checker it does not control (Chapter 11, Section 2); how far a check errs by routes other than the claim's depends on how the check was made. The point that dividing a task among copies of one make divides the task without dividing perspectives, so long as their background assumptions and routes of error have not come apart (Chapter 5, Section 6.5; Chapter 11, Section 3), and the point that one who relies cannot treat agreement among checkers as adding evidence with each, unless there is reason to think they reached it by different routes (Chapter 11, Section 5.4), extend to this case.

Who determines what counts as passing also varies with the arrangement. Where a human sets the evaluation in advance and the system is not allowed to change it, the target of a pass is set by that human, and the system's passes are passes on a question that person has set. If a system sets and rewrites the evaluation itself, the questions of Section 3.1, what part of the question a pass bears on and how firmly it settles that, become questions the system itself carries. Moreover, what counts as passing, and further what counts as a check at all, can extend beyond the choice of a procedure or a threshold (that choosing a threshold involves weighting errors is taken up in Chapter 11, Section 4) to epistemological positions on what conclusions are warranted (Chapter 2, Section 5.3) and to what checking itself presupposes, such as the uniformity of nature (Chapter 8, Section 2.1). Depending on who is A and who is S, and on who the bearer of knowledge is, such questions can arise in different forms, and at times at the root. Who can revise such presuppositions is taken up in Chapter 7, Section 4.3.

Who reads a pass also varies with the combination. If S is human, members of S read a pass against their own standards, and for that they need to be able to tell from the record who set the check and where in the inquirer it was placed. If both A and S are machines, the standards for what counts as settled may themselves belong to the artificial community (Chapter 8, Section 1.3). Where no external standard of correctness is available at all, what counts as passing becomes itself a matter for inquiry, the case to which Section 5 turns.

Claims that attribute research capacity to a system bear on the same arrangements. The more open the question, object, method, and environment, the more such a claim attributes to the system, including deciding what to check, and how that work is allocated among the system, humans, institutions, and instruments becomes part of the later attribution (Chapter 7, Sections 2 and 4.4).

4. Diagnosis across a sequence of results

Many of the cases in Sections 2 and 3 already extended over time. Localization proceeded over hours, months, and at times years (Sections 2.3 and 2.4), and what a failure or a pass was taken to show could change afterward (Sections 2.5 and 3.1). Distinguishing an anomaly from an error can likewise take a sequence of results, some of them made by others. The observers of the low ozone values at Syowa Station described in Chapter 2, Section 6, for example, repeated their readings and calculations and inspected their spectrophotometer without finding a fault, and handed the observations on to the next team without knowing whether the instrument was sound; after the observer returned to Japan, the station's earlier records showed the values to be unprecedented, and the observer recalled gaining confidence that they were not an observational error only when measurements from the South Pole showed similar values (Chubachi 1993, 816–817; Chubachi 1984, 18). In Chinn and Brewer's terms (Section 2.3), the anomaly was held in abeyance, and what moved it was a further result from elsewhere.

What this section asks about is not that there are sequences of results but how a sequence of revisions is organized and how it is assessed over time. A single result settles neither where to revise nor how long protecting a commitment remains reasonable, and whether a sequence of trials amounts to diagnosis depends on how the sequence is organized. The section takes up how long protecting a commitment remains reasonable (Section 4.1), how protection and revision are arranged when research is done by a machine (Section 4.2), and what distinguishes trial and error that diagnoses from repetition (Section 4.3).

4.1 How long protecting a commitment remains reasonable

Duhem's good sense, introduced in Section 2.2, also bore on timing. On Duhem's account, the indecision between the partisans of an old system and those of a new doctrine lasts only for a time: logic does not mark the moment at which an insufficient hypothesis must give way to a more fruitful one, recognizing that moment falls to good sense, and physicists can hasten the judgment by making their good sense more lucid and vigilant, while passions and interests, such as the vanity that makes a physicist indulgent toward their own system and severe toward that of others, delay it (Duhem 1906, Part II, ch. VI, §10). In Duhem's own example, that of Biot and Fresnel, the moment came with a new experiment. Ivanova argues that, because on Duhem's account the choice is eventually settled by experimental evidence, good sense seems to be posited after the fact, and proposes reading it as the virtues of an idealized scientist, so that practitioners can use it prospectively by asking what such a scientist would do (Ivanova 2010, §§3, 5). What a single result leaves open therefore includes not only which commitment it bears on but how long protecting a commitment remains reasonable, and whether that can be recognized before later evidence arrives. Later accounts of theory change have taken up this question. As the following shows, Kuhn and Lakatos give reasons why protecting a commitment can be reasonable for a time; they and Laudan tie the end of that protection partly to the availability of a rival; researchers' own recollections show such protection from the inside; whether protection can be assessed only in retrospect is read in two ways; Lakatos and Laudan differ over the standard by which a history of protection is to be judged; and formal learning theory examines, for idealized methods, when holding a conjecture is reliable.

Bird reads Kuhn as holding that, during normal science, anomalous results are not treated as refutations of a community's guiding theories; they are ignored or explained away, and only an accumulation of particularly troublesome anomalies produces a crisis (Bird 2025b, §2). Kuhn's reason, on this reading, is that normal science can progress only through a strong commitment to a shared disciplinary matrix of theoretical beliefs, values, instruments, and techniques; what normal science does put at risk is an investigator's speculative solution to a puzzle, a problem that the paradigm lets one assume has a solution (Chapter 4, Section 6), which can be refuted (Bird 2025b, §2). Barros Sobrinho quotes Kuhn's distinction between a scientist's inability to solve a normal-science puzzle and a test of the theory: the former counts against the scientist, while testing a paradigm requires both a crisis brought on by continuing failure on an important puzzle and a competing candidate (Barros Sobrinho 2015, 45 n. 110, 53 n. 139). Schindler reads Kuhn as allowing anomalies to be set aside for later work rather than requiring each to be resolved before other problems can be pursued (Schindler 2024, §3.2.1). Pauly quotes Kuhn's argument that confidence in the older paradigm, even when it becomes obstinate, enables normal science, and that the community can isolate a difficulty whose study may generate a new paradigm only through such work (Pauly 2022, §7.9, 173). Chang reproduces Kuhn's claim that historians can judge scientists such as Priestley unreasonably persistent without being able to identify a point at which resistance becomes illogical or unscientific (Chang 2010, §2.2). This last claim has the same form as Duhem's about Biot.

Lakatos observed that scientists do not abandon a theory merely because facts contradict it, and made the direction of blame methodological: a research program's hard core is protected from refutation by a belt of auxiliary hypotheses that absorbs anomalies. Whether that protection is reasonable is judged not at one result but over the program's development—progressive if its successive theories lead to the discovery of novel facts, degenerating if they are fabricated only to accommodate known ones (Lakatos 1973; Musgrave and Pigden 2026, §2.2). Lakatos denied that this methodology offered "instant rationality": budding programmes may take decades to become empirically progressive, and while it is a matter of intellectual honesty to keep the record public, it is not dishonest to stick to a degenerating programme and try to turn it into a progressive one (Lakatos 1973). Feyerabend's objection, as Musgrave and Pigden report it, was a dilemma. Applied strictly, the methodology would condemn programmes that might later recover; restricted, as Lakatos then restricted it, to appraisal without advice, it tells no one what to do and is scarcely distinguishable from "Anything goes." On their assessment, a Lakatosian answer to the dilemma remains an open problem (Musgrave and Pigden 2026, §4).

Several of these accounts tie the reasonableness of abandonment to the availability of a rival. Kuhn wrote that a theory is not overthrown directly by the difficulties it has met, but only when an alternate orientation is put forward that is shown to resolve them (Kuhn [1951] 2021, 86–87). Lakatos wrote that there is no refutation without a better theory (Lakatos 1973), though Musgrave and Pigden doubt that Lakatos held, as a norm, that no programme should be given up without a better one in existence (Musgrave and Pigden 2026, §2.2). Laudan and his colleagues, setting out the theses of several models of scientific change, list as one that they find in Kuhn, Lakatos, and Laudan, among others, that a set of guiding assumptions, their neutral term for paradigms, research programs, and research traditions, is never rejected unless an alternative set is available, and, among Laudan's own theses, that empirical difficulties by themselves never compel abandoning a theory and that the appraisal of a theory is relative to its existing rivals (Laudan et al. 1993, 29, 33, 73, 75). On these accounts, when protecting a commitment stops being reasonable depends partly on what candidates have been formed (Chapter 5) and on whether rival lines have been kept alive (Chapter 10, Section 4). Kuhn also placed part of this temporal work in the community: the latitude that shared values leave permits rational scientists to disagree and so to commit themselves to rival theories, which, as Bird reports, Kuhn regarded as potentially indispensable for spreading the risks of introducing or supporting novelty (Bird 2025b, §7.3).

Researchers' own recollections show such protection, and its time, from the inside. The two recollections below are those mentioned in Section 2.3 as cases in which convictions from outside the result at hand bear on where doubt goes. Looking back on graduate school at Edinburgh, Hinton recalls that the supervisor, Longuet-Higgins, held that neural networks were the wrong approach and symbolic AI the right one, and kept trying to persuade the student that this work was nonsense; Hinton kept promising to switch to symbolic AI if the approach could not be made to work in six months, and when six months had passed, another six were needed (Hinton 2024, 2). Results that did not work could have been read as evidence against the idea of neural networks itself, and the supervisor read them so; Hinton went on treating them as not yet having made it work. In the same interview Hinton says that the assumption that learning gets trapped in local optima was believed for many years without being tested (Hinton 2024, 4; compare Section 3.1), and refers to symbolic AI's neglect of computational complexity and to the brain's immense computation through its many connections (Hinton 2024, 8–9). Where to attribute the failures can be read as itself what was in dispute between Hinton and the researchers around Hinton at the time. And Hinton's continuing to commit to the idea of neural networks, whatever the results at hand, is an attribution of results in the sense of Section 2.3.

The Transformer authors' recollections show a similar shape. The first translation model built on self-attention quickly reached at least the level of the best approaches using LSTMs but was not better, and the work reached a plateau. Uszkoreit nonetheless suspected that self-attention alone could give better results, and says that such intuitive mechanisms always require very careful implementation, often by a small number of experienced people, to show any "signs of life"; the model began to work after Shazeer wrote a separate version of the code (Levy 2024). Here too the plateau can be read as having been attributed not to the idea of self-attention but to its implementation, in the sense of attribution discussed in Section 2.3.

Both are later recollections, not contemporaneous records of the judgments at the time, and ideas that were kept and paid off are more often told than ideas that were kept and did not. Still, it may be only that such cases are recorded and widely known; a long commitment to an idea, whether or not it is rewarded, may itself, to a greater or lesser degree, be nothing unusual among researchers. Whether all researchers have one, or how many do, cannot be told from these cases. Even so, these recollections may be read less as exceptional events than as conspicuous instances of something of that kind.

Where the mountain to be crossed is high, as when a problem is complex and the merits of an idea are hard to see from within the current framework, not discarding an idea at the first results may perhaps be what opens a path for exploration. Lakatos's point, seen above, that budding programmes may take decades to become empirically progressive can be read as giving one reason for this, and Feyerabend's observation that no idea is ever fully explored (Chapter 10, Section 2) touches the same circumstance from the side of how resources are allocated.

On both Kuhn's and Lakatos's accounts, protecting a focal commitment can be a defensible default. The claim that such protection is assessable only over a sequence of results can be read in two ways: that the rationality of a pattern of revision is itself settled only in retrospect, as on Lakatos's denial of instant rationality, or that the judgment is merely delayed until evidence gathered over time is in, as non-historicists, who hold that assessing rationality need not trace history, can reply (Nickles 2017, §§1.3, 3). On either reading, the object of assessment is the history of revisions rather than a single result. The caveat above, that rewarded cases are the ones more often told, is one form of this question, since an assessment made in retrospect can also depend on how the cases looked back on are selected.

What such a history shows also depends on the standard applied to it. Lakatos's test, whether protective adjustments go on to yield novel predictions, is not the only standard on offer. In the list of theses that Laudan and his colleagues drew from models of scientific change, a thesis drawn from Lakatos judges a set of guiding assumptions by its capacity to make successful predictions with its central assumptions rather than with assumptions invented for the purpose (Laudan et al. 1993, 31), whereas on Laudan's theses, a theory's capacity to turn apparent counterexamples into solved problems counts especially in its favor, as it did for Prout's hypothesis about the composition of atoms, and progress consists in replacing a theory with one that offers a net gain in solving empirical and conceptual difficulties (Laudan et al. 1993, 73–74). On such a criterion, resolving a known anomaly can count in a theory's favor without yielding a novel prediction; whether the modification is progressive depends on its net effect on empirical and conceptual problem-solving (Chapter 4, Section 9). The same list sets the thesis, drawn from Lakatos and Kuhn, that the central elements of a set of guiding assumptions are immune to refutation and never change until the whole set is abandoned against Laudan's thesis that they sometimes change piecemeal (31); among Laudan's own theses are that scientists sometimes change their minds about which elements of such a set are most central and which are open to amendment, as with Newtonian mechanics, and that, although the elements form an interconnected network, scientists treat them as individually negotiable and replaceable rather than as a package to be taken or left as a whole (74, 76). Nickles, too, reports that on Laudan's reticulated model, dramatic change in one place need not seriously disturb fixity elsewhere (Nickles 2017, §1.5). Nor, on Laudan's theses, do empirical difficulties count by their number: they count against a theory more by their importance than by their number, and the importance of an empirical difficulty grows with the discrepancy between prediction and result, with how long the difficulty has lasted, and with its past resistance to solution, while a theory that can solve difficulties its rivals face is particularly persuasive (Laudan et al. 1993, 72–73). Applied to the same repair history, the two accounts would assess it differently: repairs that only accommodate each new mismatch fail Lakatos's test, but on Laudan's account their appraisal turns on their net effect on empirical and conceptual problem-solving, and which commitments occupy the protected role can itself change.

Formal learning theory takes up the same questions for idealized methods. As Oliver Schulte presents it, it asks which methods are guaranteed to settle eventually on the true answer among hypotheses that an inquirer's background assumptions fix, and its recommendations are relative to those hypotheses, to those assumptions, and to the aims adopted (Schulte 2026, §§2.4, 3). Two of its results bear on protection. The first is that whether the rule of holding a conjecture until the evidence contradicts it is reliable depends on the structure of the hypotheses under investigation. For a universal generalization the rule is reliable. For two rival hypotheses that each allow finitely many exceptions, one of which is assumed to be true, it is not, since each counterinstance can be explained away as an exception and a false conjecture is then never abandoned; a reliable method changes its conjecture by another rule instead, for example when a clear majority of the observations goes the other way. Methods of this kind can be represented as replacing each hypothesis with strengthened versions that can be refuted, which Schulte likens to Lakatos's methodology, in which a paradigm is articulated with auxiliary hypotheses to define testable claims (Schulte 2026, §§1.4, 3.3, 7). The second is that a method can be certain to settle on the right answer eventually although it may never be certain that its current answer is the right one: "no bell tolls," in the phrase of William James that Schulte quotes (Schulte 2026, §1.1). These results can be read as assessing protection at a different object from the accounts above: not a history of revisions but the rule that governs them, judged by what it would achieve whatever evidence consistent with those assumptions arrives. The first makes how long protection remains reasonable depend on the structure of the hypotheses at issue rather than on a rule for all inquiry. The second means that, even where such a rule is reliable, the inquirer may be unable to recognize the current conjecture as the one the method will keep, and holding it then cannot be warranted by such recognition. These results concern methods whose hypotheses and background assumptions are fixed in advance. They do not show what a community or a machine inquirer whose candidates are still being formed (Chapter 5) would achieve.

4.2 Protection and revision when research is done by a machine

Most of the accounts and cases of Section 4.1 concerned human researchers and communities (formal learning theory concerns idealized methods). Duhem's good sense, Kuhn's strong commitment in normal science, the allowance Lakatos gave budding programmes, and Hinton's six-month extensions are all scenes in which how long to keep an idea is judged over a sequence of results. What bore on the judgment in these accounts and cases rests on different premises. Vanity, which on Duhem's account delays the judgment, and Hinton's promise to the supervisor concern human motives and careers. Kuhn's reason, that only through normal science does a community isolate the difficulty from which a new paradigm may emerge, and Lakatos's, that a budding programme may take long to become progressive, concern how the merits of a line become visible; the decades Lakatos mentions reflect a human pace, but the condition does not depend on it. In every case, a finite inquirer must choose how long to keep an idea, and when to let it go, before there is enough evidence to settle the matter (Chapter 5, Section 6.5). What can change is where, by whom, and in what form that choice is made, and what directs it.

When research is done by a machine, how long to keep an idea, where to attribute a difference in evaluation or a mismatch, and what to hold fixed and what to treat as open to doubt are allocated within the artificial configuration. They may be placed in rules a human wrote in advance, left to the system's own judgment, or divided among many parallel lines. Suppose, for example, a rule tries changes one at a time, keeps a change only when it improves the immediately following evaluation, retains nothing it discards, and tries no combinations. Such a rule works as if it attributed every difference in evaluation to the last change, so that changes that work only in combination with others are hard to find under it; under it, no idea is given the kind of allowance seen in the cases of Section 4.1. What makes attributing a difference to the last change unsafe is not that the rule changes one thing at a time, which is also what let Edison's records name a baseline (Section 2.4), but that it judges each change by a single following evaluation and, keeping nothing it discards and trying no combinations, never returns to a change whose effect depends on others. A rule that does not allow the evaluation itself to be changed, in turn, places the evaluation outside what a contrary result can cast doubt on. As logic can be read as lying outside what a contrary result could cast doubt on in Duhem's holism, while Quine brought it inside (Section 2.1), where that boundary is drawn changes the very candidates for localization. Who draws such a boundary connects with the question of the boundary of the actor (Chapter 7, Section 2). Outside research, a rule written in advance that fixes the locus of a fault can itself produce the failure. For Ariane 5's first flight, the system specification stated that on any kind of exception the inertial reference processor should be shut down, a provision that the Inquiry Board traced to a culture within the programme of addressing only random hardware failures, which a backup system can handle; the backup unit ran the same software, had ceased to function in the previous data cycle for the same reason, and the rule switched off two still healthy units (Ariane 501 Inquiry Board 1996, 4–5).

If S is human, members of S reading a system's results need to be able to learn from the record which ideas were discarded, or kept, under which rules or judgments, since discarded ideas often leave no trace in the record of results; this is the record's form of what Section 4.1 said about rewarded cases being the ones more often told. If both A and S are machines, the judgment of how long to keep which ideas may itself be allocated within an artificial community.

As seen in Sections 2.3 and 4.1, judgments of where to attribute and how long to keep an idea can involve convictions and expectations from outside the result at hand, and views about what is essential. The practitioners' tacit knowledge of their own models in Section 2.4, and Duhem's good sense in Section 4.1, can be read as such things. When research is done by a machine, where whatever plays the part of such convictions and expectations comes from in the inquirer, whether from the way it was made, from those who build and maintain it, or from S, and to whom it is visible, then becomes a question. If S is human, what matters for members of S is whether they can read from the record what directed the system's attributions. If both A and S are machines, the convictions that direct attribution may themselves be shared, or contested, within an artificial community (Chapter 8, Section 2.1; Chapter 11, Section 4).

The question of where to attribute a mismatch has also been raised as one of designing research by machines. An earlier analysis noted that when verification results are negative, the cause may lie in the primary hypothesis, underlying premises, auxiliary hypotheses, observations, experimental instruments, or a combination of these, and argued that an agent conducting research needs to be able to discern the likely cause among these candidates, a task that is challenging for machines to perform autonomously (Takagi 2023, §4.1.4). The problem was posed there as one of design; how far systems since have addressed it is not examined here. As Section 2.1 noted, a system that maps every mismatch directly onto the focal hypothesis does not take up this task. On accounts on which a contrary result bears on an arrangement rather than on one hypothesis (Section 2.2), a more general capacity would preserve multiple possible loci of error or revision, use subsequent inquiry to discriminate among them, and treat different dependencies differently—holding some fixed, examining others, revising some—rather than lowering confidence in the focal hypothesis by a single amount. The two positions of Section 2.3 place the weight differently within this description: on Strevens's treatment, the prior distribution of doubt among the dependencies already does part of the discriminating; on Mayo's, blaming one of them is warranted only once the claim that it is at fault has itself passed a severe test. On neither is the response a single change to the focal hypothesis. Attributing such a capacity to a system would require evidence that it can respond differently when the pattern of results changes; as Chapter 7, Section 3.1 notes, the stronger a claim about continuing capacity, the more its behavior under variation, error, and repair matters.

Such a capacity does not require treating everything as equally doubtful. Nor does it make protecting a focal commitment an error. On the Kuhnian and Lakatosian accounts of Section 4.1, protecting a shared framework or a program's hard core can be the reasonable default. Neither Kuhn nor Lakatos discusses artificial systems, but carried over to a system that protects its own framework, their reasons concerning how a line's merits become visible give that protection the same support, while vanity or a promise to a supervisor, as features of human careers, need not apply in that form (compare the question of where convictions come from, above). As Section 4.1 reported, Kuhn also gives this default communal reasons: the latitude that lets some members of a community commit to rivals may spread the risk that novelty carries. Carried over in the same way, these communal reasons make the reasonableness of that protection depend partly on whether rival lines are kept alive elsewhere, in other parts of an artificial collective or among others in the society for which the system inquires, a question Chapter 10, Section 4 takes up for artificial investigators. If the parallel lines are divided among copies of one make whose background assumptions and routes of error have not come apart, however, the task is divided without perspectives being divided (Chapter 5, Section 6.5; Chapter 11, Section 3). The copies can then keep rival lines alive, but their judgments of when to give a line up can share the same errors, so that the lines are given up together (Chapter 10, Section 4). On the argument of this section, a system that never blames its focal hypothesis presupposes the locus of the fault as much as one that always does. A system that neither represents which of its commitments play the protected role nor has a procedure that does that work has no means of distinguishing protecting them from failing to notice that they are in question. Whether its pattern of protection and revision was reasonable can be assessed, on either reading of Section 4.1, only over a sequence of revisions, and the assessment differs with the position taken, as the contrast between Lakatos and Laudan in Section 4.1 shows. Where its hypotheses and background assumptions are fixed in advance, the rule it follows can also be examined for reliability in the sense of formal learning theory (Section 4.1). Which dependencies are explicitly represented, and which are left to reliable procedures, depends on how the artificial configuration doing the research is arranged.

4.3 Trial and error as diagnosis

Treating diagnosis as ranging over several loci and over a sequence of revisions allows trial and error to be described more precisely. Trials are selected from a history-dependent inquiry configuration. Their outcomes can change the standing of a candidate, the reliability assigned to apparatus or procedure, the representation of the problem, the available next actions, and the expected value of continuing. Running many trials is not by itself epistemically productive. A failed but discriminating test can contribute more than numerous outcomes that cannot be attributed or interpreted.

Not every repetition is a trial in this sense. In the public chemistry notebook discussed in Section 2.5, for example, two single reactions were to be repeated because no photographs had been provided to show the result, the reactions on two of the pages marked "aborted" were to be attempted again, and the pair marked "Void" was redone, the redo also confirming, according to its page, that the earlier precipitate had come from the excess acid (UsefulChem 2010, Exp135–138, Exp142). Such repetitions recover a missing observation or a result whose prerequisite was not met, and a redo can also test a diagnosis; where a record states the purpose of a repetition, as these pages do, it is evidence of which of these the repetition was (compare Chapter 1, Section 3). The repetition of a fixed remedy can also turn into diagnosis. The on-call guide in the OPT-175B logbook lists restarting from the last checkpoint, which "frequently ... just kicks the can down the road 2-6 more hours," and lowering the learning rate, "our current default mode of action" (Meta AI 2022, 106–107). When runs with different optimization methods showed gradient-norm spikes in the same places, the logbook noted "(data!)" (Meta AI 2022, 45), and when a restart was found to restore state it should not, entries began with "DO NOT REQUEUE THIS RUN" (Meta AI 2022, 41–48). The repetition of a fixed remedy became evidence about where the fault lay, and people stopped the remedy when it turned out to be an intervention of its own. What the same logbook shows about localization was seen in Section 2.4.

What, then, distinguishes trial and error that diagnoses from mere repetition? What follows looks at some of what bears on this question, through cases from different fields, mostly analyzed by philosophers or sociologists of science.

How trials are sequenced bears on diagnosis. Cleland argues that experimental evaluation of a hypothesis is a series of experiments, each designed in light of earlier results, and describes the 1976 Viking labeled-release experiments as an example of what can happen when a program is designed wholly in advance of the data: the experiments and their controls produced results no one had anticipated, and no further experiments could be performed to adjudicate among the auxiliary assumptions then in dispute (Cleland 2002, 478–480).

Whether successes as well as failures are checked also bears on diagnosis. Cleland also offers an account of what such a series is for. On Cleland's account, when experimenters respond to a failed prediction by holding the test condition fixed and varying other conditions, they are not rescuing the hypothesis ad hoc but checking for a false negative, since each suspect auxiliary assumption is rejected by building a new experiment rather than merely denied; after a successful prediction they do the same, and may remove the test condition itself, to check for a false positive (Cleland 2002, 476–477). Cleland distinguishes this from Lakatos's account, read by Cleland as attending to how theories are protected from refutation but not to how hypotheses are protected from misleading confirmation, and presents this pattern of checks as an ideal to which experimenters aspire rather than a description of every experiment (Cleland 2002, 477–478). On the account given here, a diagnostic capacity of the kind this section describes would respond to an apparent success, as Section 3.2 discussed, as well as to a mismatch.

Whether the cause of a failure is sought differs between practices. Some laboratory practice does not seek localization within a procedure that fails. In a chapter written with Klaus Amann, for example, Knorr Cetina reports that the molecular biologists they observed, faced with a malfunctioning reaction or results that were hard to interpret, did not investigate the cause but varied the procedure, in several respects at once rather than one variable at a time, until it yielded results that passed critical scrutiny and could be published. The authors call this "blind" variation with selection by success, contrast it with the high-energy physicists' study of their own apparatus, and argue that the preference for it does not make it less effective; from the biologists' perspective, they report, a strategy like the physicists' might not work, since their tools were largely the cell's own machinery, of which little was known, and a faulty construct could not be corrected by calculation but had to be remade (Knorr Cetina 1999, 134–137). They qualify the label. The variations drew on embodied experience, laboratory stories, and visual reconstructions of what had happened in a reaction, but the explanations these suggested were not tested in their own right, and they call the resulting strategy semi-blind (Knorr Cetina 1999, 154–156). Read in light of their description, such a practice can be seen as still settling, at a coarse grain, what a failure bears on: it treats a failure as a matter for the procedure, to be overcome by varying it, and it judges when a result has passed critical scrutiny; the results themselves supply neither judgment. What it forgoes is localization within the procedure. It can therefore yield publishable results without exercising the diagnostic capacity described above, so that success under it is not evidence of that capacity, and the absence of that finer localization does not by itself show that its results are unreliable.

How open the arrangement is left also bears on diagnosis. Exploratory variation can be systematic without being directed by a theory. Steinle's account of exploratory experimentation, described in Chapter 5, Section 5, also bears on what an arrangement built for one hypothesis excludes. Steinle holds that exploratory and theory-oriented experimentation differ not only in their aims but also in how instruments are handled and in what the instruments are like (Steinle 2004, 51). As Karaca reports Steinle's account, in theory-driven experimentation both the set-up and the procedures are tailored to specific research questions, so that the possibilities under which unforeseen outcomes could be obtained are largely excluded, whereas exploratory work needs instruments that allow a great range of variation and remain open to a wide variety of outcomes, even unexpected ones, as when Ampère varied the battery, the needle, and above all the needle's position relative to the wire to find out which factors contributed to its deflection (Karaca 2017, 335–336). An arrangement specialized to test one hypothesis efficiently can also exclude the outcomes that would show the hypothesis to be the wrong question. The means of recording belong to the arrangement too. In 1967, for example, Jocelyn Bell (later Bell Burnell) noticed an intermittent signal on the chart records of a new radio telescope, arranged for a high-speed recorder to be run when its source crossed the beam, recorded nothing for most of November, and then obtained a train of pulses about 1.33 seconds apart, which still had to be distinguished from interference (Bell 1968, 214–216). Bell's thesis remarks that if the telescope's output had been fed directly into a computer, the sources might not have been found, because the computer would not have been programmed to look for such unexpected objects (Bell 1968, 231). The remark is the investigator's own counterfactual judgment rather than a demonstration, but it names, for the case in which records are processed automatically, the trade-off that Steinle's contrast describes. Where records are selected automatically by design, the trade-off has to be faced in advance. In the ATLAS experiment at the Large Hadron Collider, automated trigger systems reduce some forty million collision events per second to a few hundred, and the events they reject are irretrievably discarded; Karaca argues that a trigger menu built only from the signatures predicted for the Higgs boson and for models beyond the Standard Model would be biased against other events, and that the experiment diversified its menu, including triggers with lowered thresholds, so as to remain sensitive to processes not yet predicted (Karaca 2017, 341, 344–346).

There is also the question of how an arrangement that cannot be inspected directly is probed. Duhem compared the physicist repairing a faulty theory not to a watchmaker, who can take a stopped watch apart and examine its pieces one by one, but to a physician, who cannot dissect the patient and must infer the seat and cause of the illness from the disorders of the whole body (Duhem 1906, Part II, ch. VI, §2). Engineers can face a similar position with an artifact. When the high-gain antenna of the Galileo spacecraft (see Chapter 2, Section 5) failed to deploy in 1991, for example, engineers who could not inspect it inferred its state from motor currents, sun-sensor shadows, spin rates, and tests of a spare unit. Tests on the spare unit indicated that three ribs were most likely held at their stowed position; the engineers then listed four possible restraints and eliminated candidates one by one, by further tests on the spare unit, by the record of the order in which the spacecraft had been assembled, and by the forces applied in attempts to free the antenna (Johnson 1994, 366–373). The case can be taken to show trial and error becoming diagnosis where the parts cannot be examined one by one: each test was informative because it bore on a hypothesis about the state of a part that could not be observed, not because the operations were repeated. The same operations, such as calibration or a test of a spare unit, could serve as support work, evidence, or method development in another configuration; their label does not decide their role (compare Chapter 5, Section 5).

In the cases seen here, what made trials diagnostic was not their number but that each was tied to earlier results, to loci of error not yet excluded, or to hypotheses about parts that could not be observed directly. When research is done by a machine, the cost of running trials can differ greatly from a human's, and the cheaper and more numerous trials become, the harder it may be to see that their number is not the number of diagnoses. Where the inquirer itself can compose, run, and rewrite such sequences, how it chooses which trials to run can also limit in advance what results can be obtained; as Bell remarked that a computer not programmed to look for unexpected objects might have missed the sources found on the chart records, what an arrangement is set up to look for limits what can be found (compare Section 3.3), and, as the ATLAS trigger menu shows, such limits can be anticipated and partly offset by design.

5. When the standards themselves are in question

The preceding sections asked what results bear on mostly while treating the standards through which results are read as given. That the standards themselves can be in question has nonetheless already come up in several places: the Hubble mirror polished to a faulty reference standard (Section 3.2), what counts as passing and what counts as a check at all (Section 3.3), and a rule that places the evaluation outside doubt (Section 4.2). Chapter 4, Section 5 noted that, viewed from the side of unsettled matters, an inquiry's question, method, or standards can each remain unsettled, and Chapter 4, Section 4 noted that, when standards are questioned, what counts as an answer can also be revised; standards can be among what is unresolved. This section asks what happens when a result bears not only on a candidate answer but on the standards, concepts, or procedures by which answers are assessed, how revising them can proceed without an external foundation, and what separates revising a standard from moving the target. It takes up how a result can bear on a standard (Section 5.1), how standards have been revised from within inquiry under constraint (Section 5.2), and what separates improving a standard from replacing it (Section 5.3).

5.1 How a result can bear on a standard

When a result bears on standards, the standpoint from which claims are evaluated is itself among what may change in the course of inquiry. Measurement is one place where this is easy to see. A method for measuring a quantity may depend on a theory of that quantity, while evidence for the theory depends on measurements made by methods whose adequacy is still at issue. Requiring an entirely presupposition-free starting point would make the development of such a practice look viciously circular.

Tal connects this problem to the one examined in Section 2. When corrected measurements of what is supposed to be one quantity disagree beyond their stated uncertainties, Tal argues, there are three possible explanations: the instruments are not measuring the same quantity, systematic errors have been inappropriately evaluated, or measurement uncertainty has been underestimated. No amount of empirical evidence can alone decide among them, a problem Tal likens to other cases of Duhemian underdetermination. Where established methods of restoring agreement do not yet exist, the practical challenge, in Tal's words, "is to devise new methods of adjudicating agreement and error 'from within'," by extending existing theoretical presuppositions and interpretive traditions to a new domain (Tal 2012, 62–66). On this account, among the candidates for what a discrepancy bears on is a standard: what counts as measuring the same quantity. The Hubble case of Section 3.2 showed the same possibility from the side of a passed check. Chapter 8, Section 2.1 considers a related circularity at a more basic level, where a check counts as evidence only against presuppositions that it cannot itself establish.

5.2 Revision from within, under constraint

How can inquiry be held to standards that arise within it? As Henne reports Dewey's answer, logical rules regulate further inquiry because earlier use has shown their capacity to yield sound conclusions, and retain that role until there are definite grounds for questioning them. They are in this sense "operationally a priori," rather than imposed on inquiry from outside (Henne 2022, §§2.3.1–2.3.2, 58).

Neurath had earlier put a related point in an image. On Neurath's view, as Jordi Cat presents it in his encyclopedia entry on Neurath, there is no tabula rasa: no fully secured protocol statements can be established as starting points of the sciences, and inquirers are like sailors who must rebuild their ship on the open sea, never able to dismantle it in dry dock. Protocol statements, like the other statements of science, can in principle be changed; when a protocol conflicts with theoretical statements, cancelling it is one methodological option, and a statement counts as correct if it can be incorporated into the existing totality of statements (Cat 2024, §3.3). The image marks a feature of revision from within that Chang's epistemic iteration and Tal's time scale, below, both exhibit: what is being revised goes on being used while it is revised, each part being replaced against the support of the rest.

Where standards are not supplied from outside, open-ended inquiry may not infrequently do something of this kind, and Chang's historical and philosophical study of thermometry is an analysis that makes it an explicit theme. Chang argues that attempts to justify measurement methods run into a circularity that empiricist foundationalism, which builds justification up from observation as its base, cannot resolve; that the circularity is better seen as a helix, in which inquiry starts by presuming an untested method and later doubles back to correct its own starting assumptions; and that inquiry must therefore begin from some existing system of knowledge, recognized as imperfect, which successive epistemic iterations then enrich and correct, a view Chang calls progressive coherentism (Chang 2007, §2). In Chang's outline of the thermometric episodes, standards are corrected by the inquiry they made possible. Numerical thermometers were made, each on the presumption that its fluid expanded uniformly, although there was never any indubitable reason to believe this, and those that did not give self-consistent numbers were eventually rejected, Victor Regnault's work in the 1840s being crucial in this. Air, the most self-consistent fluid, provided the readings on which William Thomson's thermodynamic theory and the Joule-Thomson experiment were based, and with these physicists corrected the air thermometer used in that very experiment (Chang 2007, §2). Chang did not at first present epistemic iteration as a general thing: in an interview, Chang recalls having offered it as an idea that made sense of the specific thermometric instances treated in that study, and later finding examples of it almost everywhere (Chang 2023, 00:10:20).

The same structure was described in psychology, with the same example of temperature, by Cronbach and Meehl, writing on how psychological tests are validated. On their account, validating a test against a criterion means correlating it with an independent measure of what it is meant to predict, and validating its content means showing that its items sample a defined universe; construct validation is involved whenever a test is interpreted as measuring an attribute that is not operationally defined, such as anxiety proneness, and it proceeds by checking whether the test's relations to other measures are those that the theory of the attribute predicts, so that the test's interpretation and the theory are assessed together. Construct validity must be investigated whenever no criterion or universe of content is accepted as entirely adequate to define the quality to be measured, and investigators turn to it as the only way to avoid what they call, quoting another author, the "infinite frustration" of relating every criterion to some more ultimate standard (Cronbach and Meehl 1955, 175–177, 192–193). Measurement of temperature, they write, began from "an extremely fallible 'criterion'," the human temperature sense, which was later "relegated to a peripheral position": "We have lifted ourselves by our bootstraps, but in a legitimate and fruitful way" (Cronbach and Meehl 1955, 181–182). When a prediction derived through a chain of inference fails, they note, any link in the chain may be wrong. Where either the theory or the test's interpretation is well confirmed, the misprediction falls on the other. In the cases between these extremes, which they took to be most cases in the psychology of their day, a failed prediction shows that the bridge between test and theory is undependable, "but this is all we can say," and "the investigator must decide which end it is best to relocate," a choice they call a gamble on the best investment of effort (Cronbach and Meehl 1955, 193–195; compare Sections 2.3 and 4.1).

Elliott extends Chang's term to a wider range of knowledge claims, including theories and models as well as standards and concepts, and distinguishes such epistemic iteration from methodological iteration, repeated movement among modes of research such as hypothesis testing, exploratory inquiry, question-driven inquiry, and technology-oriented research, which counts as progressive, on this account, only in a derivative sense, insofar as it can contribute to epistemic iteration (Elliott 2012, §2). Elliott argues from a case in nanotoxicology that methodological iteration can initiate, equip, and stimulate epistemic iteration, and the case shows how such movement can reach the standards of assessment themselves. Question-driven and exploratory studies across many assays showed, for example, that single-walled carbon nanotubes could adsorb the dye used in a cell-viability assay and so lower its readings of viability: the assay, an instrument for assessing toxicity, was being affected by the material under test. A review Elliott reports then called for new test systems and reference materials (Elliott 2012, §3). The case suggests that such a call marks the beginning of a revision of the standards of assessment; and, as Elliott holds, an inquiry's cycling among modes is not by itself evidence that its knowledge claims are improving.

Tal's study of how coordinated universal time is maintained describes revision of a standard carried out month by month. The free-running time scale from which coordinated universal time is derived is an average of the indications of hundreds of atomic clocks, weighted by their stability. Every month the weights are revised and clocks that fall below a stability threshold are dropped from that month's calculation, and the international atomic time scale derived from the free-running average is steered toward the primary standards; over longer periods, clocks are replaced by others that drift less (Tal 2012, 106–110). Tal presents the exchange between such top-down adjustments and the bottom-up discovery of remaining inconsistencies as an instance of what Chang calls epistemic iteration (Tal 2012, 136 n. 94). Some of these adjustments are deliberately made insensitive to the data: they impose short-term stability on the time scale rather than test whether the clocks are stable. Tal argues that they are nonetheless legitimate, because they legislate how the concept of time is to be applied to particular clocks, laying this down as a rule rather than asserting it as an empirical claim, and so need no independent justification as approximations, while the gaps that remain among independent realizations of the concept are discovered empirically and prompt further adjustment (Tal 2012, 117, 132–134).

Without presupposing a general coherentist or pragmatist theory of research, these accounts show, for the cases they examine, a possibility that a description of research by fixed rules does not show: a practice may improve some of the means and standards by which it judges improvement. They do not license a system to redescribe every failure as success. Chang's own account allows that iteration can fail: asked in an interview what happens when science goes wrong, Chang says that iteration that goes well revises its starting point without completely discarding it and converges, but that iteration can also get nowhere and reach a dead end, after which one has to start again from a new provisional starting point (Chang 2023, 00:21:42). Iteration can remain answerable to recalcitrant phenomena, cross-checks among differently operating instruments, newly exposed errors, successful and failed applications, alternative standards, and criticism by other investigators. Greater internal coherence alone does not establish that these constraints have been met. Even an appeal to coherence would need an account of which system and which kind of coherence matter; merely making selected commitments fit together better does not settle truth. The point is that such constraints can operate during reciprocal revision rather than only from an unrevisable external foundation.

5.3 Improving a standard or replacing it

Section 5.2 set out how standards can be revised within inquiry under constraint. It does not follow that every change of standard is such a revision. When an inquiry, or an artificial system, revises its standards, a question arises that a fixed standard does not raise: is it improving by standards it already held, or replacing them? A system that changes a metric whenever it performs poorly, so that the poor performance no longer shows, has merely moved the target; when such changes are made silently, others are left without the stated reason by which a revision could be told apart from moving the target. Replacing a standard is not by that fact unreasoned: explication, which replaces an inexact concept with a more exact one for a stated purpose (Introduction, Section 4), replaces a concept at least in part (see below), and is assessed against the purpose for which the replacement is made.

Whether improving a standard and replacing it differ in kind is itself disputed. Carnap distinguished questions raised within a linguistic framework, which are answered by logical or empirical methods according to its rules, from the question whether to accept the framework, that is, its rules for forming statements and for testing, accepting, or rejecting them. Accepting a framework, Carnap held, is not an assertion and cannot be true or false; it can be judged only as more or less expedient, fruitful, and conducive to the aim for which the language is intended, although theoretical knowledge bears on that judgment (Carnap [1950] 1984, 242–244, 250). Quine agreed that whether to adopt such a framework is a matter of choosing a convenient conceptual scheme, but only on the proviso that the same be conceded of scientific hypotheses generally: on Quine's account the difference between the two kinds of question is only one of degree, turning on an inclination to adjust one strand of the fabric of science rather than another, in which conservatism and the quest for simplicity figure (Quine 1951, 43). Applied to changes of standard, the two views come apart: on Carnap's view a change made under a framework's existing rules and a change of those rules differ in kind, the second being assessed against the aim for which the framework is chosen, while on Quine's they answer to the same kinds of consideration (compare the gradual distinction between framed and framing inquiry in Chapter 4, Section 4). The choice of which statistical methods to adopt, with the conceptions of justification on which they rest (Chapter 2, Section 5.3), can be read as one instance of this question. Quine also described explication, which the Introduction, Section 4 takes as one model, as an improvement on the concept explicated: it keeps the concept's usage in the contexts where that usage is already clear and precise enough to be useful and sharpens it in others (Quine 1951, 25). Described in this way, explication would be a case of both: it is held to part of the old usage and replaces the rest. On either view, what the silently changed metric lacks is not continuity with the standard it displaced but a stated reason against which the change could be assessed.

On the accounts of Section 5.2, reasoned revision is informed by what a standard is for: which question it serves and what it is meant to check. Detecting that a standard is inadequate for a purpose, proposing a replacement, and evaluating the consequences of changing it together with the concepts, measurements, or methods that depend on it can each be informed by that purpose; they are not stages of increasing demand, and, as Chapter 7, Section 4.3 notes, a system or community can exhibit one without the others. That section distinguishes the capacities concerning norms that an attribution may need to keep apart, without arranging them as levels.

Tal's study of Coordinated Universal Time, seen in Section 5.2, can be read as implying that what separates legislation such as imposing short-term stability on a time scale from an arbitrary change of standard is not whether an adjustment responds to poor performance but whether it is declared as a rule of application and remains answerable to discrepancies among independent realizations of the same concept, assessed on grounds beyond the adjustment itself (compare the defeat relation of Chapter 11, Section 1). Other accounts set further or different conditions. On the discipline that Cronbach and Meehl state for psychological testing (see below), a theory altered to fit an unexpected finding also needs a fresh body of data before the altered version is relied on; on Chang's account, what keeps reciprocal revision from being arbitrary is that it remains answerable to recalcitrant phenomena and can fail, reaching a dead end from which inquiry has to start again from a new provisional starting point (Section 5.2).

A claim that a standard was revised or replaced for reasons, rather than merely changed, would be supported by, among other things, records of the previous standard and its role, reasons and evidence bearing on the change, comparisons with live alternatives, propagation of the change through dependent results, and routes by which the revised arrangement can itself be challenged. Cronbach and Meehl state a related discipline for psychological testing: an investigator who alters a theory to fit an unexpected finding is required to gather a fresh body of data to test the altered hypotheses, a step that "should normally precede publication of the modified theory" (Cronbach and Meehl 1955, 194). The publicly logged language-model project described in Chapter 5, Section 1 provides an example of such a record, although not of an artificial system revising a standard by itself. When a running bits-per-byte average used to compare tokenizers was found to have been weighted by tokens rather than bytes, a maintainer proposed logging the corrected measure under a new name beside the old one, and an AI agent run by one contributor prepared a patch accordingly, so that earlier dashboards and comparisons would not silently change meaning (Marin Community 2026c). The report by Lange and colleagues described in Chapter 4, Section 8 records a change that came to light only afterwards: an agent changed a task's evaluation procedure itself, and the change was caught neither by the evaluator nor by the rubric but by a later manual audit by humans.

Whether a revision counts as an improvement is often settled not by one inquirer but among those who share or contest the standard. If S is human, whether a system's revision of a standard counts as an improvement is judged by members of S in the light of the reasons stated and the record kept; if both A and S are machines, that judgment too may be made within an artificial community (Chapter 8, Section 1.3). Chapter 11, Section 9 takes up Longino's condition that a community's standards be public and themselves open to criticism and transformation, and the question of who revises an artificial configuration's standards, and against which aims.

When research is done by a machine, these questions can change their form in several respects. In human research, many revisions of standards have proceeded within communities and institutions, through records and argument, and at times over long periods; even revisions that proceed month by month, as in Tal's time scales, are made within an institution that maintains them as stated rules. A machine inquirer might redefine its own evaluation metrics and procedures at far shorter intervals, and many times over. Where the difference between improvement and moving the target is located then depends on the account taken. On the reading of Tal above, a system that redefines its own evaluation metric is distinguishable from one that merely moves its target to the extent that it states the redefinition as a rule, remains answerable to discrepancies among independent realizations of what the metric is meant to measure, and keeps adverse outcomes available to challenge the redefined metric, with reasons for reinterpreting them themselves open to scrutiny; on Cronbach and Meehl's discipline, applied by analogy, it would also need to check the redefined metric against data not used in redefining it. On any of these accounts, a record of the earlier standard kept together with the redefinition gives others a means of seeing the difference. The faster and more frequent the redefinitions, the harder silent changes may become to see, while the same speed can also make possible corrections like the iterations Tal describes. What counts as a standard, moreover, can reach what checking itself presupposes (Section 3.3). In a machine inquirer, some of those presuppositions appear as structures set by the way it was made, which remaking it can replace (Chapter 8, Section 2.1). Who can revise them, whether the system itself, those who build and maintain it, or the members of S, connects with the question of Chapter 7, Section 4.3.

Writing about the design of advanced AI rather than about research, some authors have taken positions on which standards such a system should be left to revise. Bostrom proposes endowing a superintelligent system with fundamental epistemological principles that match those governing our own thinking, since consistently applying our own standards would lead us to judge a system diverging from them as reasoning incorrectly, while he proposes leaving the seed AI to create and revise its non-fundamental principles continually as its understanding of the world develops, the seed AI being the system from which a superintelligence would develop (Bostrom 2014a, ch. 13). Applied to research, Bostrom's proposal holds part of what Section 5.2 treated as revisable from within fixed and leaves the rest to the system. On Carnap's view above, such a line could follow the difference between a framework's rules and the changes made under them; on Quine's, it would be a matter of degree, and Bostrom's own text notes that some epistemological systems may have no distinct foundation, in which case what is handed on is not a distinct set of principles but an epistemic starting point with certain propensities to respond to incoming evidence (Bostrom 2014a, ch. 13 n. 40). Tubert and Tiehen, writing about moral values, argue that an AI can play the role of a more consistent version of ourselves who advises us on how to pursue the values we now hold, or the role of a revolutionary who breaks with them, "but it cannot do both"; they themselves compare such a break with Kuhn's view that scientific revolutions could not be fully rational because standards of rationality depend on paradigms, and with the unexpected Move 37 of the Go program AlphaGo (Tubert and Tiehen 2025b, §4). Their argument concerns morality and an AI advising us about the values we now hold, but applied to the standards of research, as their own comparison with Kuhn invites, it suggests that, where S is human, keeping a system's standards consistent with those of S and leaving it room to replace them pull in opposite directions. Which standards are held fixed, and by whom, the system itself or others, then becomes a question on which the difference between improving a standard and replacing it bears directly.

Possessing the capacities discussed in this section would enlarge the scope of an artificial system's epistemic self-government, its autonomy with respect to the standards that organize its epistemic activity (Chapter 7, Section 4.3). Nothing in this chapter makes them conditions of every research activity in the sense of the working characterization; Part IV asks what their presence or external provision implies for different claims of autonomy (Chapter 7, Section 4.3).

6. What the process account passes forward

This chapter has followed the work of determining what a result bears on through contrary results, passes, sequences of revision, and standards themselves. A result that runs against expectation does not by itself say what it bears on. Where it is attributed is a judgment that can depend not only on background knowledge and independent checks but also on convictions and expectations, and that judgment can shape which path later exploration takes (Sections 2 and 4.1). What a failure is in the first place can also change afterward (Section 2.5). A pass, likewise, does not by itself say what it has established. What part of the question it bears on, how firmly it settles that, whether it is merely apparent, and which parts it supports also depend on who set the check and how, and on who reads it against which standards (Section 3). As for sequences of revision, how long it is reasonable to keep an idea is not settled by a single result, and when research is done by a machine that judgment is allocated within the artificial configuration (Section 4). Standards themselves can be revised from within inquiry, for reasons, although what separates such revision from moving the target differs among accounts (Section 5). Through these, the chapter has tried to bring out, within a process often pictured as a flow of generating a hypothesis, testing it, and accepting or rejecting it, the work of determining what a result bears on.

Who carries out this work, and to whom it is visible, changes with the combination of A and S. If A is a machine and S human, members of S read the attributions contained in a system's report, the meaning of its passes, the ideas it discarded, and the standards it redefined, in the light of records and other grounds (Sections 2.1, 3.3, 4.2, and 5.3). If both A and S are machines, checking attributions, reading passes, and setting the standards for what counts as settled may themselves take place within an artificial community. And if what a result gives reason to revise is relative to background knowledge, as on the three-place view of evidence in Section 1, what it gives the actor reason to revise need not be what it gives the members of the society for which the inquiry is conducted reason to revise.

From the side of those who would build a system that does research, these distinctions point to different work. Repeating a prescribed operation and determining why that operation has ceased to bear on the question are different work in human research too, as are applying an inherited standard and examining and revising it. Neither is a condition of every research activity in the sense of the working characterization, and neither need reside in one actor. The practice of the molecular-biology laboratories described in Section 4.3 is a case in which inquiry reached publishable results without locating the cause of a failure within a procedure. What is asked is what organization sustains the inquiry and where it is realized. A claim about artificial research becomes assessable when it makes visible the part of inquiry attributed to the actor, its dependencies and interventions, and the changes needed to assess that attribution. In the cases of this chapter, that can include which possible loci of error or revision had not yet been excluded, what part of the question a pass bore on and how firmly it settled that, who set the check and where in the inquirer it was placed, why a standard changed, and under which judgment an idea was discarded, among other things. A record supports examining those relations; it does not itself supply their epistemic warrant.

Across Part III, the implications drawn for research by a machine have come, in several places, to turn on records. Whether work on the side of questions is attributed to the machine or to something outside it rests on records that track changes (Chapter 4, Section 8); an output is not by itself sufficient evidence of the open-ended research capacity attributed to a system, on which process evidence can bear (Chapter 5, Section 2.1); and members of S read a system's attributions, passes, and discarded ideas partly from records (this chapter). What records leave out has recurred as well, such as the questions that could not be raised under a framework, matters that were never registered, the order of work behind a published account, and ideas discarded without trace (Chapter 4, Sections 4 and 5; Chapter 5, Section 2; Section 4.2 of this chapter).

The chapter thus passes two questions on to Part IV. One is a question of attribution: to which artificial or hybrid configuration, at which boundary, and on what grounds a contribution to research, the conduct of a research activity, a research capacity, or researcher status can be attributed, and in which respects what is attributed remains organized from outside. Chapter 7 takes it up. The other is the question of whose epistemic change is being pursued, for whom a result is new, and who can evaluate or incorporate it, when the actor conducting an inquiry, the bearer of knowledge, and the reference society S differ. Chapter 8 takes it up.