D. Reproducibility statements and supplementary material
Reproducibility is the Method’s function, made auditable
Section I.C assigned the Method a reconstructive function: it must let a competent reader understand and in principle reproduce what was done, and I.C warned that this is easily confused with a persuasive function, since a well-described method also looks more credible. Reproducibility statements and checklists exist because that confusion is common enough to be worth institutionalizing against. A checklist converts “in principle reproducible” into a set of specific questions a third party can verify, which is the same function the Method already had, made auditable rather than assumed.
Read this way, a checklist is not an administrative overlay on the article but a test of whether one of its components did its job. An article that cannot answer the checklist has a Method problem, and discovering that at submission is discovering it late.
What the checklist asks
ARR’s Responsible NLP Research checklist is organized in five sections, and the shape is worth knowing because it maps onto components the course has already treated. Section A covers the article’s limitations and the risks its work carries. Section B covers scientific artifacts: citing the creators of code, data, and models used; documenting licenses and terms of use; confirming that use is compatible with intended purpose; screening for personally identifiable information and offensive content; documenting the artifact’s domains, languages, and demographics; and reporting dataset statistics including splits. Section C covers computational experiments: model parameters, the total computational budget and infrastructure, the hyperparameter search procedure, descriptive statistics with error bars or confidence intervals, and the software packages relied on. Section D covers human participants — instructions, recruitment, consent, ethics board approval, and demographics. Section E covers the use of AI assistants, which V.G treats separately.
Item C3 is III.B and IV.D arriving a third time
The checklist’s requirement to report descriptive statistics with error bars or confidence intervals deserves isolating, because the course has now arrived at it three times by three different routes. Section III.B derived it as a property of sentences: the modality of a claim must match the strength of the evidence, and a single run does not license an unhedged comparative claim. Section IV.D derived it as a property of figures: a single bar per condition with no interval asserts that the value would recur, and an interval is the visual form of a hedge. The checklist arrives at the same requirement as a submission condition, from institutional rather than rhetorical premises.
Three independent derivations converging on one requirement is the strongest evidence available that the functional account is doing real work rather than redescribing convention. The requirement is not a rule the field imposes and the writer complies with; it is what honesty about evidence strength looks like, expressed once in prose, once in graphics, and once as a form.
The Appendix is not where things that did not fit go
Section I.C described the Appendix as offloading material that supports the central argument without being required to follow it on a first reading. The operative word is required: a reader must not need the appendix. This gives a routing rule that resolves most placement questions. Material a reader needs in order to follow the argument belongs in the main text, and if it does not fit, the article has a compression problem that moving it will conceal rather than solve. Material a reader needs in order to check the argument, but not to follow it, belongs in the appendix — full hyperparameter grids, additional examples, secondary proofs. Material a reader needs in order to rerun the work belongs in supplementary artifacts: code, data, configurations, and scripts.
The failure this rule catches is the load-bearing appendix, in which a definition, a crucial protocol detail, or the actual comparison against a baseline sits outside the main text. Since IV.C established that many readers enter an article at a single section and never leave it, an appendix carrying part of the argument is an argument a large fraction of readers will never encounter.
The checklist is a drafting tool
The checklist’s questions are answerable at two different times, and the difference is substantial. Answered after the experiments are complete, it is a form, and its most common output is a set of retrospective justifications and a few honest gaps. Answered before, at the point II.A describes as preparing the discourse, it changes the experiments: a writer who knows that C3 will ask for variance will run multiple seeds, and a writer who knows that B2 will ask about licenses will check them before building on an artifact rather than after.
This is the same argument II.F made about fixing the protocol before seeing the outcome, applied one level out. A requirement known in advance shapes the work; the same requirement encountered at submission can only be reported against.
Matching to the structural variant
The three variants from I.B carry the checklist’s weight in different places. The resource/dataset paper carries the heaviest burden by a wide margin, since Section B is largely about artifacts and Section D about the human annotation its central contribution depends on — PII screening, licensing, documentation of languages and demographics, and annotator recruitment and consent are all load-bearing rather than incidental for this variant. The empirical/analysis paper is most exposed on C3, because its claims are comparative across conditions and seed variance is the thing most likely to undermine them, which is also why I.B listed significance annotations among its characteristic apparatus. The system/method paper is most exposed on C1’s computational budget, since a comparative claim obtained at substantially greater compute than the baselines is a different claim from the one the results table appears to make.
A worked example
Suppose the syntactic-diversity article is prepared for submission. Section B requires citing the creators of the base model and the source corpora, and confirming that the licenses permit the use made of them — a check that can invalidate an experiment and is therefore worth running before the experiment rather than at submission. Section C requires the parameter count, the GPU hours consumed across the full experimental programme including the runs that did not appear in the paper, the hyperparameter search that produced the reported configuration, and the three-seed variance that IV.D’s caption already reported. Section A requires the limitations, which II.H said should be derived from the protocol: one language, one scale, one benchmark, and a diversity measure validated on a single corpus.
The instructive observation is how much of this the article already contains if Parts I through IV were followed. The limitations were derived in II.H, the variance was reported under III.B and displayed under IV.D, and the protocol was fixed in II.F. What the checklist adds is the artifact and compute accounting, which no earlier part of the course required and which is the genuinely new obligation.
Common failure modes
The item answered “N/A” where it applies is the most common and the least defensible, since the answers are published with accepted papers and a reader can check. Beyond it: the computational budget reported as the cost of the final run rather than of the experimental programme, which understates the resource asymmetry between the article and its baselines; the infrastructure described as “a single GPU” with no model named, which is not reproducible information; the appendix carrying a load-bearing definition, described above; supplementary code released without the configuration that produced the reported numbers, which is reproducible in principle and not in fact; and the checklist completed by one author at the deadline, which V.B’s account of seams predicts will contradict the article in at least one place.
A practical check
Give the Method, the Experiments section, and the supplementary artifacts to a competent colleague outside the project, and ask them to list every decision they would have to guess at in order to rerun the work. This is II.F’s reconstruction test with the checklist’s specificity added, and the resulting list is the article’s actual reproducibility gap, which is generally larger than the checklist’s own answers suggest and is discoverable in an afternoon.