24 Reproducible research
Reproducible research, Research software, POLARBEAR
24 Reproducible research
24.1 Introduction
Distilled to the essence, one may define science as a discipline to predict. In this aspect, reproducibility can be viewed as the cornerstone of science—without reproducibility, one cannot reliably predict even the outcome of a repeated experiment, and it is not science.
However, the term replication crisis or reproducibility crisis has been coined in the recent history of science.1
There are two layers of reproducibility. The first is reproducibility in the general sense which focuses on the reproducibility of the experiment in general, and is a standard expectation in scientific publications. The second is restricted to a subset of reproducibility in an experiment, called computationally reproducible research or often simply reproducible research.2
Regarding the general reproducibility, it is central to any CMB experiment. For example, systematic effects are an important area that many CMB physicists deeply worry about. Systematic effects can be viewed as one form of reproducibility requirement—that is, when constructing an independent experiment which comes with a different set of systematics, they better reproduce the results from each other by excluding any significant systematic effects. In POLARBEAR, the null-test framework is used because it helps to avoid these systematic effects. POLARBEAR is designed to be a “Blind Analysis as a Correction for Confirmatory Bias”,3 which is one example of many forms of bias that result in non-reproducible science. The general agreements between results from CMB experiments indicate that the general reproducibility crisis is not a major problem here.
However, in terms of computationally reproducible research, CMB experiments fail in this regard to various extents, especially in the first generations of ground-based CMB experiments including POLARBEAR.
What is (computationally) reproducible research? Peng4 said,
There are two major components to a reproducible study: that the raw data from the experiment are available; and that the statistical code and documentation to reproduce the analysis are also available.
Hutson5 mentioned in an article focusing on Artificial Intelligence (AI) with the following subtitle: “Unpublished code and sensitivity to training conditions make many claims hard to verify”, while Miyakawa6 even claim in the title: “No raw data, no science: another possible source of the reproducibility crisis”.
To summarize, (computationally) reproducible research focuses on what happens after an experiment gathers data. Reproducing an experiment is expensive, making the entry barrier to reproduce very high. Reproducible research requires everything but the building of the experiment itself to be completely reproducible in the sense that the observed data and software written to come to the conclusion of the research are open for anyone to examine and come to an independent conclusion. A prerequisite of this would require all raw data collected to be released openly, and all essential software that is used in a publication to be made at the very least source-available if not open-source7.
In this sense, none of the CMB experiments are reproducible to the best of the author’s knowledge. For example, all WMAP data are made available in LAMBDA8 but not all its associated code and recipe for complete reproducibility9. Planck releases many of their data products and software packages, but not all.10 Simons Observatory releases most of its software in a public GitHub organization,11 but no commitment has been made yet for complete (computational) reproducibility.12 Not to mention POLARBEAR has released their data products sparingly, has made no raw data available to the public, nor most, if not all, of the software they used internally.
While this is not something the author can change, the author decomposes the different extents of reproducibility in the following way, and makes every effort to make it as reproducible as humanly possible given the constraints.
First, we already mentioned that (computationally) reproducible research has components of data and software. Second, we can define the degree of reproducibility for different levels of audiences—the general public, and internal collaborators. This can be illustrated in tbl. 24.1.
| Collaborators | General public | |
|---|---|---|
| Data | Is the data accessible to internal collaborators? Is the data format documented? | Is the data available to the public? |
| Software | Is the software available to internal collaborators? Is the recipe to reproduce a product available and documented, including the software environment? | Is the software used in research released as open-source software? |
24.2 Making this research more reproducible
Software
As mentioned above, the raw data used in POLARBEAR is not available to the general public. The software that the bulk of data analysis depends on resides in a software repository named AnalysisBackend, and is not made source-available.
While there is nothing the author can do to change this, the author makes an effort to factor the code in a way that most newly written code does not reside in AnalysisBackend but in other repositories such that the author can freely open-source them without being subject to the agreements of any existing authors in the original repositories.
Because of that, most of the descriptions here focus on reproducibility to internal collaborators.
While POLARBEAR’s data is in principle available to any internal collaborators, they reside in a project directory in the Community File System (CFS) at NERSC. Due to the lack of maintainers in that space, any collaborators can write to that space with a default permission that is not readable to anyone but the one who writes the data. Hence, while all files in this space are supposed to be group-readable among all collaborators, some files such as some instrumental metadata are not readable. While this does not impact the results of analysis, it does cause trouble in, for example, staging raw data to scratch space for further analysis, and is a sign of not designing with the goal of reproducibility towards even internal collaborators. While I can volunteer to manage the project account, since there is no maintainer, there is not much I can do to change the situation.
To make my contribution reproducible to internal collaborators, I set up a GitHub organization for internal collaborators. Using that platform, I provide a repository for all the scripts I submitted to NERSC—this resembles how a lab notebook is used in a laboratory setting, whereas I recorded every script I ran at NERSC so that everything I did is traceable, a first step towards being reproducible.
Soon I found that the software environment can be part of the puzzle that makes something not reproducible. At first it was provided by another collaborator as HPCPorts, but as it is abandoned and no longer updated, I volunteered to maintain a software environment to run our software stack at NERSC. To make this software environment reproducible, I had to automate the building of the software environments (which is basically what HPCPorts did) in the form of scripts. Eventually, I rewrote it as a Python package called pmpm which stands for Python manual package manager.
Another pain point in reproducibility is the software we used—AnalysisBackend. It has no maintainer, because of that, every collaborator just creates their own branch and merging their contribution is often not done. One consequence of this is that on many occasions in our analysis meetings, after discussing an analysis problem for a long while, someone would often start to realize the root of the problem and ask, “wait a minute, which branch are you using?”. While this situation cannot be changed in AnalysisBackend without top-down management, the main collaborators working on large-patch analysis try to converge on a common branch. While each of us still maintains our own branch, we mostly keep the differences between these branches minimal.
Since I opt for writing in separate repositories as much as possible, there I can opt for semantic versioning13 with changelogs and documentations.
Regarding reproducibility between collaborators, oftentimes the pipeline scripts are passed with a long chain of command line arguments that is hard to trace the differences because there can be hundreds of options involved. A better way would be to put these configuration arguments in a format that is more tractable. In order to do this while still being able to run arbitrary command line programs with arguments from any software including AnalysisBackend, yaml2cli is written to “organize cli args by YAML” which can generate bash scripts including SLURM batch scripts from a configuration written in YAML. YAML is chosen because of its readability and features such as alias that facilitate “code reuse” within the configuration file. See Final configuration of the null-test v1.0.2 used in POLARBEAR large-patch high-\(\ell\) analysis for examples used in this analysis.
Data
Data generated in my analysis also uses the semantic versioning scheme. See Changelog of null-tests performed in POLARBEAR large-patch analysis for the versions of null-tests performed in this analysis.
The data container itself should be chosen for performance and stable archival purpose. While many collaborators like to save data in the Python pickle format due to its ease of use, “you should be wary of using pickles for long-term storage where the underlying code is not highly stable”14. HDF515 is a standard data container format designed for this purpose. It has wide adoption in the field of cosmology.
Not only is it important to use a stable container for archival purpose, how data is stored should be documented carefully to ensure it is used and interpreted correctly. In this analysis, the data released internally are documented and the documentation is made available in Data released internally in POLARBEAR.
Footnotes
Ioannidis, “Why Most Published Research Findings Are False”; Schooler, “Metascience Could Rescue the ‘Replication Crisis’”.↩︎
Alston and Rick, “A Beginner’s Guide to Conducting Reproducible Research”; Peng, “Reproducible Research in Computational Science”.↩︎
MacCoun and Perlmutter, “Blind Analysis as a Correction for Confirmatory Bias in Physics and in Psychology”.↩︎
Source-available simply means the source is available for audit, open-source has additional requirements concerning the rights of the users receiving the source codes as well.↩︎
Bennett et al., “NINE-YEAR WILKINSON MICROWAVE ANISOTROPY PROBE ( WMAP ) OBSERVATIONS”.↩︎
Incidentally, because of the availability of full WMAP data, a third party effort has been made to claim the first reproducible CMB map in Bobin, Sureau, and Starck, “Cosmic Microwave Background Reconstruction from WMAP and Planck PR2 Data”.↩︎
Koopman et al., “The Simons Observatory”; Li et al., “The Simons Observatory”.↩︎