30 The limitation of the null-test framework
POLARBEAR, Null tests, Blind analysis, Hypothesis testing, Systematics, Calibration
30 The limitation of the null-test framework
30.1 Hypothesis testing, discovery, and default action
As mentioned before, the POLARBEAR null-test framework is based on hypothesis testing in statistics where our null-hypothesis \(H_0\) is the absence of systematics. Hence, our discovery in this hypothesis testing is the detection of systematics, which contaminates our data.
But here we are using hypothesis testing in a very different context compared to how it is usually used in science. Namely, typically the null-hypothesis is the lack of new discovery leading to new science. To put it in other words, as Oleszak1 put it as “the null hypothesis reflects our default action”. In the case of making new scientific discoveries, when no discovery (null-hypothesis not rejected) is found, the default action would be not to proceed any further including publications.2
Here, as we made no discovery (null-test suite makes no discovery on systematic bias) we proceed and go to unblind the data resulting in a publication. We are taking a much more risky default action—unblind and publish.
On one hand, certainly the null-test suite is not the only procedure to guard against biases. But on the other hand, biases can arise even after all these and passing any null-test criteria.
For example, we found that the absolute angle calibration is too large after unblinding the data, which is due to not having deconvolvedSee chapter 26. the TOD by the detector time constant, which turns out to be an internal (lack of) communication error in applying a single command line argument!
Despite the presence of this systematics, the null-test suite passes anyway because it represents a lag in the measured detector signal, which has an effect that is degenerate from an additional rotation (resulting in a large apparent absolute angle in calibration.) This effect would not cause a null-test failure (as rotating null still results in null.)
Besides being an example of a limitation in the null-test framework, this perhaps reiterates the point in chapter 24 that even reproducing a result within a collaboration is crucial to avoid an error like this. It is a systemic issue that cultivates a problem as important as this to be overlooked.
30.2 The action of unblinding
Another limitation of the null-test framework is implicit in its default action—if null-test is passed, then we unblind the data.
But what is blindness? It is a somewhat arbitrary entity with no clear definitions. For example, it is argued that looking at the signal maps is ok, because the map itself is not the science product in our case. And it is abstract enough that from looking at the map, it is hard to draw conclusions. Nonetheless, this makes defining blindness rigorously impossible.
Another more subtle case would be to look at the null-spectra. Since they are expected to be zero, there is no harm done by looking at them, right? But the nature of systematics is that we may not know every source of systematics beforehand, and some, which can be as simple as a gain mismatch, can result in the signal power-spectra leaking into the null-spectra. One can be accidentally looking at a rescaled signal when looking at null-spectra.
There is an even more philosophical example, that when one runs the pipelines from AnalysisBackend, by default it will print the numerical power-spectra on screen. Those are usually scrolling so fast that a normal human’s eyes cannot catch up, but nonetheless this is an arbitrary blindness criterion which relies on the human’s incapability (in reading numbers fast enough).
Not being able to write down a formal definition of blindness has consequences. In an early study of absolute gain calibration, I accidentally used the real data to project the level of absolute gain expected. In hindsight, it clearly violates the blindness criteria, but it is easy to lump all these together and think that “since the gain is not our science product, it would be fine.”
30.3 After unblinding
Another deficiency of the null-test framework is that it implicitly assumes an unblinding procedure comes from what can be changed after unblinding.
If one changes an analysis technique for example, it can be a source of observer bias. So ideally one should not change anything after unblinding. But in our framework, a practical problem is that we do not strongly specify the analysis techniques involved after unblinding.
We already went to great lengths in chapter 28 discussing the procedure involved, but that is not a strong specification in the sense that there are devils in the details that can be changed.
To give one example, recall that we use sign-flip noise in the final analysis. It means that noise estimated from the real data is injected into maps simulated using fiducial maps, which are generated using the 2018 Planck cosmology.3 Hence, the noise components and signal components have a different Pixel Window Function (PWF). Admittedly, the implication or even existence of this has not been well thought out before unblinding. It also means after unblinding, new code was written to take this effect into account.
A solution is offered by reproducible research4, that we should have required the whole software stack and pipelines to be reproducible before unblinding. The unblinding stage should run exactly the same pipelines over the newly generated datasets.
30.4 Miscellaneous problems
Lastly, after the final analysis, I found 2 programming bugs such that the counter term of the pure \(B\)-mode estimator is incorrect.
This increases the amount of \(E\)-to-\(B\) leakage. Phenomenologically, \(\sim {20\,\%}\) of \(B\)-mode power comes from leakage peaked at the lower-\(\ell\) end. While the bug makes this larger, it is much smaller than the total statistical error in this analysis.
One solution to problems like this is to have maintainers for research software, and to require code-review before merging, which is a standard software engineering practice elsewhere.
Footnotes
See Banerjee et al., “Hypothesis Testing, Type I and Type II Errors” for more about the basics of good hypothesis testing.↩︎
Planck Collaboration, Aghanim, Akrami, Ashdown, et al., “Planck 2018 Results”.↩︎