Data released internally in POLARBEAR
POLARBEAR, Reproducible research, Pseudo-spectra and MASTER, Beam, Masks and apodization, High-performance computing
Data released internally in POLARBEAR
Data Release—Pseudo-spectra
Introduction
The data version uniquely specified the beam spectra, pseudo spectra, mode-coupling matrices, and theory spectra. The content of these files has never been changed since the simulation runs at NERSC. (While in earlier internal versioning, it is in a different container.)
3 versions are released, v1.0.0, v1.0.1, v1.0.2, together with the masks:
high_1.hdf5
high_1_0_1.hdf5
high_1_0_2.hdf5
masks.hdf5
It is released at NERSC in /global/cfs/cdirs/polar/www/khcheung/largepatch_new/.
high_1_0_1.hdf5 and high_1_0_2.hdf5 uses HDF5 external symlinks to point to data in high_1.hdf5. In practice it means both high_1.hdf5 and high_1_0_2.hdf5 have to be in the same directory in order to read v1.0.2 data.
Configurations
- The mapmaking is done using the AnalysisBackend
khcheungbranch - The post-mapmaking pipeline (masks, pseudo-spectra, mode-coupling) is done using the software TAIL below
- The configuration of the software is summarized in Final configuration of the null-test v1.0.2 used in POLARBEAR large-patch high-\(\ell\) analysis to reproduce v1.0.0. Changelog of null-tests performed in POLARBEAR large-patch analysis has recipe to reproduce v1.0.1 and v1.0.2.
Specifications—General
All l start at 0, and no binning. That is, if a shape is 4901 then it spans the l-range from 0-4900.
Specifications—Masks
p is the polarization mask. t is the temperature mask. Both in map-domain.
$> h5ls -r masks.hdf5
/ Group
/p Dataset {2880, 2880}
/t Dataset {2880, 2880}Specifications—Beam and theory spectra
The data is put in a simple HDF5 container and is programming language agnostic.
This session should be self-explanatory, the datasets within each group are all one-dimensional with the same size. l gives the l-value and for example TT gives the \(TT\) spectrum at those l-values.
$> h5ls -r high_1.hdf5
...
/beam_spectra Group
/beam_spectra/10.1 Dataset {293}
/beam_spectra/10.2 Dataset {293}
/beam_spectra/10.3 Dataset {293}
/beam_spectra/10.4 Dataset {293}
/beam_spectra/10.5 Dataset {293}
/beam_spectra/8.2.0 Dataset {293}
/beam_spectra/9.4 Dataset {293}
/beam_spectra/all Dataset {293}
/beam_spectra/l Dataset {293}
...
/theory_spectra Group
/theory_spectra/BB Dataset {4901}
/theory_spectra/EE Dataset {4901}
/theory_spectra/TE Dataset {4901}
/theory_spectra/TT Dataset {4901}
/theory_spectra/dE Dataset {4901}
/theory_spectra/dT Dataset {4901}
/theory_spectra/dd Dataset {4901}
/theory_spectra/dens[1]-dens[1] Dataset {4901}
/theory_spectra/dens[1]-lens[1] Dataset {4901}
/theory_spectra/l Dataset {4901}
/theory_spectra/lens[1]-lens[1] Dataset {4901}
/theory_spectra/phi-dens[1] Dataset {4901}
...Specifications—Mode-coupling matrices
The names should be self-explanatory. The difference between EBEB and EBEB_pseudo follows AnalysisBackend (although generated in TAIL.)
$> h5ls -r high_1.hdf5
...
/mode_coupling Group
/mode_coupling/BBBB Dataset {4901, 4901}
/mode_coupling/EBEB Dataset {4901, 4901}
/mode_coupling/EBEB_pseudo Dataset {4901, 4901}
/mode_coupling/EEEE Dataset {4901, 4901}
/mode_coupling/TBTB Dataset {4901, 4901}
/mode_coupling/TETE Dataset {4901, 4901}
/mode_coupling/TTTT Dataset {4901, 4901}
...Specifications—pseudo-spectra
- full/null indicates it is from the full map data or null-split data.
realmapsis from real map data,simmapsis from simulated map data.TTBB/TTEE/TTEEBBrepresent the power included in the input map.noise: generated from random sign-flip-noise fromrealmaps- Each of these datasets are of 6 dimensions. In this order:
- spectra: the 6 different spectra: \(TT\), \(TE\), …
- null-split: the 21 null-split:
AMP_2F_BY_BOLO, … - sub-split: 0/1/2. 0 is the 1st-half, 1 is the 2nd-half, 2 is the cross-spectrum between the 1st- and 2nd-half.
- l: self-explanatory
- sub-spectra: Cl, Nl. Cl is the signal spectra (using the cross-spectrum estimator), Nl is the noise spectra.
- n: n-th realization
See more on meta below.
$> h5ls -r high_1.hdf5
...
/full Group
/full/pseudo_spectra Group
/full/pseudo_spectra/realmaps Dataset {6, 1, 1, 4901, 2, 1}
/full/pseudo_spectra/simmaps_TTBB Dataset {6, 1, 1, 4901, 2, 128}
/full/pseudo_spectra/simmaps_TTEE Dataset {6, 1, 1, 4901, 2, 128}
/full/pseudo_spectra/simmaps_TTEEBB_noise Dataset {6, 1, 1, 4901, 2, 512}
...
/null Group
/null/pseudo_spectra Group
/null/pseudo_spectra/realmaps Dataset {6, 21, 3, 4901, 2, 1}
/null/pseudo_spectra/simmaps_TTBB Dataset {6, 21, 3, 4901, 2, 128}
/null/pseudo_spectra/simmaps_TTEE Dataset {6, 21, 3, 4901, 2, 128}
/null/pseudo_spectra/simmaps_TTEEBB_noise Dataset {6, 21, 3, 4901, 2, 512}
...Specifications—meta
null_split: the names of the 21 null-splitspectra: the names of the 6 spectrasub_spectra: the names as Cl, Nl
$> h5ls -r high_1.hdf5
...
/meta Group
/meta/null_split Dataset {21}
/meta/spectra Dataset {6}
/meta/sub_spectra Dataset {2}
...Data Release—Spectra—final freeze
Final freeze configuration
This data is released as high_1_0_2_spectra_final.hdf5. It contains both bin-width of 50 and 100.
Using TAIL,
solve_spectra high_1_0_2.hdf5 -o high_1_0_2_spectra_final.hdf5 --scaling --sim-noise-matching -b 50 100 --return-w
iterative_map high_1_0_2.hdf5 masks.hdf5 -o high_1_0_2_spectra_final.hdf5 --scaling --sim-noise-matching -b 100 --l-T-cutoff 1600 -n 100
iterative_map high_1_0_2.hdf5 masks.hdf5 -o high_1_0_2_spectra_final.hdf5 --scaling --sim-noise-matching -b 50 --l-T-cutoff 1550 -n 100Whenever MC error-bar is calculated, leave Spectra at default attributes: ev_est='signal', ddof=None.
Note: difference between final freeze and stage 2 unblinding spectra
- all options mentioned in [Changes and configurable options] is changed here comparing to those in [Reproducing null-spectra and full-spectra at unblinding].
- although between now and unblinding, sometimes bin-width of 50 is used in presentations, here bin-width of 100 is used to be consistent with the unblinding configuration. For the reason of optimizing bin-width, see the section “Choosing ℓ-min, ℓ-max, bin-width” of the internal POLARBEAR document Unblinding Proposal—Stage 1
Specifications—Filter Transfer Functions
- full/null indicates it is from the full map data or null-split data.
/full/filter_transfer/land/null/filter_transfer/lspecify the filter l-range. Here the 2 elements are 50, 4300, meaning the l-range is from 50–4300 (no binning, that is, dl=1)/null/filter_transfer/Fand/full/filter_transfer/Fare 4d-array, same as first 4 dimensions from pseudo-spectra. See more in Specifications—pseudo-spectra.- The first dimension (spectra) order is specified in
/meta/filter_transfer. For example,[..., BB, EE, TT]specifies the 3 auto-spectra cases.
$> h5ls -r high_1_0_2_spectra_final.hdf5
/ Group
/full Group
/full/filter_transfer Group
/full/filter_transfer/F Dataset {3, 1, 1, 4250}
/full/filter_transfer/l Dataset {2}
...
/meta Group
/meta/filter_transfer Dataset {5}
...
/null Group
/null/filter_transfer Group
/null/filter_transfer/F Dataset {3, 21, 3, 4250}
/null/filter_transfer/l Dataset {2}
...Specifications—spectra
- similar to pseudo-spectra, see more in Specifications—pseudo-spectra. Same 6 dimensions except that l becomes b (bin).
metais also the same. - there’s one more sub-level,
lmin_lmax_binwidthsuch as50_4300_50, indicating the l-min, l-max and bin-width used when solving for spectra. /*/spectra/*/thetas_mapare the maximum a posteriori thetas found from the likelihood code that transform the spectra (including absolute gain and angle.)/*/spectra/*/w_bl: band pass window function. 5 dimension: spectra, null-split, sub-split, b (bin), l.
$> h5ls -r high_1_0_2_spectra_final.hdf5
...
/full/spectra Group
/full/spectra/100_4300_100 Group
/full/spectra/100_4300_100/map_cases Group
/full/spectra/100_4300_100/map_cases/realmaps Dataset {6, 1, 1, 42, 2, 1}
/full/spectra/100_4300_100/map_cases/simmaps_TTBB Dataset {6, 1, 1, 42, 2, 128}
/full/spectra/100_4300_100/map_cases/simmaps_TTEE Dataset {6, 1, 1, 42, 2, 128}
/full/spectra/100_4300_100/map_cases/simmaps_TTEEBB_noise Dataset {6, 1, 1, 42, 2, 512}
/full/spectra/100_4300_100/thetas_map Dataset {6}
/full/spectra/100_4300_100/w_bl Dataset {6, 1, 1, 42, 4200}
/full/spectra/50_4300_50 Group
/full/spectra/50_4300_50/map_cases Group
/full/spectra/50_4300_50/map_cases/realmaps Dataset {6, 1, 1, 85, 2, 1}
/full/spectra/50_4300_50/map_cases/simmaps_TTBB Dataset {6, 1, 1, 85, 2, 128}
/full/spectra/50_4300_50/map_cases/simmaps_TTEE Dataset {6, 1, 1, 85, 2, 128}
/full/spectra/50_4300_50/map_cases/simmaps_TTEEBB_noise Dataset {6, 1, 1, 85, 2, 512}
/full/spectra/50_4300_50/thetas_map Dataset {6}
/full/spectra/50_4300_50/w_bl Dataset {6, 1, 1, 85, 4250}
/meta Group
...
/meta/null_split Dataset {21}
/meta/spectra Dataset {6}
/meta/sub_spectra Dataset {2}
...
/null/spectra Group
/null/spectra/100_4300_100 Group
/null/spectra/100_4300_100/map_cases Group
/null/spectra/100_4300_100/map_cases/realmaps Dataset {6, 21, 3, 42, 2, 1}
/null/spectra/100_4300_100/map_cases/simmaps_TTBB Dataset {6, 21, 3, 42, 2, 128}
/null/spectra/100_4300_100/map_cases/simmaps_TTEE Dataset {6, 21, 3, 42, 2, 128}
/null/spectra/100_4300_100/map_cases/simmaps_TTEEBB_noise Dataset {6, 21, 3, 42, 2, 512}
/null/spectra/100_4300_100/w_bl Dataset {6, 21, 3, 42, 4200}
/null/spectra/50_4300_50 Group
/null/spectra/50_4300_50/map_cases Group
/null/spectra/50_4300_50/map_cases/realmaps Dataset {6, 21, 3, 85, 2, 1}
/null/spectra/50_4300_50/map_cases/simmaps_TTBB Dataset {6, 21, 3, 85, 2, 128}
/null/spectra/50_4300_50/map_cases/simmaps_TTEE Dataset {6, 21, 3, 85, 2, 128}
/null/spectra/50_4300_50/map_cases/simmaps_TTEEBB_noise Dataset {6, 21, 3, 85, 2, 512}
/null/spectra/50_4300_50/w_bl Dataset {6, 21, 3, 85, 4250}Data Release—Real maps
It is released at NERSC in /global/cfs/cdirs/polar/www/khcheung/largepatch_null/high_1_0_2/.
It contains all the maps created from the real data (but none of the simulated data.) For example, to find the final coadded maps, it is located in /global/cfs/cdirs/polar/www/khcheung/largepatch_null/high_1_0_2/coadd_full/null_realmaps/coadd/realmap.hdf5 at NERSC.
Specifications
$> h5ls -r /global/cfs/cdirs/polar/www/khcheung/largepatch_null/high_1_0_2/coadd_full/null_realmaps/coadd/realmap.hdf5
/ Group
/d0 Dataset {1351, 1351}
/d4i Dataset {1351, 1351}
/d4r Dataset {1351, 1351}
/w0 Dataset {1351, 1351}
/w4 Dataset {1351, 1351}They are all 2D-array, with w indicating the total weights, and d indicating the coadded signal. 0 is the 0-f component, 4 is the 4-f component, r is the real part, i is the imaginary part.
For example,
\[\begin{aligned} I &= \frac{d_0}{w_0} \\ Q &= \frac{d_4^r}{w_4} \\ U &= \frac{d_4^i}{w_4} \\ \end{aligned}\]
These maps are projected using the OLEA projection, as requested from the high-\(\ell\) \(EE\) pipeline. An example implementation of the projector is in https://github.com/ickc/TAIL/blob/3b4e49d6d5a4/tail/pointsource.py#L10-L22.
Also, for null-maps, there may be additional HDF5 group level with the name of respective null-split, which would be self-explanatory.