Skip to main content

Data released internally in POLARBEAR

Keywords

POLARBEAR, Reproducible research, Pseudo-spectra and MASTER, Beam, Masks and apodization, High-performance computing

Data released internally in POLARBEAR

Data Release—Pseudo-spectra

Introduction

The data version uniquely specified the beam spectra, pseudo spectra, mode-coupling matrices, and theory spectra. The content of these files has never been changed since the simulation runs at NERSC. (While in earlier internal versioning, it is in a different container.)

3 versions are released, v1.0.0, v1.0.1, v1.0.2, together with the masks:

high_1.hdf5
high_1_0_1.hdf5
high_1_0_2.hdf5
masks.hdf5

It is released at NERSC in /global/cfs/cdirs/polar/www/khcheung/largepatch_new/.

high_1_0_1.hdf5 and high_1_0_2.hdf5 uses HDF5 external symlinks to point to data in high_1.hdf5. In practice it means both high_1.hdf5 and high_1_0_2.hdf5 have to be in the same directory in order to read v1.0.2 data.

Configurations

Specifications—General

All l start at 0, and no binning. That is, if a shape is 4901 then it spans the l-range from 0-4900.

Specifications—Masks

p is the polarization mask. t is the temperature mask. Both in map-domain.

$> h5ls -r masks.hdf5 
/                        Group
/p                       Dataset {2880, 2880}
/t                       Dataset {2880, 2880}

Specifications—Beam and theory spectra

The data is put in a simple HDF5 container and is programming language agnostic.

This session should be self-explanatory, the datasets within each group are all one-dimensional with the same size. l gives the l-value and for example TT gives the \(TT\) spectrum at those l-values.

$> h5ls -r high_1.hdf5
...
/beam_spectra            Group
/beam_spectra/10.1       Dataset {293}
/beam_spectra/10.2       Dataset {293}
/beam_spectra/10.3       Dataset {293}
/beam_spectra/10.4       Dataset {293}
/beam_spectra/10.5       Dataset {293}
/beam_spectra/8.2.0      Dataset {293}
/beam_spectra/9.4        Dataset {293}
/beam_spectra/all        Dataset {293}
/beam_spectra/l          Dataset {293}
...
/theory_spectra          Group
/theory_spectra/BB       Dataset {4901}
/theory_spectra/EE       Dataset {4901}
/theory_spectra/TE       Dataset {4901}
/theory_spectra/TT       Dataset {4901}
/theory_spectra/dE       Dataset {4901}
/theory_spectra/dT       Dataset {4901}
/theory_spectra/dd       Dataset {4901}
/theory_spectra/dens[1]-dens[1] Dataset {4901}
/theory_spectra/dens[1]-lens[1] Dataset {4901}
/theory_spectra/l        Dataset {4901}
/theory_spectra/lens[1]-lens[1] Dataset {4901}
/theory_spectra/phi-dens[1] Dataset {4901}
...

Specifications—Mode-coupling matrices

The names should be self-explanatory. The difference between EBEB and EBEB_pseudo follows AnalysisBackend (although generated in TAIL.)

$> h5ls -r high_1.hdf5
...
/mode_coupling           Group
/mode_coupling/BBBB      Dataset {4901, 4901}
/mode_coupling/EBEB      Dataset {4901, 4901}
/mode_coupling/EBEB_pseudo Dataset {4901, 4901}
/mode_coupling/EEEE      Dataset {4901, 4901}
/mode_coupling/TBTB      Dataset {4901, 4901}
/mode_coupling/TETE      Dataset {4901, 4901}
/mode_coupling/TTTT      Dataset {4901, 4901}
...

Specifications—pseudo-spectra

  • full/null indicates it is from the full map data or null-split data.
  • realmaps is from real map data, simmaps is from simulated map data.
  • TTBB/TTEE/TTEEBB represent the power included in the input map.
  • noise: generated from random sign-flip-noise from realmaps
  • Each of these datasets are of 6 dimensions. In this order:
    • spectra: the 6 different spectra: \(TT\), \(TE\), …
    • null-split: the 21 null-split: AMP_2F_BY_BOLO, …
    • sub-split: 0/1/2. 0 is the 1st-half, 1 is the 2nd-half, 2 is the cross-spectrum between the 1st- and 2nd-half.
    • l: self-explanatory
    • sub-spectra: Cl, Nl. Cl is the signal spectra (using the cross-spectrum estimator), Nl is the noise spectra.
    • n: n-th realization

See more on meta below.

$> h5ls -r high_1.hdf5
...
/full                    Group
/full/pseudo_spectra     Group
/full/pseudo_spectra/realmaps Dataset {6, 1, 1, 4901, 2, 1}
/full/pseudo_spectra/simmaps_TTBB Dataset {6, 1, 1, 4901, 2, 128}
/full/pseudo_spectra/simmaps_TTEE Dataset {6, 1, 1, 4901, 2, 128}
/full/pseudo_spectra/simmaps_TTEEBB_noise Dataset {6, 1, 1, 4901, 2, 512}
...
/null                    Group
/null/pseudo_spectra     Group
/null/pseudo_spectra/realmaps Dataset {6, 21, 3, 4901, 2, 1}
/null/pseudo_spectra/simmaps_TTBB Dataset {6, 21, 3, 4901, 2, 128}
/null/pseudo_spectra/simmaps_TTEE Dataset {6, 21, 3, 4901, 2, 128}
/null/pseudo_spectra/simmaps_TTEEBB_noise Dataset {6, 21, 3, 4901, 2, 512}
...

Specifications—meta

  • null_split: the names of the 21 null-split
  • spectra: the names of the 6 spectra
  • sub_spectra: the names as Cl, Nl
$> h5ls -r high_1.hdf5
...
/meta                    Group
/meta/null_split         Dataset {21}
/meta/spectra            Dataset {6}
/meta/sub_spectra        Dataset {2}
...

Data Release—Spectra—final freeze

Final freeze configuration

This data is released as high_1_0_2_spectra_final.hdf5. It contains both bin-width of 50 and 100.

Using TAIL,

solve_spectra high_1_0_2.hdf5 -o high_1_0_2_spectra_final.hdf5 --scaling --sim-noise-matching -b 50 100 --return-w
iterative_map high_1_0_2.hdf5 masks.hdf5 -o high_1_0_2_spectra_final.hdf5 --scaling --sim-noise-matching -b 100 --l-T-cutoff 1600 -n 100
iterative_map high_1_0_2.hdf5 masks.hdf5 -o high_1_0_2_spectra_final.hdf5 --scaling --sim-noise-matching -b  50 --l-T-cutoff 1550 -n 100

Whenever MC error-bar is calculated, leave Spectra at default attributes: ev_est='signal', ddof=None.

Note: difference between final freeze and stage 2 unblinding spectra

  • all options mentioned in [Changes and configurable options] is changed here comparing to those in [Reproducing null-spectra and full-spectra at unblinding].
  • although between now and unblinding, sometimes bin-width of 50 is used in presentations, here bin-width of 100 is used to be consistent with the unblinding configuration. For the reason of optimizing bin-width, see the section “Choosing ℓ-min, ℓ-max, bin-width” of the internal POLARBEAR document Unblinding Proposal—Stage 1

Specifications—Filter Transfer Functions

  • full/null indicates it is from the full map data or null-split data.
  • /full/filter_transfer/l and /null/filter_transfer/l specify the filter l-range. Here the 2 elements are 50, 4300, meaning the l-range is from 50–4300 (no binning, that is, dl=1)
  • /null/filter_transfer/F and /full/filter_transfer/F are 4d-array, same as first 4 dimensions from pseudo-spectra. See more in Specifications—pseudo-spectra.
  • The first dimension (spectra) order is specified in /meta/filter_transfer. For example, [..., BB, EE, TT] specifies the 3 auto-spectra cases.
$> h5ls -r high_1_0_2_spectra_final.hdf5 
/                        Group
/full                    Group
/full/filter_transfer    Group
/full/filter_transfer/F  Dataset {3, 1, 1, 4250}
/full/filter_transfer/l  Dataset {2}
...
/meta                    Group
/meta/filter_transfer    Dataset {5}
...
/null                    Group
/null/filter_transfer    Group
/null/filter_transfer/F  Dataset {3, 21, 3, 4250}
/null/filter_transfer/l  Dataset {2}
...

Specifications—spectra

  • similar to pseudo-spectra, see more in Specifications—pseudo-spectra. Same 6 dimensions except that l becomes b (bin). meta is also the same.
  • there’s one more sub-level, lmin_lmax_binwidth such as 50_4300_50, indicating the l-min, l-max and bin-width used when solving for spectra.
  • /*/spectra/*/thetas_map are the maximum a posteriori thetas found from the likelihood code that transform the spectra (including absolute gain and angle.)
  • /*/spectra/*/w_bl: band pass window function. 5 dimension: spectra, null-split, sub-split, b (bin), l.
$> h5ls -r high_1_0_2_spectra_final.hdf5 
...
/full/spectra            Group
/full/spectra/100_4300_100 Group
/full/spectra/100_4300_100/map_cases Group
/full/spectra/100_4300_100/map_cases/realmaps Dataset {6, 1, 1, 42, 2, 1}
/full/spectra/100_4300_100/map_cases/simmaps_TTBB Dataset {6, 1, 1, 42, 2, 128}
/full/spectra/100_4300_100/map_cases/simmaps_TTEE Dataset {6, 1, 1, 42, 2, 128}
/full/spectra/100_4300_100/map_cases/simmaps_TTEEBB_noise Dataset {6, 1, 1, 42, 2, 512}
/full/spectra/100_4300_100/thetas_map Dataset {6}
/full/spectra/100_4300_100/w_bl Dataset {6, 1, 1, 42, 4200}
/full/spectra/50_4300_50 Group
/full/spectra/50_4300_50/map_cases Group
/full/spectra/50_4300_50/map_cases/realmaps Dataset {6, 1, 1, 85, 2, 1}
/full/spectra/50_4300_50/map_cases/simmaps_TTBB Dataset {6, 1, 1, 85, 2, 128}
/full/spectra/50_4300_50/map_cases/simmaps_TTEE Dataset {6, 1, 1, 85, 2, 128}
/full/spectra/50_4300_50/map_cases/simmaps_TTEEBB_noise Dataset {6, 1, 1, 85, 2, 512}
/full/spectra/50_4300_50/thetas_map Dataset {6}
/full/spectra/50_4300_50/w_bl Dataset {6, 1, 1, 85, 4250}
/meta                    Group
...
/meta/null_split         Dataset {21}
/meta/spectra            Dataset {6}
/meta/sub_spectra        Dataset {2}
...
/null/spectra            Group
/null/spectra/100_4300_100 Group
/null/spectra/100_4300_100/map_cases Group
/null/spectra/100_4300_100/map_cases/realmaps Dataset {6, 21, 3, 42, 2, 1}
/null/spectra/100_4300_100/map_cases/simmaps_TTBB Dataset {6, 21, 3, 42, 2, 128}
/null/spectra/100_4300_100/map_cases/simmaps_TTEE Dataset {6, 21, 3, 42, 2, 128}
/null/spectra/100_4300_100/map_cases/simmaps_TTEEBB_noise Dataset {6, 21, 3, 42, 2, 512}
/null/spectra/100_4300_100/w_bl Dataset {6, 21, 3, 42, 4200}
/null/spectra/50_4300_50 Group
/null/spectra/50_4300_50/map_cases Group
/null/spectra/50_4300_50/map_cases/realmaps Dataset {6, 21, 3, 85, 2, 1}
/null/spectra/50_4300_50/map_cases/simmaps_TTBB Dataset {6, 21, 3, 85, 2, 128}
/null/spectra/50_4300_50/map_cases/simmaps_TTEE Dataset {6, 21, 3, 85, 2, 128}
/null/spectra/50_4300_50/map_cases/simmaps_TTEEBB_noise Dataset {6, 21, 3, 85, 2, 512}
/null/spectra/50_4300_50/w_bl Dataset {6, 21, 3, 85, 4250}

Data Release—Real maps

It is released at NERSC in /global/cfs/cdirs/polar/www/khcheung/largepatch_null/high_1_0_2/.

It contains all the maps created from the real data (but none of the simulated data.) For example, to find the final coadded maps, it is located in /global/cfs/cdirs/polar/www/khcheung/largepatch_null/high_1_0_2/coadd_full/null_realmaps/coadd/realmap.hdf5 at NERSC.

Specifications

$> h5ls -r /global/cfs/cdirs/polar/www/khcheung/largepatch_null/high_1_0_2/coadd_full/null_realmaps/coadd/realmap.hdf5
/                        Group
/d0                      Dataset {1351, 1351}
/d4i                     Dataset {1351, 1351}
/d4r                     Dataset {1351, 1351}
/w0                      Dataset {1351, 1351}
/w4                      Dataset {1351, 1351}

They are all 2D-array, with w indicating the total weights, and d indicating the coadded signal. 0 is the 0-f component, 4 is the 4-f component, r is the real part, i is the imaginary part.

For example,

\[\begin{aligned} I &= \frac{d_0}{w_0} \\ Q &= \frac{d_4^r}{w_4} \\ U &= \frac{d_4^i}{w_4} \\ \end{aligned}\]

These maps are projected using the OLEA projection, as requested from the high-\(\ell\) \(EE\) pipeline. An example implementation of the projector is in https://github.com/ickc/TAIL/blob/3b4e49d6d5a4/tail/pointsource.py#L10-L22.

Also, for null-maps, there may be additional HDF5 group level with the name of respective null-split, which would be self-explanatory.