Skip to main content

35 Simulating crosstalk systematics in the time-domain

Keywords

LiteBIRD, Crosstalk, Systematics, Time-ordered data, Simulations, High-performance computing, Research software

35 Simulating crosstalk systematics in the time-domain

In this section, we will focus on how to simulate crosstalk systematics from time domain simulations.

35.1 Crosstalk as a computational problem

Fundamentally, crosstalk simulation is a matrix multiplication problem. So let us take a look at in the computational details first.

The software of choice in our simulation is Time Ordered Astrophysics Scalable Tools (TOAST), a software framework simulating and processing TODs in CMB observations and performing data compression into maps. It is designed with High-Performance Computing (HPC) in mind and is endorsed by NERSC, and for example, is used in the processing of the Planck satellite mission data.

TOAST data distribution
Figure 35.1: TOAST data distribution1

Needless to say, TOAST is a massively parallel application. For example, fig. 35.1 shows the data distribution across MPI groups and ranks. The main takeaway here is that per group, TODs from different detectors are distributed across different ranks, where each rank may hold data from a different total number of detectors. Say if there are a total of 10 detectors and 4 ranks, some ranks may hold data from 2 or 3 detectors.

Our task is to apply a matrix multiplication on all detectors under this data distribution pattern. And from the hardware simulation using SQUID parameters in chapter 33, the crosstalk matrix is dense2. We should note that block-diagonal structure can be broken down into separate dense matrix operations, as we can loop over each block (typically a wafer) and perform the dense matrix multiplication there.

Since we are having a massively parallel application, and the fact that TOAST is already memory limited for other operations such as the Madam maker, the goal is to minimize the amount of MPI communications needed and the memory overhead in computing the intermediate results.

35.2 OpCrosstalk

Each detector owner, the rank which holds the detector TOD locally, eventually needs the TODs from all other detectors and hence all other ranks within a group, as per detector this is just a weighted sum of all detectors. Because of this, we have to loop over each detector, corresponding to a row in the crosstalk matrix.

Per row, to minimize the MPI communication, a weighted sum is first performed within a local rank who owns a couple of detectors already, and after that, a single MPI-Reduce is called to perform the global sum, which costs around \(O(\log n_\text{rank})\) communication time.

Since we loop over the number of detectors, the total communication cost is \(O(n_\text{det} \log n_\text{rank})\).

To minimize memory use, note that as someone once said, “the Python garbage collector is garbage.” We use an internal facility from TOAST to create new arrays. While it has a Numpy array interface, it is allocated in C++ directly where we can manually free them to avoid memory overhead.

With this, \((2 n_\text{local dets} + 1) \times n_\text{samples}\) memory is needed, comparing to \(n_\text{local dets} \times n_\text{samples}\) originally needed, this requires double memory as each rank needs to hold the intermediate results during the whole matrix multiplication process.

However, with an implementation detail in TOAST that typically \(n_\text{obs}\) observations are performed in a run, such as \(n_\text{obs} = 365\) for a year of satellite mission, this doubling of memory requirements only occur within each observation, so the actual memory increase is negligible.

Currently, technological limit places \(n_\text{det}\) per SQUID to be around ~100. And since \(n_\text{rank} \leq n_\text{det}\), the computational aspect appears to be well under control. An example run below with \(n_\text{det} \sim 60\) takes a negligible amount of compute time compared to the total duration with other expensive operations such as the Madam mapmaker.

OpCrosstalk is also general purpose and is not tied to a particular CMB experiment, and will benefit any future CMB experiments that use TOAST such as SO and CMB-S4. The pull request has been merged to TOAST and is available in https://github.com/hpc4cmb/toast/pull/380.

Footnotes

  1. Kisner, “TOAST Workshop”.↩︎

  2. While the \(n\)-th diagonal in the crosstalk matrix is roughly \(n\)-th order, later we will see that the first few leading order effects can be approximately canceling. Hence, Without Loss of Generality (WLOG), it should be treated as dense and can only be treated as sparse when justified by simulations.↩︎