NPIPE Introduction

The NPIPE data processing pipeline is the first pipeline capable of reducing raw time-ordered data from both Planck instruments, LFI and HFI, into calibrated frequency maps. Unlike the instrument-specific pipelines that were developed to run at the Data Processing Centres and that interface the local databases, NPIPE can be installed and run on any machine that runs a Linux-like operating system. The input and output file format is based on the flexible image transport system (FITS).

NPIPE combines the most powerful aspects of the instrument-specific pipelines and introduces some new innovations. Here is a list of the main differences between the 2018 (PR3) release and NPIPE (PR4) processing.

• A hybrid calibration scheme is used to work around a degeneracy in the Planck calibration process. The 30- and 353-GHz data are first calibrated along with the polarized sky, much like the way that HFI is calibrated in PR3. The PR3 LFI calibration scheme is mimicked and the intermediate CMB frequencies (44–217 GHz) are calibrated, using an approximation that the sky polarization can be captured with templates derived from the other (foreground) frequency channels. The calibration for the CMB frequencies is thus based on temperature-only sky and polarized foreground templates. The use of this polarization prior significantly reduces the uncertainty in gain and hence the large-scale polarization systematics, but also requires the determination of a measurable and correctable transfer function that affects the large-scale CMB polarization.
• The Planck repointing manoeuvre data are included in all of our processing, leading to a roughly 8% increase in total integration time .
• The Solar dipole is retained in the frequency maps for component separation.
• The pointing solution includes the latest star-camera distortion corrections that were developed for Herschel data processing.
• All LFI radiometers are corrected for 1-Hz housekeeping spikes, rather than just the 44-GHz ones.
• More 4-K cooler lines are fit, and the global signal estimate is used for signal removal.
• The LFI sky–load differencing is modified to include a low-pass filter that reduces the injection of uncorrelated noise into the differenced signal.
• A new signal estimate is used during HFI glitch detection and removal.
• The HFI glitch flags are extended during transfer function deconvolution, which avoids leaking constrained-realization power from the gaps into the surrounding data. This leads to less small-scale noise and lower noise correlations between half-rings.
• More HFI transfer-function harmonics are fit for, to better address the odd-even survey differences.
• Only the seasonally varying part of zodiacal emission is subtracted, in order not to bias the dust component in the maps.
• Polarization-corrected, single-detector, and single-horn temperature maps are provided at all frequencies.
• The HFI polarization angles and efficiencies are corrected, to address polarized signal-like residuals in single-detector maps.
• The single-detector maps are binned from 217 to 857GHz at N_side = 4096, to fully sample the narrow beam.
• Detector-set maps are made for cross-correlation analysis and systematics-level studies that are processed in a fully independent way.
• A consistent, low-resolution data set is provided, with estimates of pixel-pixel noise covariance across all Planck frequencies.
• Signal distortions are fit for, to address second-order ADC nonlinearity (ADCNL) in the HFI CMB channels from 100 to 217GHz.
• The HFI approach of correcting for bandpass mismatch in the time domain is extended to the LFI data.
• All data are destriped with Madam, using extremely short 167-ms baselines.
• Calibration is performed with apodized masks that allow most of the sky to be accessed, while down-weighting Galactic regions where even small issues in the signal model can cause problems with calibration.

---

In what follows we quote verbatim from A&A 643, A42 (2020) Conclusions section:

We summarize here what we consider the benefits of the NPIPE processing and products, and also give several cautionary comments. Advantages of the NPIPE release:

• reduced levels of noise and systematics at all angular scales;
• improved consistency across frequencies, particularly in polarization;
• more Monte Carlo realizations of simulated data, and better agreement between the simulated maps and flight data;
• availability of single-detector temperature maps from 100 to 857 GHz, with resolution of Nside = 4096 at 217 GHz and above;
• absence of certain HFI analysis artefacts, in particular “zebra” stripes and CO-template pixel boundaries;
• one publicly-released pipeline to process LFI and HFI data, accompanied with the release of the raw timestreams for future analysis and improvement.

The following are some cautionary comments about the NPIPE release.

• Quantitative use of the NPIPE CMB polarization data on large angular scales (l < 20) requires accounting for the non-negligible transfer function. This will often necessitate processing the large (36TB) body of NPIPE simulations to determine the degree of signal suppression in the cosmological observables of concern, at considerable computational expense.
• At the time of writing, the NPIPE data products have been extensively tested and validated through to the map level. With the exception of the demonstration cases of foreground separation and the determination of τ given in this paper, extraction of the full range of science results represented in the PR2 and PR3 releases remains for the future. While ongoing tests of cosmological parameter solutions indicate no disparity with the Planck 2018 results, presentation of work on this subject is deferred to a future publication. Although unlikely, it is possible that future analysis beyond what is presented here will uncover unidentified issues that impact the estimation of cosmological observables from NPIPE data.
• The component separation and sky model described in Sect.7 use only Planck data, and do not include external data at lower frequencies that help break degeneracies between synchrotron, free-free, and AME, as was done in PR2. We leave this for a future study. For now, the limitations of the low- frequency foreground model should be recognized

The NPIPE time-ordered-data processing can be divided into two major parts.

1. Preprocessing includes all of the data-processing steps needed to prepare the data for calibration. Preprocessing modules only require data for one detector at a time.
2. Reprocessing consists of calibration and systematics corrections that rely on full frequency data. Corrections are based on mismatch between individual detectors and the frequency average.

Reprocessing also contains the mapmaking step that is repeated after each iteration of TOD processing. NPIPE uses the Madam generalized destriper code to find a maximum likelihood map in the presence of 1/f noise fluctuations.

Time-ordered data resulting from both steps are written to disk and made available on the Planck Legacy Archive.

NPIPE was developed and operated using the computing resources hosted by the U.S. National Energy Research Scientific Computing Center (NERSC). Due to volume constrains, only a subset of all simulated NPIPE maps is available on the Planck Legacy Archive. Researchers intending to study the full simulation set are encouraged to apply for an account at NERSC. The full NPIPE release is available to registered users in the CMB project area at

 /global/cfs/cdirs/cmb/data/planck2020