The HFI DPC
The first stage of HFI(Planck) High Frequency Instrument data processing is performed on-board in order to generate the telemetry, as described in the Data compression section. On the ground, the HFI(Planck) High Frequency Instrument DPCData Processing Center has been organized into different "Levels": 1, 2, 3, 4, and "S". In brief, during operations, L1 feeds the data base, resulting in time-ordered information (TOI) objects. L2 is the core of the processing, which turns TOIs into clean calibrated sky maps. L3 transforms these maps at specific frequencies into more scientific products, like catalogues, maps, and spectra of astrophysical components. L3 can rely on simulation provided by the LS, while L4 refers to delivering the DPCData Processing Center products to ESAEuropean Space Agency. This processing relies on dedicated software and hardware infrastructures developed pre-launch.
The data processing applied for the "Early Planck results" series of publications was described in Planck-Early-VI. The Planck-2013-VI and its companion papers provide the reference for the processing done for the 2013 data release.
Level 1: building the reference data base during flight operations
(L1): consists in receiving the telemetry and ancillary data files and ingesting them into the DPCData Processing Center data base. This involves decompressing, in some cases changing data formats, computing the time of individual data samples from the time of the compression slices, but otherwise no processing proper. Other steps are:
- science, housekeeping, and ancillary data ingestion;
- timing and pointing interpolation.
This is further described in the Pre-processing section.
Level 2: converting temporal information into clean calibrated maps
(L2): this is where the data are processed from timelines into maps. The main processing steps are:
- timeline (or Time-Ordered Information = TOI) processing, which includes conversion from ADUs to engineering units (volts), demodulation, deglitching, conversion from engineering to physical units (watts), removal of known systematic effects (non-linearities, 4-K lines, jumps, ring flagging), removal of the instrumental signature (time transfer function), and temporal noise estimation (see section TOI_processing);
- pointing and beam construction of each detector (see sections Detector pointing and Beams);
- mapmaking and photometric calibration[LFI meaning]: absolute calibration refers to the 0th order calibration for each channel, 1 single number, while the relative calibration refers to the component of the calibration that varies pointing period by pointing period., projecting the TOIs onto all-sky maps, etc (see section Map-making);
- characterization/validation through angular power spectra (see section PowerSpectra);
- overal HFI(Planck) High Frequency Instrument data validation, through difference tests, comparison to detailed simulations, etc. (see section HFI-Validation).
The resulting data characteristics are given in the Summary section.
Level 3: basic analyses of (Level 2) sky temperature maps
(L3): This is where the data in the form of frequency maps are converted to catalogues and full-sky astrophysical component maps. Much of this is done in common with the LFI(Planck) Low Frequency Instrument DPCData Processing Center and is further described in the HFI/LFI common sections.
Level S : a common HFI(Planck) High Frequency Instrument/LFI(Planck) Low Frequency Instrument simulation software
Level S is the so-called "Simulation Level" software suite, common to both consortia, which, given a sky model (generated by the Planck sky model, PSMPlanck Sky Model), detectors pointing and beams, generates the power falling on each detector. It can also provide a simplified description of the noise. It is further described in the HFI(Planck) High Frequency Instrument/LFI(Planck) Low Frequency Instrument common section. HFI(Planck) High Frequency Instrument specific developments (configuration control and MC bench, specific effects like 4-K lines, glitches, ADCanalog to digital converter nonlinearity, etc.) are described in the HFI data validation section.
HFI(Planck) High Frequency Instrument DPCData Processing Center Infrastructures
The HFI(Planck) High Frequency Instrument Data Processing Centre can be thought of as a centralized backbone, providing hardware and software infrastructure to a relatively large number of geographically distributed groups of developers and other research groups in the HFI(Planck) High Frequency Instrument and LFI(Planck) Low Frequency Instrument core teams. An overview was given in Planck-Early-VI. Particular tasks include:
- code and configuration management;
- data management;
- instrument model (IMO) data base;
- data flow management;
- Planck early results. VI. The High Frequency Instrument data processing, Planck HFI Core Team, A&A, 536, A6, (2011).
- Planck 2013 results. VI. High Frequency Instrument Data Processing, Planck Collaboration, 2014, A&A, 571, A6