Difference between revisions of "The LFI DPC"

From Planck Legacy Archive Wiki
Jump to: navigation, search
 
(30 intermediate revisions by 5 users not shown)
Line 1: Line 1:
[[Category:Data processing]]
+
<!--
 +
---------
 +
<br>
 +
<span style="font-size:150%">'''Contents of this chapter'''</span>
  
== Overview ==
+
<br>
The LFI DPC is organized into different "Levels":
+
; [[The_LFI_DPC#overview|Overview]]
  
* Level-1, which scope is principally to analyze the data in a daily base and trasforma the telemetry packets in to timelines in enginnering value.
+
; [[Pre-processing LFI| Pre-processing]]: ''[[Pre-processing LFI#Overview|Overview]] • [[Pre-processing LFI#Scientific telemetry|Scientific telemetry]] • [[Pre-processing LFI#From packets to raw TOIs|From packets to raw TOIs]] • [[Pre-processing LFI#On-board time reconstruction|On-board time reconstruction]] • [[Pre-processing LFI#Housekeeping telemetry handling|Housekeeping telemetry handling]] • [[Pre-processing LFI#Auxiliary data handling|Auxiliary data handling]]''
 +
; [[TOI_processing LFI| TOI processing]]: ''[[TOI_processing LFI#Overview|Overview]] • [[TOI_processing LFI#ADC Correction|ADC Correction]] • [[TOI_processing LFI#Spikes Removal|Spikes Removal]] • [[TOI_processing LFI#Gaps Filling|Gaps Filling]] • [[TOI_processing LFI#Gain Modulation Factor|Gain Modulation Factor]] • [[TOI_processing LFI#Diode Combination|Diode Combination]] • [[TOI_processing LFI#Planet Flagging|Planet Flagging]] • [[TOI_processing LFI#Photometric Calibration|Photometric Calibration]] • [[TOI_processing LFI#Noise|Noise]] • [[TOI_processing LFI#References|References]] ''
 +
; [[Pointing LFI|Pointing]]: ''[[Pointing LFI#Detector Pointing|Detector Pointing]]''
  
* Level-2, start with the output of the Level-1, apply the differetiation, combine the diode and remove at timeline level the systematics that has been identified. It is too responsible to compute the instrument attitude and then compute the detector pointing. Scientific timelines are then calibrated in K_CMB unit and are the input to the MapMaker. Maps at frequency leveland varios combination are the principal output .
+
; [[Map-making| Map-Making and photometric calibration]]: ''[[Map-making#Introduction|Introduction]] • [[Map-making#Photometric_calibration|Photometric calibration]] • [[Map-making#Building_of_Maps|Building of Maps]] • [[Map-making#Noise_properties|Noise properties]] •  [[Map-making#Zodi_correction|Zodi correction]] • [[Map-making#Far_Sidelobe_Correction|Far Sidelobe Correction]] • [[Map-making#CO_Correction|CO Correction]] • [[Map-making#Map_validation|Map validation]]''
  
* Level-3, is a COMMON level between the DPC , it takes as input the maps produced at Level-2 and then transforms these maps at specific frequencies into more scientific products, like catalogues, maps and spectra of astrophysical components.
+
; [[Spectral response| Spectral response]]: TBW
 +
; [[HFI-Validation| Internal overall validation]]: ''[[HFI-Validation#Expected_systematics_and_tests_(bottom-up approach)|Expected systematics and tests (bottom-up approach)]] • [[HFI-Validation#Generic_approach_to_systematics|Generic approach to systematics]] • [[HFI-Validation#HFI_simulations|HFI simulations]] • [[HFI-Validation#Simulations_versus_data|Simulations versus data]] • [[HFI-Validation#Systematics_Impact_Estimates|Systematics Impact Estimates]]''
 +
; [[Power spectra| Power spectra]]: TBW
 +
; [[Summary| Summary of HFI data characteristics]]
  
* Level-S, is the COMMON simulation software used to validate all the algorithms used in the process.
+
---------
 +
-->
  
 +
<span id="overview" style="font-size:200%">'''Overview'''</span>
  
The description of the pipeline applied for the ''"Early Planck results"'' can be found at the following link http://www.aanda.org/index.php?option=com_article&access=standard&Itemid=129&url=/articles/aa/abs/2011/12/aa16484-11/aa16484-11.html
+
LFI DPC processing is organized into several different levels, numbered 1 to 4 and S. In brief, Level 1 has the purpose of analysing the data received from the satellite on a daily basis, transforming the telemetry packets into timelines containing engineering values and feeding the results to the DPC database. Level 2 uses the output of Level 1, transforming raw TOI into calibrated timelines with all the known systematic effects removed; those timelines are then used in the mapmaking process to create all desired map combinations. Level 3 uses the Level 2 output of both HFI and LFI DPCs to derive astrophysical results such as source catalogues, CMB maps, and foreground maps.  Level S is the common simulation pipeline used to validate the results as well as to test any algorithm before its introduction into the official pipeline. Finally, Level 4 merely acts to reformat, document, and deliver the products to the final archive (PLA).
  
  
data is
+
<span id="overview" style="font-size:150%">'''Level 1: From packets to TOI '''</span>
  
== Level 1: From packets to TOI ==
+
Level 1 takes input from the Mission Operation Centre's (MOC's) Data Distribution System (DDS), decompresses the raw data, and outputs time-ordered information for Level 2. The inputs to Level 1 are the telemetry (T/M) and auxiliary data as they are released by the MOC. Level 1 uses T/M data for performing a routine analysis of the S/C and P/L
 +
with the aim of monitoring the overall health of the payload and detecting possible anomalies.  Level 1 also includes a quick-look analysis of the science T/M, to monitor the operation of the observation plan and to verify the behaviour of the instrument. Additional tasks of Level 1 relate to its role for instrument control and as the DPC interface with the MOC. Level-1 processing is described in detail in the [[Pre-processing_LFI|pre-processing]] section.
  
The first processing level of the LFI DPC is the so called Level 1. The source data of the Level 1 software includes:
 
* raw housekeeping telemetry packets retrieved from different satellite subsystems: the LFI instrument, the Sorption cooler, the HFI instrument and the Central Data Management Unit (CDMU).
 
* the LFI raw scientific telemetry
 
* Additional auxiliary data provided by the MOC and the Flight dynamics:
 
** The Attitude History File (AHF)
 
** Time correlation data (time correlation coefficients and time couples)
 
** The Sorption cooler out of limit data
 
  
Scope of the Level-1 is to  
+
<span id="overview" style="font-size:150%">'''Level 2: From TOI to maps'''</span>
  
Only a subset of the raw housekeeping telemetry packets is daily processed and converted into TOIs, i.e. those relevant to the LFI DQR production and the estimation of the LFI instrument systematic effects.
+
The DPC Level 2 has many tasks. The first one is the creation of differenced data. Level 1 stores measurements made from both the sky and the 4-K reference load.  These have to be properly combined to produce differenced data, greatly reducing the impact of 1/<i>f</i> noise. This is done via the computation of the so-called gain modulation factor “<i>R</i>,” which is derived by taking the ratio of the mean signals from both sky and load. After differenced data are produced, the next step is the removal of known systematic effects followed by photometric calibration, where "calibrated" means essentially that the TOD are in physical units instead of engineering units.  The next major task is the production of frequency maps calibrated and free from systematic effects (which is a complex task and involves several sub-pipelines). Level 2 processing is described in detail in the [[TOI processing_LFI|TOI processing]] section.
 +
 
 +
 
 +
<span id="overview" style="font-size:150%">'''Level 3: From maps to components'''</span>
 +
 
 +
The aim of Level 3 is to transform the frequency maps produced by both instruments into preliminary maps of the underlying astrophysical components, including the CMB, by means of pipeline processing, as well as to provide other data sets including catalogues of astrophysical sources (compact source lists, extended source maps and catalogues, a description of global or statistical properties, etc.). Data from both HFI and LFI are analysed jointly to reach the final results. Level-3 processing is described in detail in the [[L3_LFI|Power Spectra]] and [[HFI/LFI joint data processing|HFI/LFI joint data processing]] sections.
 +
 
 +
 
 +
<span id="overview" style="font-size:150%">'''Level S: Common HFI/LFI simulation software'''</span>
 +
 +
Level S is the so-called "Simulation Level" software suite common to both consortia, which, given a sky model (generated by the Planck sky model, PSM), detector pointing, and beams, generates a model of the sky and hence of the power reaching each detector.  It can also provide a simplified description of quantities such as the noise. It is further described in the [[HFI/LFI joint data processing|HFI/LFI joint data processing]] section.
 +
 
 +
 
 +
<span id="overview" style="font-size:150%">'''LFI DPC infrastructure'''</span>
 +
 
 +
The LFI DPC provides centralized hardware and software infrastructure to a large number of geographically distributed institutions participating in the Planck mission. Briefly, the data are interfaced to a database, where only meta-information is stored. This allows for very high flexibility to enable modification of the products that are to be delivered.
 +
 
 +
 
 +
<span id="overview" style="font-size:150%">'''Published paper'''</span>
 +
 
 +
A description of the pipeline applied to the ''Early Planck results'' can be found in {{PlanckPapers|planck2013-p02b}}, while instrument performance is reported in {{PlanckPapers|planck2013-p02a}}.
 +
 
 +
The description of the pipeline applied to the ''2015 Planck release'' can be found in {{PlanckPapers|planck2014-a03||Planck-2015-A03}}, systematic effects are detailed in {{PlanckPapers|planck2014-a04||Planck-2015-A04}}, beams and window functions in {{PlanckPapers|planck2014-a05||Planck-2015-A05}}, and calibration in {{PlanckPapers|planck2014-a06||Planck-2015-A06}}. Finally the LFI mapmaking procedure is described in {{PlanckPapers|planck2014-a07||Planck-2015-A07}}.
 +
 
 +
Final description of the pipeline applied to the ''2018 Planck release'' can be found in {{PlanckPapers|planck2016-l02}}.
 +
 
 +
==References==
 +
<references />
 +
 
 +
[[Category:LFI data processing|000]]

Latest revision as of 15:32, 4 July 2018


Overview

LFI DPC processing is organized into several different levels, numbered 1 to 4 and S. In brief, Level 1 has the purpose of analysing the data received from the satellite on a daily basis, transforming the telemetry packets into timelines containing engineering values and feeding the results to the DPC database. Level 2 uses the output of Level 1, transforming raw TOI into calibrated timelines with all the known systematic effects removed; those timelines are then used in the mapmaking process to create all desired map combinations. Level 3 uses the Level 2 output of both HFI and LFI DPCs to derive astrophysical results such as source catalogues, CMB maps, and foreground maps. Level S is the common simulation pipeline used to validate the results as well as to test any algorithm before its introduction into the official pipeline. Finally, Level 4 merely acts to reformat, document, and deliver the products to the final archive (PLA).


Level 1: From packets to TOI

Level 1 takes input from the Mission Operation Centre's (MOC's) Data Distribution System (DDS), decompresses the raw data, and outputs time-ordered information for Level 2. The inputs to Level 1 are the telemetry (T/M) and auxiliary data as they are released by the MOC. Level 1 uses T/M data for performing a routine analysis of the S/C and P/L with the aim of monitoring the overall health of the payload and detecting possible anomalies. Level 1 also includes a quick-look analysis of the science T/M, to monitor the operation of the observation plan and to verify the behaviour of the instrument. Additional tasks of Level 1 relate to its role for instrument control and as the DPC interface with the MOC. Level-1 processing is described in detail in the pre-processing section.


Level 2: From TOI to maps

The DPC Level 2 has many tasks. The first one is the creation of differenced data. Level 1 stores measurements made from both the sky and the 4-K reference load. These have to be properly combined to produce differenced data, greatly reducing the impact of 1/f noise. This is done via the computation of the so-called gain modulation factor “R,” which is derived by taking the ratio of the mean signals from both sky and load. After differenced data are produced, the next step is the removal of known systematic effects followed by photometric calibration, where "calibrated" means essentially that the TOD are in physical units instead of engineering units. The next major task is the production of frequency maps calibrated and free from systematic effects (which is a complex task and involves several sub-pipelines). Level 2 processing is described in detail in the TOI processing section.


Level 3: From maps to components

The aim of Level 3 is to transform the frequency maps produced by both instruments into preliminary maps of the underlying astrophysical components, including the CMB, by means of pipeline processing, as well as to provide other data sets including catalogues of astrophysical sources (compact source lists, extended source maps and catalogues, a description of global or statistical properties, etc.). Data from both HFI and LFI are analysed jointly to reach the final results. Level-3 processing is described in detail in the Power Spectra and HFI/LFI joint data processing sections.


Level S: Common HFI/LFI simulation software

Level S is the so-called "Simulation Level" software suite common to both consortia, which, given a sky model (generated by the Planck sky model, PSM), detector pointing, and beams, generates a model of the sky and hence of the power reaching each detector. It can also provide a simplified description of quantities such as the noise. It is further described in the HFI/LFI joint data processing section.


LFI DPC infrastructure

The LFI DPC provides centralized hardware and software infrastructure to a large number of geographically distributed institutions participating in the Planck mission. Briefly, the data are interfaced to a database, where only meta-information is stored. This allows for very high flexibility to enable modification of the products that are to be delivered.


Published paper

A description of the pipeline applied to the Early Planck results can be found in Planck-2013-V[1], while instrument performance is reported in Planck-2013-III[2].

The description of the pipeline applied to the 2015 Planck release can be found in Planck-2015-A03[3], systematic effects are detailed in Planck-2015-A04[4], beams and window functions in Planck-2015-A05[5], and calibration in Planck-2015-A06[6]. Finally the LFI mapmaking procedure is described in Planck-2015-A07[7].

Final description of the pipeline applied to the 2018 Planck release can be found in Planck-2020-A2[8].

References[edit]

  1. Planck 2013 results. V. LFI Calibration, Planck Collaboration, 2014, A&A, 571, A5.
  2. Planck 2013 results. III. Low Frequency Instrument systematic uncertainties, Planck Collaboration, 2014, A&A, 571, A3.
  3. Planck 2015 results. II. LFI processing, Planck Collaboration, 2016, A&A, 594, A2.
  4. Planck 2015 results. III. LFI systematics, Planck Collaboration, 2016, A&A, 594, A3.
  5. Planck 2015 results. IV. LFI beams and window functions, Planck Collaboration, 2016, A&A, 594, A4.
  6. Planck 2015 results. V. LFI calibration, Planck Collaboration, 2016, A&A, 594, A5.
  7. Planck 2015 results. VI. LFI mapmaking, Planck Collaboration, 2016, A&A, 594, A6.
  8. Planck 2018 results. II. Low Frequency Instrument data processing, Planck Collaboration, 2020, A&A, 641, A2.

(Planck) Low Frequency Instrument

Data Processing Center

(Planck) High Frequency Instrument

Cosmic Microwave background

Planck Legacy Archive

[ESA's] Mission Operation Center [Darmstadt, Germany]

MOC's Data Distribution System

Spacecraft

Payload

Planck Sky Model