Difference between revisions of "The LFI DPC"

From Planck PLA 2015 Wiki
Jump to: navigation, search
 
(14 intermediate revisions by 2 users not shown)
Line 1: Line 1:
 +
<!--
 
---------
 
---------
 
<br>
 
<br>
Line 6: Line 7:
 
; [[The_LFI_DPC#overview|Overview]]
 
; [[The_LFI_DPC#overview|Overview]]
  
; [[Pre-processing LFI| Pre-processing ]]: ''[[Pre-processing LFI#Overview|Overview]] • [[Pre-processing#Telemetry_data|Telemetry data]] • [[Pre-processing#Pointing_data|Pointing data]] • [[Pre-processing#Orbit_data|Orbit data]] • [[Pre-processing#Time_correlation_data|Time correlation data]]''
+
; [[Pre-processing LFI| Pre-processing]]: ''[[Pre-processing LFI#Overview|Overview]] • [[Pre-processing LFI#Scientific telemetry|Scientific telemetry]] • [[Pre-processing LFI#From packets to raw TOIs|From packets to raw TOIs]] • [[Pre-processing LFI#On-board time reconstruction|On-board time reconstruction]] • [[Pre-processing LFI#Housekeeping telemetry handling|Housekeeping telemetry handling]] • [[Pre-processing LFI#Auxiliary data handling|Auxiliary data handling]]''
; [[TOI_processing| TOI processing]]: ''[[TOI_processing#Overview|Overview]] • [[TOI_processing#Input_TOI|Input TOI]] • [[TOI_processing#General_Pipeline_Structure|General Pipeline Structure]] • [[TOI_processing#Output_TOIs_and_products|Output TOIs and products]] • [[TOI_processing#Examples_of_clean_TOIs|Examples of clean TOIs]] • [[TOI_processing#Trends_in_the_output_processing_variables|Trends in the output processing variables]] • [[TOI_processing#Flag_description|Flag description]] ''
+
; [[TOI_processing LFI| TOI processing]]: ''[[TOI_processing LFI#Overview|Overview]] • [[TOI_processing LFI#ADC Correction|ADC Correction]] • [[TOI_processing LFI#Spikes Removal|Spikes Removal]] • [[TOI_processing LFI#Gaps Filling|Gaps Filling]] • [[TOI_processing LFI#Gain Modulation Factor|Gain Modulation Factor]] • [[TOI_processing LFI#Diode Combination|Diode Combination]] • [[TOI_processing LFI#Planet Flagging|Planet Flagging]] [[TOI_processing LFI#Photometric Calibration|Photometric Calibration]] [[TOI_processing LFI#Noise|Noise]] • [[TOI_processing LFI#References|References]] ''
; [[Pointing&Beams| Pointing&Beams]]: ''[[Pointing&Beams#Detector_Pointing|Detector Pointing]] • [[Pointing&Beams#Scanning_Beams|Scanning Beams]] [[Pointing&Beams#Effective_Beams|Effective Beams]]''
+
; [[Pointing LFI|Pointing]]: ''[[Pointing LFI#Detector Pointing|Detector Pointing]]''
 +
 
 
; [[Map-making| Map-Making and photometric calibration]]: ''[[Map-making#Introduction|Introduction]] • [[Map-making#Photometric_calibration|Photometric calibration]] • [[Map-making#Building_of_Maps|Building of Maps]] • [[Map-making#Noise_properties|Noise properties]] •  [[Map-making#Zodi_correction|Zodi correction]] • [[Map-making#Far_Sidelobe_Correction|Far Sidelobe Correction]] • [[Map-making#CO_Correction|CO Correction]] • [[Map-making#Map_validation|Map validation]]''
 
; [[Map-making| Map-Making and photometric calibration]]: ''[[Map-making#Introduction|Introduction]] • [[Map-making#Photometric_calibration|Photometric calibration]] • [[Map-making#Building_of_Maps|Building of Maps]] • [[Map-making#Noise_properties|Noise properties]] •  [[Map-making#Zodi_correction|Zodi correction]] • [[Map-making#Far_Sidelobe_Correction|Far Sidelobe Correction]] • [[Map-making#CO_Correction|CO Correction]] • [[Map-making#Map_validation|Map validation]]''
 +
 
; [[Spectral response| Spectral response]]: TBW
 
; [[Spectral response| Spectral response]]: TBW
 
; [[HFI-Validation| Internal overall validation]]: ''[[HFI-Validation#Expected_systematics_and_tests_(bottom-up approach)|Expected systematics and tests (bottom-up approach)]] • [[HFI-Validation#Generic_approach_to_systematics|Generic approach to systematics]] • [[HFI-Validation#HFI_simulations|HFI simulations]] • [[HFI-Validation#Simulations_versus_data|Simulations versus data]] • [[HFI-Validation#Systematics_Impact_Estimates|Systematics Impact Estimates]]''  
 
; [[HFI-Validation| Internal overall validation]]: ''[[HFI-Validation#Expected_systematics_and_tests_(bottom-up approach)|Expected systematics and tests (bottom-up approach)]] • [[HFI-Validation#Generic_approach_to_systematics|Generic approach to systematics]] • [[HFI-Validation#HFI_simulations|HFI simulations]] • [[HFI-Validation#Simulations_versus_data|Simulations versus data]] • [[HFI-Validation#Systematics_Impact_Estimates|Systematics Impact Estimates]]''  
Line 16: Line 19:
  
 
---------
 
---------
 +
-->
  
 
<span id="overview" style="font-size:200%">'''Overview'''</span>
 
<span id="overview" style="font-size:200%">'''Overview'''</span>
  
The LFI DPC processing is organized into different "Levels": 1,2,3,4 and S. In brief the Level-1 has the scope to analyze the data in a daily base, transform the telemetry packets in to timelines containing engineering values and feed it in the DPC database. Level-2 use the output of the Level-1 transforming raw TOI in to calibrated timelines with all the know systematic sources removed, those timelines are then used in the mapmaking process to create all the possible maps combinations. Level-3 use the Level-2 output of both DPC to derive astrophysical results like Catalogue, CMB map, foreground etc…. Level-S is the common simulation pipeline used to validate the results and any algorithm before its introduction into the official pipeline. Finally the Level-4 just act as a collector pointing used to reformat, document and deliver the products to the final archive.
+
LFI DPC processing is organized into several different levels, numbered 1 to 4 and S. In brief, Level 1 has the purpose of analyzing the data received from the satellite on a daily basis, transforming the telemetry packets into timelines containing engineering values and feeding the results to the DPC database. Level 2 uses the output of Level1, transforming raw TOI into calibrated timelines with all the known systematic effects removed; those timelines are then used in the mapmaking process to create all desired map combinations. Level 3 uses the Level 2 output of both HFI and LFI DPCs to derive astrophysical results like source catalogues, CMB maps, foreground maps, etc. Level S is the common simulation pipeline used to validate the results and any algorithm before its introduction into the official pipeline. Finally Level 4 merely acts to reformat, document and deliver the products to the final archive (PLA).
 +
 
  
<span style="font-size:150%">''' Level 1: From packets to TOI ''' </span>
+
<span id="overview" style="font-size:150%">'''From packets to TOI '''</span>
  
Level 1 takes input from the MOC’s Data Distribution System (DDS), decompresses the raw data, and outputs Time Ordered Information for Level 2. This will be done by using software. The input to Level 1 are the telemetry (TM) and auxiliary data as they are released by the MOC (Mission Operation Centre). Level 1 will use TM data for performing a routine analysis (RTA) of the S/C and P/L with the aim of monitoring the overall health of the payload and detecting possible anomalies, and performing a quick-look data analysis of the science TM to monitor the operation of the observation plan and to verify the behavior of the instrument.  
+
Level 1 takes input from the MOC’s Data Distribution System (DDS), decompresses the raw data, and outputs Time Ordered Information for Level 2. The inputs to Level 1 are the telemetry (TM) and auxiliary data as they are released by the MOC (Mission Operation Centre). Level 1 uses TM data for performing a routine analysis of the S/C and P/L  
Additional tasks of Level 1 relate to its role of instrument control and as the DPC interface with the MOC. Level-1 processing is described in detail in the [[Pre-processing_LFI | Pre-processing]] section
+
with the aim of monitoring the overall health of the payload and detecting possible anomalies.  Level 1 also includes a quick-look analysis of the science TM to monitor the operation of the observation plan and to verify the behavior of the instrument. Additional tasks of Level 1 relate to its role of instrument control and as the DPC interface with the MOC. Level-1 processing is described in detail in the Pre-processing section.
  
<span style="font-size:150%">'''Level 2: From TOI to Maps''' </span>
 
  
The DPC Level 2 has many tasks. The first one is the creation of differenced data. Level 1 stores data from both Sky and Load. These two have to be properly combined to produce differenced data therefore reducing the impact of 1/f noise. This is done via the computation of the so-called gain modulation factor “R” which is derived taking the ratio of the mean signals from both Sky and Load. After differenced data are produced, the next step are the removal of know systematic effects and then the photometric calibration where calibrated means essentially that TOD are in physical units instead of engineering units; the following major task is the production of frequency maps calibrated and free from systematic effects (which is a complex task and involves several sub-pipelines). Level-2 processing is described in detail in the [[TOI processing_LFI|TOI processing]] section
 
  
<span style="font-size:150%">'''Level 3: From Maps to Component''' </span>
+
<span id="overview" style="font-size:150%">'''Level 2: From TOI to Maps'''</span>
  
The aim of Level 3 is to transform the frequency maps produced by both instruments into preliminary maps of the underlying astrophysical components by means of pipeline processing and to provide other data sets including description of astrophysical sources (final catalogue of point sources, extended source maps and catalogues, description of global or statistical properties etc …). Data from both HFI and LFI are analyzed jointly to reach the final expected result. Level-3 processing is described in detail in the [[L3_LFI]] and [[HFI/LFI joint data processing|HFI/LFI joint data processing]] section
 
  
<span style="font-size:150%">'''Level S : A common HFI/LFI simulation software''' </span>
+
The DPC Level 2 has many tasks. The first one is the creation of differenced data. Level 1 stores measurements made from both the sky and the 4K reference load.  These two have to be properly combined to produce differenced data, greatly reducing the impact of 1/f noise. This is done via the computation of the so-called gain modulation factor “R” which is derived taking the ratio of the mean signals from both sky and load. After differenced data are produced, the next step is the removal of known systematic effects followed by photometric calibration, where ‘calibrated’ means essentially that the TOD are in physical units instead of engineering units.  The next major task is the production of frequency maps calibrated and free from systematic effects (which is a complex task and involves several sub-pipelines). Level 2 processing is described in detail in the [[TOI processing_LFI|TOI processing]] section.
  
Level S is the so-called "Simulation Level"  software suite common to both consortia, which, given a sky model (generated by the Planck sky model, <tt>PSM</tt>), detectors pointing and beams, generates the infalling power on each detector. It can also provide a simplified description of eg. the noise. It is further described in the HFI/LFI [[ HL | common section]].
 
  
<span style="font-size:150%">''' LFI DPC Infrastructures''' </span>
+
<span id="overview" style="font-size:150%">'''Level 3: From Maps to Component'''</span>
The LFI DPC provide a centralized hardware and software infrastructure to a large number of geographically distributed institution participating to the Planck mission. In few word the data are interfaced to a database where only meta-information are stored. This allow very high flexibility to eventually modify the product to be delivered.
+
 
 +
The aim of Level 3 is to transform the frequency maps produced by both instruments into preliminary maps of the underlying astrophysical components, including the CMB, by means of pipeline processing, as well as to provide other data sets including catalogues of astrophysical sources (compact source lists, extended source maps and catalogues, description of global or statistical properties, etc …). Data from both HFI and LFI are analyzed jointly to reach the final expected results. Level-3 processing is described in detail in the [[L3_LFI|Power Spectra]] and [[HFI/LFI joint data processing|HFI/LFI joint data processing]] section.
 +
 
 +
 
 +
<span id="overview" style="font-size:150%">'''Level S : A common HFI/LFI simulation software'''</span>
 
   
 
   
<span style="font-size:150%">'''Published Paper ''' </span>
 
  
The description of the pipeline applied to the ''"Early Planck results"'' can be found at the following link
+
Level S is the so-called "Simulation Level"  software suite common to both consortia, which, given a sky model (generated by the Planck sky model, <tt>PSM</tt>), detector pointing and beams, generates a model of the sky and hence of the power reaching each detector.  It can also provide a simplified description of quantities such as the noise. It is further described in the [[HFI/LFI joint data processing|HFI/LFI joint data processing]].
[http://www.aanda.org/index.php?option=com_article&access=standard&Itemid=129&url=/articles/aa/abs/2011/12/aa16484-11/aa16484-11.html Planck early results. V. The Low Frequency Instrument data processing] and instrument performances are reported [http://www.aanda.org/index.php?option=com_article&access=standard&Itemid=129&url=/articles/aa/abs/2011/12/aa16480-11/aa16480-11.html Planck early results. III. First assessment of the Low Frequency Instrument in-flight performance]
+
 
 +
 
 +
<span id="overview" style="font-size:150%">'''LFI DPC Infrastructures'''</span>
 +
 
 +
The LFI DPC provides a centralized hardware and software infrastructure to a large number of geographically distributed institutions participating to the Planck mission. Briefly, the data are interfaced to a database where only meta-information is stored. This allows very high flexibility to eventually modify the product to be delivered
 +
 
 +
 
 +
<span id="overview" style="font-size:150%">'''Published Paper'''</span>
 +
 
 +
The description of the pipeline applied to the ''Early Planck results'' can be found in {{PlanckPapers|planck2013-p02b}}, and instrument performances are reported in {{PlanckPapers|planck2013-p02a}}.
 +
 
 +
The description of the pipeline applied to the ''2015 Planck release'' can be found in {{PlanckPapers|planck2014-a03||Planck-2015-A03}}, systematic effects are detailed in {{PlanckPapers|planck2014-a04||Planck-2015-A04}}, beams and window functions in {{PlanckPapers|planck2014-a05||Planck-2015-A05}} and calibration in {{PlanckPapers|planck2014-a06||Planck-2015-A06}}. Finally the LFI map making is described in {{PlanckPapers|planck2014-a07||Planck-2015-A07}}.
 +
 
 +
==References==
 +
<references />
  
[[Category:Data processing|005]]
+
[[Category:LFI data processing|000]]

Latest revision as of 20:27, 4 February 2015


Overview

LFI DPC processing is organized into several different levels, numbered 1 to 4 and S. In brief, Level 1 has the purpose of analyzing the data received from the satellite on a daily basis, transforming the telemetry packets into timelines containing engineering values and feeding the results to the DPC database. Level 2 uses the output of Level1, transforming raw TOI into calibrated timelines with all the known systematic effects removed; those timelines are then used in the mapmaking process to create all desired map combinations. Level 3 uses the Level 2 output of both HFI and LFI DPCs to derive astrophysical results like source catalogues, CMB maps, foreground maps, etc. Level S is the common simulation pipeline used to validate the results and any algorithm before its introduction into the official pipeline. Finally Level 4 merely acts to reformat, document and deliver the products to the final archive (PLA).


From packets to TOI

Level 1 takes input from the MOC’s Data Distribution System (DDS), decompresses the raw data, and outputs Time Ordered Information for Level 2. The inputs to Level 1 are the telemetry (TM) and auxiliary data as they are released by the MOC (Mission Operation Centre). Level 1 uses TM data for performing a routine analysis of the S/C and P/L with the aim of monitoring the overall health of the payload and detecting possible anomalies. Level 1 also includes a quick-look analysis of the science TM to monitor the operation of the observation plan and to verify the behavior of the instrument. Additional tasks of Level 1 relate to its role of instrument control and as the DPC interface with the MOC. Level-1 processing is described in detail in the Pre-processing section.


Level 2: From TOI to Maps


The DPC Level 2 has many tasks. The first one is the creation of differenced data. Level 1 stores measurements made from both the sky and the 4K reference load. These two have to be properly combined to produce differenced data, greatly reducing the impact of 1/f noise. This is done via the computation of the so-called gain modulation factor “R” which is derived taking the ratio of the mean signals from both sky and load. After differenced data are produced, the next step is the removal of known systematic effects followed by photometric calibration, where ‘calibrated’ means essentially that the TOD are in physical units instead of engineering units. The next major task is the production of frequency maps calibrated and free from systematic effects (which is a complex task and involves several sub-pipelines). Level 2 processing is described in detail in the TOI processing section.


Level 3: From Maps to Component

The aim of Level 3 is to transform the frequency maps produced by both instruments into preliminary maps of the underlying astrophysical components, including the CMB, by means of pipeline processing, as well as to provide other data sets including catalogues of astrophysical sources (compact source lists, extended source maps and catalogues, description of global or statistical properties, etc …). Data from both HFI and LFI are analyzed jointly to reach the final expected results. Level-3 processing is described in detail in the Power Spectra and HFI/LFI joint data processing section.


Level S : A common HFI/LFI simulation software


Level S is the so-called "Simulation Level" software suite common to both consortia, which, given a sky model (generated by the Planck sky model, PSM), detector pointing and beams, generates a model of the sky and hence of the power reaching each detector. It can also provide a simplified description of quantities such as the noise. It is further described in the HFI/LFI joint data processing.


LFI DPC Infrastructures

The LFI DPC provides a centralized hardware and software infrastructure to a large number of geographically distributed institutions participating to the Planck mission. Briefly, the data are interfaced to a database where only meta-information is stored. This allows very high flexibility to eventually modify the product to be delivered


Published Paper

The description of the pipeline applied to the Early Planck results can be found in Planck-2013-V[1], and instrument performances are reported in Planck-2013-III[2].

The description of the pipeline applied to the 2015 Planck release can be found in Planck-2015-A03[3], systematic effects are detailed in Planck-2015-A04[4], beams and window functions in Planck-2015-A05[5] and calibration in Planck-2015-A06[6]. Finally the LFI map making is described in Planck-2015-A07[7].

References[edit]

  1. Planck 2013 results. V. LFI Calibration, Planck Collaboration, 2014, A&A, 571, A5
  2. Planck 2013 results. III. Low Frequency Instrument systematic uncertainties, Planck Collaboration, 2014, A&A, 571, A3
  3. Planck 2015 results. II. LFI processing, Planck Collaboration, 2016, A&A, 594, A2.
  4. Planck 2015 results. III. LFI systematics, Planck Collaboration, 2016, A&A, 594, A3.
  5. Planck 2015 results. IV. LFI beams and window functions, Planck Collaboration, 2016, A&A, 594, A4.
  6. Planck 2015 results. V. LFI calibration, Planck Collaboration, 2016, A&A, 594, A5.
  7. Planck 2015 results. VI. LFI mapmaking, Planck Collaboration, 2016, A&A, 594, A6.

(Planck) Low Frequency Instrument

Data Processing Center

(Planck) High Frequency Instrument

Cosmic Microwave background

Planck Legacy Archive

[ESA's] Mission Operation Center [Darmstadt, Germany]

MOC's Data Distribution System

Spacecraft

Payload

[LFI meaning]: absolute calibration refers to the 0th order calibration for each channel, 1 single number, while the relative calibration refers to the component of the calibration that varies pointing period by pointing period.

Planck Sky Model