Difference between revisions of "TOI processing"

From Planck Legacy Archive Wiki
Jump to: navigation, search
m
 
(189 intermediate revisions by 14 users not shown)
Line 1: Line 1:
This Section kept with a low profile as this Planck release does not contain TOIs.
+
==Overview==
  
==Overview==
+
We describe here how the TOIs are processed in order to be used for map production. They are first corrected for non-linear effects in the ADCs, then processed with a pipeline whose general features are given in the HFI Data Processing paper {{PlanckPapers|planck2013-p03}} for PR1 and  {{PlanckPapers|planck2014-a08}} for PR2. For PR3 (2018 release), the main change with respect to 2015 has been in the selection of data: the last 955 pointings (or rings) have not been used because they have been found to be affected by thermal fluctuations (see {{PlanckPapers|planck2016-l03}}).
 +
 
 +
Detailed explanations of each stage of the TOI processing are given in various papers, referenced in the diagram below (taken from {{PlanckPapers|planck2016-l03}}).
 +
 
 +
[[File:Screen Shot 2018-07-02 at 16.08.52.png|thumb|500px|center]]
 +
 
 +
Here we give complementary explanations in some detail. The TOI of each bolometer is processed independently of the other bolometers, so as to keep the noise properties as uncorrelated as possible. The pipeline processing involves modifying the TOI itself for the conversion to absorbed power and the correction of glitch tails. It also adds a flag TOI that masks the TOI samples that are not to be projected onto maps for various reasons.
  
We describe here the how the TOIs are processed in order to be used for map production. We do not repeat the general features of the pipeline which are given in the HFI Data Processing article (REF). Here we give complementary explanations on some details. The TOI of each bolometer is processed independently of the other bolometers, so as to keep the noise properties as uncorrelated as possible. The processing involves modifying the TOI itself for what concerns the conversion to absorbed power and the correction of glitch tails. It also adds a flag TOI that masks the TOI samples that are not to be projected on maps for various reasons.
+
The input TOI consists of the AC-modulated voltage output read out of each bolometer. The input has previously been decompressed, and converted from internal digital units to voltages via a constant factor. The TOI has a regular sampling at the acquisition frequency of <i>f</i><sub>acq</sub>=(180.373700&plusmn;0.000050)Hz. There are almost no missing data samples in the TOIs, except for a few hundred samples of 545 and 857 GHz TOIs, which were lost in the on-board compression due to saturation on the Galactic centre crossings.
  
 +
== ADC non-linearity correction ==
  
== Input TOI ==
+
On-board, the modulated bolometer outputs are sampled 20x during each half-cycle, and the 20 samples are then averaged into the individual samples that are compressed and transmitted to ground. The ADC non-linearity affects the signals before the averaging, and a complicated process is necessary to determine and remove the systematic effects induced by that non-linearity, a process that is further complicated by the 40-Hz parasitics associated with the 4-K cooler drive electronics.
  
The input TOI consists in the AC modulated voltage output of the readout of each bolometer. The input has previously been decompressed, and converted from internal digital units to voltage via a constant factor. The TOI has a regular sampling at the acquisition frequency of facq=180.373700+-0.000050 Hz. There are almost no missing data in the TOIs, except for few hundred samples of 545 and 857GHz TOIs which are lost in the on-board compression due to saturation on the Galactic Center crossings.  
+
The principles of the correction are detailed in {{PlanckPapers|planck2014-a08}}. The correction cannot be done using only the TOI values, but requires ancillary data from ground-based tests and additional tests done during the LFI extension of the mission, when the dilution cooler no longer operated. For this reason, we do not describe here any details that are not already in the paper, since the users should not expect to be able to improve this part of the processing.
  
== General Pipeline Structure ==
+
The result of this processing is a corrected TOI, always in ADUs, which is then fed to the TOI processing pipeline described below.
  
The next figure shows how the initial ccTOI is transformed and how flags are produced:
+
== General pipeline structure ==
  
[[File:HFI_4_4_2_ToiProc_ExplS.png|thumb|600px|A schematic of the TOI processing pipeline]]
+
The following figure shows how the initial TOI is transformed and how flags are produced.
  
 +
[[File:HFI_4_4_2_ToiProc_ExplS.png|thumb|500px|center|A schematic of the TOI processing pipeline.]]
  
 
== Output TOIs and products ==
 
== Output TOIs and products ==
  
A TOI of clean calibrated samples (ccTOI) and a combined flag TOI (fTOI) are the outputs of the processing. The ccTOI is calibrated so as to represent the instantaneous power absorbed by the detector.  It is worth mentioning how the ccTOI is changed with respect to the input TOI, beyond the harmless constant conversion factor from voltage to absorbed power. The demodulation stage allows to get the demodulated bolometer voltage. The non-linearity correction is a second-order polynomial correction based on the physical but static bolometer model. In order to avoid too much masking after glitches, a glitch tail is subtracted after an occurrence of a glitch in the TOI. Finally, the temporal response of the bolometer is deconvolved. This affects mostly the high-temporal frequency part of the TOI, although a small but significant low frequency tail is corrected too. Although flagged samples are not projected, their value influences the valid samples somehow. Hence interpolation procedures introduce some indirect modifications of the TOI.
+
A TOI of clean calibrated samples and a combined flag TOI are the outputs of the processing. The clean calibrated TOI is calibrated so as to represent the instantaneous power absorbed by the detector, up to a constant (which will be determined by the mapmaking destriper).  It is worth mentioning how the clean calibrated TOI is changed with respect to the input TOI, beyond the basic constant conversion factor from voltage to absorbed power. The demodulation stage allows one to obtain the demodulated bolometer voltage. The nonlinearity correction is a second-order polynomial correction based on the physical but static bolometer model. In order to avoid too much masking after glitches, a glitch tail is subtracted after an occurrence of a glitch in the TOI. The 4-K cooler lines are removed at a series of nine single temporal frequencies. Finally, the temporal response of the bolometer is deconvolved. This affects mostly the high-temporal frequency part of the TOI, although a small but significant low frequency tail (the long time response) is corrected too. Although flagged samples are not projected, their value still influences the valid samples. Hence interpolation procedures introduce some indirect modifications of the TOI.
The flag TOI is a combination a dozen flags with an OR logic. Only unflagged data are projected. The exhaustive list of flags is given here: CompressionError, NoData, SSO, UnstablePointing, Glitch, BoloPlateFluctuation, RTS, Jump, PSBab. A complete qualification of the data is obtained at the ring level. If the TOI shows an anomalous behaviour during the duration of a ring, this ring is discarded from projection.
+
The flag TOI is a combination of flags with an OR logic. Only unflagged data are projected. The exhaustive list of flags is: CompressionError; SSO; UnstablePointing; Glitch; BoloPlateFluctuation; RTS; Jump; and PSBab. A complete qualification of the data is obtained at the ring level. If the TOI shows any anomalous behaviour on a given ring, this ring is discarded from projection.  A special TOI is also produced as an input to the beam analysis for Mars, Jupiter, and Saturn.
 +
 
 +
== Examples of clean TOIs ==
 +
<center>
 +
<gallery heights=200px widths=300px perrow=3>
 +
File:10_143_5_PBR_10800RING_LFER4_JC.jpg|Phase-binned signal (PBR) for six consecutive rings from 4000 to 4005.
 +
File:10_143_5_rcircles_LFER4_JC.jpg| Valid signal (up) and noise (bottom) for two consecutive circles of ring 4000.
 +
File:10_143_5_fft_LFER4_JC.jpg| Fourier power spectral density of the first 30 minutes of ring 4005 with signal (top) before (pink) and after (black) deconvolution and with noise (bottom) also shown.
 +
File:14_545_1_PBR_10800RING_LFER4_JC.jpg|Phase-binned signal (PBR) for six consecutive rings from 4000 to 4005.
 +
File:14_545_1_rcircles_LFER4_JC.jpg| Valid signal (up) and noise (bottom) for two consecutive circles of ring 4000.
 +
File:14_545_1_fft_LFER4_JC.jpg| Fourier power spectral density of the first 30 minutes of ring 4005, with signal (top) before (pink) and after (black) deconvolution and with noise (bottom) also shown.
 +
</gallery>
 +
</center>
 +
Samples of PBR, TOIs, and PSDs of all detectors are shown in [[Media:check_ring.pdf|this file]].
  
 
== Trends in the output processing variables ==
 
== Trends in the output processing variables ==
  
Here we intend to show the trend of the systematic effects that are dealt with in the TOI processing
+
Here we show trends in the systematic effects that are dealt with in the TOI processing. The full impact of each of them is analysed in the [[HFI-Validation]] section.
 +
 
 +
=== ADC baseline ===
 +
The following figure shows the ADC baseline that is used prior to demodulation (a constant offset is removed for clarity). This baseline is obtained by smoothing on an hour timescale by block-averaging the undemodulated TOI on unflagged samples.
 +
 
 +
[[File:baseline.jpg|thumb|500px|center|ADC baseline for all bolometers.]]
  
 
=== Glitch statistics ===
 
=== Glitch statistics ===
 +
The glitch rate per channel is shown in the figure below. For details, see {{PlanckPapers|planck2013-p03e}}
  
=== 4K cooler lines variability ===
+
[[File:figIntermPaperGR.jpg|thumb|500px|center|Glitch rate evolution.]]
 +
 
 +
The percentage of flagged data (mostly due to cosmic rays) at the ring level is shown in these examples. No smoothing was applied and only valid rings are shown.
 +
<center>
 +
<gallery heights=300px widths=300px perrow=2>
 +
File:group10_143_5.jpg | 143GHz bolometers.
 +
File:group14_545_1.jpg | 545GHz bolometers.
 +
</gallery>
 +
'''Percentage of flagged data'''
 +
</center>
 +
The complete set of plots is attached [[Media:PercentUnvalid2.pdf|here]].
 +
 
 +
=== Thermal template for decorrelation ===
 +
[[File:T90.jpg|thumb|center|500px|Thermal template used for the decorrelation from bolometer plate temperature fluctuations.]]
 +
 
 +
A simple linear decorrelation is performed using the two "dark" bolometers as a proxy of the bolometer plate temperature. Coupling coefficients were measured during the CPV phase.
 +
 
 +
=== 4-K cooler lines variability ===
 +
The amplitude of the nine 4-K cooler lines at 10, 20, 30, 40, 50, 60, 70, 80, and 17Hz is shown (in aW) for two bolometers in the following figures. The trend is smoothed over 31 ring values after having discarded measurements done at a ring that is discarded for all bolometers.
 +
<center>
 +
<gallery heights=300px widths=300px perrow=2>
 +
File:10_143_5_4Klines.jpg | 143_5 bolometer.
 +
File:14_545_1_4Klines.jpg | 545_1 bolometer.
 +
</gallery>
 +
'''Amplitude of the nine 4-K cooler lines'''
 +
</center>
 +
 
 +
The 4-K cooler line coefficients of all bolometers are shown in [[Media:Lines4K.pdf|this file]].
 +
 
 +
The 4-K cooler lines project onto the maps only for a limited fraction of rings, the so-called "resonant rings." This is graphically shown in the following figure. For each ring (fixed pointing period), the spin rate is very stable at about 1rpm. From one ring to another, the spin frequency (shown as diamonds) changes around that value. The sky signal is imprinted at the corresponding spin frequency and its 5400 (60&times;90) harmonics. If one of the nine 4-K cooler lines happens to coincide with one of the spin frequency harmonics (a resonant ring), it will project a sine-wave systematic onto the maps. The horizontal coloured bars show the zone of influence of a particular 4-K line (labelled on the left side of the plot), when folded around 16.666mHz. When the spin frequency hits one of these zones, we have a resonant ring. The 4-K line coefficient is interpolated for this ring and an estimate of the systematic effect is subtracted from the TOI. Resonant rings are different for different 4-K lines. Note the two-level oscillation pattern of the spin frequency, which is due to the satellite attitude control system.
 +
 
 +
[[File:spinfreq1.jpg|thumb|center|500px|Spin frequency and 4-K line zones of influence.]]
 +
 
 +
=== Jump correction ===
 +
A piecewise constant value is removed from the TOI if a jump is detected. An example of a jump is shown in the following figure:
 +
 
 +
[[File:jump_exe.png|thumb|500px|center|An example of a jump seen on the noise TOI.]]
 +
 
 +
The number of jumps per day (all bolometers included) is shown in this figure:
 +
 
 +
[[File:jumps_per_day.png|thumb|500px|center|Evolution of jump number during the mission.]]
 +
 
 +
The jumps are uncorrelated from bolometer to bolometer. The total number of jumps detected in the nominal and full mission is shown below:
 +
 
 +
[[File:jumps_per_bolometer.png|thumb|500px|center|Number of jumps per bolometer.]]
 +
 
 +
 
 +
=== Trends in noise and signal ===
 +
<center>
 +
<gallery heights=300px widths=600px perrow=2 >
 +
File:10_143_5_smooth_Watt.jpg | 143_5 bolometer.
 +
File:14_545_1_smooth_Watt.jpg | 545_1 bolometer.
 +
</gallery>
 +
<small>'''Signal (top) and noise (bottom) smoothed over 1 minute. All values falling in a discarded ring are not plotted.'''</small>
 +
</center>
 +
 
 +
The smooth TOIs of all detectors are shown in [[Media:check_smooth.pdf|this file]].
  
 
=== Noise stationarity ===
 
=== Noise stationarity ===
 +
This is not the final version but gives a good idea of power spectra at the detector level of noise TOIs. All PSDs can be seen in [[Media:v53_meanSpectra_bySurvey.pdf|this file]].
 +
 +
The standard deviation per ring corrected for ring duration bias is given here, one per bolometer using only valid rings. No smoothing is applied (except a 31-point smoothing for the 545 and 857 GHz channels), but values for rings discarded for all bolometers (see below) are not used. The standard deviation is computed on samples valid for mapmaking that are also not affected by the Galaxy or the point-sources using the usual flags. Two examples are shown below.
 +
 +
<center>
 +
<gallery heights=300px widths=300px perrow=2>
 +
File:stddev_group10_143_5.jpg | 143_5 bolometer.
 +
File:stddev_group_14_545_1.jpg | 545_1 bolometer.
 +
</gallery>
 +
'''Standard deviation per ring'''
 +
</center>
 +
 +
The full series of plots of the Standard deviation of noise TOIs at the ring level is
 +
[[Media:StationarityInROIoutput2.pdf|here.]]
 +
 +
Note the presence for three bolometers of a two-level noise system. No correction has been made for that effect. See one example below.
 +
 +
[[File:23_353_TwoLevel.jpg|thumb|500px|center|An example of a two-level noise system is seen in bolometer 23_353_3a.]]
  
Noise power spectrum
 
  
rms per ring
+
An example of the higher-order statistics that are used to unveil rings affected by RTS problems is below.
  
 +
[[File:HFI_4_4_2_RTSexample13.jpg|thhumb|600px|center|Example of RTS detection.]]
  
[[File:HFI_4_4_2_RTSexample13.jpg|thumb|400px|Example of RTS detection]]
+
=== Discarded rings ===
  
 +
Some rings are discarded (flagged) from further use (e.g., beam-making, and mapmaking) by using ring statistics (see above and {{PlanckPapers|planck2013-p03}}). For each statistic, we compare each ring value to the ring values averaged (RVA) over a large selection of rings (between 3000 and 21700). We also define the modified standard deviation (MSD) of a ring quantity as the standard deviation of that quantity over the rings that deviate by less than five nominal standard deviations. This truncation is necessary to be robust against extremely deviant rings.
  
=== Flag description ===
+
More specifically, a ring is discarded if it matches one of the following criteria:
  
==== Input flags ====
+
* the | mean-median | deviates from the RVA by more than fifteen times the MSD;
 +
* the standard deviation deviates from the RVA by more than -5 times the MSD (all cases corresponding to almost empty rings) or +15 times the MSD;
 +
* the Kolmogorov-Smirnov test deviates from the RVA by more than 15 times the MSD;
 +
* the ring duration is more than 90 min;
 +
* the ring is contaminated by RTS with an amplitude of more than one standard deviation of the noise.
 +
 
 +
This last item concerns a few hundred rings for three bolometers (44_217_6b, 71_217_8a, 74_857_4). Notice that two bolometers are completely discarded for maps, namely 55_545_3 and 70_143_8, which present RTS at all times.
 +
 
 +
For the three first criteria, a visual inspection of the noise TOI at each of the incriminated rings has shown that all such anomalous rings are due to either a drift, a small jump in the TOI trend, or a sudden change of noise level, the origin of which is unknown at present.
 +
 
 +
Once the list of discarded rings per bolometer is produced, a common list of discarded rings can be extracted for all bolometers (by using discarded rings for at least half the bolometers). Such rings correspond to identified phenomena, as can be seen in the following table.
 +
 
 +
Furthermore, an isolated valid ring sandwiched between two common discarded rings becomes discarded as well.
 +
 
 +
Table of "common discarded rings" of the full mission (rings 240-27005).
 +
 
 +
<center>
 +
{| class="wikitable"  align="center" style="text-align:center" border="1" cellpadding="5" cellspacing="0"  width=500px
 +
|- bgcolor="ffdead"
 +
! '''Cause''' !! '''Ring number'''
 +
|-
 +
|align="left"| Manoeuvre || 304 1312 3590 3611 3642 3922 4949 6379 8456 11328
 +
|-
 +
| align="left"| Sorption Cooler switchover || 11149 11150 11151 11152
 +
|-
 +
| align="left"| Rings too long || 440 474 509 544 897 898 3589 13333 14627 14653 
 +
|-
 +
| align="left"| Star tracker switchover || 14628 14654 14842 16407 18210
 +
|-
 +
| align="left"| Massive glitch event || 7665 
 +
|-
 +
| align="left"| Solar flare || 11235  20451 20452 20453 20454 20455 20456
 +
|-
 +
| align="left"| End-Of-Life tests || 24185-24187 24206 24782 24806-24810 24973-24976 25074-25176 25850-25863 26231-26235 26520-26619
 +
|}
 +
</center>
  
These flags are used as inputs to the TOI processing
 
  
- The point-source flag (PSflag):
 
  
An earlier version of HFI point-source catalog is read back into a flag TOIs, at a given frequency. In practice, 5 sigma sources are masked within a radius of 1.3 FWHM
+
The following figure is a summary of the impact of the discarding process <i>for each bolometer</i> (the solid black line is the common discarded ring percentage). The outlier bolometers have some RTS problems, as mentioned above.
  
- the galactic flag (Galflag):
+
<center>
 +
<gallery heights=300 widths=600px perrow=4 caption="">
 +
File:Fraction_discarded_rings_v53.png | Fraction of discarded rings.
 +
File:Fraction_discarded_time_v53.png | Fraction of time due to discarded rings.
 +
</gallery>
 +
</center>
  
An earlier version of HFI maps is thresholded and apodized. The produced masks are read into flag TOIs. The retained threshold corresponds to a sky coverage of respectively 90, 90, 90, 80, 70, 70% at 100,143,217,353,545,857 GHz (TBC????)
+
=== Effective integration time ===
 +
The following figure summarizes the effective integration time per bolometer. For that purpose the number of unflagged samples in non-discarded pointing periods have been used within the nominal mission. The average value is about 335 days of effective integration time.
  
- Solar System Object flag (SSO flag)
+
[[File:HFInominal_IntegrationTime.jpg|thumb|500px|center|Effective integration time per bolometer.]]
  
4 toi, 4map, seen
+
== Flag description ==
beam FWHM = 9,7,5,5,5,5 a 100,143,217,353,545,857 GHz
 
  
TOI : Mars, Jupiter, Saturn flagges a 2,3,3,4,4,4 FWHM
+
==== Input flags ====
  
MAP :  Mars, Jupiter, Saturn flagges a Factor_Source x  NbBeam_freq FWHM
+
The following flags are used as inputs to the TOI processing.
avec
 
Factor_per_Source1 = 1.1 (Mars)
 
Factor_per_Source2 = 2.25 (Jupiter)
 
Factor_per_Source3 = 1.25 (Saturn)
 
NbBeam2Flag_list1 = 2.25 100 GHz
 
NbBeam2Flag_list2 = 4.25
 
NbBeam2Flag_list3 = 4.0
 
NbBeam2Flag_list4 = 5.0
 
NbBeam2Flag_list5 = 6.0 857 GHz
 
  
en plus une petite queue pour prendre en compte la non deconvolution du signal de la planete interpolee par le signal "de fond"
+
; The point-source flag (PSflag) -
largeur = 10 % du diametre du flag principal
+
: An earlier version of HFI point-source catalogue is read back in to flag TOIs, at a given frequency. In practice, 5&sigma; sources are masked within a radius of 1.3 FWHM (which is 9, 7, 5, 5, 5, 5 arcmin at 100, 143, 217, 353, 545, and 857 GHz).
Nb de SN flaggues en plus du flag principal = Factor_Source x  AddSNafter1
 
AddSNafter1 = 10  100 GHz
 
AddSNafter2 = 30
 
AddSNafter3 = 20
 
AddSNafter4 = 20
 
AddSNafter5 = 30
 
AddSNafter6 = 40 857 GHz
 
  
seen :  
+
; The Galactic flag (Galflag) -
 +
: An earlier version of HFI maps is thresholded and apodized. The produced masks are read into flag TOIs. The retained threshold corresponds to a sky coverage of, respectively, 70, 70, 80, 90, 90, and 90% at 100, 143, 217, 353, 545, and 857GHz.
  
 +
; Solar System Object flag -
 +
: For the TOI flag, Mars, Jupiter, and Saturn are flagged up to a radius of <i>N</i><sub>Beam</sub>= 2, 3, 3, 4, 4, and 4 times the fiducial SSO_FWHM, with SSO_FWHM=  9, 7, 5, 5, 5, and 5 arcmin at 100, 143, 217, 353, 545, and 857GHz, respectively.
  
 +
: As an input to the planet mask for maps, Mars, Jupiter, and Saturn are flagged with a radius computed as a coefficient depending on the planet (Factor_per_source) times <i>N</i><sub>Beam</sub> times SSO_FWHM, with Factor_per_source = 1.1, 2.25, and 1.25 for Mars, Jupiter, and Saturn, respectively, and <i>N</i><sub>Beam</sub> = 2.25, 4.25, 4.0, 5.0, 6.0, and 8.0 at 100, 143, 217, 353, 545, and 857GHz. This flag is called SSOflag4map.
  
 +
: A small trailing tail is added to the mask to take into account the non-deconvolution of the planet signal, which has been replaced by background values. The width of this tail is 10% of the main flag diameter. The number of samples which are additionally flagged are Factor_per_Source times AddSNafter with "trail" &times; (Factor_per_source)<sup>3</sup> samples, where trail = 10, 30, 20, 20, 30, and 40 at 100, 143, 217, 353, 545, and 857GHz, respectively.
  
==== Output flags ====
+
: Uranus and Neptune, together with detected asteroids, are masked by HFI. They are masked at the TOI level using an exclusion radius of 1.5 times SSO_FWHM. At 857GHz, 24 asteroids have been detected with HFI, namely 1Ceres, 2Pallas, 3Juno, 4Vesta, 7Iris, 8Flora, 9Metis, 10Hygiea, 11Parthenope, 12Victoria, 13Egeria, 14Irene, 15Eunomia, 16Psyche, 18Melpomene, 19Fortuna, 20Massalia, 29Amphitrite, 41Daphne, 45Eugenia, 52Europa, 88Thisbe, 704Interamnia, and 324Bamberga.
  
with and without point-sources, identical to a and b PSBs.
+
<center>
 +
<gallery heights=200px widths=200px perrow=4 caption="">
 +
File:SSOflag_10_143_5J.png | 143 SWB row, around Jupiter first crossing.
 +
File:SSOflag_10_143_5S.png | 143 SWB row, around Saturn.
 +
File:SSOflag_10_143_5M.png | 143 SWB row, around Mars.
 +
File:SSOflag_10_143_5a.png | 143 SWB row, random field in the ecliptic plane.
 +
File:SSOflag_25_857_1J.png | 545/857 row, around Jupiter first crossing.
 +
File:SSOflag_25_857_1S.png | 545/857 row, around Saturn.
 +
File:SSOflag_25_857_1M.png | 545/857 row, around Mars.
 +
File:SSOflag_25_857_1a.png | 545/857 row, random field in the ecliptic plane.
 +
</gallery>
 +
Local maps showing the SSO flag. The colours correspond to the surveys involved in the nominal mission (green for Survey I, yellow for Survey 2, and red for Survey 3).
 +
</center>
  
 +
=== Output flags ===
  
Flags produced for the map making :
+
A FlagTOIproc is produced by the TOI processing. It marks measurements which are not reliable for any of the following reasons.
    UnstablePointing Flag OR FlagTOIproc OR SSOflag 4 map OR SSOflag seen
+
* Gap (no valid input data), enlarged by one sample on each side. This flags less than 0.00044% (respectively 0.00062%) of the nominal (respectively complete) mission. It is equivalent to less than 3 (respectively 8) minutes of data.
    FlagTOIproc = glitch OR jump OR flag thermal template  
+
* "Glitch" on dark bolometers: as the thermal template used for decorrelation is computed from these bolometer data, chunks of 1 minute-length data are discarded for all bolometers if at least 50% of the data for at least one dark bolometer are flagged during this time. This is efficient for flagging the data around the maximum of thermal events.
If PSB flag glitch = flag glitch A OR flag glitch B
+
* "Glitch" on individual bolometers: samples where the signal from a cosmic ray hit dominates the sky signal at more than 3&sigma; are discarded.
 +
* Jump with 100 samples flagged around the computed position of the jump, to take into account the error on this reconstructed position.
  
Flag used in the destriping: same + point-source flag
+
The flag produced for the map making, called Total_flag, is therefore defined by
 +
* Total_flag = UnstablePointing Flag OR FlagTOIproc OR SSOflag4map OR SSOflag, seen,
 +
where FlagTOIproc =  gap OR flag thermal template OR glitch OR jump
 +
and for PSB bolometers,  FlagTOIproc_AB =  FlagTOIproc_A OR  FlagTOIproc_B.
 +
Note that the Total_flag is then identical for the A and B bolometers of a PSB pair.
  
 +
At the destriping stage, a more restricted flag, called Total_flag_PS, is used. It is defined by
 +
Total_flag_PS = Total_flag OR PS_flag.
  
=== output TOI description ===
+
== References ==
 +
<References />
 +
  
  
  
[[Category:Data processing]]
+
[[Category:HFI data processing|002]]

Latest revision as of 14:18, 2 July 2018

Overview[edit]

We describe here how the TOIs are processed in order to be used for map production. They are first corrected for non-linear effects in the ADCs, then processed with a pipeline whose general features are given in the HFI Data Processing paper Planck-2013-VI[1] for PR1 and Planck-2015-A07[2] for PR2. For PR3 (2018 release), the main change with respect to 2015 has been in the selection of data: the last 955 pointings (or rings) have not been used because they have been found to be affected by thermal fluctuations (see Planck-2020-A3[3]).

Detailed explanations of each stage of the TOI processing are given in various papers, referenced in the diagram below (taken from Planck-2020-A3[3]).

Screen Shot 2018-07-02 at 16.08.52.png

Here we give complementary explanations in some detail. The TOI of each bolometer is processed independently of the other bolometers, so as to keep the noise properties as uncorrelated as possible. The pipeline processing involves modifying the TOI itself for the conversion to absorbed power and the correction of glitch tails. It also adds a flag TOI that masks the TOI samples that are not to be projected onto maps for various reasons.

The input TOI consists of the AC-modulated voltage output read out of each bolometer. The input has previously been decompressed, and converted from internal digital units to voltages via a constant factor. The TOI has a regular sampling at the acquisition frequency of facq=(180.373700±0.000050)Hz. There are almost no missing data samples in the TOIs, except for a few hundred samples of 545 and 857 GHz TOIs, which were lost in the on-board compression due to saturation on the Galactic centre crossings.

ADC non-linearity correction[edit]

On-board, the modulated bolometer outputs are sampled 20x during each half-cycle, and the 20 samples are then averaged into the individual samples that are compressed and transmitted to ground. The ADC non-linearity affects the signals before the averaging, and a complicated process is necessary to determine and remove the systematic effects induced by that non-linearity, a process that is further complicated by the 40-Hz parasitics associated with the 4-K cooler drive electronics.

The principles of the correction are detailed in Planck-2015-A07[2]. The correction cannot be done using only the TOI values, but requires ancillary data from ground-based tests and additional tests done during the LFI extension of the mission, when the dilution cooler no longer operated. For this reason, we do not describe here any details that are not already in the paper, since the users should not expect to be able to improve this part of the processing.

The result of this processing is a corrected TOI, always in ADUs, which is then fed to the TOI processing pipeline described below.

General pipeline structure[edit]

The following figure shows how the initial TOI is transformed and how flags are produced.

A schematic of the TOI processing pipeline.

Output TOIs and products[edit]

A TOI of clean calibrated samples and a combined flag TOI are the outputs of the processing. The clean calibrated TOI is calibrated so as to represent the instantaneous power absorbed by the detector, up to a constant (which will be determined by the mapmaking destriper). It is worth mentioning how the clean calibrated TOI is changed with respect to the input TOI, beyond the basic constant conversion factor from voltage to absorbed power. The demodulation stage allows one to obtain the demodulated bolometer voltage. The nonlinearity correction is a second-order polynomial correction based on the physical but static bolometer model. In order to avoid too much masking after glitches, a glitch tail is subtracted after an occurrence of a glitch in the TOI. The 4-K cooler lines are removed at a series of nine single temporal frequencies. Finally, the temporal response of the bolometer is deconvolved. This affects mostly the high-temporal frequency part of the TOI, although a small but significant low frequency tail (the long time response) is corrected too. Although flagged samples are not projected, their value still influences the valid samples. Hence interpolation procedures introduce some indirect modifications of the TOI. The flag TOI is a combination of flags with an OR logic. Only unflagged data are projected. The exhaustive list of flags is: CompressionError; SSO; UnstablePointing; Glitch; BoloPlateFluctuation; RTS; Jump; and PSBab. A complete qualification of the data is obtained at the ring level. If the TOI shows any anomalous behaviour on a given ring, this ring is discarded from projection. A special TOI is also produced as an input to the beam analysis for Mars, Jupiter, and Saturn.

Examples of clean TOIs[edit]

Samples of PBR, TOIs, and PSDs of all detectors are shown in this file.

Trends in the output processing variables[edit]

Here we show trends in the systematic effects that are dealt with in the TOI processing. The full impact of each of them is analysed in the HFI-Validation section.

ADC baseline[edit]

The following figure shows the ADC baseline that is used prior to demodulation (a constant offset is removed for clarity). This baseline is obtained by smoothing on an hour timescale by block-averaging the undemodulated TOI on unflagged samples.

ADC baseline for all bolometers.

Glitch statistics[edit]

The glitch rate per channel is shown in the figure below. For details, see Planck-2013-X[4]

Glitch rate evolution.

The percentage of flagged data (mostly due to cosmic rays) at the ring level is shown in these examples. No smoothing was applied and only valid rings are shown.

Percentage of flagged data

The complete set of plots is attached here.

Thermal template for decorrelation[edit]

Thermal template used for the decorrelation from bolometer plate temperature fluctuations.

A simple linear decorrelation is performed using the two "dark" bolometers as a proxy of the bolometer plate temperature. Coupling coefficients were measured during the CPV phase.

4-K cooler lines variability[edit]

The amplitude of the nine 4-K cooler lines at 10, 20, 30, 40, 50, 60, 70, 80, and 17Hz is shown (in aW) for two bolometers in the following figures. The trend is smoothed over 31 ring values after having discarded measurements done at a ring that is discarded for all bolometers.

Amplitude of the nine 4-K cooler lines

The 4-K cooler line coefficients of all bolometers are shown in this file.

The 4-K cooler lines project onto the maps only for a limited fraction of rings, the so-called "resonant rings." This is graphically shown in the following figure. For each ring (fixed pointing period), the spin rate is very stable at about 1rpm. From one ring to another, the spin frequency (shown as diamonds) changes around that value. The sky signal is imprinted at the corresponding spin frequency and its 5400 (60×90) harmonics. If one of the nine 4-K cooler lines happens to coincide with one of the spin frequency harmonics (a resonant ring), it will project a sine-wave systematic onto the maps. The horizontal coloured bars show the zone of influence of a particular 4-K line (labelled on the left side of the plot), when folded around 16.666mHz. When the spin frequency hits one of these zones, we have a resonant ring. The 4-K line coefficient is interpolated for this ring and an estimate of the systematic effect is subtracted from the TOI. Resonant rings are different for different 4-K lines. Note the two-level oscillation pattern of the spin frequency, which is due to the satellite attitude control system.

Spin frequency and 4-K line zones of influence.

Jump correction[edit]

A piecewise constant value is removed from the TOI if a jump is detected. An example of a jump is shown in the following figure:

An example of a jump seen on the noise TOI.

The number of jumps per day (all bolometers included) is shown in this figure:

Evolution of jump number during the mission.

The jumps are uncorrelated from bolometer to bolometer. The total number of jumps detected in the nominal and full mission is shown below:

Number of jumps per bolometer.


Trends in noise and signal[edit]

Signal (top) and noise (bottom) smoothed over 1 minute. All values falling in a discarded ring are not plotted.

The smooth TOIs of all detectors are shown in this file.

Noise stationarity[edit]

This is not the final version but gives a good idea of power spectra at the detector level of noise TOIs. All PSDs can be seen in this file.

The standard deviation per ring corrected for ring duration bias is given here, one per bolometer using only valid rings. No smoothing is applied (except a 31-point smoothing for the 545 and 857 GHz channels), but values for rings discarded for all bolometers (see below) are not used. The standard deviation is computed on samples valid for mapmaking that are also not affected by the Galaxy or the point-sources using the usual flags. Two examples are shown below.

Standard deviation per ring

The full series of plots of the Standard deviation of noise TOIs at the ring level is here.

Note the presence for three bolometers of a two-level noise system. No correction has been made for that effect. See one example below.

An example of a two-level noise system is seen in bolometer 23_353_3a.


An example of the higher-order statistics that are used to unveil rings affected by RTS problems is below.

Example of RTS detection.

Discarded rings[edit]

Some rings are discarded (flagged) from further use (e.g., beam-making, and mapmaking) by using ring statistics (see above and Planck-2013-VI[1]). For each statistic, we compare each ring value to the ring values averaged (RVA) over a large selection of rings (between 3000 and 21700). We also define the modified standard deviation (MSD) of a ring quantity as the standard deviation of that quantity over the rings that deviate by less than five nominal standard deviations. This truncation is necessary to be robust against extremely deviant rings.

More specifically, a ring is discarded if it matches one of the following criteria:

  • the | mean-median | deviates from the RVA by more than fifteen times the MSD;
  • the standard deviation deviates from the RVA by more than -5 times the MSD (all cases corresponding to almost empty rings) or +15 times the MSD;
  • the Kolmogorov-Smirnov test deviates from the RVA by more than 15 times the MSD;
  • the ring duration is more than 90 min;
  • the ring is contaminated by RTS with an amplitude of more than one standard deviation of the noise.

This last item concerns a few hundred rings for three bolometers (44_217_6b, 71_217_8a, 74_857_4). Notice that two bolometers are completely discarded for maps, namely 55_545_3 and 70_143_8, which present RTS at all times.

For the three first criteria, a visual inspection of the noise TOI at each of the incriminated rings has shown that all such anomalous rings are due to either a drift, a small jump in the TOI trend, or a sudden change of noise level, the origin of which is unknown at present.

Once the list of discarded rings per bolometer is produced, a common list of discarded rings can be extracted for all bolometers (by using discarded rings for at least half the bolometers). Such rings correspond to identified phenomena, as can be seen in the following table.

Furthermore, an isolated valid ring sandwiched between two common discarded rings becomes discarded as well.

Table of "common discarded rings" of the full mission (rings 240-27005).

Cause Ring number
Manoeuvre 304 1312 3590 3611 3642 3922 4949 6379 8456 11328
Sorption Cooler switchover 11149 11150 11151 11152
Rings too long 440 474 509 544 897 898 3589 13333 14627 14653
Star tracker switchover 14628 14654 14842 16407 18210
Massive glitch event 7665
Solar flare 11235 20451 20452 20453 20454 20455 20456
End-Of-Life tests 24185-24187 24206 24782 24806-24810 24973-24976 25074-25176 25850-25863 26231-26235 26520-26619


The following figure is a summary of the impact of the discarding process for each bolometer (the solid black line is the common discarded ring percentage). The outlier bolometers have some RTS problems, as mentioned above.

Effective integration time[edit]

The following figure summarizes the effective integration time per bolometer. For that purpose the number of unflagged samples in non-discarded pointing periods have been used within the nominal mission. The average value is about 335 days of effective integration time.

Effective integration time per bolometer.

Flag description[edit]

Input flags[edit]

The following flags are used as inputs to the TOI processing.

The point-source flag (PSflag) -
An earlier version of HFI point-source catalogue is read back in to flag TOIs, at a given frequency. In practice, 5σ sources are masked within a radius of 1.3 FWHM (which is 9, 7, 5, 5, 5, 5 arcmin at 100, 143, 217, 353, 545, and 857 GHz).
The Galactic flag (Galflag) -
An earlier version of HFI maps is thresholded and apodized. The produced masks are read into flag TOIs. The retained threshold corresponds to a sky coverage of, respectively, 70, 70, 80, 90, 90, and 90% at 100, 143, 217, 353, 545, and 857GHz.
Solar System Object flag -
For the TOI flag, Mars, Jupiter, and Saturn are flagged up to a radius of NBeam= 2, 3, 3, 4, 4, and 4 times the fiducial SSO_FWHM, with SSO_FWHM= 9, 7, 5, 5, 5, and 5 arcmin at 100, 143, 217, 353, 545, and 857GHz, respectively.
As an input to the planet mask for maps, Mars, Jupiter, and Saturn are flagged with a radius computed as a coefficient depending on the planet (Factor_per_source) times NBeam times SSO_FWHM, with Factor_per_source = 1.1, 2.25, and 1.25 for Mars, Jupiter, and Saturn, respectively, and NBeam = 2.25, 4.25, 4.0, 5.0, 6.0, and 8.0 at 100, 143, 217, 353, 545, and 857GHz. This flag is called SSOflag4map.
A small trailing tail is added to the mask to take into account the non-deconvolution of the planet signal, which has been replaced by background values. The width of this tail is 10% of the main flag diameter. The number of samples which are additionally flagged are Factor_per_Source times AddSNafter with "trail" × (Factor_per_source)3 samples, where trail = 10, 30, 20, 20, 30, and 40 at 100, 143, 217, 353, 545, and 857GHz, respectively.
Uranus and Neptune, together with detected asteroids, are masked by HFI. They are masked at the TOI level using an exclusion radius of 1.5 times SSO_FWHM. At 857GHz, 24 asteroids have been detected with HFI, namely 1Ceres, 2Pallas, 3Juno, 4Vesta, 7Iris, 8Flora, 9Metis, 10Hygiea, 11Parthenope, 12Victoria, 13Egeria, 14Irene, 15Eunomia, 16Psyche, 18Melpomene, 19Fortuna, 20Massalia, 29Amphitrite, 41Daphne, 45Eugenia, 52Europa, 88Thisbe, 704Interamnia, and 324Bamberga.

Local maps showing the SSO flag. The colours correspond to the surveys involved in the nominal mission (green for Survey I, yellow for Survey 2, and red for Survey 3).

Output flags[edit]

A FlagTOIproc is produced by the TOI processing. It marks measurements which are not reliable for any of the following reasons.

  • Gap (no valid input data), enlarged by one sample on each side. This flags less than 0.00044% (respectively 0.00062%) of the nominal (respectively complete) mission. It is equivalent to less than 3 (respectively 8) minutes of data.
  • "Glitch" on dark bolometers: as the thermal template used for decorrelation is computed from these bolometer data, chunks of 1 minute-length data are discarded for all bolometers if at least 50% of the data for at least one dark bolometer are flagged during this time. This is efficient for flagging the data around the maximum of thermal events.
  • "Glitch" on individual bolometers: samples where the signal from a cosmic ray hit dominates the sky signal at more than 3σ are discarded.
  • Jump with 100 samples flagged around the computed position of the jump, to take into account the error on this reconstructed position.

The flag produced for the map making, called Total_flag, is therefore defined by

  • Total_flag = UnstablePointing Flag OR FlagTOIproc OR SSOflag4map OR SSOflag, seen,

where FlagTOIproc = gap OR flag thermal template OR glitch OR jump and for PSB bolometers, FlagTOIproc_AB = FlagTOIproc_A OR FlagTOIproc_B. Note that the Total_flag is then identical for the A and B bolometers of a PSB pair.

At the destriping stage, a more restricted flag, called Total_flag_PS, is used. It is defined by Total_flag_PS = Total_flag OR PS_flag.

References[edit]

(Planck) High Frequency Instrument

analog to digital converter

(Planck) Low Frequency Instrument

Solar System Object

random telegraphic signal

Calibration and Performance Verification

sudden change of the baseline level inside a ring

Full-Width-at-Half-Maximum