Difference between revisions of "TOI processing"

From Planck PLA 2015 Wiki
Jump to: navigation, search
(Input flags)
(Input flags)
Line 208: Line 208:
 
: Uranus and Neptune together with detected asteroids are masked by HFI. They are masked at the TOI level using an exclusion radius of 1.5 SSO_FWHM. At 857 GHz, 24 asteroids have been detected with HFI : 1Ceres, 2Pallas, 3Juno, 4Vesta, 7Iris, 8Flora, 9Metis, 10Hygiea, 11Parthenope, 12Victoria, 13Egeria, 14Irene, 15Eunomia, 16Psyche, 18Melpomene, 19Fortuna, 20Massalia, 29Amphitrite, 41Daphne, 45Eugenia, 52Europa, 88Thisbe, 704Interamnia, 324Bamberga.
 
: Uranus and Neptune together with detected asteroids are masked by HFI. They are masked at the TOI level using an exclusion radius of 1.5 SSO_FWHM. At 857 GHz, 24 asteroids have been detected with HFI : 1Ceres, 2Pallas, 3Juno, 4Vesta, 7Iris, 8Flora, 9Metis, 10Hygiea, 11Parthenope, 12Victoria, 13Egeria, 14Irene, 15Eunomia, 16Psyche, 18Melpomene, 19Fortuna, 20Massalia, 29Amphitrite, 41Daphne, 45Eugenia, 52Europa, 88Thisbe, 704Interamnia, 324Bamberga.
  
<gallery widths=160px perrow=4 caption="Local maps showing the SSO flag. The colors correspond to the surveys involved in the nominal mission (green = Survey I, yellow = Survey II, red = Survey III).">
+
<gallery widths=140px perrow=4 caption="Local maps showing the SSO flag. The colors correspond to the surveys involved in the nominal mission (green = Survey I, yellow = Survey II, red = Survey III).">
 
File:SSOflag_10_143_5J.png | 143 SWB row, around Jupiter 1st crossing
 
File:SSOflag_10_143_5J.png | 143 SWB row, around Jupiter 1st crossing
 
File:SSOflag_10_143_5S.png | 143 SWB row, around Saturn
 
File:SSOflag_10_143_5S.png | 143 SWB row, around Saturn

Revision as of 14:55, 14 March 2013

This Section is kept short as this Planck release does not contain TOIs. The main information in the HFI data processing paper #planck2013-p03 is not duplicated here.

Overview[edit]

We describe here how the TOIs are processed in order to be used for map production. We do not repeat the general features of the pipeline which are given in the HFI Data Processing article #planck2013-p03. Here we give complementary explanations on some details. The TOI of each bolometer is processed independently of the other bolometers, so as to keep the noise properties as uncorrelated as possible. The processing involves modifying the TOI itself for what concerns the conversion to absorbed power and the correction of glitch tails. It also adds a flag TOI that masks the TOI samples that are not to be projected on maps for various reasons.


Acronyms and definitions[edit]

  • 4K lines: EMI/EMC influence of the 4K cooler mechanical motion on the bolometer readout electronics.
  • ADC : analog to digital converter
  • IMO : instrument model
  • Jump : sudden change of the baseline level inside a ring
  • LFER : low frequency excess response
  • PBR : phase binned ring
  • RIMO : reduced IMO
  • Ring : pointing period
  • ROI : ring ordered information
  • RTS : random telegraphic signal
  • SSO : solar system object
  • TOI : time ordered information

Input TOI[edit]

The input TOI consists in the AC modulated voltage output of the readout of each bolometer. The input has previously been decompressed, and converted from internal digital units to voltage via a constant factor. The TOI has a regular sampling at the acquisition frequency of facq=180.373700+-0.000050 Hz. There are almost no missing data in the TOIs, except for few hundred samples of 545 and 857GHz TOIs which are lost in the on-board compression due to saturation on the Galactic Center crossings.

General Pipeline Structure[edit]

The figure on the right shows how the initial TOI is transformed and how flags are produced:

A schematic of the TOI processing pipeline


Output TOIs and products[edit]

A TOI of clean calibrated samples (ccTOI) and a combined flag TOI (fTOI) are the outputs of the processing. The ccTOI is calibrated so as to represent the instantaneous power absorbed by the detector up to a constant (which will be determined by the map-making destriper). It is worth mentioning how the ccTOI is changed with respect to the input TOI, beyond the harmless constant conversion factor from voltage to absorbed power. The demodulation stage allows to get the demodulated bolometer voltage. The non-linearity correction is a second-order polynomial correction based on the physical but static bolometer model. In order to avoid too much masking after glitches, a glitch tail is subtracted after an occurrence of a glitch in the TOI. The 4K cooler lines noise is substituted at a series of 9 single temporal frequencies. Finally, the temporal response of the bolometer is deconvolved. This affects mostly the high-temporal frequency part of the TOI, although a small but significant low frequency (the long time response) tail is corrected too. Although flagged samples are not projected, their value influences the valid samples somehow. Hence interpolation procedures introduce some indirect modifications of the TOI. The flag TOI is a combination a dozen flags with an OR logic. Only unflagged data are projected. The exhaustive list of flags is given here: CompressionError, NoData, SSO, UnstablePointing, Glitch, BoloPlateFluctuation, RTS, Jump, PSBab. A complete qualification of the data is obtained at the ring level. If the TOI shows an anomalous behaviour on a given ring, this ring is discarded from projection. A special production of TOIs is also made as an input to the beam analysis with Mars, Jupiter and Saturn.


Examples of clean TOIs[edit]

Samples of PBR, TOIs, and PSDs of all detectors are shown in this file

Trends in the output processing variables[edit]

Here we intend to show the trend of the systematic effects that are dealt with in the TOI processing. The full impact of each of them is analyzed in HFI-Validation.

ADC baseline[edit]

The following figure shows the ADC baseline which is used prior to demodulation (a constant offset is removed for clarity). This baseline is obtained by smoothing on an hour block average the undemodulated TOI on unflagged samples.

ADC baseline for all bolometers

Glitch statistics[edit]

The glitch rate per channel is shown in this figure. For details, see copap.

Glitch rate evolution

The percentage of flagged data (mostly due to Cosmic Rays) at the ring level is shown in these examples. No smoothing was applied. Only valid rings are shown.

The complete set of plots is here

Thermal template for decorrelation[edit]

Thermal template used for the decorrelation from bolometer plate temperature fluctuations

A simple linear decorrelation is performed using the 2 dark bolometers as a proxy of the bolometer plate temperature. Coupling coefficients were measured during the CPV phase.

4K cooler lines variability[edit]

The amplitude of the nine 4K cooler lines in aW at 10, 20, 30, 40, 50, 60, 70, 80 and 17 Hz is shown for 2 bolometers in the following figures. The trend is smoothed over 31 ring values after having discarded measurements done at a ring which is discarded for all bolometers.

The 4K cooler line coefficients of all bolometers are shown in this file

The 4K cooler lines project onto the maps only for a limited fraction of rings, the so-called resonant rings. This is graphically shown in the following figure. For each ring (stable pointing period), the spin rate is very stable at about 1 rpm. From one ring to another, the spin frequency (shown as diamonds) changes around that value. The sky signal is imprinted at the corresponding spin frequency and its 5400 (60x90) harmonics. If one of the nine 4K cooler lines happens to coincide with one of the spin frequency harmonics (a resonant ring), it will project a sine-wave systematic on the maps. The horizontal coloured bars show the zone of influence of a particular 4K line (labeled on the left side of the plot), when folded around 16.666 mHz. When the spin frequency hits one of these zones, we have a resonant ring. The 4K line coefficient is interpolated for this ring and an estimate of the systematic effect is subtracted from the TOI. Resonant rings are different for different 4K lines. Note the two-level oscillation pattern of the spin frequency is due to the satellite attitude control system.

Spin frequency and 4K line zones of influence

jump correction[edit]

A piecewise constant value is removed to the TOI if a jump is detected. See a jump example in this figure:

An example of a jump seen on the rmsigTOI

The number of jumps per day (all bolometers included) is shown in this figure:

Evolution of jump number during the mission

The jumps are uncorrelated from bolometer to bolometer. The total number of jumps detected in the nominal and full mission is shown here:

Number of jumps per bolometer


Trends in noise and signal[edit]

The smooth TOIs of all detectors are shown in this file

Noise stationarity[edit]

This is not the final version but gives a good idea of power spectra at the detector level of rmsigTOIs. All PSDs can be seen in this file

The standard deviation per ring corrected by ring duration bias is given here, one per bolometer using only valid rings. No smoothing is applied (except a 31-point smoothing for the 545 and 857 GHz channels) but values for rings discarded for all bolometers (see below) are not used. The standard deviation is computed on samples valid for map-making which are also not affected by the Galaxy or the point-sources using the usual flags. Two examples are given here.


The full series of plots is here: Standard deviation of rmsig TOIs at the ring level

Note the presence for 3 bolometers of a two-level noise system. No correction can be done for that effect. See one example here:

An example of two-level noise system is seen in bolometer 23_353_3a


An example of the higher order statistics which are used to unveil rings affected by RTS problems.

Example of RTS detection

Discarded rings[edit]

Some rings are discarded (flagged) from further use (beam making, map making) by using ring statistics (see above and #planck2013-p03). For each statistic, we compare each ring value to the ring values averaged (RVA) over a large selection of rings (between 3000 and 21700). We also define the modified standard deviation (MSD) of a ring quantity as the standard deviation of that quantity over the rings that deviate by less than five nominal standard deviations. This truncation is necessary to be robust against extreme deviant rings.

More specifically, a ring is discarded if it matches one of the following criteria:

  • the | mean-median | deviates from the RVA by more than fifteen times the MSD.
  • the standard deviation deviates from the RVA by more than -5 times the MSD (all cases corresponding to almost empty rings) or +15 times the MSD.
  • the Kolmogorov-Smirnov test deviates from the RVA by more than 15 times the MSD.
  • the ring duration is more than 90 min.
  • the ring is contaminated by RTS with an amplitude of more than one standard deviation of the noise. It concerns a few hundreds of rings for 3 bolometers (44_217_6b, 71_217_8a, 74_857_4). Notice that two bolometers are completely discarded for maps: 55_545_3 and 70_143_8, which present RTS at all time.

For the three first criteria, a visual inspection of the rmsigTOI at each of the incriminated rings has shown that all these anomalous rings are due to either a drift, a small jump in the TOI trend or a sudden change of noise level, the origin of which is unknown at present.

Once the list of discarded rings per bolometer is produced, a common list of discarded rings can be extracted for all bolometers (by using discarded rings for at least half the bolometers). Such rings correspond to identified phenomena, as can be seen on the following table.

Furthermore, an isolated valid ring stuck between two common discarded rings becomes discarded as well.

Table of common discarded rings of the nominal mission (rings 240-14723).

Cause ring number
manoeuvre 304 1312 3590 3611 3642 3922 4949 6379 8456 11328
Sorption Cooler Switchover 11149 11150 11151 11152
Rings too long 440 474 509 544 897 898 3589 13333 14627 14653
Star tracker switchover 14628 14654
Massive Glitch Event 7665
Solar Flare 11235


The following figure is a summary of the impact of the discarding process for each bolometer (the solid black line is the common discarded ring percentage). The outlier bolometers have some RTS problems as mentionned above.

Fraction of discarded rings Fraction of time due to discarded rings

Effective integration time[edit]

The following figure summarizes the effective integration time per bolometer. For that purpose the number of unflagged samples in non-discarded pointing periods have been used within the nominal mission. The average value is of about 335 days of effective integration time.

Effective integration time per bolometer

Flag description[edit]

Input flags[edit]

These flags are used as inputs to the TOI processing

The point-source flag (PSflag)
An earlier version of HFI point-source catalog is read back into a flag TOIs, at a given frequency. In practice, 5 sigma sources are masked within a radius of 1.3 FWHM (9, 7, 5, 5, 5, 5 arcmin at 100,143,217,353,545,857 GHz) TBC.
the galactic flag (Galflag
An earlier version of HFI maps is thresholded and apodized. The produced masks are read into flag TOIs. The retained threshold corresponds to a sky coverage of respectively 70, 70, 80, 90, 90, 90% at 100,143,217,353,545,857 GHz.
Solar System Object flag
For the TOI flag, Mars, Jupiter, Saturn are flagged up to a radius of NbBeam= 2,3,3,4,4,4 times the fiducial SSO_FWHM with SSO_FWHM= 9, 7, 5, 5, 5, 5 arcmin at 100,143,217,353,545,857 GHz.
As an input to planet mask for maps, Mars, Jupiter, Saturn are flagged with a radius computed as a coefficient depending on the planet (Factor_per_source) times NbBeam times SSO_FWHM, with Factor_per_source = 1.1, 2.25, 1.25 for Mars, Jupiter, Saturn respectively and NbBeam = 2.25, 4.25, 4.0, 5.0,.6.0, 8.0 at 100, 143, 217, 353, 545, 857 GHz. This flag is called SSOflag4map.
A small trailing tail is added to the mask to take into account the non-deconvolution of the planet signal which has been replaced by background values. The width of that tail is 10 % of the main flag diameter. The number of samples which are additionnaly flagged are the Factor_per_Source times AddSNafter with (trail) times (Factor_per_source)^3 samples where trail = 10, 30, 20, 20, 30, 40 at 100, 143, 217, 353, 545, 857 GHz.
Uranus and Neptune together with detected asteroids are masked by HFI. They are masked at the TOI level using an exclusion radius of 1.5 SSO_FWHM. At 857 GHz, 24 asteroids have been detected with HFI : 1Ceres, 2Pallas, 3Juno, 4Vesta, 7Iris, 8Flora, 9Metis, 10Hygiea, 11Parthenope, 12Victoria, 13Egeria, 14Irene, 15Eunomia, 16Psyche, 18Melpomene, 19Fortuna, 20Massalia, 29Amphitrite, 41Daphne, 45Eugenia, 52Europa, 88Thisbe, 704Interamnia, 324Bamberga.

Output flags[edit]

A FlagTOIproc is produced by the TOIprocessing. It marks measurements which are not reliable for any of the following reasons:

  • gap (no valid input data), enlarged by one sample on each side. It flags less than 0.00044% (resp. 0.00062%) of the nominal (resp. complete) mission. It is equivalent to less than 3 (resp. 8) minutes of data.
  • "glitch" on dark bolometers: as the thermal template used for decorrelation is computed from these bolometer data, chunks of one minute-length data are discarded for all bolometers if at least 50\% of the data for at least one dark bolometer are flagged during this time. It is efficient to flag the data around the maximum of thermal events.
  • "glitch" on individual bolometers : samples where the signal from a cosmic ray hit dominates the sky signal at more than 3 $\sigma$ are discarded.
  • jump as 100 samples are flagged around the computed position of the jump to take into account the error on this reconstructed position.


So the flag produced for the map making, called Total_flag, is defined by:

  • Total_flag = UnstablePointing Flag OR FlagTOIproc OR SSOflag4map OR SSOflag seen

where FlagTOIproc = gap OR flag thermal template OR glitch OR jump and for PSB bolometers, FlagTOIproc_AB = FlagTOIproc_A OR FlagTOIproc_B. Note that the Total_flag is then identical for the A and B bolometers of a PSB pair.

At the destriping stage, a more restricted flag, called Total_flag_PS, is used. It is defined by Total_flag_PS = Total_flag OR PS_flag.


References[edit]

<biblio force=false>

  1. References

</biblio>

(Planck) High Frequency Instrument

EMI/EMC influence of the 4K cooler mechanical motion on the bolometer readout electronics.

analog to digital converter

low frequency excess response

reduced IMO

Ring-Ordered Information (DMC group/object)

random telegraphic signal

Solar System Object

Calibration and Performance Verification

revolutions per minute

sudden change of the baseline level inside a ring

Full-Width-at-Half-Maximum

To be confirmed