https://wiki.cosmos.esa.int/planckpla2015/api.php?action=feedcontributions&user=Fdesert&feedformat=atom
Planck PLA 2015 Wiki - User contributions [en-gb]
2024-03-29T14:32:29Z
User contributions
MediaWiki 1.31.6
https://wiki.cosmos.esa.int/planckpla2015/index.php?title=Timelines&diff=11208
Timelines
2015-02-04T16:11:25Z
<p>Fdesert: /* HFI processing */</p>
<hr />
<div>==General description==<br />
<br />
The timelines, or TOIs for ''Time-Ordered Information'', are vectors of signal or of pointing or of some other quantity giving the value of that quantity as a function of time during the mission. The TOIs described here are sampled regularly at the (instrument dependent) detector sampling frequency and span the full science mission. They thus consist of a large amount of data. This section is concerned with the signal TOIs and their associated pointing TOIs. For selected detectors the DPCs provide a single timeline of cleaned and calibrated signal and 3 timelines of coordinates, corresponding to the two angular coordinates and one position angle. <br />
<br />
Obviously these vectors are very long (~1.38E10 samples for HFI, from ~2.5E06 to ~5.5E06 for LFI) and thus need to be split into multiple files for export. Here the data are split by Operational Day (OD) as follows:<br />
* HFI: 12 files per OD, one for pointing and one for signal for each channel, covering ODs 91 -- 974, all sampled at F<sub>samp</sub> = 180.3737 Hz. <br />
* LFI: 6 files per OD, 3 each for signal and pointing, since the sampling frequency is channel dependent, and covering ODs 91 -- 1543, sampled at F<sub>samp</sub> = 32.5079, F<sub>samp</sub> = 46.5455 and F<sub>samp</sub> = 78.7692 Hz.<br />
<br />
The signal timelines are encoded as Real*4, but the pointing vectors had to be encoded as Real*8 to maintain the required accuracy. The result is that the total volume of the full dataset is ~30 TB. All files also contain the OBT.<br />
<br />
The signal timelines have been cleaned of all known instrumental systematic effects, they have been calibrated in flux, and the solar and earth motion dipole signals have been removed. But they have not been cleaned of the low-frequency, ''1/f'' noise that needs to be removed via a destriping tool. The methods consists of removing offsets or ''baselines'' determined by minimizing the differences in the signals at points on the sky where they intersect. For HFI an offset per ring is determined; for LFI the the baseline is computed every 0.246, 0.988 and 1.000 second for the 30, 44 and 70 GHz respectively and maintain the same structure of the signal timelines.<br />
In case of HFI these offsets are determined using the full mission and all the valid detectors per channel, and they are then used for all the maps produced, i.e., for those maps using any fraction of the mission (year, survey) or any subset of the detectors (single detector, detector set), and it was shown that using offsets determined from a limited part of the mission and/or a subset of the detectors yields maps that are less consistent than otherwise. <br />
<br />
In case of LFI these offsets are determined using the full mission and all the valid detectors per channel, those values has been used for the production of the full mission period maps. Note that baseline used for shorter period maps are determined on those data period to avoid noise cross-correlation effect and those are not delivered.<br />
<br />
The offsets are delivered separately, as described below.<br />
<br />
<span style="color:red"> Maybe insert a figure for example? </span><br />
<br />
Furthermore, for HFI and LFI, there are three sets of offsets produced: the primary set using the full rings, and the secondary ones using the first and second half rings, only. The difference between the primary and the secondary sets are fairly minor, but they are necessary to rebuild the maps as they were built by the HFI-DPC and LFI-DPC. Overall the offsets are useful for users wishing to build a map of a small area of the sky.<br />
<br />
The HFI delivers its offsets in a ROI, or "Ring-Ordered Information" file. That file contains a table of N<sub>rings</sub> = 26766 rows by N<sub>bolometers</sub> in which each cell contains a 3-elements vector containing the full-ring, first half-ring, and second half-ring offsets. The indices of the signal TOI delimiting the rings are given in a second ROI file with the same N<sub>rings</sub> rows containing global parameters.<br />
<br />
The LFI delivers its offsets in a TOI format, the structure is EXACTLY the same used by the Science Timelines. The user can simply subtract one-by-one the offset timelines from the Science Timelines and then generate a map with the result.<br />
In case of the half-ring baseline, a vector has been added in the OBT extension; this vector contains 1 or 2 depending to which half ring should be applied.<br />
<br />
===HFI Indexing===<br />
<br />
Each FITS file contains keywords giving the first and last index of the section of the full TOI contained in the file. These indices are those of the internal DPC representation; they begin at 0 corresponding to the switch-on of the instrument, and run to some very large number (~25 billion for HFI) representing the full mission. Only the ''science'' part of the mission is exported, which is about 60% of the total for HFI. Are left out the early mission phase (cool-down and validation phases) and the warm phase of HFI, during which LFI continued to collect data. The HFI keywords also indicated which rings are included in each OD, but be aware that normally an OD does not begin at the same time (with the same index) as a ring. <br />
<br />
Note that for historical reasons, the OD definitions of the two DPCs differ: for LFI they occur at the transition between pointing periods, whereas for HFI they do not. This has however no significant importance for the user, as this splitting is somewhat arbitrary anyway, and what counts is the full vector once it is rebuilt in the user's own work space.<br />
<br />
HFI timelines at the DPC are indexed from 0 to ~25E9, that correspond to instrument switch-on to switch-off. Of these the indices ~1.4E9 to 151.5E9 correspond to the science mission and are exported and delivered. Each file contains a keyword giving the first and last index of the data in that file, and EndIndex(OD)+1 = BeginIndex (OD+1). The ROI ''Global'' file gives the Begin/End Indices of each Ring, or Pointing Period, and can be used to destripe the signal TOIs with the offsets provided in the ''Destriping-Offsets'' ROI file.<br />
<br />
==Production process==<br />
<br />
=== HFI processing ===<br />
<br />
The input TOIs are in ADUs representing the voltage signal at the end of the electronics (see [[HFI_detection_chain | Detection chain]] for details). The processing applied to remove instrumental effects and to calibrate them is described in detail in the [[TOI processing | HFI TOI processing]] section, and in <span style="color:Red">Planck-2015-VII ref</span> we give a very brief summary here for convenience. That pipeline performs the following operations:<br />
<br />
; ADC correction: corrects for the uneven size of the ADC bins (see [[ADC correction]]).<br />
; demodulation: this is performed around a variable level which is determined from the valid input data (using a validity flag from a previous version of the processing), and the data are converted to engineering units (V) using known conversion coefficients.<br />
; despiking: the redundancy within a ring is used to determine where glitches occur. Once identified, the glitches are fitted with templates. A glitch flag is produced that identifies the strongest part of the glitches, and a timeline of glitch tails. produced from the template fits, is subtracted from the demodulated timeline. Finally, the flagged regions are replaced with data from an average over the pointing period in order to not leave holes in the timeline that would perturb the usage of Fourier methods on them<br />
; dark template removal: the two dark bolometers are demodulated and despiked as above; the resulting timelines are then smoothed and used as an indicator of the overall temperature variations of the bolometer plate. Where the variations are consistent with each other, they are combined and removed from the bolometer signal timelines using appropriate coupling coefficients. The few percent of the data where they are not consistent are flagged as not valid on the timelines.<br />
; conversion to absorbed power: the timeline is converted to watts of absorbed power using the bolometer function. This includes a non-linearity correction; removal of the 4K cooler lines: the electromagnetic interference of the 4K cooler with the bolometer readout wires induces some sharp lines in the signal power spectra at frequencies of the 4K cooler's fundamental and its multiples, folded by the signal modulations. Fourier coefficients of the relevant lines are determined on a per-ring basis, and then removed from the data. The quality of the removal depends on the bolometer.<br />
; deconvolution by the time transfer function: this is done to correct for the non-instantaneous time response of the bolometers. The function itself is modeled using up to 8 time constants which are adjusted primarily on the planet data and also from comparisons of the northward and southward scans of the Galactic Plane. It is then removed using Fourier techniques.<br />
; jump correction: corrects some pretty rare jumps in the signal baseline (there are on average ~ 0.3 jumps per bolometer per pointing period). The jumps are detected and characterised on smoothed TOIs, and corrected by adding a constant to part of the signal timeline. The origin of the jumps is not known.<br />
<br />
The results of this processing are a timeline of signal (in absorbed watts) and a ''valid data'' flag timeline for each of the 50 valid bolometers which are processed, out of the 52 HFI bolometers; these timelines contain the full sky signal, i.e., including the solar and orbital dipoles, the Zodiacal light, and contributions from the Far-Side lobes. The dipoles are necessary for the flux calibration and are removed at the mapmaking stage. The remaining two bolometers (143-8 and 545-3) show semi-random jumps in the signal level, typically jumping over 2-5 different ''pseudo-baseline'' levels, a behavior known as ''Random Telegraphic Signal'', so that these are commonly called the RTS bolometers. Finally, ring-level statistics of different types (mean, median, rms, kurtosis, etc.) are determined on a per-ring basis for all timelines, and a selection based on these statistics is used to discard anomalous rings, which are recorded in a ring-level flag for each valid bolometer timeline (see the [[TOI_processing#Discarded_rings | Discarded rings]] section). <br />
<br />
Throughout this processing, some planet signals are masked in the timeline in order to avoid ringing effects in the processing. It concerns only Jupiter at 100 and 143 GHz, Jupiter and Saturn at 217 GHz, Jupiter, Saturn and Mars at 353, 545 and 857 GHz (thanks to various improvements in the despiking and in the transfer functions it is no longer necessary to flag these three planets in the CMB channels as was done in Release 1). Since they move on the sky, the portion of the sky masked during one survey is observed during another one, and no hole is left in the final map. In parallel, the planet data are processed in a similar way, but with different parameters for the despiking step. These results are processed separately to determine the beam shapes and the focal plane geometry.<br />
<br />
Next, the TOIs are calibrated in astrophysical units using the results of the calibration pipeline (see the [[Map-making | Map-making and calibration]] section) and the best estimate of the zero-point offsets (a constant level for each bolometer). These values are given in the RIMO. Also, the solar and earth-motion dipole signals are computed and removed. <br />
<br />
These TOIs are accompanied by several flags that are described below. The most important one is the ''Total'' flag, which identifies all the samples that were discarded by the DPC mapmaking, ad described in [[TOI processing | HFI TOI processing]]. This flag includes the portion beyond 72 min for the rings that are longer, which is not used because of the slight drift of the satellite pointing direction (spin axis) during these long acquisition periods.<br />
<br />
At this point the TOIs still contain the low frequency (1/f) noise which should be removed before projection onto a map. That cleaning step is called "destriping" "baseline removal". The HFI-DPC does its destriping at ring-level, meaning that a constant is added to the signal of each ring in order to minimise the difference at the ring crossings, where the signal should be the same for all detectors (with the necessary precautions as described in [[Map-making | Map-making and calibration]] section); should the user want to use the DPC's offsets, they are provided separately (see [[Timelines#ROI_files | ROI files]] below). Maps produced from these TOIs, and after subtraction of the DPC's destriping offsets, are not identical to the maps delivered. This is discussed in Section A.2 by <span style="color:Red">Planck-2015-VII ref</span>.<br />
<br />
The pointing (see common [[Detector_pointing | Detector Pointing]] section for details) is determined from the AHF produced by MOC, which gives the direction and orientation of the LOS of a fiducial position in the focal plane at frequencies of 8Hz during stable pointing and 4 Hz during maneuvers. This is interpolated to the timestamps of data samples, corrected for the wobble and other time-dependent offsets determined from the observed positions of a large number of sources around the sky, and finally converted to the LOS of each detector using the quaternions in the IMO.<br />
<br />
=== LFI processing ===<br />
The input TOIs are in ADUs representing the voltage signal at the end of the electronics (see [[LFI design, qualification, and performance#Radiometer Chain Assembly (RCA) | Radiometer Chain Assembly (RCA)]] for details). The processing applied to remove instrumental effects and to calibrate them is described in detail in the [[TOI processing | LFI TOI processing]] section, and in {{PlanckPapers|planck2014-a03||Planck-2015-A03}} we give a very brief summary here for convenience. That pipeline performs the following operations:<br />
; ADC correction: due to ADC not linearity under certain condition, this instrumental effect is removed applying well know templates directly to the diode signal.<br />
; Electronics Spikes: caused by the interaction between the electronics clock and the scientific data lines. The<br />
signal is detected in all the LFI radiometers time-domain outputs as a 1s square wave with a rising edge near 0.5s <br />
and a falling edge near 0.75s synchronous with the on-board time signal. In the frequency domain it appears as a spike signal at multiples of 1 Hz. The 44 GHz channels are the only LFI outputs significantly affected by this effect. Consequently the spike signal is removed from the data only in these channels.<br />
; Demodulation: each LFI diode switches at 4096 Hz between the sky and the 4K reference load. The data acquired in this way is dominated by 1\f noise that is highly correlated between the two streams; differencing those streams results in a strong reduction of 1\f noise. The procedure applied is described in [[TOI processing | LFI TOI processing]] taking in mind that the gain modulation factor R was computed on times stream with the Galaxy and point sources masked to avoid strong sky signals.<br />
; Diode combination: two detector diodes provide the output for each LFI receiver channel. To minimize the impact of imperfect isolation the data stream of each diode, we perform a weighted average of the time-ordered data from the two diodes of each receiver. The procedure applied is detailed in [[TOI processing | LFI TOI processing]]; the weights used are kept fixed for the entire mission.<br />
; Scientific Calibration: calibrate the timelines to physical units K<sub>CMB</sub>, fitting the total CMB dipole convolved with the 4pi beam representation, without taking into account the signature due to Galactic straylight;<br />
; Gain regularization: the calibration constants computed using the model of the dipole signal suffer from large uncertainties when the Planck spacecraft is badly aligned with the dipole axis. To reduce the noise, we apply an adaptive smoothing algorithm<br />
that is also designed to preserve the discontinuities caused by abrupt changes in the working configuration of the radiometers<br />
(e.g., sudden temperature changes in the focal plane).<br />
; Removal of solar and orbital dipole: solar and orbital dipole are convolved with the 4pi beam representation of each radiometer and the removed from its timeline.<br />
; Removal of Galactic Strylight: the light incident on the focal plane without reflecting on the primary mirror (straylight) is a major source of systematic effects, especially when the Galactic plane intersects the direction of the main spillover. This effect is corrected by removing the estimated straylight signal from the timelines. This signal is computed as the convolution of Galaxy simulation with the beams sidelobes, see details in [[TOI processing | LFI TOI processing]].<br />
<br />
At this point the timelines are used for the production of the maps.<br />
<br />
The pointing (see common [[Detector_pointing | Detector Pointing]] section for details) is determined from the AHF produced by MOC, which gives the direction and orientation of the LOS of a fiducial position in the focal plane at frequencies of 8Hz during stable pointing and 4 Hz during maneuvers. This is interpolated to the timestamps of data samples, corrected for the wobble and other time-dependent offsets determined from the observed positions of a large number of sources around the sky and temperature sensor, and finally converted to the LOS of each detector.<br />
<br />
==File Names==<br />
<br />
The file names are of the form:<br />
<br />
''{H,L}FI_TOI_{fff}-{SCI,PTG,TAI,OFF}_R2.nn_ODxxxx.fits''<br />
<br />
where<br />
* ''fff'' denotes the frequency<br />
* SCI denote signal TOIs<br />
* PTG denote pointing TOIs<br />
* TAI denote OBT-MJD correlation TOI<br />
* OFF denote Baseline TOI<br />
* R2.nn is the version, and<br />
* ODxxxx indicates the OD.<br />
<br />
<br />
At the present time, HFI is providing signal TOIs for all the SWB (non-polarised) bolometers and for the PSB (polarised) bolometers at 353GHz only. The reasons for this are detailed in [###REF###]. On the other hand, the pointing TOIs are available for all bolometers. LFI is providing all signal TOI.<br />
<br />
==FITS file structure==<br />
<br />
All FITS files begin with a primary extension containing no data and a minimal header, which is followed by one or more ''BINTABLE'' extensions with data, and with a description of the data in the header keywords. In what follows will usually ignore the primary extension, and count only the extensions containing data.<br />
<br />
===TOI files===<br />
<br />
The signal FITS files contain N ,'BINTABLE', data extensions, where N is the number of detectors in that frequency channel. The first extension contains the OBT and its flags, called the ''global'' flags since they apply to all detectors, which is followed by one extension for each detector, containing the signal and its ''local'' flag. The flags columns are written as ''byte'' in which each of the 8 bits (max) encodes one flag timeline. The meaning of each bit is given in the header comments and is subject to change.<br />
For the global flag they include:<br />
* HFI<br />
** Unstable pointing: 1= pointing is not stable (e.g., during repointing maneuvers)<br />
** Dark correlation: 1 = darks are uncorrelated and data are flagged<br />
** First/Second half ring: which samples are in which half (only covers the stable pointing part of the ring)<br />
** HCM: in HCM mode (unstable pointing)<br />
** and more<br />
<br />
* LFI<br />
** Bit 0, unstable pointing: 1= pointing is not stable<br />
** Bit 1, time correlation quality: 1= outside specification<br />
** Bit 2, special observation: 1= special observation like deep scan<br />
<br />
And for the local flag they include<br />
<br />
* HFI<br />
** Total Flag: a combination of the various flags that is the one finally used in the map-making (all samples with Total Flag different from zero should not be used)<br />
** Data not valid: glitched samples<br />
** Despike Common: (for PSBs only) glitch on current or other of PSB pair<br />
** StrongSignal: on Galactic Plane<br />
** Strong Source: on point source<br />
** other<br />
<br />
* LFI<br />
** Bit 0, Data not valid: 1= Science sample not valid<br />
** Bit 2: Planet crossing: 1= Science Sample containing planet<br />
** Bit 3: Moving objects: 1= minor Solar System object (not yet used)<br />
** Bit 4: Gap: 1= this sample was artificially included due to gap in the data<br />
<br />
<br />
<br />
{| border="1" cellpadding="3" cellspacing="0" align="center" style="text-align:left" width=800px<br />
|+ '''TOI file data structure'''<br />
<br />
|- bgcolor="ffdead" <br />
!colspan="5" | 1. EXTNAME = 'OBT' : Data columns<br />
|- bgcolor="ffdead" <br />
! Column Name || Data Type || Units || Description || Comment<br />
|-<br />
|OBT || Int*8 || 2<sup>-16</sup> sec || On-board time ||<br />
|-<br />
|FLAG || Byte || none || the various bit-level flags ||<br />
<br />
|- bgcolor="ffdead" <br />
! Keyword || Data Type || Value || Description || Comment<br />
|-<br />
|OD || Int || || OD covered (as in filename) ||<br />
|-<br />
|BEGIDX || Int || || first sample index of given OD || Only HFI<br />
|-<br />
|ENDIDX || Int || || last sample index of given OD || Only HFI<br />
|-<br />
|BEGRING || Int || || first ring in given OD || Only HFI<br />
|-<br />
|ENDRING || Int || || last ring in given OD || Only HFI<br />
|-<br />
|TIMEZERO || String || 1958-01-01z00:00 || Origin of OBT || Only HFI<br />
|-<br />
|TIMEZERO || Float || 106744000000000. || Origin of OBT || Only LFI<br />
<br />
|- bgcolor="ffdead" <br />
!colspan="5" | n. EXTNAME = ''DETNAME'' : Data columns<br />
<br />
|- bgcolor="ffdead" <br />
! Column Name || Data Type || Units || Description || Comment<br />
|-<br />
|SIGNAL || Real*4 || K<sub>cmb</sub> or MJy/sr || Value of signal || Comment<br />
|-<br />
|FLAG || Byte || none || the various bit-level flags ||<br />
|- bgcolor="ffdead" <br />
<br />
! Keyword || Data Type || Value || Description || Comment<br />
|-<br />
|UNIT || String || || Units of signal || Only HFI, LFI always K<sub>cmb</sub><br />
|-<br />
|DESTRIPE || 1/0 || || whether timeline is destriped || Only HFI<br />
|-<br />
|OD || Int || || OD covered (as in filename) ||<br />
|-<br />
|BEGIDX || Int || || first sample index of given OD || Only HFI<br />
|-<br />
|ENDIDX || Int || || last sample index of given OD || Only HFI<br />
|-<br />
|BEGRING || Int || || first ring in given OD || Only HFI<br />
|-<br />
|ENDRING || Int || || last ring in given OD || Only HFI<br />
|}<br />
<br />
<br />
The pointing files have a similar structure, except that ''DETNAME'' extensions contain 3 columns of Real*8 with the phi, theta, and psi Galactic spherical coordinates of each sample in radians, as shown in the table below. There is no local flag for the coordinates.<br />
<br />
{| border="1" cellpadding="3" cellspacing="0" align="center" style="text-align:left" width=800px<br />
|+ '''Pointing TOI file DETNAME extension structure'''<br />
<br />
|- bgcolor="ffdead" <br />
!colspan="4" | n. EXTNAME = ''DETNAME'' : Data columns<br />
<br />
|- bgcolor="ffdead" <br />
! Column Name || Data Type || Units || Description<br />
|-<br />
|PHI || Real*8 || radian || longitude<br />
|-<br />
|THETA || Real*8 || radian || colatitude<br />
|-<br />
|PSI || Real*8 || radian || roll angle<br />
|}<br />
<br />
===HFI ROI files===<br />
<br />
The files provided by HFI are<br />
* ''HFI_ROI_GlobalParams_RelNum_full.fits''<br />
* ''HFI_ROI_DestripeOffsets_RelNum_full.fits''<br />
which are described below.<br />
<br />
;Global parameters: this ROI file contains a single ''BINTABLE'' extension with 3 columns that give, for each ring, the index number of the first sample, the ESA pointing ID of the ring, and the ring start time. All three are encoded as Int*8, and the time is given in units of 2<sup>-16</sup> sec from ''TIMEZERO'', 1958-01-01z00:00, as in the TOI files. Rings are numbered from 240 to 27005, as given in the ''BEGRING'' and ''ENDRING'' keywords<br />
<br />
; Destriping offsets: this ROI file contains a single ''BINTABLE'' extension with 3 columns that give, for each ring, the offsets to subtract from each bolometer signal timeline or vector. There are three offsets available for each bolometer: full-ring, first half-ring, and second half-ring, for building the full-ring or the two half-ring maps, respectively. The offsets must be subtracted from Index(N) to Index(N+1)-1 of the corresponding signal timeline, where N is the ring number, and the indices are given in the Global parameters file. The offsets are given in the same units as the signal vectors, that is K<sub>cmb</sub> for the 100-353 GHz channels, and MJy/sr for the 545 and 857 GHz channels. All values are encoded as Real*4. The rings for which the offsets are set to 0.000 are [[TOI_processing#Discarded_rings | ''badrings'' ]] which are rejected in the mapmaking process. These rings are also flagged by the ''TotalFlag'', as are the portions of rings longer than ~72.5 min, where the drift in the satellite's spin axis becomes important.<br />
<br />
===TAI TOI files===<br />
<br />
The TAI TOI files contains one extension with two column, the first is the OBT value (exactly the same reported in the SCI TOI), the second the corresponding Modified Julian day. Note that Leap second where not added.<br />
<br />
{| border="1" cellpadding="3" cellspacing="0" align="center" style="text-align:left" width=800px<br />
|+ '''Time TAI TOI file data structure'''<br />
<br />
|- bgcolor="ffdead" <br />
!colspan="5" | 1. EXTNAME = 'OBT-MJD' : Data columns<br />
|- bgcolor="ffdead" <br />
! Column Name || Data Type || Units || Description || Comment<br />
|-<br />
|OBT || Int*8 || 2<sup>-16</sup> sec || On-board time ||<br />
|-<br />
|MJD || Real*8 || day || Modified Julian day ||<br />
|}<br />
<br />
===LFI OFF TOI files===<br />
The OFF (baseline) TOI files adopt the same file structure of the Science TOI files. Note that in case of OFF timelines related to the half-ring, and addition column in included in the OBT extension to define for each sample if it belong to half-ring 1 or half-ring2.<br />
<br />
===LFI HouseKeeping files===<br />
House keeping timelines are:<br />
* LFI_TOI_DAE-FAST-HK_R2.nn_ODxxxx.fits<br />
* LFI_TOI_DAE-SlowConfiguration-HK_R2.nn_ODxxxx.fits<br />
* LFI_TOI_DAE-SlowCurrent-HK_R2.nn_ODxxxx.fits<br />
* LFI_TOI_DAE-SlowPhaseSwitch-HK_R2.nn_ODxxxx.fits<br />
* LFI_TOI_DAE-SlowVoltage-HK_R2.nn_ODxxxx.fits<br />
* LFI_TOI_REBA-HK_R2.nn_ODxxxx.fits<br />
* SCS_TOI_EssentialASW-HK_R2.nn_ODxxxx.fits<br />
* SCS_TOI_NonEssentialASW-HK_R2.nn_ODxxxx.fits<br />
<br />
Each file contains two extension, the first is the OBT (value are sampled at 1 or 10 seconds), the latter contain a variable number of column = 2* number of HouseKeeping stored. Each Housekeeping is accompanied by its flag (normally 0 is NOT 0 the value was considerate invalid or Out Of Limit). The Housekeeping name are the once defined in the LFI Instrnument Operation Manual.<br />
<br />
<br />
<span style="color:red"> TBW: Note on half-ring offsets, which differ by 1-2% from full ring offsets</span> <br />
<br />
== References ==<br />
<br />
<References /><br />
<br />
[[Category:Mission products|001]]</div>
Fdesert
https://wiki.cosmos.esa.int/planckpla2015/index.php?title=Timelines&diff=11207
Timelines
2015-02-04T16:08:41Z
<p>Fdesert: /* HFI processing */</p>
<hr />
<div>==General description==<br />
<br />
The timelines, or TOIs for ''Time-Ordered Information'', are vectors of signal or of pointing or of some other quantity giving the value of that quantity as a function of time during the mission. The TOIs described here are sampled regularly at the (instrument dependent) detector sampling frequency and span the full science mission. They thus consist of a large amount of data. This section is concerned with the signal TOIs and their associated pointing TOIs. For selected detectors the DPCs provide a single timeline of cleaned and calibrated signal and 3 timelines of coordinates, corresponding to the two angular coordinates and one position angle. <br />
<br />
Obviously these vectors are very long (~1.38E10 samples for HFI, from ~2.5E06 to ~5.5E06 for LFI) and thus need to be split into multiple files for export. Here the data are split by Operational Day (OD) as follows:<br />
* HFI: 12 files per OD, one for pointing and one for signal for each channel, covering ODs 91 -- 974, all sampled at F<sub>samp</sub> = 180.3737 Hz. <br />
* LFI: 6 files per OD, 3 each for signal and pointing, since the sampling frequency is channel dependent, and covering ODs 91 -- 1543, sampled at F<sub>samp</sub> = 32.5079, F<sub>samp</sub> = 46.5455 and F<sub>samp</sub> = 78.7692 Hz.<br />
<br />
The signal timelines are encoded as Real*4, but the pointing vectors had to be encoded as Real*8 to maintain the required accuracy. The result is that the total volume of the full dataset is ~30 TB. All files also contain the OBT.<br />
<br />
The signal timelines have been cleaned of all known instrumental systematic effects, they have been calibrated in flux, and the solar and earth motion dipole signals have been removed. But they have not been cleaned of the low-frequency, ''1/f'' noise that needs to be removed via a destriping tool. The methods consists of removing offsets or ''baselines'' determined by minimizing the differences in the signals at points on the sky where they intersect. For HFI an offset per ring is determined; for LFI the the baseline is computed every 0.246, 0.988 and 1.000 second for the 30, 44 and 70 GHz respectively and maintain the same structure of the signal timelines.<br />
In case of HFI these offsets are determined using the full mission and all the valid detectors per channel, and they are then used for all the maps produced, i.e., for those maps using any fraction of the mission (year, survey) or any subset of the detectors (single detector, detector set), and it was shown that using offsets determined from a limited part of the mission and/or a subset of the detectors yields maps that are less consistent than otherwise. <br />
<br />
In case of LFI these offsets are determined using the full mission and all the valid detectors per channel, those values has been used for the production of the full mission period maps. Note that baseline used for shorter period maps are determined on those data period to avoid noise cross-correlation effect and those are not delivered.<br />
<br />
The offsets are delivered separately, as described below.<br />
<br />
<span style="color:red"> Maybe insert a figure for example? </span><br />
<br />
Furthermore, for HFI and LFI, there are three sets of offsets produced: the primary set using the full rings, and the secondary ones using the first and second half rings, only. The difference between the primary and the secondary sets are fairly minor, but they are necessary to rebuild the maps as they were built by the HFI-DPC and LFI-DPC. Overall the offsets are useful for users wishing to build a map of a small area of the sky.<br />
<br />
The HFI delivers its offsets in a ROI, or "Ring-Ordered Information" file. That file contains a table of N<sub>rings</sub> = 26766 rows by N<sub>bolometers</sub> in which each cell contains a 3-elements vector containing the full-ring, first half-ring, and second half-ring offsets. The indices of the signal TOI delimiting the rings are given in a second ROI file with the same N<sub>rings</sub> rows containing global parameters.<br />
<br />
The LFI delivers its offsets in a TOI format, the structure is EXACTLY the same used by the Science Timelines. The user can simply subtract one-by-one the offset timelines from the Science Timelines and then generate a map with the result.<br />
In case of the half-ring baseline, a vector has been added in the OBT extension; this vector contains 1 or 2 depending to which half ring should be applied.<br />
<br />
===HFI Indexing===<br />
<br />
Each FITS file contains keywords giving the first and last index of the section of the full TOI contained in the file. These indices are those of the internal DPC representation; they begin at 0 corresponding to the switch-on of the instrument, and run to some very large number (~25 billion for HFI) representing the full mission. Only the ''science'' part of the mission is exported, which is about 60% of the total for HFI. Are left out the early mission phase (cool-down and validation phases) and the warm phase of HFI, during which LFI continued to collect data. The HFI keywords also indicated which rings are included in each OD, but be aware that normally an OD does not begin at the same time (with the same index) as a ring. <br />
<br />
Note that for historical reasons, the OD definitions of the two DPCs differ: for LFI they occur at the transition between pointing periods, whereas for HFI they do not. This has however no significant importance for the user, as this splitting is somewhat arbitrary anyway, and what counts is the full vector once it is rebuilt in the user's own work space.<br />
<br />
HFI timelines at the DPC are indexed from 0 to ~25E9, that correspond to instrument switch-on to switch-off. Of these the indices ~1.4E9 to 151.5E9 correspond to the science mission and are exported and delivered. Each file contains a keyword giving the first and last index of the data in that file, and EndIndex(OD)+1 = BeginIndex (OD+1). The ROI ''Global'' file gives the Begin/End Indices of each Ring, or Pointing Period, and can be used to destripe the signal TOIs with the offsets provided in the ''Destriping-Offsets'' ROI file.<br />
<br />
==Production process==<br />
<br />
=== HFI processing ===<br />
<br />
The input TOIs are in ADUs representing the voltage signal at the end of the electronics (see [[HFI_detection_chain | Detection chain]] for details). The processing applied to remove instrumental effects and to calibrate them is described in detail in the [[TOI processing | HFI TOI processing]] section, and in <span style="color:Red">Planck-2015-VII ref</span> we give a very brief summary here for convenience. That pipeline performs the following operations:<br />
<br />
; ADC correction: corrects for the uneven size of the ADC bins (see [[ADC correction]]).<br />
; demodulation: this is performed around a variable level which is determined from the valid input data (using a validity flag from a previous version of the processing), and the data are converted to engineering units (V) using known conversion coefficients.<br />
; despiking: the redundancy within a ring is used to determine where glitches occur. Once identified, the glitches are fitted with templates. A glitch flag is produced that identifies the strongest part of the glitches, and a timeline of glitch tails. produced from the template fits, is subtracted from the demodulated timeline. Finally, the flagged regions are replaced with data from an average over the pointing period in order to not leave holes in the timeline that would perturb the usage of Fourier methods on them<br />
; dark template removal: the two dark bolometers are demodulated and despiked as above; the resulting timelines are then smoothed and used as an indicator of the overall temperature variations of the bolometer plate. Where the variations are consistent with each other, they are combined and removed from the bolometer signal timelines using appropriate coupling coefficients. The few percent of the data where they are not consistent are flagged as not valid on the timelines.<br />
; conversion to absorbed power: the timeline is converted to watts of absorbed power using the bolometer function. This includes a non-linearity correction; removal of the 4K cooler lines: the electromagnetic interference of the 4K cooler with the bolometer readout wires induces some sharp lines in the signal power spectra at frequencies of the 4K cooler's fundamental and its multiples, folded by the signal modulations. Fourier coefficients of the relevant lines are determined on a per-ring basis, and then removed from the data. The quality of the removal depends on the bolometer.<br />
; deconvolution by the time transfer function: this is done to correct for the non-instantaneous time response of the bolometers. The function itself is modeled using up to 8 time constants which are adjusted primarily on the planet data and also from comparisons of the northward and southward scans of the Galactic Plane. It is then removed using Fourier techniques.<br />
; jump correction: corrects some pretty rare jumps in the signal baseline (there are on average ~ 0.3 jumps per bolometer per pointing period). The jumps are detected and characterised on smoothed TOIs, and corrected by adding a constant to part of the signal timeline. The origin of the jumps is not known.<br />
<br />
The results of this processing are a timeline of signal (in absorbed watts) and a ''valid data'' flag timeline for each of the 50 valid bolometers which are processed, out of the 52 HFI bolometers; these timelines contain the full sky signal, i.e., including the solar and orbital dipoles, the Zodiacal light, and contributions from the Far-Side lobes. The dipoles are necessary for the flux calibration and are removed at the mapmaking stage. The remaining two bolometers (143-8 and 545-3) show semi-random jumps in the signal level, typically jumping over 2-5 different ''pseudo-baseline'' levels, a behavior known as ''Random Telegraphic Signal'', so that these are commonly called the RTS bolometers. Finally, ring-level statistics of different types (mean, median, rms, kurtosis, etc.) are determined on a per-ring basis for all timelines, and a selection based on these statistics is used to discard anomalous rings, which are recorded in a ring-level flag for each valid bolometer timeline (see the [[TOI_processing#Discarded_rings | Discarded rings]] section). <br />
<br />
Throughout this processing, some planet signals are masked in the timeline in order to avoid ringing effects in the processing. It concerns only Jupiter at 100 and 143 GHz, Jupiter and Saturn at 217 GHz, Jupiter, Saturn and Mars at 353, 545 and 857 GHz (thanks to various improvements in the despiking and in the transfer functions it is no longer necessary to flag these three planets in the CMB channels as was done in Release 1). Since they move on the sky, the portion of the sky masked during one survey is observed during another one, and no hole is left in the final map. In parallel, the planet data are processed in a similar way, but with different parameters for the despiking step. These results are processed separately to determine the beam shapes and the focal plane geometry.<br />
<br />
Next, the TOIs are calibrated in astrophysical units using the results of the calibration pipeline (see the [[Map-making | Map-making and calibration]] section) and the best estimate of the zero-point offsets (a constant level for each bolometer). These values are given in the RIMO. Also, the solar and earth-motion dipole signals are computed and removed. <br />
<br />
These TOIs are accompanied by several flags that are described below. The most important one is the ''Total'' flag, which identifies all the samples that were discarded by the DPC mapmaking, ad described in [[TOI processing | HFI TOI processing]]. This flag includes the portion beyond 72 min for the rings that are longer, which is not used because of the slight drift of the satellite pointing direction (spin axis) during these long acquisition periods.<br />
<br />
At this point the TOIs still contain the low frequency (1/f) noise which should be removed before projection onto a map. That cleaning step is called "destriping" "baseline removal". The HFI-DPC does its destriping at ring-level, meaning that a constant is added to the signal of each ring in order to minimise the difference at the ring crossings, where the signal should be the same for all detectors (with the necessary precautions as described in [[Map-making | Map-making and calibration]] section); should the user want to use the DPC's offsets, they are provided separately (see [[Timelines#ROI_files | ROI files]] below). Maps produced from these TOIs, and after subtraction of the DPC's destriping offsets, are not identical to the maps delivered, the differences being of order XXXXX. The reasons for these differences are described in [###???? SECTION]<br />
<br />
The pointing (see common [[Detector_pointing | Detector Pointing]] section for details) is determined from the AHF produced by MOC, which gives the direction and orientation of the LOS of a fiducial position in the focal plane at frequencies of 8Hz during stable pointing and 4 Hz during maneuvers. This is interpolated to the timestamps of data samples, corrected for the wobble and other time-dependent offsets determined from the observed positions of a large number of sources around the sky, and finally converted to the LOS of each detector using the quaternions in the IMO.<br />
<br />
=== LFI processing ===<br />
The input TOIs are in ADUs representing the voltage signal at the end of the electronics (see [[LFI design, qualification, and performance#Radiometer Chain Assembly (RCA) | Radiometer Chain Assembly (RCA)]] for details). The processing applied to remove instrumental effects and to calibrate them is described in detail in the [[TOI processing | LFI TOI processing]] section, and in {{PlanckPapers|planck2014-a03||Planck-2015-A03}} we give a very brief summary here for convenience. That pipeline performs the following operations:<br />
; ADC correction: due to ADC not linearity under certain condition, this instrumental effect is removed applying well know templates directly to the diode signal.<br />
; Electronics Spikes: caused by the interaction between the electronics clock and the scientific data lines. The<br />
signal is detected in all the LFI radiometers time-domain outputs as a 1s square wave with a rising edge near 0.5s <br />
and a falling edge near 0.75s synchronous with the on-board time signal. In the frequency domain it appears as a spike signal at multiples of 1 Hz. The 44 GHz channels are the only LFI outputs significantly affected by this effect. Consequently the spike signal is removed from the data only in these channels.<br />
; Demodulation: each LFI diode switches at 4096 Hz between the sky and the 4K reference load. The data acquired in this way is dominated by 1\f noise that is highly correlated between the two streams; differencing those streams results in a strong reduction of 1\f noise. The procedure applied is described in [[TOI processing | LFI TOI processing]] taking in mind that the gain modulation factor R was computed on times stream with the Galaxy and point sources masked to avoid strong sky signals.<br />
; Diode combination: two detector diodes provide the output for each LFI receiver channel. To minimize the impact of imperfect isolation the data stream of each diode, we perform a weighted average of the time-ordered data from the two diodes of each receiver. The procedure applied is detailed in [[TOI processing | LFI TOI processing]]; the weights used are kept fixed for the entire mission.<br />
; Scientific Calibration: calibrate the timelines to physical units K<sub>CMB</sub>, fitting the total CMB dipole convolved with the 4pi beam representation, without taking into account the signature due to Galactic straylight;<br />
; Gain regularization: the calibration constants computed using the model of the dipole signal suffer from large uncertainties when the Planck spacecraft is badly aligned with the dipole axis. To reduce the noise, we apply an adaptive smoothing algorithm<br />
that is also designed to preserve the discontinuities caused by abrupt changes in the working configuration of the radiometers<br />
(e.g., sudden temperature changes in the focal plane).<br />
; Removal of solar and orbital dipole: solar and orbital dipole are convolved with the 4pi beam representation of each radiometer and the removed from its timeline.<br />
; Removal of Galactic Strylight: the light incident on the focal plane without reflecting on the primary mirror (straylight) is a major source of systematic effects, especially when the Galactic plane intersects the direction of the main spillover. This effect is corrected by removing the estimated straylight signal from the timelines. This signal is computed as the convolution of Galaxy simulation with the beams sidelobes, see details in [[TOI processing | LFI TOI processing]].<br />
<br />
At this point the timelines are used for the production of the maps.<br />
<br />
The pointing (see common [[Detector_pointing | Detector Pointing]] section for details) is determined from the AHF produced by MOC, which gives the direction and orientation of the LOS of a fiducial position in the focal plane at frequencies of 8Hz during stable pointing and 4 Hz during maneuvers. This is interpolated to the timestamps of data samples, corrected for the wobble and other time-dependent offsets determined from the observed positions of a large number of sources around the sky and temperature sensor, and finally converted to the LOS of each detector.<br />
<br />
==File Names==<br />
<br />
The file names are of the form:<br />
<br />
''{H,L}FI_TOI_{fff}-{SCI,PTG,TAI,OFF}_R2.nn_ODxxxx.fits''<br />
<br />
where<br />
* ''fff'' denotes the frequency<br />
* SCI denote signal TOIs<br />
* PTG denote pointing TOIs<br />
* TAI denote OBT-MJD correlation TOI<br />
* OFF denote Baseline TOI<br />
* R2.nn is the version, and<br />
* ODxxxx indicates the OD.<br />
<br />
<br />
At the present time, HFI is providing signal TOIs for all the SWB (non-polarised) bolometers and for the PSB (polarised) bolometers at 353GHz only. The reasons for this are detailed in [###REF###]. On the other hand, the pointing TOIs are available for all bolometers. LFI is providing all signal TOI.<br />
<br />
==FITS file structure==<br />
<br />
All FITS files begin with a primary extension containing no data and a minimal header, which is followed by one or more ''BINTABLE'' extensions with data, and with a description of the data in the header keywords. In what follows will usually ignore the primary extension, and count only the extensions containing data.<br />
<br />
===TOI files===<br />
<br />
The signal FITS files contain N ,'BINTABLE', data extensions, where N is the number of detectors in that frequency channel. The first extension contains the OBT and its flags, called the ''global'' flags since they apply to all detectors, which is followed by one extension for each detector, containing the signal and its ''local'' flag. The flags columns are written as ''byte'' in which each of the 8 bits (max) encodes one flag timeline. The meaning of each bit is given in the header comments and is subject to change.<br />
For the global flag they include:<br />
* HFI<br />
** Unstable pointing: 1= pointing is not stable (e.g., during repointing maneuvers)<br />
** Dark correlation: 1 = darks are uncorrelated and data are flagged<br />
** First/Second half ring: which samples are in which half (only covers the stable pointing part of the ring)<br />
** HCM: in HCM mode (unstable pointing)<br />
** and more<br />
<br />
* LFI<br />
** Bit 0, unstable pointing: 1= pointing is not stable<br />
** Bit 1, time correlation quality: 1= outside specification<br />
** Bit 2, special observation: 1= special observation like deep scan<br />
<br />
And for the local flag they include<br />
<br />
* HFI<br />
** Total Flag: a combination of the various flags that is the one finally used in the map-making (all samples with Total Flag different from zero should not be used)<br />
** Data not valid: glitched samples<br />
** Despike Common: (for PSBs only) glitch on current or other of PSB pair<br />
** StrongSignal: on Galactic Plane<br />
** Strong Source: on point source<br />
** other<br />
<br />
* LFI<br />
** Bit 0, Data not valid: 1= Science sample not valid<br />
** Bit 2: Planet crossing: 1= Science Sample containing planet<br />
** Bit 3: Moving objects: 1= minor Solar System object (not yet used)<br />
** Bit 4: Gap: 1= this sample was artificially included due to gap in the data<br />
<br />
<br />
<br />
{| border="1" cellpadding="3" cellspacing="0" align="center" style="text-align:left" width=800px<br />
|+ '''TOI file data structure'''<br />
<br />
|- bgcolor="ffdead" <br />
!colspan="5" | 1. EXTNAME = 'OBT' : Data columns<br />
|- bgcolor="ffdead" <br />
! Column Name || Data Type || Units || Description || Comment<br />
|-<br />
|OBT || Int*8 || 2<sup>-16</sup> sec || On-board time ||<br />
|-<br />
|FLAG || Byte || none || the various bit-level flags ||<br />
<br />
|- bgcolor="ffdead" <br />
! Keyword || Data Type || Value || Description || Comment<br />
|-<br />
|OD || Int || || OD covered (as in filename) ||<br />
|-<br />
|BEGIDX || Int || || first sample index of given OD || Only HFI<br />
|-<br />
|ENDIDX || Int || || last sample index of given OD || Only HFI<br />
|-<br />
|BEGRING || Int || || first ring in given OD || Only HFI<br />
|-<br />
|ENDRING || Int || || last ring in given OD || Only HFI<br />
|-<br />
|TIMEZERO || String || 1958-01-01z00:00 || Origin of OBT || Only HFI<br />
|-<br />
|TIMEZERO || Float || 106744000000000. || Origin of OBT || Only LFI<br />
<br />
|- bgcolor="ffdead" <br />
!colspan="5" | n. EXTNAME = ''DETNAME'' : Data columns<br />
<br />
|- bgcolor="ffdead" <br />
! Column Name || Data Type || Units || Description || Comment<br />
|-<br />
|SIGNAL || Real*4 || K<sub>cmb</sub> or MJy/sr || Value of signal || Comment<br />
|-<br />
|FLAG || Byte || none || the various bit-level flags ||<br />
|- bgcolor="ffdead" <br />
<br />
! Keyword || Data Type || Value || Description || Comment<br />
|-<br />
|UNIT || String || || Units of signal || Only HFI, LFI always K<sub>cmb</sub><br />
|-<br />
|DESTRIPE || 1/0 || || whether timeline is destriped || Only HFI<br />
|-<br />
|OD || Int || || OD covered (as in filename) ||<br />
|-<br />
|BEGIDX || Int || || first sample index of given OD || Only HFI<br />
|-<br />
|ENDIDX || Int || || last sample index of given OD || Only HFI<br />
|-<br />
|BEGRING || Int || || first ring in given OD || Only HFI<br />
|-<br />
|ENDRING || Int || || last ring in given OD || Only HFI<br />
|}<br />
<br />
<br />
The pointing files have a similar structure, except that ''DETNAME'' extensions contain 3 columns of Real*8 with the phi, theta, and psi Galactic spherical coordinates of each sample in radians, as shown in the table below. There is no local flag for the coordinates.<br />
<br />
{| border="1" cellpadding="3" cellspacing="0" align="center" style="text-align:left" width=800px<br />
|+ '''Pointing TOI file DETNAME extension structure'''<br />
<br />
|- bgcolor="ffdead" <br />
!colspan="4" | n. EXTNAME = ''DETNAME'' : Data columns<br />
<br />
|- bgcolor="ffdead" <br />
! Column Name || Data Type || Units || Description<br />
|-<br />
|PHI || Real*8 || radian || longitude<br />
|-<br />
|THETA || Real*8 || radian || colatitude<br />
|-<br />
|PSI || Real*8 || radian || roll angle<br />
|}<br />
<br />
===HFI ROI files===<br />
<br />
The files provided by HFI are<br />
* ''HFI_ROI_GlobalParams_RelNum_full.fits''<br />
* ''HFI_ROI_DestripeOffsets_RelNum_full.fits''<br />
which are described below.<br />
<br />
;Global parameters: this ROI file contains a single ''BINTABLE'' extension with 3 columns that give, for each ring, the index number of the first sample, the ESA pointing ID of the ring, and the ring start time. All three are encoded as Int*8, and the time is given in units of 2<sup>-16</sup> sec from ''TIMEZERO'', 1958-01-01z00:00, as in the TOI files. Rings are numbered from 240 to 27005, as given in the ''BEGRING'' and ''ENDRING'' keywords<br />
<br />
; Destriping offsets: this ROI file contains a single ''BINTABLE'' extension with 3 columns that give, for each ring, the offsets to subtract from each bolometer signal timeline or vector. There are three offsets available for each bolometer: full-ring, first half-ring, and second half-ring, for building the full-ring or the two half-ring maps, respectively. The offsets must be subtracted from Index(N) to Index(N+1)-1 of the corresponding signal timeline, where N is the ring number, and the indices are given in the Global parameters file. The offsets are given in the same units as the signal vectors, that is K<sub>cmb</sub> for the 100-353 GHz channels, and MJy/sr for the 545 and 857 GHz channels. All values are encoded as Real*4. The rings for which the offsets are set to 0.000 are [[TOI_processing#Discarded_rings | ''badrings'' ]] which are rejected in the mapmaking process. These rings are also flagged by the ''TotalFlag'', as are the portions of rings longer than ~72.5 min, where the drift in the satellite's spin axis becomes important.<br />
<br />
===TAI TOI files===<br />
<br />
The TAI TOI files contains one extension with two column, the first is the OBT value (exactly the same reported in the SCI TOI), the second the corresponding Modified Julian day. Note that Leap second where not added.<br />
<br />
{| border="1" cellpadding="3" cellspacing="0" align="center" style="text-align:left" width=800px<br />
|+ '''Time TAI TOI file data structure'''<br />
<br />
|- bgcolor="ffdead" <br />
!colspan="5" | 1. EXTNAME = 'OBT-MJD' : Data columns<br />
|- bgcolor="ffdead" <br />
! Column Name || Data Type || Units || Description || Comment<br />
|-<br />
|OBT || Int*8 || 2<sup>-16</sup> sec || On-board time ||<br />
|-<br />
|MJD || Real*8 || day || Modified Julian day ||<br />
|}<br />
<br />
===LFI OFF TOI files===<br />
The OFF (baseline) TOI files adopt the same file structure of the Science TOI files. Note that in case of OFF timelines related to the half-ring, and addition column in included in the OBT extension to define for each sample if it belong to half-ring 1 or half-ring2.<br />
<br />
===LFI HouseKeeping files===<br />
House keeping timelines are:<br />
* LFI_TOI_DAE-FAST-HK_R2.nn_ODxxxx.fits<br />
* LFI_TOI_DAE-SlowConfiguration-HK_R2.nn_ODxxxx.fits<br />
* LFI_TOI_DAE-SlowCurrent-HK_R2.nn_ODxxxx.fits<br />
* LFI_TOI_DAE-SlowPhaseSwitch-HK_R2.nn_ODxxxx.fits<br />
* LFI_TOI_DAE-SlowVoltage-HK_R2.nn_ODxxxx.fits<br />
* LFI_TOI_REBA-HK_R2.nn_ODxxxx.fits<br />
* SCS_TOI_EssentialASW-HK_R2.nn_ODxxxx.fits<br />
* SCS_TOI_NonEssentialASW-HK_R2.nn_ODxxxx.fits<br />
<br />
Each file contains two extension, the first is the OBT (value are sampled at 1 or 10 seconds), the latter contain a variable number of column = 2* number of HouseKeeping stored. Each Housekeeping is accompanied by its flag (normally 0 is NOT 0 the value was considerate invalid or Out Of Limit). The Housekeeping name are the once defined in the LFI Instrnument Operation Manual.<br />
<br />
<br />
<span style="color:red"> TBW: Note on half-ring offsets, which differ by 1-2% from full ring offsets</span> <br />
<br />
== References ==<br />
<br />
<References /><br />
<br />
[[Category:Mission products|001]]</div>
Fdesert
https://wiki.cosmos.esa.int/planckpla2015/index.php?title=Main_Page&diff=11204
Main Page
2015-02-04T16:04:18Z
<p>Fdesert: </p>
<hr />
<div><br />
<br />
'''<span style="font-size:180%"> <span style="color:Blue"> This is the Explanatory Supplement development page for the Planck 2015 data release </span><br />
<br />
<br />
<br />
* Instructions for new users: [[Help:READ ME FIRST|Read me first]]<br />
* See [http://meta.wikimedia.org/wiki/Help:Contents User's Guide] for a detailed User Guide of the MediaWiki software;<br />
* See [[Help:Contents|Explanatory Supplement Help page]] for Planck-specific guidelines.<br />
<br />
<br />
== [[:Category:Explanatory Supplement|Explanatory Supplement]] ==<br />
<br />
By the [[Planck Collaboration]]<br />
<br />
The Explanatory Supplement is a reference text accompanying the public data delivered from the operations of the European Space Agency’s Planck satellite during its mission.<br />
*[[Questions and Answers|Q&A from PR1]]<br />
<!--- ############# ---><br />
#[[Introduction_WiP|Introduction]]<br />
##[[The Planck mission_WiP|The Planck mission]] <br />
##[[The satellite_WiP|The spacecraft]]<br />
##[[Ground Segment and Operations|Ground segment and operations]]<br />
##[[Survey_scanning_and_performance|Survey scanning and performance]]<br />
<!--- ############# ---><br />
#[[The Instruments_WiP|The Instruments]]<br />
##[[HFI design, qualification, and performance|HFI design, qualification, and performance]]<br />
###[[HFI_cryogenics | Cryogenics]]<br />
###[[HFI_cold_optics | Cold optics]]<br />
###[[HFI_detection_chain | Detection chain]]<br />
###[[HFI_operations | Operations]]<br />
###[[HFI_performance_summary | Performance summary]]<br />
###[[HFI_instrument_annexes | Annexes]]<br />
##[[LFI overview|LFI design, qualification, and performance]]<span style="color:red"></span><br />
###[[LFI design, qualification, and performance#LFIDescription| Instrument description]]<br />
###[[LFI design, qualification, and performance#LFITests| Ground tests]]<br />
###[[LFI design, qualification, and performance#LFICalibration| In-flight calibration]]<br />
###[[LFI design, qualification, and performance#LFIPerformance| Performance summary]]<br />
###[[LFI design, qualification, and performance#LFISystematics| Systematic effects]]<br />
###[[LFI design, qualification, and performance#SCS| Sorption cooler]]<br />
###[[LFIAppendix| Annexes]]<br />
<!--- ############# ---><br />
#[[Data processing]]<br />
##[[The HFI DPC| HFI Data Processing]]<br />
###[[Pre-processing | Pre-processing]]<br />
###[[ADC correction]]<br />
###[[TOI processing|TOI processing]]<br />
###[[Beams | Beams]]<br />
###[[Map-making | Mapmaking]]<br />
###[[Spectral response | Spectral response]]<br />
###[[HFI-Validation | Internal overall validation]]<br />
###[[Power spectra | Power spectra]]<br />
###[[Summary_of_HFI_data_characteristics | Summary of HFI data characteristics]]<br />
##[[The LFI DPC| LFI data processing]] <span style="color:red"></span><br />
###[[Pre-processing_LFI| Pre-processing]]<br />
###[[TOI processing_LFI| TOI processing]] <span style="color:red"></span><br />
###[[Beams_LFI | Beams]] <span style="color:red"></span><br />
###[[Galactic stray light removal]]<br />
###[[Map-making_LFI | Mapmaking]] <span style="color:red"></span><br />
###[[LFI systematic effect uncertainties | Systematic effects uncertainties]]<br />
###[[LFI-Validation | Internal overall validation]] <span style="color:red"></span><br />
###[[L3_LFI | Power spectra]] <br />
###[[Summary_LFI | Summary of LFI data characteristics ]]<br />
##[[HFI/LFI joint data processing]]<br />
###[[Detector pointing| Detector pointing]]<br />
<!--- ###[[NoiseCovarMatrices | Noise covariance matrices and low-resolution maps ]] ---><br />
###[[Compact Source catalogues| Compact Source Catalogues]]<br />
###[[Astrophysical component separation]]<br />
###[[C2 | CMB Power spectra and Planck likelihood code]]<br />
<!--- ############# ---><br />
#[[Mission products]]<br />
##[[Timelines | Time-ordered data]]<br />
##[[Frequency Maps | Sky temperature and polarization maps]]<br />
<!--- ##[[NoiseCovariance | Noise covariance matrices and low-resolution maps ]]<span style="color:red">(Keskitalo)</span ---><br />
##[[The RIMO|Instrument model]] <br />
##[[Scanning Beams | Scanning Beams]]<br />
##[[Effective Beams | Effective beams]]<br />
##[[Catalogues|Catalogues]]: [[Catalogues#Catalogue of Compact Sources|PCCS]] • [[Catalogues#SZ Catalogue|PSZ]] • [[Catalogues#Catalogue_of_Planck_Galactic_Cold_Clumps|PGCC]]<br />
##[[CMB_and_astrophysical_component maps | CMB and astrophysical component maps]]<br />
##[[CMB spectrum & Likelihood Code | CMB spectrum and likelihood code]]<br />
##[[Cosmological Parameters | Cosmological parameters and MC chains]]<br />
##[[Specially processed maps | Additional maps]]: [[Specially processed maps#Lensing map | Lensing map]] • [[Specially processed maps#Compton parameter map | Compton parameter map]] <br />
##[[Scientific data used to generate Planck products | Scientific data used to generate Planck products]]<br />
##[[Simulation data | Simulation data]] <!--: [[Simulation data#The Planck Sky Model|The Planck Sky Model]] • [[Simulation data#The Planck Simulator | The Planck Simulator]] • [[Simulation data#Products delivered | Products delivered]] --><br />
##[[DatesObs|Dates of observations]]<br />
<!--- ############# ---><br />
#[[Software utilities|Software utilities]]<br />
##[[Unit conversion and Color correction|Unit conversion and Colour correction]] <br />
<!--- ############# ---><br />
#[[Operational data]]<br />
<!---##[[Thermal|Thermal and cooler system]]---><br />
##[[Survey history | Survey history data]]<br />
##[[Planck operational state history]]<br />
<!---##[[FOG|Fibre-optic gyro]]---><br />
##[[SREM|Space radiation environment monitor]]<br />
#[[Appendix]]<br />
##[[Glossary]]<br />
##[[List of acronyms]]<br />
[[Category:PSOBook]]</div>
Fdesert
https://wiki.cosmos.esa.int/planckpla2015/index.php?title=TOI_processing&diff=11203
TOI processing
2015-02-04T16:02:21Z
<p>Fdesert: /* Overview */</p>
<hr />
<div>==Overview==<br />
<br />
We describe here how the TOIs are processed in order to be used for map production. We do not repeat the general features of the pipeline, which are given in the HFI Data Processing paper {{PlanckPapers|planck2013-p03}} for PR1 and <span style="color:Red">Planck-2015-VII ref</span> for PR2. Here we give complementary explanations in some detail. The TOI of each bolometer is processed independently of the other bolometers, so as to keep the noise properties as uncorrelated as possible. The processing involves modifying the TOI itself for the conversion to absorbed power and the correction of glitch tails. It also adds a flag TOI that masks the TOI samples that are not to be projected onto maps for various reasons.<br />
<br />
== Input TOI ==<br />
<br />
The input TOI consists of the AC-modulated voltage output read out of each bolometer. The input has previously been decompressed, and converted from internal digital units to voltages via a constant factor. The TOI has a regular sampling at the acquisition frequency of <i>f</i><sub>acq</sub>=(180.373700&plusmn;0.000050)Hz. There are almost no missing data in the TOIs, except for a few hundred samples of 545 and 857 GHz TOIs, which were lost in the on-board compression due to saturation on the Galactic centre crossings.<br />
<br />
== General pipeline structure ==<br />
<br />
The following figure shows how the initial TOI is transformed and how flags are produced.<br />
<br />
[[File:HFI_4_4_2_ToiProc_ExplS.png|thumb|500px|center|A schematic of the TOI processing pipeline.]]<br />
<br />
== Output TOIs and products ==<br />
<br />
A TOI of clean calibrated samples and a combined flag TOI are the outputs of the processing. The clean calibrated TOI is calibrated so as to represent the instantaneous power absorbed by the detector, up to a constant (which will be determined by the mapmaking destriper). It is worth mentioning how the clean calibrated TOI is changed with respect to the input TOI, beyond the basic constant conversion factor from voltage to absorbed power. The demodulation stage allows one to obtain the demodulated bolometer voltage. The nonlinearity correction is a second-order polynomial correction based on the physical but static bolometer model. In order to avoid too much masking after glitches, a glitch tail is subtracted after an occurrence of a glitch in the TOI. The 4-K cooler lines are removed at a series of nine single temporal frequencies. Finally, the temporal response of the bolometer is deconvolved. This affects mostly the high-temporal frequency part of the TOI, although a small but significant low frequency tail (the long time response) is corrected too. Although flagged samples are not projected, their value still influences the valid samples. Hence interpolation procedures introduce some indirect modifications of the TOI.<br />
The flag TOI is a combination of a dozen flags with an OR logic. Only unflagged data are projected. The exhaustive list of flags is: CompressionError; NoData; SSO; UnstablePointing; Glitch; BoloPlateFluctuation; RTS; Jump; and PSBab. A complete qualification of the data is obtained at the ring level. If the TOI shows any anomalous behaviour on a given ring, this ring is discarded from projection. A special TOI is also produced as an input to the beam analysis for Mars, Jupiter, and Saturn.<br />
<br />
== Examples of clean TOIs ==<br />
<center><br />
<gallery heights=200px widths=300px perrow=3><br />
File:10_143_5_PBR_10800RING_LFER4_JC.jpg|Phase-binned signal (PBR) for six consecutive rings from 4000 to 4005.<br />
File:10_143_5_rcircles_LFER4_JC.jpg| Valid signal (up) and noise (bottom) for two consecutive circles of ring 4000.<br />
File:10_143_5_fft_LFER4_JC.jpg| Fourier power spectral density of the first 30 minutes of ring 4005 with signal (top) before (pink) and after (black) deconvolution and with noise (bottom) also shown.<br />
File:14_545_1_PBR_10800RING_LFER4_JC.jpg|Phase-binned signal (PBR) for six consecutive rings from 4000 to 4005.<br />
File:14_545_1_rcircles_LFER4_JC.jpg| Valid signal (up) and noise (bottom) for two consecutive circles of ring 4000.<br />
File:14_545_1_fft_LFER4_JC.jpg| Fourier power spectral density of the first 30 minutes of ring 4005, with signal (top) before (pink) and after (black) deconvolution and with noise (bottom) also shown.<br />
</gallery><br />
</center><br />
Samples of PBR, TOIs, and PSDs of all detectors are shown in [[Media:check_ring.pdf|this file]].<br />
<br />
== Trends in the output processing variables ==<br />
<br />
Here we show trends in the systematic effects that are dealt with in the TOI processing. The full impact of each of them is analysed in the [[HFI-Validation]] section.<br />
<br />
=== ADC baseline ===<br />
The following figure shows the ADC baseline that is used prior to demodulation (a constant offset is removed for clarity). This baseline is obtained by smoothing on an hourtimescale by block-averaging the undemodulated TOI on unflagged samples.<br />
<br />
[[File:baseline.jpg|thumb|500px|center|ADC baseline for all bolometers.]]<br />
<br />
=== Glitch statistics ===<br />
The glitch rate per channel is shown in the figure below. For details, see {{PlanckPapers|planck2013-p03e}}<br />
<br />
[[File:figIntermPaperGR.jpg|thumb|500px|center|Glitch rate evolution.]]<br />
<br />
The percentage of flagged data (mostly due to cosmic rays) at the ring level is shown in these examples. No smoothing was applied and only valid rings are shown.<br />
<center><br />
<gallery heights=300px widths=300px perrow=2><br />
File:group10_143_5.jpg | 143GHz bolometers.<br />
File:group14_545_1.jpg | 545GHz bolometers.<br />
</gallery><br />
'''Percentage of flagged data'''<br />
</center><br />
The complete set of plots is attached [[Media:PercentUnvalid2.pdf|here]].<br />
<br />
=== Thermal template for decorrelation ===<br />
[[File:T90.jpg|thumb|center|500px|Thermal template used for the decorrelation from bolometer plate temperature fluctuations.]]<br />
<br />
A simple linear decorrelation is performed using the two "dark" bolometers as a proxy of the bolometer plate temperature. Coupling coefficients were measured during the CPV phase.<br />
<br />
=== 4-K cooler lines variability ===<br />
The amplitude of the nine 4-K cooler lines at 10, 20, 30, 40, 50, 60, 70, 80, and 17Hz is shown (in aW) for two bolometers in the following figures. The trend is smoothed over 31 ring values after having discarded measurements done at a ring that is discarded for all bolometers.<br />
<center><br />
<gallery heights=300px widths=300px perrow=2><br />
File:10_143_5_4Klines.jpg | 143_5 bolometer<br />
File:14_545_1_4Klines.jpg | 545_1 bolometer<br />
</gallery><br />
'''Amplitude of the nine 4-K cooler lines'''<br />
</center><br />
<br />
The 4-K cooler line coefficients of all bolometers are shown in [[Media:Lines4K.pdf|this file]].<br />
<br />
The 4-K cooler lines project onto the maps only for a limited fraction of rings, the so-called "resonant rings." This is graphically shown in the following figure. For each ring (fixed pointing period), the spin rate is very stable at about 1rpm. From one ring to another, the spin frequency (shown as diamonds) changes around that value. The sky signal is imprinted at the corresponding spin frequency and its 5400 (60&times;90) harmonics. If one of the nine 4-K cooler lines happens to coincide with one of the spin frequency harmonics (a resonant ring), it will project a sine-wave systematic onto the maps. The horizontal coloured bars show the zone of influence of a particular 4-K line (labelled on the left side of the plot), when folded around 16.666mHz. When the spin frequency hits one of these zones, we have a resonant ring. The 4-K line coefficient is interpolated for this ring and an estimate of the systematic effect is subtracted from the TOI. Resonant rings are different for different 4-K lines. Note the two-level oscillation pattern of the spin frequency, which is due to the satellite attitude control system.<br />
<br />
[[File:spinfreq1.jpg|thumb|center|500px|Spin frequency and 4-K line zones of influence.]]<br />
<br />
=== Jump correction ===<br />
A piecewise constant value is removed from the TOI if a jump is detected. An example of a jump is shown in the following figure:<br />
<br />
[[File:jump_exe.png|thumb|500px|center|An example of a jump seen on the noise TOI.]]<br />
<br />
The number of jumps per day (all bolometers included) is shown in this figure.<br />
<br />
[[File:jumps_per_day.png|thumb|500px|center|Evolution of jump number during the mission.]]<br />
<br />
The jumps are uncorrelated from bolometer to bolometer. The total number of jumps detected in the nominal and full mission is shown below.<br />
<br />
[[File:jumps_per_bolometer.png|thumb|500px|center|Number of jumps per bolometer.]]<br />
<br />
<br />
=== Trends in noise and signal ===<br />
<center><br />
<gallery heights=300px widths=600px perrow=2 ><br />
File:10_143_5_smooth_Watt.jpg | 143_5 bolometer<br />
File:14_545_1_smooth_Watt.jpg | 545_1 bolometer<br />
</gallery><br />
<small>'''Signal (top) and noise (bottom) smoothed over 1 minute. All values falling in a discarded ring are not plotted.'''</small><br />
</center><br />
<br />
The smooth TOIs of all detectors are shown in [[Media:check_smooth.pdf|this file]].<br />
<br />
=== Noise stationarity ===<br />
This is not the final version but gives a good idea of power spectra at the detector level of noise TOIs. All PSDs can be seen in [[Media:v53_meanSpectra_bySurvey.pdf|this file]].<br />
<br />
The standard deviation per ring corrected for ring duration bias is given here, one per bolometer using only valid rings. No smoothing is applied (except a 31-point smoothing for the 545 and 857 GHz channels), but values for rings discarded for all bolometers (see below) are not used. The standard deviation is computed on samples valid for mapmaking that are also not affected by the Galaxy or the point-sources using the usual flags. Two examples are shown below.<br />
<br />
<center><br />
<gallery heights=300px widths=300px perrow=2><br />
File:stddev_group10_143_5.jpg | 143_5 bolometer.<br />
File:stddev_group_14_545_1.jpg | 545_1 bolometer.<br />
</gallery><br />
'''Standard deviation per ring'''<br />
</center><br />
<br />
The full series of plots is below.<br />
[[Media:StationarityInROIoutput2.pdf|Standard deviation of noise TOIs at the ring level.]]<br />
<br />
Note the presence for three bolometers of a two-level noise system. No correction has been made for that effect. See one example below.<br />
<br />
[[File:23_353_TwoLevel.jpg|thumb|500px|center|An example of a two-level noise system is seen in bolometer 23_353_3a]]<br />
<br />
<br />
An example of the higher-order statistics that are used to unveil rings affected by RTS problems is below.<br />
<br />
[[File:HFI_4_4_2_RTSexample13.jpg|thhumb|600px|center|Example of RTS detection]]<br />
<br />
=== Discarded rings ===<br />
<br />
Some rings are discarded (flagged) from further use (e.g., beam-making, and mapmaking) by using ring statistics (see above and {{PlanckPapers|planck2013-p03}}). For each statistic, we compare each ring value to the ring values averaged (RVA) over a large selection of rings (between 3000 and 21700). We also define the modified standard deviation (MSD) of a ring quantity as the standard deviation of that quantity over the rings that deviate by less than five nominal standard deviations. This truncation is necessary to be robust against extremely deviant rings.<br />
<br />
More specifically, a ring is discarded if it matches one of the following criteria:<br />
<br />
* the | mean-median | deviates from the RVA by more than fifteen times the MSD;<br />
* the standard deviation deviates from the RVA by more than -5 times the MSD (all cases corresponding to almost empty rings) or +15 times the MSD;<br />
* the Kolmogorov-Smirnov test deviates from the RVA by more than 15 times the MSD;<br />
* the ring duration is more than 90 min;<br />
* the ring is contaminated by RTS with an amplitude of more than one standard deviation of the noise.<br />
<br />
This last item concerns a few hundreds rings for three bolometers (44_217_6b, 71_217_8a, 74_857_4). Notice that two bolometers are completely discarded for maps, namely 55_545_3 and 70_143_8, which present RTS at all times.<br />
<br />
For the three first criteria, a visual inspection of the noise TOI at each of the incriminated rings has shown that all such anomalous rings are due to either a drift, a small jump in the TOI trend, or a sudden change of noise level, the origin of which is unknown at present.<br />
<br />
Once the list of discarded rings per bolometer is produced, a common list of discarded rings can be extracted for all bolometers (by using discarded rings for at least half the bolometers). Such rings correspond to identified phenomena, as can be seen in the following table.<br />
<br />
Furthermore, an isolated valid ring sandwiched between two common discarded rings becomes discarded as well.<br />
<br />
Table of "common discarded rings" of the nominal mission (rings 240-14723).<br />
<br />
<center><br />
{| class="wikitable" align="center" style="text-align:center" border="1" cellpadding="5" cellspacing="0" width=500px<br />
|- bgcolor="ffdead"<br />
! '''Cause''' !! '''Ring number'''<br />
|-<br />
|align="left"| Manoeuvre || 304 1312 3590 3611 3642 3922 4949 6379 8456 11328<br />
|-<br />
| align="left"| Sorption Cooler switchover || 11149 11150 11151 11152 <br />
|-<br />
| align="left"| Rings too long || 440 474 509 544 897 898 3589 13333 14627 14653 <br />
|-<br />
| align="left"| Star tracker switchover || 14628 14654 <br />
|-<br />
| align="left"| Massive glitch event || 7665 <br />
|-<br />
| align="left"| Solar flare || 11235 <br />
|}<br />
</center><br />
<br />
<br />
<br />
The following figure is a summary of the impact of the discarding process <i>for each bolometer</i> (the solid black line is the common discarded ring percentage). The outlier bolometers have some RTS problems, as mentionned above.<br />
<br />
<center><br />
<gallery heights=300 widths=600px perrow=4 caption=""><br />
File:Fraction_discarded_rings_v53.png | Fraction of discarded rings.<br />
File:Fraction_discarded_time_v53.png | Fraction of time due to discarded rings.<br />
</gallery><br />
</center><br />
<br />
=== Effective integration time ===<br />
The following figure summarizes the effective integration time per bolometer. For that purpose the number of unflagged samples in non-discarded pointing periods have been used within the nominal mission. The average value is about 335 days of effective integration time.<br />
<br />
[[File:HFInominal_IntegrationTime.jpg|thumb|500px|center|Effective integration time per bolometer.]]<br />
<br />
== Flag description ==<br />
<br />
==== Input flags ====<br />
<br />
The following flags are used as inputs to the TOI processing.<br />
<br />
; The point-source flag (PSflag) -<br />
: An earlier version of HFI point-source catalogue is read back in to flag TOIs, at a given frequency. In practice, 5&sigma; sources are masked within a radius of 1.3 FWHM (which is 9, 7, 5, 5, 5, 5 arcmin at 100,143,217,353,545,857 GHz) TBC.<br />
<br />
; The Galactic flag (Galflag) -<br />
: An earlier version of HFI maps is thresholded and apodized. The produced masks are read into flag TOIs. The retained threshold corresponds to a sky coverage of, respectively, 70, 70, 80, 90, 90, and 90% at 100,143, 217, 353, 545, and 857GHz.<br />
<br />
; Solar System Object flag -<br />
: For the TOI flag, Mars, Jupiter, and Saturn are flagged up to a radius of <i>N</i><sub>Beam</sub>= 2, 3, 3, 4, 4, and 4 times the fiducial SSO_FWHM, with SSO_FWHM= 9, 7, 5, 5, 5, and 5 arcmin at 100, 143, 217, 353, 545, and 857GHz, respectively.<br />
<br />
: As an input to the planet mask for maps, Mars, Jupiter, and Saturn are flagged with a radius computed as a coefficient depending on the planet (Factor_per_source) times <i>N</i><sub>Beam</sub> times SSO_FWHM, with Factor_per_source = 1.1, 2.25, and 1.25 for Mars, Jupiter, and Saturn, respectively, and <i>N</i><sub>Beam</sub> = 2.25, 4.25, 4.0, 5.0,.6.0, and 8.0 at 100, 143, 217, 353, 545, and 857GHz. This flag is called SSOflag4map.<br />
<br />
: A small trailing tail is added to the mask to take into account the non-deconvolution of the planet signal, which has been replaced by background values. The width of this tail is 10% of the main flag diameter. The number of samples which are additionnaly flagged are Factor_per_Source times AddSNafter with "trail" &times; (Factor_per_source)<sup>3</sup> samples, where trail = 10, 30, 20, 20, 30, and 40 at 100, 143, 217, 353, 545, and 857GHz, respectively.<br />
<br />
: Uranus and Neptune together with detected asteroids are masked by HFI. They are masked at the TOI level using an exclusion radius of 1.5 times SSO_FWHM. At 857GHz, 24 asteroids have been detected with HFI, namely 1Ceres, 2Pallas, 3Juno, 4Vesta, 7Iris, 8Flora, 9Metis, 10Hygiea, 11Parthenope, 12Victoria, 13Egeria, 14Irene, 15Eunomia, 16Psyche, 18Melpomene, 19Fortuna, 20Massalia, 29Amphitrite, 41Daphne, 45Eugenia, 52Europa, 88Thisbe, 704Interamnia, and 324Bamberga.<br />
<br />
<center><br />
<gallery heights=200px widths=200px perrow=4 caption=""><br />
File:SSOflag_10_143_5J.png | 143 SWB row, around Jupiter first crossing.<br />
File:SSOflag_10_143_5S.png | 143 SWB row, around Saturn.<br />
File:SSOflag_10_143_5M.png | 143 SWB row, around Mars.<br />
File:SSOflag_10_143_5a.png | 143 SWB row, random field in the ecliptic plane.<br />
File:SSOflag_25_857_1J.png | 545/857 row, around Jupiter first crossing.<br />
File:SSOflag_25_857_1S.png | 545/857 row, around Saturn.<br />
File:SSOflag_25_857_1M.png | 545/857 row, around Mars.<br />
File:SSOflag_25_857_1a.png | 545/857 row, random field in the ecliptic plane.<br />
</gallery><br />
Local maps showing the SSO flag. The colours correspond to the surveys involved in the nominal mission (green for Survey I, yellow for Survey 2, and red for Survey 3).<br />
</center><br />
<br />
=== Output flags ===<br />
<br />
A FlagTOIproc is produced by the TOI processing. It marks measurements which are not reliable for any of the following reasons.<br />
* Gap (no valid input data), enlarged by one sample on each side. This flags less than 0.00044% (respectively 0.00062%) of the nominal (respectively complete) mission. It is equivalent to less than 3 (respectively 8) minutes of data.<br />
* "Glitch" on dark bolometers: as the thermal template used for decorrelation is computed from these bolometer data, chunks of 1 minute-length data are discarded for all bolometers if at least 50% of the data for at least one dark bolometer are flagged during this time. This is efficient for flagging the data around the maximum of thermal events.<br />
* "Glitch" on individual bolometers : samples where the signal from a cosmic ray hit dominates the sky signal at more than 3&sigma; are discarded.<br />
* Jump with 100 samples flagged around the computed position of the jump, to take into account the error on this reconstructed position.<br />
<br />
The flag produced for the map making, called Total_flag, is therefore defined by<br />
* Total_flag = UnstablePointing Flag OR FlagTOIproc OR SSOflag4map OR SSOflag seen<br />
where FlagTOIproc = gap OR flag thermal template OR glitch OR jump<br />
and for PSB bolometers, FlagTOIproc_AB = FlagTOIproc_A OR FlagTOIproc_B.<br />
Note that the Total_flag is then identical for the A and B bolometers of a PSB pair.<br />
<br />
At the destriping stage, a more restricted flag, called Total_flag_PS, is used. It is defined by<br />
Total_flag_PS = Total_flag OR PS_flag.<br />
<br />
== References ==<br />
<References /><br />
<br />
<br />
<br />
<br />
[[Category:HFI data processing|002]]</div>
Fdesert
https://wiki.cosmos.esa.int/planckpla2015/index.php?title=ADC_correction&diff=11200
ADC correction
2015-02-04T16:01:13Z
<p>Fdesert: </p>
<hr />
<div>The ADC non linearity correction is affecting the modulated signal of the bolometers before 40 of these fast samples are averaged and transmitted to the ground. The TOI data delivered in the HFI products are made of these average values. The raw signal of a bolometer for one modulation period is transmitted only at sparce intervals. This results in a complicated process to remove the systematic effects associated with ADC non linearity on the modulated signal taking into account the 40 Hz parasitics associated with the 4K cooler drive electronics.<br />
<br />
The principles of the correction are detailed in the <span style="color:Red">Planck-2015-VII ref</span>. The correction cannot be done using only the TOI values but requires ancillary data from ground based tests and additional tests done during the LFI extension of the mission when the dilution cooler did not operate anymore.<br />
For this reason, there is no reason to detail more than what was described in the paper as the users cannot expect to improve this part of the processing.</div>
Fdesert
https://wiki.cosmos.esa.int/planckpla2015/index.php?title=Map-making&diff=11199
Map-making
2015-02-04T16:00:17Z
<p>Fdesert: /* Introduction */</p>
<hr />
<div>{{DISPLAYTITLE:Map-making and photometric calibration}}<br />
== Introduction ==<br />
<br />
This page will give an overview of the map-making and photometric calibration procedures used by the HFI DPC to build detector and frequency maps for the 2015 data release. <br />
They are described in <span style="color:Red">Planck-2015-VIII ref</span>.<br />
These have common elements with the tools used for the 2013 release that are described in {{PlanckPapers|planck2013-p03}} and {{PlanckPapers|planck2013-p03b}}. <br />
<br />
To build HFI maps, we use the destriping approximation, in which noise is assumed to decompose into two components : white noise plus low frequency drifts. Using the sky redundancy, the low frequency drifts are modelled as one constant, or offset, per pointing period. To speed up the ulterior processing we first build intermediate products, by taking advantage of redundancies : we average signal and detector orientation on healpix pixels visited during <br />
each fixed pointing period, which we call hereafter 'ring'. Detector's pointing are corrected for slow drifts and aberration (displacement on the sky indouced by the satellite's motion). This intermediate product is called HPR for healpix pixel ring. They have been constructed using the same map resolution as the final HFI products (corresponding to <math>N_{\mathrm side}</math>=2048). <br />
This new dataset is used as input in the following steps.<br />
<br />
== Photometric calibration ==<br />
<br />
=== Dipole calibration (100 to 353 GHz) ===<br />
<br />
For the 2015 data release, the HFI CMB channels were calibrated using the orbital dipole modulation. This time-variable anisotropy results from the motion of the spacecraft in the solar system, which is precisely measured. Thus it provides an absolute calibrator for orbital CMB missions. Its measurement is now used to calibrate HFI data thanks to the improvements in the timne stability of the data <br />
brought by the ADC non-linearities corrections and a better control for the detectors time response. <br />
<br />
Residual time response slow components are modeled as a dipole shifted by 90 degrees in phase, whose amplitude is fitted bolometer per bolometer. To mitigate residual systematics, we perform a simultaneous fit of the detectors gains on the orbital dipole. This amounts to solve the non-linear equation : <br />
<math> \displaystyle{d\ =\ g_i.(S+D) + O_i + n}<br />
\label{nlequat}</math><br />
<br />
where d is s the detector measurement, D the total dipole component, n the (white) noise, and both S the sky signal, g the detector gain, O the offset (for ring no i) are the unknowns to be determined. <br />
This is done by linearizing it to look for gains and sky variations, and iterating by updating the approximate sky and gains. <br />
<br />
<br />
=== Higher frequency calibration (545 and 857 GHz) ===<br />
<br />
We therefore the sub-mm channels' calibration for the 2015 Planck data release from the comparison of measurements of the Neptune and Uranus fluxes (with aperture photometry) with their expectations from the Moreno et al model of their atmospheres' emission. This procedure is justified, since for both planets, at the lower frequencies (100-353 Ghz), the fluxes we recover are in agreement within ~ +/-5\% with what is expected from the planet spectral model, and the HFI detector's band-passes.<br />
<br />
=== Zero levels === <br />
We determined zero-level for the released maps in selected regions of the sky where dust emissions are low and well correlated with HI. We may thus estimate and subtract dust emissions using the HI template, and CMB from a Planck component-separated template. The remaining astrophysical zero level is that of CIB. By imposing that the level we find is equal to that of the CIB model of Bethermin et al, we set the zero level of our maps.<br />
<br />
<br />
== Building of maps == <br />
<br />
Using the photometric calibration parameters, we build maps in two steps : <br />
<br />
* we determine the destriping offsets using the full mission data for all detectors of a given frequency<br />
* we build the maps, using these offsets, by inverting the photometric equation :<br />
<br />
<math> \displaystyle{d_i = g(I^p+\eta [Q^p cos(2\psi_i) + U^p sin(2\psi_i)]) + n}<br />
\label{photeq}</math><br />
where d is the destriped and calibrated signal at the HPR level. Detector's data are combined with an inverse noise weights derived from each detector's NEP. Q and U maps are build whenever possible. We propagate the white noise by building the 3x3 (or 1x1 if only I is reconstructed) covariance matrices in each pixel. <br />
At each frequency we build maps combining all detectors and independent detector sets. <br />
We use the offsets build for the full mission for building maps for each scan survey, year (sombineation of surveys 1 and 2 or 3 and 4 respectively) and for the full, nominal mission duration and its two halves. <br />
We also build maps from the two independent halves of each rings. Altogether, more than 8000 maps are built at each release. <br />
<br />
HPR and Maps are built in galactic coordinates.<br />
<br />
== Noise properties ==<br />
<br />
Map noise properties can be evaluated using several methods, thanks to the high level of observation redundancies. <br />
We can use the maps built from the difference between the first and second half of each rings, or compare individual sky scans, years, half-mission or independent detector sets with each other. <br />
Some of these tests are described in <span style="color:Red">Planck-2015-VIII ref</span>.<br />
<br />
Low resolution (nside = 8, 16) pixel-to-pixel noise covariance matrices are build using an analytic approach from the measured noise power spectra. <span style="color:Red">CONFIRM THIS NO: THESE MATRICES ARE NOT DELIVERED</span>.<br />
<br />
== Zodiacal light correction ==<br />
<br />
At the highest Planck frequencies, zodiacal light emission is visible in a survey difference map:<br />
<br />
[[File:Z857SurveyJackknifeWiZodi.png|500px|thumb|center|2013 Release 857 GHz Survey 2 - Survey 1 Difference]]<br />
<br />
This map is a difference between the 857 GHz Survey 2 map and the 857 GHz Survey 1 map. This difference effectively removes Galactic and other emissions which originate far from Planck. As the Solar elongation is different for measurements of the same point on the sky for the two surveys, we see zodiacal light emission, while all emission from further sources is removed. The zodiacal light emission follows the Ecliptic plane, which starts at the lower left of the image, then crosses the center of the plot towards the upper right. Note that the "arcs" at the top and bottom of the image are images of the Galactic center in the Far Sidelobes, which are discussed in the section below. Similar plots for other HFI frequencies, for maps both before and after removal, are shown [[beforeAndAfterSurveyDifferences|here]].<br />
<br />
For the 2015 Planck release, zodiacal light emission is removed from all HFI channels. The general procedure is described in {{PlanckPapers|planck2013-p03}}, but a synopsis of the procedure is as follows:<br />
* During each survey, a large fraction of the sky has observations which all fall within a week of each other. That is, during a single survey, most pixels are observed during a short, well-defined period. The contribution from zodiacal light emission to the total brightness seen, then, is well defined. <br />
* We use the the COBE model of the zodiacal light emission to make predictions for this zodiacal light emission for those pixels observed over a span of one week or less. The templates from the COBE model are shown [[COBEZodiModelTemplates|here]].<br />
* We fit the survey difference maps with these model templates to estimate the emissivity of each zodi component at the Planck wavelengths. The results of these fits at each frequency are given in <span style="color:red">Planck-2015-VIII</span>. <br />
* We reconstruct each ring of the the full mission using the combination of the COBE geometric model with the emissivities determined above. <br />
* We remove the reconstruction above from each ring of data. <br />
* We then make maps as described previously in this section.<br />
<br />
== Far SideLobes (FSL)==<br />
<br />
Contrary to the 2013 Planck release, Far Sidelobes were not removed from the HFI data for the 2015 Planck release. <br />
The change of the gain due to neglect of the far sidelobes is calculated by fitting the dipole to full timeline simulations of the dipole convolved by the FSL. The correction factors applied to the data are 0.09 % at 100 GHz, 0.05 % at 143 GHz, 0.04 % at 217 GHz and negligible at 353 GHz. Corrections were not made at 545 and 857 GHz <span style="color:red">(see Planck-2015-VIII for details)</span>.<br />
<br />
== CO maps ==<br />
<br />
Carbon monoxide rotational transition line emission is present in all HFI bands but for the 143 GHz<br />
channel. It is especially significant in the 100, 217 and 353 GHz channels (due to the 115<br />
(1-0), 230 (2-1) and 345 GHz (3-2) CO transitions). This emission comes essentially from<br />
the Galactic interstellar medium and is mainly located at low and intermediate Galactic<br />
latitudes. Three approaches (summarised below) have been used to extract CO velocity<br />
integrated emission maps from HFI maps and to generate the CO products. See<br />
Planck-2013-XIII and Planck-2015-X for a full description.<br />
<br />
*Type 1 product: it is based on a single channel approach using the fact that each CO<br />
line has a slightly different transmission in each bolometer at a given frequency channel.<br />
From this, the J=1-0, J=2-1 and J=3-2 CO lines can be extracted independently. As this<br />
approach is based on individual bolometer maps of a single channel, the resulting<br />
Signal-to-Noise ratio (SNR) is relatively low. The benefit, however, is that these maps do<br />
not suffer from contamination from other HFI channels (as is the case for the other<br />
approaches) and are more reliable, especially in the Galactic plane. <br />
<br />
*Type 2 product: this product is obtained using a multi-frequency approach. Three<br />
frequency channel maps are combined to extract the J=1-0 (using the 100, 143 and 353<br />
GHz channels) and J=2-1 (using the 143, 217 and 353 GHz channels) CO maps. As<br />
frequency maps are combined, the spectral behaviour of other foregrounds influences<br />
the result. The two Type 2 CO maps produced in this way have a higher SNR than the<br />
Type 1 maps at the cost of a larger residual contamination from other diffuse<br />
foregrounds.<br />
<br />
* Type 3 product: no Type 3 product (as defined in 2013) has been produced for the<br />
2015 release. Instead, it is superseded by a high-resolution CO(2-1) map (FWHM=7.5<br />
arcmin) produced by the Commander component separation pipeline. Note that low<br />
resolution (FWHM=1 deg) CO maps of the three lines have also been produced using<br />
Commander. See [le lien vers la section avant-plans de Commander] for a complete<br />
description.<br />
<br />

The 2015 Type 1 and Type 2 CO maps have been produced using the same procedure as<br />
for the 2013 results. Very similar to their 2013 counterparts, the 2015 maps benefit from<br />
an increased SNR due to the use of the full, rather than nominal, mission data. <br />

Characteristics of the released maps are the following. We provide Healpix maps with<br />
Nside=2048. For one transition, the CO velocity-integrated line signal map is given in<br />
K_RJ.km/s units. A conversion factor from this unit to the native unit of HFI maps (K_CMB)<br />
is provided in the header of the data files and in the RIMO. Four maps are given per<br />
transition and per type:<br />

* The signal map<br />
* The standard deviation map (same unit as the signal),<br />
* A null test noise map (same unit as the signal) with similar statistical properties. It is<br />
made out of half the difference of half-ring maps.<br />
* A mask map (0B or 1B) giving the regions (1B) where the CO measurement is not<br />
reliable because of some severe identified foreground contamination. <br />

All products of a given type belong to a single file. Type 1 products have the native HFI<br />
resolution i.e. approximately 10, 5 and 5 arcminutes for the CO 1-0, 2-1, 3-2 transitions<br />
respectively. Type 2 products have a 15 arcminute resolution.<br />

We refer the reader to [le lien vers la section avant-plans de Commander] for a<br />
description and characteristics of the Commander CO products.<br />
<br />
== Map validation ==<br />
<br />
Several validations of HFI maps are described in <span style="color:Red">Planck-2015-VII ref</span> and in <span style="color:Red">Planck-2015-VIII ref</span>. <br />
<br />
Further checks are presented in the <span style="color:Red"> likelihood, params, comsep and commander papers refs </span>.<br />
<br />
== References ==<br />
<References /><br />
<br />
<br />
[[Category:HFI data processing|004]]</div>
Fdesert
https://wiki.cosmos.esa.int/planckpla2015/index.php?title=Map-making&diff=11197
Map-making
2015-02-04T15:57:45Z
<p>Fdesert: /* Far SideLobes (FSL) */</p>
<hr />
<div>{{DISPLAYTITLE:Map-making and photometric calibration}}<br />
== Introduction ==<br />
<br />
This page will give an overview of the map-making and photometric calibration procedures used by the HFI DPC to build detector and frequency maps for the 2015 data release. <br />
They are described in <span style="color:Red">A09 ref</span>.<br />
These have common elements with the tools used for the 2013 release that are described in {{PlanckPapers|planck2013-p03}} and {{PlanckPapers|planck2013-p03b}}. <br />
<br />
To build HFI maps, we use the destriping approximation, in which noise is assumed to decompose into two components : white noise plus low frequency drifts. Using the sky redundancy, the low frequency drifts are modelled as one constant, or offset, per pointing period. To speed up the ulterior processing we first build intermediate products, by taking advantage of redundancies : we average signal and detector orientation on healpix pixels visited during <br />
each fixed pointing period, which we call hereafter 'ring'. Detector's pointing are corrected for slow drifts and aberration (displacement on the sky indouced by the satellite's motion). This intermediate product is called HPR for healpix pixel ring. They have been constructed using the same map resolution as the final HFI products (corresponding to <math>N_{\mathrm side}</math>=2048). <br />
This new dataset is used as input in the following steps.<br />
<br />
== Photometric calibration ==<br />
<br />
=== Dipole calibration (100 to 353 GHz) ===<br />
<br />
For the 2015 data release, the HFI CMB channels were calibrated using the orbital dipole modulation. This time-variable anisotropy results from the motion of the spacecraft in the solar system, which is precisely measured. Thus it provides an absolute calibrator for orbital CMB missions. Its measurement is now used to calibrate HFI data thanks to the improvements in the timne stability of the data <br />
brought by the ADC non-linearities corrections and a better control for the detectors time response. <br />
<br />
Residual time response slow components are modeled as a dipole shifted by 90 degrees in phase, whose amplitude is fitted bolometer per bolometer. To mitigate residual systematics, we perform a simultaneous fit of the detectors gains on the orbital dipole. This amounts to solve the non-linear equation : <br />
<math> \displaystyle{d\ =\ g_i.(S+D) + O_i + n}<br />
\label{nlequat}</math><br />
<br />
where d is s the detector measurement, D the total dipole component, n the (white) noise, and both S the sky signal, g the detector gain, O the offset (for ring no i) are the unknowns to be determined. <br />
This is done by linearizing it to look for gains and sky variations, and iterating by updating the approximate sky and gains. <br />
<br />
<br />
=== Higher frequency calibration (545 and 857 GHz) ===<br />
<br />
We therefore the sub-mm channels' calibration for the 2015 Planck data release from the comparison of measurements of the Neptune and Uranus fluxes (with aperture photometry) with their expectations from the Moreno et al model of their atmospheres' emission. This procedure is justified, since for both planets, at the lower frequencies (100-353 Ghz), the fluxes we recover are in agreement within ~ +/-5\% with what is expected from the planet spectral model, and the HFI detector's band-passes.<br />
<br />
=== Zero levels === <br />
We determined zero-level for the released maps in selected regions of the sky where dust emissions are low and well correlated with HI. We may thus estimate and subtract dust emissions using the HI template, and CMB from a Planck component-separated template. The remaining astrophysical zero level is that of CIB. By imposing that the level we find is equal to that of the CIB model of Bethermin et al, we set the zero level of our maps.<br />
<br />
<br />
== Building of maps == <br />
<br />
Using the photometric calibration parameters, we build maps in two steps : <br />
<br />
* we determine the destriping offsets using the full mission data for all detectors of a given frequency<br />
* we build the maps, using these offsets, by inverting the photometric equation :<br />
<br />
<math> \displaystyle{d_i = g(I^p+\eta [Q^p cos(2\psi_i) + U^p sin(2\psi_i)]) + n}<br />
\label{photeq}</math><br />
where d is the destriped and calibrated signal at the HPR level. Detector's data are combined with an inverse noise weights derived from each detector's NEP. Q and U maps are build whenever possible. We propagate the white noise by building the 3x3 (or 1x1 if only I is reconstructed) covariance matrices in each pixel. <br />
At each frequency we build maps combining all detectors and independent detector sets. <br />
We use the offsets build for the full mission for building maps for each scan survey, year (sombineation of surveys 1 and 2 or 3 and 4 respectively) and for the full, nominal mission duration and its two halves. <br />
We also build maps from the two independent halves of each rings. Altogether, more than 8000 maps are built at each release. <br />
<br />
HPR and Maps are built in galactic coordinates.<br />
<br />
== Noise properties ==<br />
<br />
Map noise properties can be evaluated using several methods, thanks to the high level of observation redundancies. <br />
We can use the maps built from the difference between the first and second half of each rings, or compare individual sky scans, years, half-mission or independent detector sets with each other. <br />
Some of these tests are described in <span style="color:Red">Planck-2015-VIII ref</span>.<br />
<br />
Low resolution (nside = 8, 16) pixel-to-pixel noise covariance matrices are build using an analytic approach from the measured noise power spectra. <span style="color:Red">CONFIRM THIS NO: THESE MATRICES ARE NOT DELIVERED</span>.<br />
<br />
== Zodiacal light correction ==<br />
<br />
At the highest Planck frequencies, zodiacal light emission is visible in a survey difference map:<br />
<br />
[[File:Z857SurveyJackknifeWiZodi.png|500px|thumb|center|2013 Release 857 GHz Survey 2 - Survey 1 Difference]]<br />
<br />
This map is a difference between the 857 GHz Survey 2 map and the 857 GHz Survey 1 map. This difference effectively removes Galactic and other emissions which originate far from Planck. As the Solar elongation is different for measurements of the same point on the sky for the two surveys, we see zodiacal light emission, while all emission from further sources is removed. The zodiacal light emission follows the Ecliptic plane, which starts at the lower left of the image, then crosses the center of the plot towards the upper right. Note that the "arcs" at the top and bottom of the image are images of the Galactic center in the Far Sidelobes, which are discussed in the section below. Similar plots for other HFI frequencies, for maps both before and after removal, are shown [[beforeAndAfterSurveyDifferences|here]].<br />
<br />
For the 2015 Planck release, zodiacal light emission is removed from all HFI channels. The general procedure is described in {{PlanckPapers|planck2013-p03}}, but a synopsis of the procedure is as follows:<br />
* During each survey, a large fraction of the sky has observations which all fall within a week of each other. That is, during a single survey, most pixels are observed during a short, well-defined period. The contribution from zodiacal light emission to the total brightness seen, then, is well defined. <br />
* We use the the COBE model of the zodiacal light emission to make predictions for this zodiacal light emission for those pixels observed over a span of one week or less. The templates from the COBE model are shown [[COBEZodiModelTemplates|here]].<br />
* We fit the survey difference maps with these model templates to estimate the emissivity of each zodi component at the Planck wavelengths. The results of these fits at each frequency are given in <span style="color:red">Planck-2015-VIII</span>. <br />
* We reconstruct each ring of the the full mission using the combination of the COBE geometric model with the emissivities determined above. <br />
* We remove the reconstruction above from each ring of data. <br />
* We then make maps as described previously in this section.<br />
<br />
== Far SideLobes (FSL)==<br />
<br />
Contrary to the 2013 Planck release, Far Sidelobes were not removed from the HFI data for the 2015 Planck release. <br />
The change of the gain due to neglect of the far sidelobes is calculated by fitting the dipole to full timeline simulations of the dipole convolved by the FSL. The correction factors applied to the data are 0.09 % at 100 GHz, 0.05 % at 143 GHz, 0.04 % at 217 GHz and negligible at 353 GHz. Corrections were not made at 545 and 857 GHz <span style="color:red">(see Planck-2015-VIII for details)</span>.<br />
<br />
== CO maps ==<br />
<br />
Carbon monoxide rotational transition line emission is present in all HFI bands but for the 143 GHz<br />
channel. It is especially significant in the 100, 217 and 353 GHz channels (due to the 115<br />
(1-0), 230 (2-1) and 345 GHz (3-2) CO transitions). This emission comes essentially from<br />
the Galactic interstellar medium and is mainly located at low and intermediate Galactic<br />
latitudes. Three approaches (summarised below) have been used to extract CO velocity<br />
integrated emission maps from HFI maps and to generate the CO products. See<br />
Planck-2013-XIII and Planck-2015-X for a full description.<br />
<br />
*Type 1 product: it is based on a single channel approach using the fact that each CO<br />
line has a slightly different transmission in each bolometer at a given frequency channel.<br />
From this, the J=1-0, J=2-1 and J=3-2 CO lines can be extracted independently. As this<br />
approach is based on individual bolometer maps of a single channel, the resulting<br />
Signal-to-Noise ratio (SNR) is relatively low. The benefit, however, is that these maps do<br />
not suffer from contamination from other HFI channels (as is the case for the other<br />
approaches) and are more reliable, especially in the Galactic plane. <br />
<br />
*Type 2 product: this product is obtained using a multi-frequency approach. Three<br />
frequency channel maps are combined to extract the J=1-0 (using the 100, 143 and 353<br />
GHz channels) and J=2-1 (using the 143, 217 and 353 GHz channels) CO maps. As<br />
frequency maps are combined, the spectral behaviour of other foregrounds influences<br />
the result. The two Type 2 CO maps produced in this way have a higher SNR than the<br />
Type 1 maps at the cost of a larger residual contamination from other diffuse<br />
foregrounds.<br />
<br />
* Type 3 product: no Type 3 product (as defined in 2013) has been produced for the<br />
2015 release. Instead, it is superseded by a high-resolution CO(2-1) map (FWHM=7.5<br />
arcmin) produced by the Commander component separation pipeline. Note that low<br />
resolution (FWHM=1 deg) CO maps of the three lines have also been produced using<br />
Commander. See [le lien vers la section avant-plans de Commander] for a complete<br />
description.<br />
<br />

The 2015 Type 1 and Type 2 CO maps have been produced using the same procedure as<br />
for the 2013 results. Very similar to their 2013 counterparts, the 2015 maps benefit from<br />
an increased SNR due to the use of the full, rather than nominal, mission data. <br />

Characteristics of the released maps are the following. We provide Healpix maps with<br />
Nside=2048. For one transition, the CO velocity-integrated line signal map is given in<br />
K_RJ.km/s units. A conversion factor from this unit to the native unit of HFI maps (K_CMB)<br />
is provided in the header of the data files and in the RIMO. Four maps are given per<br />
transition and per type:<br />

* The signal map<br />
* The standard deviation map (same unit as the signal),<br />
* A null test noise map (same unit as the signal) with similar statistical properties. It is<br />
made out of half the difference of half-ring maps.<br />
* A mask map (0B or 1B) giving the regions (1B) where the CO measurement is not<br />
reliable because of some severe identified foreground contamination. <br />

All products of a given type belong to a single file. Type 1 products have the native HFI<br />
resolution i.e. approximately 10, 5 and 5 arcminutes for the CO 1-0, 2-1, 3-2 transitions<br />
respectively. Type 2 products have a 15 arcminute resolution.<br />

We refer the reader to [le lien vers la section avant-plans de Commander] for a<br />
description and characteristics of the Commander CO products.<br />
<br />
== Map validation ==<br />
<br />
Several validations of HFI maps are described in <span style="color:Red">Planck-2015-VII ref</span> and in <span style="color:Red">Planck-2015-VIII ref</span>. <br />
<br />
Further checks are presented in the <span style="color:Red"> likelihood, params, comsep and commander papers refs </span>.<br />
<br />
== References ==<br />
<References /><br />
<br />
<br />
[[Category:HFI data processing|004]]</div>
Fdesert
https://wiki.cosmos.esa.int/planckpla2015/index.php?title=Map-making&diff=11196
Map-making
2015-02-04T15:57:02Z
<p>Fdesert: /* Noise properties */</p>
<hr />
<div>{{DISPLAYTITLE:Map-making and photometric calibration}}<br />
== Introduction ==<br />
<br />
This page will give an overview of the map-making and photometric calibration procedures used by the HFI DPC to build detector and frequency maps for the 2015 data release. <br />
They are described in <span style="color:Red">A09 ref</span>.<br />
These have common elements with the tools used for the 2013 release that are described in {{PlanckPapers|planck2013-p03}} and {{PlanckPapers|planck2013-p03b}}. <br />
<br />
To build HFI maps, we use the destriping approximation, in which noise is assumed to decompose into two components : white noise plus low frequency drifts. Using the sky redundancy, the low frequency drifts are modelled as one constant, or offset, per pointing period. To speed up the ulterior processing we first build intermediate products, by taking advantage of redundancies : we average signal and detector orientation on healpix pixels visited during <br />
each fixed pointing period, which we call hereafter 'ring'. Detector's pointing are corrected for slow drifts and aberration (displacement on the sky indouced by the satellite's motion). This intermediate product is called HPR for healpix pixel ring. They have been constructed using the same map resolution as the final HFI products (corresponding to <math>N_{\mathrm side}</math>=2048). <br />
This new dataset is used as input in the following steps.<br />
<br />
== Photometric calibration ==<br />
<br />
=== Dipole calibration (100 to 353 GHz) ===<br />
<br />
For the 2015 data release, the HFI CMB channels were calibrated using the orbital dipole modulation. This time-variable anisotropy results from the motion of the spacecraft in the solar system, which is precisely measured. Thus it provides an absolute calibrator for orbital CMB missions. Its measurement is now used to calibrate HFI data thanks to the improvements in the timne stability of the data <br />
brought by the ADC non-linearities corrections and a better control for the detectors time response. <br />
<br />
Residual time response slow components are modeled as a dipole shifted by 90 degrees in phase, whose amplitude is fitted bolometer per bolometer. To mitigate residual systematics, we perform a simultaneous fit of the detectors gains on the orbital dipole. This amounts to solve the non-linear equation : <br />
<math> \displaystyle{d\ =\ g_i.(S+D) + O_i + n}<br />
\label{nlequat}</math><br />
<br />
where d is s the detector measurement, D the total dipole component, n the (white) noise, and both S the sky signal, g the detector gain, O the offset (for ring no i) are the unknowns to be determined. <br />
This is done by linearizing it to look for gains and sky variations, and iterating by updating the approximate sky and gains. <br />
<br />
<br />
=== Higher frequency calibration (545 and 857 GHz) ===<br />
<br />
We therefore the sub-mm channels' calibration for the 2015 Planck data release from the comparison of measurements of the Neptune and Uranus fluxes (with aperture photometry) with their expectations from the Moreno et al model of their atmospheres' emission. This procedure is justified, since for both planets, at the lower frequencies (100-353 Ghz), the fluxes we recover are in agreement within ~ +/-5\% with what is expected from the planet spectral model, and the HFI detector's band-passes.<br />
<br />
=== Zero levels === <br />
We determined zero-level for the released maps in selected regions of the sky where dust emissions are low and well correlated with HI. We may thus estimate and subtract dust emissions using the HI template, and CMB from a Planck component-separated template. The remaining astrophysical zero level is that of CIB. By imposing that the level we find is equal to that of the CIB model of Bethermin et al, we set the zero level of our maps.<br />
<br />
<br />
== Building of maps == <br />
<br />
Using the photometric calibration parameters, we build maps in two steps : <br />
<br />
* we determine the destriping offsets using the full mission data for all detectors of a given frequency<br />
* we build the maps, using these offsets, by inverting the photometric equation :<br />
<br />
<math> \displaystyle{d_i = g(I^p+\eta [Q^p cos(2\psi_i) + U^p sin(2\psi_i)]) + n}<br />
\label{photeq}</math><br />
where d is the destriped and calibrated signal at the HPR level. Detector's data are combined with an inverse noise weights derived from each detector's NEP. Q and U maps are build whenever possible. We propagate the white noise by building the 3x3 (or 1x1 if only I is reconstructed) covariance matrices in each pixel. <br />
At each frequency we build maps combining all detectors and independent detector sets. <br />
We use the offsets build for the full mission for building maps for each scan survey, year (sombineation of surveys 1 and 2 or 3 and 4 respectively) and for the full, nominal mission duration and its two halves. <br />
We also build maps from the two independent halves of each rings. Altogether, more than 8000 maps are built at each release. <br />
<br />
HPR and Maps are built in galactic coordinates.<br />
<br />
== Noise properties ==<br />
<br />
Map noise properties can be evaluated using several methods, thanks to the high level of observation redundancies. <br />
We can use the maps built from the difference between the first and second half of each rings, or compare individual sky scans, years, half-mission or independent detector sets with each other. <br />
Some of these tests are described in <span style="color:Red">Planck-2015-VIII ref</span>.<br />
<br />
Low resolution (nside = 8, 16) pixel-to-pixel noise covariance matrices are build using an analytic approach from the measured noise power spectra. <span style="color:Red">CONFIRM THIS NO: THESE MATRICES ARE NOT DELIVERED</span>.<br />
<br />
== Zodiacal light correction ==<br />
<br />
At the highest Planck frequencies, zodiacal light emission is visible in a survey difference map:<br />
<br />
[[File:Z857SurveyJackknifeWiZodi.png|500px|thumb|center|2013 Release 857 GHz Survey 2 - Survey 1 Difference]]<br />
<br />
This map is a difference between the 857 GHz Survey 2 map and the 857 GHz Survey 1 map. This difference effectively removes Galactic and other emissions which originate far from Planck. As the Solar elongation is different for measurements of the same point on the sky for the two surveys, we see zodiacal light emission, while all emission from further sources is removed. The zodiacal light emission follows the Ecliptic plane, which starts at the lower left of the image, then crosses the center of the plot towards the upper right. Note that the "arcs" at the top and bottom of the image are images of the Galactic center in the Far Sidelobes, which are discussed in the section below. Similar plots for other HFI frequencies, for maps both before and after removal, are shown [[beforeAndAfterSurveyDifferences|here]].<br />
<br />
For the 2015 Planck release, zodiacal light emission is removed from all HFI channels. The general procedure is described in {{PlanckPapers|planck2013-p03}}, but a synopsis of the procedure is as follows:<br />
* During each survey, a large fraction of the sky has observations which all fall within a week of each other. That is, during a single survey, most pixels are observed during a short, well-defined period. The contribution from zodiacal light emission to the total brightness seen, then, is well defined. <br />
* We use the the COBE model of the zodiacal light emission to make predictions for this zodiacal light emission for those pixels observed over a span of one week or less. The templates from the COBE model are shown [[COBEZodiModelTemplates|here]].<br />
* We fit the survey difference maps with these model templates to estimate the emissivity of each zodi component at the Planck wavelengths. The results of these fits at each frequency are given in <span style="color:red">Planck-2015-VIII</span>. <br />
* We reconstruct each ring of the the full mission using the combination of the COBE geometric model with the emissivities determined above. <br />
* We remove the reconstruction above from each ring of data. <br />
* We then make maps as described previously in this section.<br />
<br />
== Far SideLobes (FSL)==<br />
<br />
Contrary to the 2013 Planck release, Far Sidelobes were not removed from the HFI data for the 2015 Planck release. <br />
The change of the gain due to neglect of the far sidelobes is calculated by fitting the dipole to full timeline simulations of the dipole convolved by the FSL. The correction factors applied to the data are 0.09 % at 100 GHz, 0.05 % at 143 GHz, 0.04 % at 217 GHz and negligible at 353 GHz. Corrections were not made at 545 and 857 GHz <span style="color:red">(see the 2015 Planck/HFI Map-Making paper, A09, for details)</span>.<br />
<br />
== CO maps ==<br />
<br />
Carbon monoxide rotational transition line emission is present in all HFI bands but for the 143 GHz<br />
channel. It is especially significant in the 100, 217 and 353 GHz channels (due to the 115<br />
(1-0), 230 (2-1) and 345 GHz (3-2) CO transitions). This emission comes essentially from<br />
the Galactic interstellar medium and is mainly located at low and intermediate Galactic<br />
latitudes. Three approaches (summarised below) have been used to extract CO velocity<br />
integrated emission maps from HFI maps and to generate the CO products. See<br />
Planck-2013-XIII and Planck-2015-X for a full description.<br />
<br />
*Type 1 product: it is based on a single channel approach using the fact that each CO<br />
line has a slightly different transmission in each bolometer at a given frequency channel.<br />
From this, the J=1-0, J=2-1 and J=3-2 CO lines can be extracted independently. As this<br />
approach is based on individual bolometer maps of a single channel, the resulting<br />
Signal-to-Noise ratio (SNR) is relatively low. The benefit, however, is that these maps do<br />
not suffer from contamination from other HFI channels (as is the case for the other<br />
approaches) and are more reliable, especially in the Galactic plane. <br />
<br />
*Type 2 product: this product is obtained using a multi-frequency approach. Three<br />
frequency channel maps are combined to extract the J=1-0 (using the 100, 143 and 353<br />
GHz channels) and J=2-1 (using the 143, 217 and 353 GHz channels) CO maps. As<br />
frequency maps are combined, the spectral behaviour of other foregrounds influences<br />
the result. The two Type 2 CO maps produced in this way have a higher SNR than the<br />
Type 1 maps at the cost of a larger residual contamination from other diffuse<br />
foregrounds.<br />
<br />
* Type 3 product: no Type 3 product (as defined in 2013) has been produced for the<br />
2015 release. Instead, it is superseded by a high-resolution CO(2-1) map (FWHM=7.5<br />
arcmin) produced by the Commander component separation pipeline. Note that low<br />
resolution (FWHM=1 deg) CO maps of the three lines have also been produced using<br />
Commander. See [le lien vers la section avant-plans de Commander] for a complete<br />
description.<br />
<br />

The 2015 Type 1 and Type 2 CO maps have been produced using the same procedure as<br />
for the 2013 results. Very similar to their 2013 counterparts, the 2015 maps benefit from<br />
an increased SNR due to the use of the full, rather than nominal, mission data. <br />

Characteristics of the released maps are the following. We provide Healpix maps with<br />
Nside=2048. For one transition, the CO velocity-integrated line signal map is given in<br />
K_RJ.km/s units. A conversion factor from this unit to the native unit of HFI maps (K_CMB)<br />
is provided in the header of the data files and in the RIMO. Four maps are given per<br />
transition and per type:<br />

* The signal map<br />
* The standard deviation map (same unit as the signal),<br />
* A null test noise map (same unit as the signal) with similar statistical properties. It is<br />
made out of half the difference of half-ring maps.<br />
* A mask map (0B or 1B) giving the regions (1B) where the CO measurement is not<br />
reliable because of some severe identified foreground contamination. <br />

All products of a given type belong to a single file. Type 1 products have the native HFI<br />
resolution i.e. approximately 10, 5 and 5 arcminutes for the CO 1-0, 2-1, 3-2 transitions<br />
respectively. Type 2 products have a 15 arcminute resolution.<br />

We refer the reader to [le lien vers la section avant-plans de Commander] for a<br />
description and characteristics of the Commander CO products.<br />
<br />
== Map validation ==<br />
<br />
Several validations of HFI maps are described in <span style="color:Red">Planck-2015-VII ref</span> and in <span style="color:Red">Planck-2015-VIII ref</span>. <br />
<br />
Further checks are presented in the <span style="color:Red"> likelihood, params, comsep and commander papers refs </span>.<br />
<br />
== References ==<br />
<References /><br />
<br />
<br />
[[Category:HFI data processing|004]]</div>
Fdesert
https://wiki.cosmos.esa.int/planckpla2015/index.php?title=Map-making&diff=11195
Map-making
2015-02-04T15:54:49Z
<p>Fdesert: /* Zodiacal light correction */</p>
<hr />
<div>{{DISPLAYTITLE:Map-making and photometric calibration}}<br />
== Introduction ==<br />
<br />
This page will give an overview of the map-making and photometric calibration procedures used by the HFI DPC to build detector and frequency maps for the 2015 data release. <br />
They are described in <span style="color:Red">A09 ref</span>.<br />
These have common elements with the tools used for the 2013 release that are described in {{PlanckPapers|planck2013-p03}} and {{PlanckPapers|planck2013-p03b}}. <br />
<br />
To build HFI maps, we use the destriping approximation, in which noise is assumed to decompose into two components : white noise plus low frequency drifts. Using the sky redundancy, the low frequency drifts are modelled as one constant, or offset, per pointing period. To speed up the ulterior processing we first build intermediate products, by taking advantage of redundancies : we average signal and detector orientation on healpix pixels visited during <br />
each fixed pointing period, which we call hereafter 'ring'. Detector's pointing are corrected for slow drifts and aberration (displacement on the sky indouced by the satellite's motion). This intermediate product is called HPR for healpix pixel ring. They have been constructed using the same map resolution as the final HFI products (corresponding to <math>N_{\mathrm side}</math>=2048). <br />
This new dataset is used as input in the following steps.<br />
<br />
== Photometric calibration ==<br />
<br />
=== Dipole calibration (100 to 353 GHz) ===<br />
<br />
For the 2015 data release, the HFI CMB channels were calibrated using the orbital dipole modulation. This time-variable anisotropy results from the motion of the spacecraft in the solar system, which is precisely measured. Thus it provides an absolute calibrator for orbital CMB missions. Its measurement is now used to calibrate HFI data thanks to the improvements in the timne stability of the data <br />
brought by the ADC non-linearities corrections and a better control for the detectors time response. <br />
<br />
Residual time response slow components are modeled as a dipole shifted by 90 degrees in phase, whose amplitude is fitted bolometer per bolometer. To mitigate residual systematics, we perform a simultaneous fit of the detectors gains on the orbital dipole. This amounts to solve the non-linear equation : <br />
<math> \displaystyle{d\ =\ g_i.(S+D) + O_i + n}<br />
\label{nlequat}</math><br />
<br />
where d is s the detector measurement, D the total dipole component, n the (white) noise, and both S the sky signal, g the detector gain, O the offset (for ring no i) are the unknowns to be determined. <br />
This is done by linearizing it to look for gains and sky variations, and iterating by updating the approximate sky and gains. <br />
<br />
<br />
=== Higher frequency calibration (545 and 857 GHz) ===<br />
<br />
We therefore the sub-mm channels' calibration for the 2015 Planck data release from the comparison of measurements of the Neptune and Uranus fluxes (with aperture photometry) with their expectations from the Moreno et al model of their atmospheres' emission. This procedure is justified, since for both planets, at the lower frequencies (100-353 Ghz), the fluxes we recover are in agreement within ~ +/-5\% with what is expected from the planet spectral model, and the HFI detector's band-passes.<br />
<br />
=== Zero levels === <br />
We determined zero-level for the released maps in selected regions of the sky where dust emissions are low and well correlated with HI. We may thus estimate and subtract dust emissions using the HI template, and CMB from a Planck component-separated template. The remaining astrophysical zero level is that of CIB. By imposing that the level we find is equal to that of the CIB model of Bethermin et al, we set the zero level of our maps.<br />
<br />
<br />
== Building of maps == <br />
<br />
Using the photometric calibration parameters, we build maps in two steps : <br />
<br />
* we determine the destriping offsets using the full mission data for all detectors of a given frequency<br />
* we build the maps, using these offsets, by inverting the photometric equation :<br />
<br />
<math> \displaystyle{d_i = g(I^p+\eta [Q^p cos(2\psi_i) + U^p sin(2\psi_i)]) + n}<br />
\label{photeq}</math><br />
where d is the destriped and calibrated signal at the HPR level. Detector's data are combined with an inverse noise weights derived from each detector's NEP. Q and U maps are build whenever possible. We propagate the white noise by building the 3x3 (or 1x1 if only I is reconstructed) covariance matrices in each pixel. <br />
At each frequency we build maps combining all detectors and independent detector sets. <br />
We use the offsets build for the full mission for building maps for each scan survey, year (sombineation of surveys 1 and 2 or 3 and 4 respectively) and for the full, nominal mission duration and its two halves. <br />
We also build maps from the two independent halves of each rings. Altogether, more than 8000 maps are built at each release. <br />
<br />
HPR and Maps are built in galactic coordinates.<br />
<br />
== Noise properties ==<br />
<br />
Map noise properties can be evaluated using several methods, thanks to the high level of observation redundancies. <br />
We can use the maps built from the difference between the first and second half of each rings, or compare individual sky scans, years, half-mission or independent detector sets with each other. <br />
Some of these tests are described in <span style="color:Red">A09 ref</span>.<br />
<br />
Low resolution (nside = 8, 16) pixel-to-pixel noise covariance matrices are build using an analytic approach from the measured noise power spectra. <span style="color:Red">CONFIRM THIS</span>.<br />
<br />
== Zodiacal light correction ==<br />
<br />
At the highest Planck frequencies, zodiacal light emission is visible in a survey difference map:<br />
<br />
[[File:Z857SurveyJackknifeWiZodi.png|500px|thumb|center|2013 Release 857 GHz Survey 2 - Survey 1 Difference]]<br />
<br />
This map is a difference between the 857 GHz Survey 2 map and the 857 GHz Survey 1 map. This difference effectively removes Galactic and other emissions which originate far from Planck. As the Solar elongation is different for measurements of the same point on the sky for the two surveys, we see zodiacal light emission, while all emission from further sources is removed. The zodiacal light emission follows the Ecliptic plane, which starts at the lower left of the image, then crosses the center of the plot towards the upper right. Note that the "arcs" at the top and bottom of the image are images of the Galactic center in the Far Sidelobes, which are discussed in the section below. Similar plots for other HFI frequencies, for maps both before and after removal, are shown [[beforeAndAfterSurveyDifferences|here]].<br />
<br />
For the 2015 Planck release, zodiacal light emission is removed from all HFI channels. The general procedure is described in {{PlanckPapers|planck2013-p03}}, but a synopsis of the procedure is as follows:<br />
* During each survey, a large fraction of the sky has observations which all fall within a week of each other. That is, during a single survey, most pixels are observed during a short, well-defined period. The contribution from zodiacal light emission to the total brightness seen, then, is well defined. <br />
* We use the the COBE model of the zodiacal light emission to make predictions for this zodiacal light emission for those pixels observed over a span of one week or less. The templates from the COBE model are shown [[COBEZodiModelTemplates|here]].<br />
* We fit the survey difference maps with these model templates to estimate the emissivity of each zodi component at the Planck wavelengths. The results of these fits at each frequency are given in <span style="color:red">Planck-2015-VIII</span>. <br />
* We reconstruct each ring of the the full mission using the combination of the COBE geometric model with the emissivities determined above. <br />
* We remove the reconstruction above from each ring of data. <br />
* We then make maps as described previously in this section.<br />
<br />
== Far SideLobes (FSL)==<br />
<br />
Contrary to the 2013 Planck release, Far Sidelobes were not removed from the HFI data for the 2015 Planck release. <br />
The change of the gain due to neglect of the far sidelobes is calculated by fitting the dipole to full timeline simulations of the dipole convolved by the FSL. The correction factors applied to the data are 0.09 % at 100 GHz, 0.05 % at 143 GHz, 0.04 % at 217 GHz and negligible at 353 GHz. Corrections were not made at 545 and 857 GHz <span style="color:red">(see the 2015 Planck/HFI Map-Making paper, A09, for details)</span>.<br />
<br />
== CO maps ==<br />
<br />
Carbon monoxide rotational transition line emission is present in all HFI bands but for the 143 GHz<br />
channel. It is especially significant in the 100, 217 and 353 GHz channels (due to the 115<br />
(1-0), 230 (2-1) and 345 GHz (3-2) CO transitions). This emission comes essentially from<br />
the Galactic interstellar medium and is mainly located at low and intermediate Galactic<br />
latitudes. Three approaches (summarised below) have been used to extract CO velocity<br />
integrated emission maps from HFI maps and to generate the CO products. See<br />
Planck-2013-XIII and Planck-2015-X for a full description.<br />
<br />
*Type 1 product: it is based on a single channel approach using the fact that each CO<br />
line has a slightly different transmission in each bolometer at a given frequency channel.<br />
From this, the J=1-0, J=2-1 and J=3-2 CO lines can be extracted independently. As this<br />
approach is based on individual bolometer maps of a single channel, the resulting<br />
Signal-to-Noise ratio (SNR) is relatively low. The benefit, however, is that these maps do<br />
not suffer from contamination from other HFI channels (as is the case for the other<br />
approaches) and are more reliable, especially in the Galactic plane. <br />
<br />
*Type 2 product: this product is obtained using a multi-frequency approach. Three<br />
frequency channel maps are combined to extract the J=1-0 (using the 100, 143 and 353<br />
GHz channels) and J=2-1 (using the 143, 217 and 353 GHz channels) CO maps. As<br />
frequency maps are combined, the spectral behaviour of other foregrounds influences<br />
the result. The two Type 2 CO maps produced in this way have a higher SNR than the<br />
Type 1 maps at the cost of a larger residual contamination from other diffuse<br />
foregrounds.<br />
<br />
* Type 3 product: no Type 3 product (as defined in 2013) has been produced for the<br />
2015 release. Instead, it is superseded by a high-resolution CO(2-1) map (FWHM=7.5<br />
arcmin) produced by the Commander component separation pipeline. Note that low<br />
resolution (FWHM=1 deg) CO maps of the three lines have also been produced using<br />
Commander. See [le lien vers la section avant-plans de Commander] for a complete<br />
description.<br />
<br />

The 2015 Type 1 and Type 2 CO maps have been produced using the same procedure as<br />
for the 2013 results. Very similar to their 2013 counterparts, the 2015 maps benefit from<br />
an increased SNR due to the use of the full, rather than nominal, mission data. <br />

Characteristics of the released maps are the following. We provide Healpix maps with<br />
Nside=2048. For one transition, the CO velocity-integrated line signal map is given in<br />
K_RJ.km/s units. A conversion factor from this unit to the native unit of HFI maps (K_CMB)<br />
is provided in the header of the data files and in the RIMO. Four maps are given per<br />
transition and per type:<br />

* The signal map<br />
* The standard deviation map (same unit as the signal),<br />
* A null test noise map (same unit as the signal) with similar statistical properties. It is<br />
made out of half the difference of half-ring maps.<br />
* A mask map (0B or 1B) giving the regions (1B) where the CO measurement is not<br />
reliable because of some severe identified foreground contamination. <br />

All products of a given type belong to a single file. Type 1 products have the native HFI<br />
resolution i.e. approximately 10, 5 and 5 arcminutes for the CO 1-0, 2-1, 3-2 transitions<br />
respectively. Type 2 products have a 15 arcminute resolution.<br />

We refer the reader to [le lien vers la section avant-plans de Commander] for a<br />
description and characteristics of the Commander CO products.<br />
<br />
== Map validation ==<br />
<br />
Several validations of HFI maps are described in <span style="color:Red">Planck-2015-VII ref</span> and in <span style="color:Red">Planck-2015-VIII ref</span>. <br />
<br />
Further checks are presented in the <span style="color:Red"> likelihood, params, comsep and commander papers refs </span>.<br />
<br />
== References ==<br />
<References /><br />
<br />
<br />
[[Category:HFI data processing|004]]</div>
Fdesert
https://wiki.cosmos.esa.int/planckpla2015/index.php?title=Map-making&diff=11194
Map-making
2015-02-04T15:53:46Z
<p>Fdesert: /* Map validation */</p>
<hr />
<div>{{DISPLAYTITLE:Map-making and photometric calibration}}<br />
== Introduction ==<br />
<br />
This page will give an overview of the map-making and photometric calibration procedures used by the HFI DPC to build detector and frequency maps for the 2015 data release. <br />
They are described in <span style="color:Red">A09 ref</span>.<br />
These have common elements with the tools used for the 2013 release that are described in {{PlanckPapers|planck2013-p03}} and {{PlanckPapers|planck2013-p03b}}. <br />
<br />
To build HFI maps, we use the destriping approximation, in which noise is assumed to decompose into two components : white noise plus low frequency drifts. Using the sky redundancy, the low frequency drifts are modelled as one constant, or offset, per pointing period. To speed up the ulterior processing we first build intermediate products, by taking advantage of redundancies : we average signal and detector orientation on healpix pixels visited during <br />
each fixed pointing period, which we call hereafter 'ring'. Detector's pointing are corrected for slow drifts and aberration (displacement on the sky indouced by the satellite's motion). This intermediate product is called HPR for healpix pixel ring. They have been constructed using the same map resolution as the final HFI products (corresponding to <math>N_{\mathrm side}</math>=2048). <br />
This new dataset is used as input in the following steps.<br />
<br />
== Photometric calibration ==<br />
<br />
=== Dipole calibration (100 to 353 GHz) ===<br />
<br />
For the 2015 data release, the HFI CMB channels were calibrated using the orbital dipole modulation. This time-variable anisotropy results from the motion of the spacecraft in the solar system, which is precisely measured. Thus it provides an absolute calibrator for orbital CMB missions. Its measurement is now used to calibrate HFI data thanks to the improvements in the timne stability of the data <br />
brought by the ADC non-linearities corrections and a better control for the detectors time response. <br />
<br />
Residual time response slow components are modeled as a dipole shifted by 90 degrees in phase, whose amplitude is fitted bolometer per bolometer. To mitigate residual systematics, we perform a simultaneous fit of the detectors gains on the orbital dipole. This amounts to solve the non-linear equation : <br />
<math> \displaystyle{d\ =\ g_i.(S+D) + O_i + n}<br />
\label{nlequat}</math><br />
<br />
where d is s the detector measurement, D the total dipole component, n the (white) noise, and both S the sky signal, g the detector gain, O the offset (for ring no i) are the unknowns to be determined. <br />
This is done by linearizing it to look for gains and sky variations, and iterating by updating the approximate sky and gains. <br />
<br />
<br />
=== Higher frequency calibration (545 and 857 GHz) ===<br />
<br />
We therefore the sub-mm channels' calibration for the 2015 Planck data release from the comparison of measurements of the Neptune and Uranus fluxes (with aperture photometry) with their expectations from the Moreno et al model of their atmospheres' emission. This procedure is justified, since for both planets, at the lower frequencies (100-353 Ghz), the fluxes we recover are in agreement within ~ +/-5\% with what is expected from the planet spectral model, and the HFI detector's band-passes.<br />
<br />
=== Zero levels === <br />
We determined zero-level for the released maps in selected regions of the sky where dust emissions are low and well correlated with HI. We may thus estimate and subtract dust emissions using the HI template, and CMB from a Planck component-separated template. The remaining astrophysical zero level is that of CIB. By imposing that the level we find is equal to that of the CIB model of Bethermin et al, we set the zero level of our maps.<br />
<br />
<br />
== Building of maps == <br />
<br />
Using the photometric calibration parameters, we build maps in two steps : <br />
<br />
* we determine the destriping offsets using the full mission data for all detectors of a given frequency<br />
* we build the maps, using these offsets, by inverting the photometric equation :<br />
<br />
<math> \displaystyle{d_i = g(I^p+\eta [Q^p cos(2\psi_i) + U^p sin(2\psi_i)]) + n}<br />
\label{photeq}</math><br />
where d is the destriped and calibrated signal at the HPR level. Detector's data are combined with an inverse noise weights derived from each detector's NEP. Q and U maps are build whenever possible. We propagate the white noise by building the 3x3 (or 1x1 if only I is reconstructed) covariance matrices in each pixel. <br />
At each frequency we build maps combining all detectors and independent detector sets. <br />
We use the offsets build for the full mission for building maps for each scan survey, year (sombineation of surveys 1 and 2 or 3 and 4 respectively) and for the full, nominal mission duration and its two halves. <br />
We also build maps from the two independent halves of each rings. Altogether, more than 8000 maps are built at each release. <br />
<br />
HPR and Maps are built in galactic coordinates.<br />
<br />
== Noise properties ==<br />
<br />
Map noise properties can be evaluated using several methods, thanks to the high level of observation redundancies. <br />
We can use the maps built from the difference between the first and second half of each rings, or compare individual sky scans, years, half-mission or independent detector sets with each other. <br />
Some of these tests are described in <span style="color:Red">A09 ref</span>.<br />
<br />
Low resolution (nside = 8, 16) pixel-to-pixel noise covariance matrices are build using an analytic approach from the measured noise power spectra. <span style="color:Red">CONFIRM THIS</span>.<br />
<br />
== Zodiacal light correction ==<br />
<br />
At the highest Planck frequencies, zodiacal light emission is visible in a survey difference map:<br />
<br />
[[File:Z857SurveyJackknifeWiZodi.png|500px|thumb|center|2013 Release 857 GHz Survey 2 - Survey 1 Difference]]<br />
<br />
This map is a difference between the 857 GHz Survey 2 map and the 857 GHz Survey 1 map. This difference effectively removes Galactic and other emissions which originate far from Planck. As the Solar elongation is different for measurements of the same point on the sky for the two surveys, we see zodiacal light emission, while all emission from further sources is removed. The zodiacal light emission follows the Ecliptic plane, which starts at the lower left of the image, then crosses the center of the plot towards the upper right. Note that the "arcs" at the top and bottom of the image are images of the Galactic center in the Far Sidelobes, which are discussed in the section below. Similar plots for other HFI frequencies, for maps both before and after removal, are shown [[beforeAndAfterSurveyDifferences|here]].<br />
<br />
For the 2015 Planck release, zodiacal light emission is removed from all HFI channels. The general procedure is described in {{PlanckPapers|planck2013-p03}}, but a synopsis of the procedure is as follows:<br />
* During each survey, a large fraction of the sky has observations which all fall within a week of each other. That is, during a single survey, most pixels are observed during a short, well-defined period. The contribution from zodiacal light emission to the total brightness seen, then, is well defined. <br />
* We use the the COBE model of the zodiacal light emission to make predictions for this zodiacal light emission for those pixels observed over a span of one week or less. The templates from the COBE model are shown [[COBEZodiModelTemplates|here]].<br />
* We fit the survey difference maps with these model templates to estimate the emissivity of each zodi component at the Planck wavelengths. The results of these fits at each frequency are given in <span style="color:red">the Planck/HFI Map-Making paper</span>. <br />
* We reconstruct each ring of the the full mission using the combination of the COBE geometric model with the emissivities determined above. <br />
* We remove the reconstruction above from each ring of data. <br />
* We then make maps as described previously in this section.<br />
<br />
== Far SideLobes (FSL)==<br />
<br />
Contrary to the 2013 Planck release, Far Sidelobes were not removed from the HFI data for the 2015 Planck release. <br />
The change of the gain due to neglect of the far sidelobes is calculated by fitting the dipole to full timeline simulations of the dipole convolved by the FSL. The correction factors applied to the data are 0.09 % at 100 GHz, 0.05 % at 143 GHz, 0.04 % at 217 GHz and negligible at 353 GHz. Corrections were not made at 545 and 857 GHz <span style="color:red">(see the 2015 Planck/HFI Map-Making paper, A09, for details)</span>.<br />
<br />
== CO maps ==<br />
<br />
Carbon monoxide rotational transition line emission is present in all HFI bands but for the 143 GHz<br />
channel. It is especially significant in the 100, 217 and 353 GHz channels (due to the 115<br />
(1-0), 230 (2-1) and 345 GHz (3-2) CO transitions). This emission comes essentially from<br />
the Galactic interstellar medium and is mainly located at low and intermediate Galactic<br />
latitudes. Three approaches (summarised below) have been used to extract CO velocity<br />
integrated emission maps from HFI maps and to generate the CO products. See<br />
Planck-2013-XIII and Planck-2015-X for a full description.<br />
<br />
*Type 1 product: it is based on a single channel approach using the fact that each CO<br />
line has a slightly different transmission in each bolometer at a given frequency channel.<br />
From this, the J=1-0, J=2-1 and J=3-2 CO lines can be extracted independently. As this<br />
approach is based on individual bolometer maps of a single channel, the resulting<br />
Signal-to-Noise ratio (SNR) is relatively low. The benefit, however, is that these maps do<br />
not suffer from contamination from other HFI channels (as is the case for the other<br />
approaches) and are more reliable, especially in the Galactic plane. <br />
<br />
*Type 2 product: this product is obtained using a multi-frequency approach. Three<br />
frequency channel maps are combined to extract the J=1-0 (using the 100, 143 and 353<br />
GHz channels) and J=2-1 (using the 143, 217 and 353 GHz channels) CO maps. As<br />
frequency maps are combined, the spectral behaviour of other foregrounds influences<br />
the result. The two Type 2 CO maps produced in this way have a higher SNR than the<br />
Type 1 maps at the cost of a larger residual contamination from other diffuse<br />
foregrounds.<br />
<br />
* Type 3 product: no Type 3 product (as defined in 2013) has been produced for the<br />
2015 release. Instead, it is superseded by a high-resolution CO(2-1) map (FWHM=7.5<br />
arcmin) produced by the Commander component separation pipeline. Note that low<br />
resolution (FWHM=1 deg) CO maps of the three lines have also been produced using<br />
Commander. See [le lien vers la section avant-plans de Commander] for a complete<br />
description.<br />
<br />

The 2015 Type 1 and Type 2 CO maps have been produced using the same procedure as<br />
for the 2013 results. Very similar to their 2013 counterparts, the 2015 maps benefit from<br />
an increased SNR due to the use of the full, rather than nominal, mission data. <br />

Characteristics of the released maps are the following. We provide Healpix maps with<br />
Nside=2048. For one transition, the CO velocity-integrated line signal map is given in<br />
K_RJ.km/s units. A conversion factor from this unit to the native unit of HFI maps (K_CMB)<br />
is provided in the header of the data files and in the RIMO. Four maps are given per<br />
transition and per type:<br />

* The signal map<br />
* The standard deviation map (same unit as the signal),<br />
* A null test noise map (same unit as the signal) with similar statistical properties. It is<br />
made out of half the difference of half-ring maps.<br />
* A mask map (0B or 1B) giving the regions (1B) where the CO measurement is not<br />
reliable because of some severe identified foreground contamination. <br />

All products of a given type belong to a single file. Type 1 products have the native HFI<br />
resolution i.e. approximately 10, 5 and 5 arcminutes for the CO 1-0, 2-1, 3-2 transitions<br />
respectively. Type 2 products have a 15 arcminute resolution.<br />

We refer the reader to [le lien vers la section avant-plans de Commander] for a<br />
description and characteristics of the Commander CO products.<br />
<br />
== Map validation ==<br />
<br />
Several validations of HFI maps are described in <span style="color:Red">Planck-2015-VII ref</span> and in <span style="color:Red">Planck-2015-VIII ref</span>. <br />
<br />
Further checks are presented in the <span style="color:Red"> likelihood, params, comsep and commander papers refs </span>.<br />
<br />
== References ==<br />
<References /><br />
<br />
<br />
[[Category:HFI data processing|004]]</div>
Fdesert
https://wiki.cosmos.esa.int/planckpla2015/index.php?title=Map-making&diff=11193
Map-making
2015-02-04T15:52:27Z
<p>Fdesert: /* Map validation */</p>
<hr />
<div>{{DISPLAYTITLE:Map-making and photometric calibration}}<br />
== Introduction ==<br />
<br />
This page will give an overview of the map-making and photometric calibration procedures used by the HFI DPC to build detector and frequency maps for the 2015 data release. <br />
They are described in <span style="color:Red">A09 ref</span>.<br />
These have common elements with the tools used for the 2013 release that are described in {{PlanckPapers|planck2013-p03}} and {{PlanckPapers|planck2013-p03b}}. <br />
<br />
To build HFI maps, we use the destriping approximation, in which noise is assumed to decompose into two components : white noise plus low frequency drifts. Using the sky redundancy, the low frequency drifts are modelled as one constant, or offset, per pointing period. To speed up the ulterior processing we first build intermediate products, by taking advantage of redundancies : we average signal and detector orientation on healpix pixels visited during <br />
each fixed pointing period, which we call hereafter 'ring'. Detector's pointing are corrected for slow drifts and aberration (displacement on the sky indouced by the satellite's motion). This intermediate product is called HPR for healpix pixel ring. They have been constructed using the same map resolution as the final HFI products (corresponding to <math>N_{\mathrm side}</math>=2048). <br />
This new dataset is used as input in the following steps.<br />
<br />
== Photometric calibration ==<br />
<br />
=== Dipole calibration (100 to 353 GHz) ===<br />
<br />
For the 2015 data release, the HFI CMB channels were calibrated using the orbital dipole modulation. This time-variable anisotropy results from the motion of the spacecraft in the solar system, which is precisely measured. Thus it provides an absolute calibrator for orbital CMB missions. Its measurement is now used to calibrate HFI data thanks to the improvements in the timne stability of the data <br />
brought by the ADC non-linearities corrections and a better control for the detectors time response. <br />
<br />
Residual time response slow components are modeled as a dipole shifted by 90 degrees in phase, whose amplitude is fitted bolometer per bolometer. To mitigate residual systematics, we perform a simultaneous fit of the detectors gains on the orbital dipole. This amounts to solve the non-linear equation : <br />
<math> \displaystyle{d\ =\ g_i.(S+D) + O_i + n}<br />
\label{nlequat}</math><br />
<br />
where d is s the detector measurement, D the total dipole component, n the (white) noise, and both S the sky signal, g the detector gain, O the offset (for ring no i) are the unknowns to be determined. <br />
This is done by linearizing it to look for gains and sky variations, and iterating by updating the approximate sky and gains. <br />
<br />
<br />
=== Higher frequency calibration (545 and 857 GHz) ===<br />
<br />
We therefore the sub-mm channels' calibration for the 2015 Planck data release from the comparison of measurements of the Neptune and Uranus fluxes (with aperture photometry) with their expectations from the Moreno et al model of their atmospheres' emission. This procedure is justified, since for both planets, at the lower frequencies (100-353 Ghz), the fluxes we recover are in agreement within ~ +/-5\% with what is expected from the planet spectral model, and the HFI detector's band-passes.<br />
<br />
=== Zero levels === <br />
We determined zero-level for the released maps in selected regions of the sky where dust emissions are low and well correlated with HI. We may thus estimate and subtract dust emissions using the HI template, and CMB from a Planck component-separated template. The remaining astrophysical zero level is that of CIB. By imposing that the level we find is equal to that of the CIB model of Bethermin et al, we set the zero level of our maps.<br />
<br />
<br />
== Building of maps == <br />
<br />
Using the photometric calibration parameters, we build maps in two steps : <br />
<br />
* we determine the destriping offsets using the full mission data for all detectors of a given frequency<br />
* we build the maps, using these offsets, by inverting the photometric equation :<br />
<br />
<math> \displaystyle{d_i = g(I^p+\eta [Q^p cos(2\psi_i) + U^p sin(2\psi_i)]) + n}<br />
\label{photeq}</math><br />
where d is the destriped and calibrated signal at the HPR level. Detector's data are combined with an inverse noise weights derived from each detector's NEP. Q and U maps are build whenever possible. We propagate the white noise by building the 3x3 (or 1x1 if only I is reconstructed) covariance matrices in each pixel. <br />
At each frequency we build maps combining all detectors and independent detector sets. <br />
We use the offsets build for the full mission for building maps for each scan survey, year (sombineation of surveys 1 and 2 or 3 and 4 respectively) and for the full, nominal mission duration and its two halves. <br />
We also build maps from the two independent halves of each rings. Altogether, more than 8000 maps are built at each release. <br />
<br />
HPR and Maps are built in galactic coordinates.<br />
<br />
== Noise properties ==<br />
<br />
Map noise properties can be evaluated using several methods, thanks to the high level of observation redundancies. <br />
We can use the maps built from the difference between the first and second half of each rings, or compare individual sky scans, years, half-mission or independent detector sets with each other. <br />
Some of these tests are described in <span style="color:Red">A09 ref</span>.<br />
<br />
Low resolution (nside = 8, 16) pixel-to-pixel noise covariance matrices are build using an analytic approach from the measured noise power spectra. <span style="color:Red">CONFIRM THIS</span>.<br />
<br />
== Zodiacal light correction ==<br />
<br />
At the highest Planck frequencies, zodiacal light emission is visible in a survey difference map:<br />
<br />
[[File:Z857SurveyJackknifeWiZodi.png|500px|thumb|center|2013 Release 857 GHz Survey 2 - Survey 1 Difference]]<br />
<br />
This map is a difference between the 857 GHz Survey 2 map and the 857 GHz Survey 1 map. This difference effectively removes Galactic and other emissions which originate far from Planck. As the Solar elongation is different for measurements of the same point on the sky for the two surveys, we see zodiacal light emission, while all emission from further sources is removed. The zodiacal light emission follows the Ecliptic plane, which starts at the lower left of the image, then crosses the center of the plot towards the upper right. Note that the "arcs" at the top and bottom of the image are images of the Galactic center in the Far Sidelobes, which are discussed in the section below. Similar plots for other HFI frequencies, for maps both before and after removal, are shown [[beforeAndAfterSurveyDifferences|here]].<br />
<br />
For the 2015 Planck release, zodiacal light emission is removed from all HFI channels. The general procedure is described in {{PlanckPapers|planck2013-p03}}, but a synopsis of the procedure is as follows:<br />
* During each survey, a large fraction of the sky has observations which all fall within a week of each other. That is, during a single survey, most pixels are observed during a short, well-defined period. The contribution from zodiacal light emission to the total brightness seen, then, is well defined. <br />
* We use the the COBE model of the zodiacal light emission to make predictions for this zodiacal light emission for those pixels observed over a span of one week or less. The templates from the COBE model are shown [[COBEZodiModelTemplates|here]].<br />
* We fit the survey difference maps with these model templates to estimate the emissivity of each zodi component at the Planck wavelengths. The results of these fits at each frequency are given in <span style="color:red">the Planck/HFI Map-Making paper</span>. <br />
* We reconstruct each ring of the the full mission using the combination of the COBE geometric model with the emissivities determined above. <br />
* We remove the reconstruction above from each ring of data. <br />
* We then make maps as described previously in this section.<br />
<br />
== Far SideLobes (FSL)==<br />
<br />
Contrary to the 2013 Planck release, Far Sidelobes were not removed from the HFI data for the 2015 Planck release. <br />
The change of the gain due to neglect of the far sidelobes is calculated by fitting the dipole to full timeline simulations of the dipole convolved by the FSL. The correction factors applied to the data are 0.09 % at 100 GHz, 0.05 % at 143 GHz, 0.04 % at 217 GHz and negligible at 353 GHz. Corrections were not made at 545 and 857 GHz <span style="color:red">(see the 2015 Planck/HFI Map-Making paper, A09, for details)</span>.<br />
<br />
== CO maps ==<br />
<br />
Carbon monoxide rotational transition line emission is present in all HFI bands but for the 143 GHz<br />
channel. It is especially significant in the 100, 217 and 353 GHz channels (due to the 115<br />
(1-0), 230 (2-1) and 345 GHz (3-2) CO transitions). This emission comes essentially from<br />
the Galactic interstellar medium and is mainly located at low and intermediate Galactic<br />
latitudes. Three approaches (summarised below) have been used to extract CO velocity<br />
integrated emission maps from HFI maps and to generate the CO products. See<br />
Planck-2013-XIII and Planck-2015-X for a full description.<br />
<br />
*Type 1 product: it is based on a single channel approach using the fact that each CO<br />
line has a slightly different transmission in each bolometer at a given frequency channel.<br />
From this, the J=1-0, J=2-1 and J=3-2 CO lines can be extracted independently. As this<br />
approach is based on individual bolometer maps of a single channel, the resulting<br />
Signal-to-Noise ratio (SNR) is relatively low. The benefit, however, is that these maps do<br />
not suffer from contamination from other HFI channels (as is the case for the other<br />
approaches) and are more reliable, especially in the Galactic plane. <br />
<br />
*Type 2 product: this product is obtained using a multi-frequency approach. Three<br />
frequency channel maps are combined to extract the J=1-0 (using the 100, 143 and 353<br />
GHz channels) and J=2-1 (using the 143, 217 and 353 GHz channels) CO maps. As<br />
frequency maps are combined, the spectral behaviour of other foregrounds influences<br />
the result. The two Type 2 CO maps produced in this way have a higher SNR than the<br />
Type 1 maps at the cost of a larger residual contamination from other diffuse<br />
foregrounds.<br />
<br />
* Type 3 product: no Type 3 product (as defined in 2013) has been produced for the<br />
2015 release. Instead, it is superseded by a high-resolution CO(2-1) map (FWHM=7.5<br />
arcmin) produced by the Commander component separation pipeline. Note that low<br />
resolution (FWHM=1 deg) CO maps of the three lines have also been produced using<br />
Commander. See [le lien vers la section avant-plans de Commander] for a complete<br />
description.<br />
<br />

The 2015 Type 1 and Type 2 CO maps have been produced using the same procedure as<br />
for the 2013 results. Very similar to their 2013 counterparts, the 2015 maps benefit from<br />
an increased SNR due to the use of the full, rather than nominal, mission data. <br />

Characteristics of the released maps are the following. We provide Healpix maps with<br />
Nside=2048. For one transition, the CO velocity-integrated line signal map is given in<br />
K_RJ.km/s units. A conversion factor from this unit to the native unit of HFI maps (K_CMB)<br />
is provided in the header of the data files and in the RIMO. Four maps are given per<br />
transition and per type:<br />

* The signal map<br />
* The standard deviation map (same unit as the signal),<br />
* A null test noise map (same unit as the signal) with similar statistical properties. It is<br />
made out of half the difference of half-ring maps.<br />
* A mask map (0B or 1B) giving the regions (1B) where the CO measurement is not<br />
reliable because of some severe identified foreground contamination. <br />

All products of a given type belong to a single file. Type 1 products have the native HFI<br />
resolution i.e. approximately 10, 5 and 5 arcminutes for the CO 1-0, 2-1, 3-2 transitions<br />
respectively. Type 2 products have a 15 arcminute resolution.<br />

We refer the reader to [le lien vers la section avant-plans de Commander] for a<br />
description and characteristics of the Commander CO products.<br />
<br />
== Map validation ==<br />
<br />
Several validations of HFI maps are described in <span style="color:Red">Planck-2015-IIX ref</span> and in <span style="color:Red">Planck-2015-IX ref</span>. <br />
<br />
Further checks are presented in the <span style="color:Red"> likelihood, params, comsep and commander papers refs </span>.<br />
<br />
== References ==<br />
<References /><br />
<br />
<br />
[[Category:HFI data processing|004]]</div>
Fdesert
https://wiki.cosmos.esa.int/planckpla2015/index.php?title=Timelines&diff=9607
Timelines
2014-09-29T14:05:37Z
<p>Fdesert: /* General description */</p>
<hr />
<div>==General description==<br />
<br />
The timelines, or TOIs for ''Time-Ordered Information'', are vectors of signal or of pointing or of some other quantity giving the value of that quantity as a function of time during the mission. The TOIs described here are sampled regularly at the (instrument dependent) detector sampling frequency and span the full science mission. They thus consist of a large amount of data. This section is concerned with the signal TOIs and their associated pointing TOIs. For each bolometer the DPCs provide a single timeline of calibrated signal and 3 timelines of coordinates, corresponding to the two angular coordinates and one position angle. <br />
<br />
Obviously these vectors are very long (~1.38E10 samples for HFI, nnnn for LFI) and thus need to be split into multiple files for export. Here the data are split by Operational Day (OD) as follows:<br />
* HFI: 12 files per OD, one for pointing and one for signal for each channel, covering ODs 91 -- 974, all sampled at F<sub>samp</sub> = 180.3737 Hz. <br />
* LFI: 6 files per OD, 3 each for signal and pointing, since the sampling frequency is channel dependent, and covering ODs 91--<span style="color:red">NNN</span>.<br />
<br />
The signal timelines are encoded as Real*4, but the pointing vectors had to be encoded as Real*8 to maintain the required accuracy. The result is that the total volume of the full dataset is ~30 TB. All files also contain the OBT.<br />
<br />
The signal timelines have been cleaned of all known instrumental systematic effects, they have been calibrated in flux, and the solar and earth motion dipole signals have been removed. But they have not been cleaned of the low-frequency, ''1/f'' noise that needs to be removed via a destriping tool. The methods consists of removing offsets or ''baselines'' determined by minimizing the differences in the signals at points on the sky where they intersect. For HFI, an offset per ring is determined; <span style="color:red"> For LFI ..... </span>. These offsets are determined using the full mission and all the valid detectors per channel, and they are then used for all the maps produced, i.e., for those maps using any fraction of the mission (year, survey) or any subset of the detectors (single detector, detector set), and it was shown that using offsets determined from a limited part of the mission and/or a subset of the detectors yields maps that are less consistent than otherwise. The offsets are delivered separately, as described below.<br />
<br />
<span style="color:red"> Maybe insert a figure for example? </span><br />
<br />
Furthermore, for HFI, there are three sets of offsets produced: the primary set using the full rings, and the secondary ones using the first and second half rings, only. The difference between the primary and the secondary sets are fairly minor, but they are necessary to rebuild the maps as they were built by the HFI-DPC. Overall the offsets are useful for users wishing to build a map of a small area of the sky. <br />
<br />
<span style="color:red"> TBW: baselines for LFI</span> <br />
<br />
The HFI delivers its offsets in a ROI, or "Ring-Ordered Information" file. That file contains a table of N<sub>rings</sub> = 26766 rows by N<sub>bolometers</sub> = 50 columns in which each cell contains a 3-elements vector containing the full-ring, first half-ring, and second half-ring offsets. The indices of the signal TOI delimiting the rings are given in a second ROI file with the same N<sub>rings</sub> rows containing global parameters.<br />
<br />
===Indexing===<br />
<br />
Each FITS file contains keywords giving the first and last index of the section of the full TOI contained in the file. These indices are those of the internal DPC representation; they begin at 0 corresponding to the switch-on of the instrument, and run to some very large number (~25 billion for HFI) representing the full mission. Only the ''science'' part of the mission is exported, which is about 60% of the total for HFI. Are left out the early mission phase (cool-down and validation phases) and the warm phase of HFI, during which LFI continued to collect data. The HFI keywords also indicated which rings are included in each OD, but be aware that normally an OD does not begin at the same time (with the same index) as a ring. <br />
The HFI offsets are determined on a per-ring basis, and must be applied ring-by-ring, and for this purpose the begin and end ring indices are given in <br />
<br />
<br />
<span style="color:red"> TBW: HFI: explain TOI indices and (BEG/ENDINDEX kwds) and same for ROI </span><br />
<br />
==Production process==<br />
<br />
=== HFI processing ===<br />
<br />
The input TOIs are in ADUs representing the voltage signal at the end of the electronics (see [[HFI_detection_chain | Detection chain]] for details). The processing applied to remove instrumental effects and to calibrate them is described in detail in the [[TOI processing | HFI TOI processing]] section, and we give a very brief summary here for convenience. That pipeline performs the following operations:<br />
<br />
; ADC correction: corrects for the uneven size of the ADC bins.<br />
; demodulation: this is performed around a variable level which is determined from the valid input data (using a validity flag from a previous version of the processing), and the data are converted to engineering units (V) using known conversion coefficients.<br />
; despiking: the redundancy within a ring is used to determine where glitches occur. Once identified, the glitches are fitted with templates. A glitch flag is produced that identifies the strongest part of the glitches, and a timeline of glitch tails. produced from the template fits, is subtracted from the demodulated timeline. Finally, the flagged regions are replaced with data from an average over the pointing period in order to not leave holes in the timeline that would perturb the usage of Fourier methods on them<br />
; dark template removal: the two dark bolometers are demodulated and despiked as above; the resulting timelines are then smoothed and used as an indicator of the overall temperature variations of the bolometer plate. Where the variations are consistent with each other, they are combined and removed from the bolometer signal timelines using appropriate coupling coefficients. The few percent of the data where they are not consistent are flagged as not valid on the timelines.<br />
; conversion to absorbed power: the timeline is converted to watts of absorbed power using the bolometer function. This includes a non-linearity correction; removal of the 4K cooler lines: the electromagnetic interference of the 4K cooler with the bolometer readout wires induces some sharp lines in the signal power spectra at frequencies of the 4K cooler's fundamental and its multiples, folded by the signal modulations. Fourier coefficients of the relevant lines are determined on a per-ring basis, and then removed from the data. The quality of the removal depends on the bolometer.<br />
; deconvolution by the time transfer function: this is done to correct for the non-instantaneous time response of the bolometers. The function itself is modeled using up to 8 time constants which are adjusted primarily on the planet data and also from comparisons of the northward and southward scans of the Galactic Plane. It is then removed using Fourier techniques, which has the side-effect of increasing the noise at high frequencies.<br />
; jump correction: removes some (relatively rare: 0.3 jumps per bolometer per pointing period, on average) jumps in the signal baseline. The jumps are detected characterized on smoothed TOIs, and corrected by adding a constant to part of the signal timeline. The origin of the jumps is not known.<br />
<br />
The results of this processing are a timeline of signal (in absorbed watts) and a ''valid data'' flag timeline for each of the 50 valid bolometers which are processed, out of the 52 HFI bolometers; these timelines contain the full sky signal, i.e., including the solar and orbital dipoles, the Zodiacal light, and contributions from the Far-Side lobes. The dipoles are necessary for the flux calibration and are removed at the mapmaking stage. The remaining two bolometers (143-8 and 545-3) show semi-random jumps in the signal level, typically jumping over 2-5 different ''pseudo-baseline'' levels, a behavior known as ''Random Telegraphic Signal'', so that these are commonly called the RTS bolometers. Finally, ring-level statistics of different types (mean, median, rms, kurtosis, etc.) are determined on a per-ring basis for all timelines, and a selection based on these statistics is used to discard anomalous rings, which are recorded in a ring-level flag for each valid bolometer timeline (see the [[TOI_processing#Discarded_rings | Discarded rings]] section). <br />
<br />
Throughout this processing, some planet signals are masked in the timeline in order to avoid ringing effects in the processing. It concerns only Jupiter at 100 and 143 GHZ, Jupiter and Saturn at 217 GHz, Jupiter, Saturn and Mars at 353, 545 and 857 GHz (thanks to various improvements in the despiking and in the transfer functions it is no longer necessary to flag these three planets in the CMB channels as was done in Release 1). Since they move on the sky, the portion of the sky masked during one survey is observed during one, and no hole is left in the final map. In parallel, the planet data are processed in a similar way, but with different parameters for the despiking step. These results are processed separately to determine the beam shapes and the focal plane geometry.<br />
<br />
Finally, the TOIs are calibrated in astrophysical units using the results of the calibration the calibration pipeline, described in the [[Map-making | Map-making and calibration]] section, which are stored in the IMO, and the best estimate of the zero-point offsets (a constant level for each bolometer) and of the far-side lobes (another signal TOI determined from the input TOI) are removed. These TOIs are ready for projection onto a map. That projection normally requires that the low-frequency noise be removed via a "destriping" (sometimes called "baseline removal") step. The HFI-DPC does its destriping at ring-level; should the user want to use the DPC's offsets, they are provided separately ([##REF##]). Due to a technical reason, for a ring with a duration larger than 72 minutes, only the first 72 minutes of the ring are used to make the HFI maps which are officially delivered (the loss of data only amounts to a total of 1/2 hour for the 29 months of HFI data).<br />
<br />
The pointing (see common [[Detector_pointing | Detector Pointing]] section for details) is determined from the AHF produced by MOC, which gives the direction and orientation of the LOS of a fiducial position in the focal plane at frequencies of 8Hz during stable pointing and 4 Hz during maneuvers. This is interpolated to the timestamps of data samples, corrected for the wobble and other time-dependent offsets determined from the observed positions of a large number of sources around the sky, and finally converted to the LOS of each detector using the quaternions in the IMO.<br />
<br />
==File Names==<br />
<br />
The file names are of the form:<br />
<br />
''{H,L}FI_TOI_{fff}-{SCI,PTG}_R2.nn_ODxxxx.fits''<br />
<br />
where<br />
* ''fff'' denotes the frequency<br />
* SCI or PTG denote signal or pointing TOIs<br />
* R2.nn is the version, and<br />
* ODxxxx indicates the OD.<br />
<br />
==FITS file structure==<br />
<br />
All FITS files begin with a primary extension containing no data and a minimal header, which is followed by one or more ''BINTABLE'' extensions with data, and with a description of the data in the header keywords. In what follows will usually ignore the primary extension, and count only the extensions containing data.<br />
<br />
===TOI files===<br />
<br />
The signal FITS files contain N ,'BINTABLE', data extensions, where N is stye number of detectors in that frequency channel. The first extension contains the OBT and its flags, called the ''global'' flags since they apply to all detectors, which is followed by one extension for each detector, containing the signal and its ''local'' flag. The flags columns are written as ''byte'' in which each of the 8 bits (max) encodes one flag timeline. The meaning of each bit is given in the header comments and is subject to change. For the global flag they include:<br />
* Unstable pointing: 1= pointing is not stable (e.g., during repointing maneuvers)<br />
* Dark correlation: 1 = darks are uncorrelated and data are flagged<br />
* First/Second half ring: which samples are in which half (only covers the stable pointing part of the ring)<br />
* HCM: in HCM mode (unstable pointing)<br />
* and more<br />
<br />
And for the local flag they include<br />
<br />
* Total Flag: a combination of the various flags that is the one finally used in the map-making (all samples with Total Flag different from zero should not be used)<br />
* Data not valid: glitched samples<br />
* Despike Common: (for PSBs only) glitch on current or other of PSB pair<br />
* StrongSignal: on Galactic Plane<br />
* Strong Source: on point source<br />
* other<br />
<br />
<br />
{| border="1" cellpadding="3" cellspacing="0" align="center" style="text-align:left" width=800px<br />
|+ '''TOI file data structure'''<br />
<br />
|- bgcolor="ffdead" <br />
!colspan="4" | 1. EXTNAME = 'OBT' : Data columns<br />
|- bgcolor="ffdead" <br />
! Column Name || Data Type || Units || Description<br />
|-<br />
|OBT || Int*8 || 2<sup>-16</sup> sec || On-board time<br />
|-<br />
|FLAG || Byte || none || the various bit-level flags<br />
<br />
|- bgcolor="ffdead" <br />
! Keyword || Data Type || Value || Description<br />
|-<br />
|OD || Int || || OD covered (as in filename)<br />
|-<br />
|BEGIDX || Int || || first sample index of given OD<br />
|-<br />
|ENDIDX || Int || || last sample index of given OD<br />
|-<br />
|BEGRING || Int || || first ring in given OD<br />
|-<br />
|ENDRING || Int || || last ring in given OD<br />
|-<br />
|TIMEZERO || String || 1958-01-01z00:00 || Origin of OBT<br />
<br />
|- bgcolor="ffdead" <br />
!colspan="4" | n. EXTNAME = ''DETNAME'' : Data columns<br />
<br />
|- bgcolor="ffdead" <br />
! Column Name || Data Type || Units || Description<br />
|-<br />
|SIGNAL || Real*4 || K<sub>cmb</sub> or MJy/sr || Value of signal<br />
|-<br />
|FLAG || Byte || none || the various bit-level flags<br />
|- bgcolor="ffdead" <br />
<br />
! Keyword || Data Type || Value || Description<br />
|-<br />
|UNIT || String || || Units of signal<br />
|-<br />
|DESTRIPE || 1/0 || || whether timeline is destriped<br />
|-<br />
|OD || Int || || OD covered (as in filename)<br />
|-<br />
|BEGIDX || Int || || first sample index of given OD<br />
|-<br />
|ENDIDX || Int || || last sample index of given OD<br />
|-<br />
|BEGRING || Int || || first ring in given OD<br />
|-<br />
|ENDRING || Int || || last ring in given OD<br />
|}<br />
<br />
<br />
The pointing files have a similar structure, except that ''DETNAME'' extensions contain 3 columns of Real*8 with the phi, theta, and psi Galactic spherical coordinates of each sample in radians. There is no local flag for the coordinates.<br />
<br />
===ROI files===<br />
<br />
The files provided by HFI are<br />
* ''HFI_ROI_GlobalParams_RelNum_full.fits''<br />
* ''HFI_ROI_DestripeOffsets_RelNum_full.fits''<br />
which are described below.<br />
<br />
;Global parameters: this ROI file contains a single ''BINTABLE'' extension with 3 columns that give, for each ring, the index number of the first sample, the ESA pointing ID of the ring, and the ring start time. All three are encoded as Int*8, and the time is given in units of 2<sup>-16</sup> sec from ''TIMEZERO'', 1958-01-01z00:00, as in the TOI files. Rings are numbered from 240 to 27005, as given in the ''BEGRING'' and ''ENDRING'' keywords<br />
<br />
; Destriping offsets: this ROI file contains a single ''BINTABLE'' extension with 3 columns that give, for each ring, the offsets to subtract from each bolometer signal timeline or vector. There are three offsets available for each bolometer: full-ring, first half-ring, and second half-ring, for building the full-ring or the two half-ring maps, respectively. The offsets must be subtracted from Index(N) to Index(N+1)-1 of the corresponding signal timeline, where N is stye ring number, and the indices are given in the Global parameters file. The offsets are given in the same units as the signal vectors, that is K<sub>cmv</sub> for the 100-353 GHz channels, and MJy/sr for the 545 and 857 GHz channels. All values are encoded as Real*4. The rings for which the offsets are set to 0.000 are [[TOI_processing#Discarded_rings | ''badrings'' ]] which are rejected in the mapmaking process.<br />
<br />
<span style="color:red"> TBW: Note on half-ring offsets, which differ by 1-2% from full ring offsets</span> <br />
<br />
[[Category:Mission products|001]]</div>
Fdesert
https://wiki.cosmos.esa.int/planckpla2015/index.php?title=Timelines&diff=9606
Timelines
2014-09-29T14:03:51Z
<p>Fdesert: /* TOI files */</p>
<hr />
<div>==General description==<br />
<br />
The timelines, or TOIs for ''Time-Ordered Information'', are vectors of signal or of pointing or of some other quantity giving the value of that quantity as a function of time during the mission. The TOIs described here are sampled regularly at the (instrument dependent) detector sampling frequency and span the full science mission. They thus consists of a large amount of data. This section is concerned with the signal TOIs and their associated pointing TOIs. For each bolometer the DPCs provide a single timeline of calibrated signal and 3 timelines of coordinates, corresponding to the two angular coordinates and one position angle. <br />
<br />
Obviously these vectors are very long (~1.38E10 samples for HFI, nnnn for LFI) and thus need to be split into multiple files for export. Here the data are split by Operational Day (OD) as follows:<br />
* HFI: 12 files per OD, one for pointing and one for signal for each channel, covering ODs 91 -- 974, all sampled at F<sub>samp</sub> = 180.3737 Hz. <br />
* LFI: 6 files per OD, 3 each for signal and pointing, since the sampling frequency is channel dependent, and covering ODs 91--<span style="color:red">NNN</span>.<br />
<br />
The signal timelines are encoded as Real*4, but the pointing vectors had to be encoded as Real*8 to maintain the required accuracy. The result is that the total volume of the full dataset is ~30 TB. All files also contain the OBT.<br />
<br />
The signal timelines have been cleaned of all known instrumental systematic effects, they have been calibrated in flux, and the solar and earth motion dipole signals have been removed. But they have not been cleaned of the low-frequency, ''1/f'' noise that needs to be removed via a destriping tool. The methods consists of removing offsets or ''baselines'' determined by minimizing the differences in the signals at points on the sky where they intersect. For HFI, an offset per ring is determined; <span style="color:red"> For LFI ..... </span>. These offsets are determined using the full mission and all the valid detectors per channel, and they are then used for all the maps produced, i.e., for those maps using any fraction of the mission (year, survey) or any subset of the detectors (single detector, detector set), and it was shown that using offsets determined from a limited part of the mission and/or a subset of the detectors yields maps that are less consistent than otherwise. The offsets are delivered separately, as described below.<br />
<br />
<span style="color:red"> Maybe insert a figure for example? </span><br />
<br />
Furthermore, for HFI, there are three sets of offsets produced: the primary set using the full rings, and the secondary ones using the first and second half rings, only. The difference between the primary and the secondary sets are fairly minor, but they are necessary to rebuild the maps as they were built by the HFI-DPC. Overall the offsets are useful for users wishing to build a map of a small area of the sky. <br />
<br />
<span style="color:red"> TBW: baselines for LFI</span> <br />
<br />
The HFI delivers its offsets in a ROI, or "Ring-Ordered Information" file. That file contains a table of N<sub>rings</sub> = 26766 rows by N<sub>bolometers</sub> = 50 columns in which each cell contains a 3-elements vector containing the full-ring, first half-ring, and second half-ring offsets. The indices of the signal TOI delimiting the rings are given in a second ROI file with the same N<sub>rings</sub> rows containing global parameters.<br />
<br />
===Indexing===<br />
<br />
Each FITS file contains keywords giving the first and last index of the section of the full TOI contained in the file. These indices are those of the internal DPC representation; they begin at 0 corresponding to the switch-on of the instrument, and run to some very large number (~25 billion for HFI) representing the full mission. Only the ''science'' part of the mission is exported, which is about 60% of the total for HFI. Are left out the early mission phase (cool-down and validation phases) and the warm phase of HFI, during which LFI continued to collect data. The HFI keywords also indicated which rings are included in each OD, but be aware that normally an OD does not begin at the same time (with the same index) as a ring. <br />
The HFI offsets are determined on a per-ring basis, and must be applied ring-by-ring, and for this purpose the begin and end ring indices are given in <br />
<br />
<br />
<span style="color:red"> TBW: HFI: explain TOI indices and (BEG/ENDINDEX kwds) and same for ROI </span><br />
<br />
==Production process==<br />
<br />
=== HFI processing ===<br />
<br />
The input TOIs are in ADUs representing the voltage signal at the end of the electronics (see [[HFI_detection_chain | Detection chain]] for details). The processing applied to remove instrumental effects and to calibrate them is described in detail in the [[TOI processing | HFI TOI processing]] section, and we give a very brief summary here for convenience. That pipeline performs the following operations:<br />
<br />
; ADC correction: corrects for the uneven size of the ADC bins.<br />
; demodulation: this is performed around a variable level which is determined from the valid input data (using a validity flag from a previous version of the processing), and the data are converted to engineering units (V) using known conversion coefficients.<br />
; despiking: the redundancy within a ring is used to determine where glitches occur. Once identified, the glitches are fitted with templates. A glitch flag is produced that identifies the strongest part of the glitches, and a timeline of glitch tails. produced from the template fits, is subtracted from the demodulated timeline. Finally, the flagged regions are replaced with data from an average over the pointing period in order to not leave holes in the timeline that would perturb the usage of Fourier methods on them<br />
; dark template removal: the two dark bolometers are demodulated and despiked as above; the resulting timelines are then smoothed and used as an indicator of the overall temperature variations of the bolometer plate. Where the variations are consistent with each other, they are combined and removed from the bolometer signal timelines using appropriate coupling coefficients. The few percent of the data where they are not consistent are flagged as not valid on the timelines.<br />
; conversion to absorbed power: the timeline is converted to watts of absorbed power using the bolometer function. This includes a non-linearity correction; removal of the 4K cooler lines: the electromagnetic interference of the 4K cooler with the bolometer readout wires induces some sharp lines in the signal power spectra at frequencies of the 4K cooler's fundamental and its multiples, folded by the signal modulations. Fourier coefficients of the relevant lines are determined on a per-ring basis, and then removed from the data. The quality of the removal depends on the bolometer.<br />
; deconvolution by the time transfer function: this is done to correct for the non-instantaneous time response of the bolometers. The function itself is modeled using up to 8 time constants which are adjusted primarily on the planet data and also from comparisons of the northward and southward scans of the Galactic Plane. It is then removed using Fourier techniques, which has the side-effect of increasing the noise at high frequencies.<br />
; jump correction: removes some (relatively rare: 0.3 jumps per bolometer per pointing period, on average) jumps in the signal baseline. The jumps are detected characterized on smoothed TOIs, and corrected by adding a constant to part of the signal timeline. The origin of the jumps is not known.<br />
<br />
The results of this processing are a timeline of signal (in absorbed watts) and a ''valid data'' flag timeline for each of the 50 valid bolometers which are processed, out of the 52 HFI bolometers; these timelines contain the full sky signal, i.e., including the solar and orbital dipoles, the Zodiacal light, and contributions from the Far-Side lobes. The dipoles are necessary for the flux calibration and are removed at the mapmaking stage. The remaining two bolometers (143-8 and 545-3) show semi-random jumps in the signal level, typically jumping over 2-5 different ''pseudo-baseline'' levels, a behavior known as ''Random Telegraphic Signal'', so that these are commonly called the RTS bolometers. Finally, ring-level statistics of different types (mean, median, rms, kurtosis, etc.) are determined on a per-ring basis for all timelines, and a selection based on these statistics is used to discard anomalous rings, which are recorded in a ring-level flag for each valid bolometer timeline (see the [[TOI_processing#Discarded_rings | Discarded rings]] section). <br />
<br />
Throughout this processing, some planet signals are masked in the timeline in order to avoid ringing effects in the processing. It concerns only Jupiter at 100 and 143 GHZ, Jupiter and Saturn at 217 GHz, Jupiter, Saturn and Mars at 353, 545 and 857 GHz (thanks to various improvements in the despiking and in the transfer functions it is no longer necessary to flag these three planets in the CMB channels as was done in Release 1). Since they move on the sky, the portion of the sky masked during one survey is observed during one, and no hole is left in the final map. In parallel, the planet data are processed in a similar way, but with different parameters for the despiking step. These results are processed separately to determine the beam shapes and the focal plane geometry.<br />
<br />
Finally, the TOIs are calibrated in astrophysical units using the results of the calibration the calibration pipeline, described in the [[Map-making | Map-making and calibration]] section, which are stored in the IMO, and the best estimate of the zero-point offsets (a constant level for each bolometer) and of the far-side lobes (another signal TOI determined from the input TOI) are removed. These TOIs are ready for projection onto a map. That projection normally requires that the low-frequency noise be removed via a "destriping" (sometimes called "baseline removal") step. The HFI-DPC does its destriping at ring-level; should the user want to use the DPC's offsets, they are provided separately ([##REF##]). Due to a technical reason, for a ring with a duration larger than 72 minutes, only the first 72 minutes of the ring are used to make the HFI maps which are officially delivered (the loss of data only amounts to a total of 1/2 hour for the 29 months of HFI data).<br />
<br />
The pointing (see common [[Detector_pointing | Detector Pointing]] section for details) is determined from the AHF produced by MOC, which gives the direction and orientation of the LOS of a fiducial position in the focal plane at frequencies of 8Hz during stable pointing and 4 Hz during maneuvers. This is interpolated to the timestamps of data samples, corrected for the wobble and other time-dependent offsets determined from the observed positions of a large number of sources around the sky, and finally converted to the LOS of each detector using the quaternions in the IMO.<br />
<br />
==File Names==<br />
<br />
The file names are of the form:<br />
<br />
''{H,L}FI_TOI_{fff}-{SCI,PTG}_R2.nn_ODxxxx.fits''<br />
<br />
where<br />
* ''fff'' denotes the frequency<br />
* SCI or PTG denote signal or pointing TOIs<br />
* R2.nn is the version, and<br />
* ODxxxx indicates the OD.<br />
<br />
==FITS file structure==<br />
<br />
All FITS files begin with a primary extension containing no data and a minimal header, which is followed by one or more ''BINTABLE'' extensions with data, and with a description of the data in the header keywords. In what follows will usually ignore the primary extension, and count only the extensions containing data.<br />
<br />
===TOI files===<br />
<br />
The signal FITS files contain N ,'BINTABLE', data extensions, where N is stye number of detectors in that frequency channel. The first extension contains the OBT and its flags, called the ''global'' flags since they apply to all detectors, which is followed by one extension for each detector, containing the signal and its ''local'' flag. The flags columns are written as ''byte'' in which each of the 8 bits (max) encodes one flag timeline. The meaning of each bit is given in the header comments and is subject to change. For the global flag they include:<br />
* Unstable pointing: 1= pointing is not stable (e.g., during repointing maneuvers)<br />
* Dark correlation: 1 = darks are uncorrelated and data are flagged<br />
* First/Second half ring: which samples are in which half (only covers the stable pointing part of the ring)<br />
* HCM: in HCM mode (unstable pointing)<br />
* and more<br />
<br />
And for the local flag they include<br />
<br />
* Total Flag: a combination of the various flags that is the one finally used in the map-making (all samples with Total Flag different from zero should not be used)<br />
* Data not valid: glitched samples<br />
* Despike Common: (for PSBs only) glitch on current or other of PSB pair<br />
* StrongSignal: on Galactic Plane<br />
* Strong Source: on point source<br />
* other<br />
<br />
<br />
{| border="1" cellpadding="3" cellspacing="0" align="center" style="text-align:left" width=800px<br />
|+ '''TOI file data structure'''<br />
<br />
|- bgcolor="ffdead" <br />
!colspan="4" | 1. EXTNAME = 'OBT' : Data columns<br />
|- bgcolor="ffdead" <br />
! Column Name || Data Type || Units || Description<br />
|-<br />
|OBT || Int*8 || 2<sup>-16</sup> sec || On-board time<br />
|-<br />
|FLAG || Byte || none || the various bit-level flags<br />
<br />
|- bgcolor="ffdead" <br />
! Keyword || Data Type || Value || Description<br />
|-<br />
|OD || Int || || OD covered (as in filename)<br />
|-<br />
|BEGIDX || Int || || first sample index of given OD<br />
|-<br />
|ENDIDX || Int || || last sample index of given OD<br />
|-<br />
|BEGRING || Int || || first ring in given OD<br />
|-<br />
|ENDRING || Int || || last ring in given OD<br />
|-<br />
|TIMEZERO || String || 1958-01-01z00:00 || Origin of OBT<br />
<br />
|- bgcolor="ffdead" <br />
!colspan="4" | n. EXTNAME = ''DETNAME'' : Data columns<br />
<br />
|- bgcolor="ffdead" <br />
! Column Name || Data Type || Units || Description<br />
|-<br />
|SIGNAL || Real*4 || K<sub>cmb</sub> or MJy/sr || Value of signal<br />
|-<br />
|FLAG || Byte || none || the various bit-level flags<br />
|- bgcolor="ffdead" <br />
<br />
! Keyword || Data Type || Value || Description<br />
|-<br />
|UNIT || String || || Units of signal<br />
|-<br />
|DESTRIPE || 1/0 || || whether timeline is destriped<br />
|-<br />
|OD || Int || || OD covered (as in filename)<br />
|-<br />
|BEGIDX || Int || || first sample index of given OD<br />
|-<br />
|ENDIDX || Int || || last sample index of given OD<br />
|-<br />
|BEGRING || Int || || first ring in given OD<br />
|-<br />
|ENDRING || Int || || last ring in given OD<br />
|}<br />
<br />
<br />
The pointing files have a similar structure, except that ''DETNAME'' extensions contain 3 columns of Real*8 with the phi, theta, and psi Galactic spherical coordinates of each sample in radians. There is no local flag for the coordinates.<br />
<br />
===ROI files===<br />
<br />
The files provided by HFI are<br />
* ''HFI_ROI_GlobalParams_RelNum_full.fits''<br />
* ''HFI_ROI_DestripeOffsets_RelNum_full.fits''<br />
which are described below.<br />
<br />
;Global parameters: this ROI file contains a single ''BINTABLE'' extension with 3 columns that give, for each ring, the index number of the first sample, the ESA pointing ID of the ring, and the ring start time. All three are encoded as Int*8, and the time is given in units of 2<sup>-16</sup> sec from ''TIMEZERO'', 1958-01-01z00:00, as in the TOI files. Rings are numbered from 240 to 27005, as given in the ''BEGRING'' and ''ENDRING'' keywords<br />
<br />
; Destriping offsets: this ROI file contains a single ''BINTABLE'' extension with 3 columns that give, for each ring, the offsets to subtract from each bolometer signal timeline or vector. There are three offsets available for each bolometer: full-ring, first half-ring, and second half-ring, for building the full-ring or the two half-ring maps, respectively. The offsets must be subtracted from Index(N) to Index(N+1)-1 of the corresponding signal timeline, where N is stye ring number, and the indices are given in the Global parameters file. The offsets are given in the same units as the signal vectors, that is K<sub>cmv</sub> for the 100-353 GHz channels, and MJy/sr for the 545 and 857 GHz channels. All values are encoded as Real*4. The rings for which the offsets are set to 0.000 are [[TOI_processing#Discarded_rings | ''badrings'' ]] which are rejected in the mapmaking process.<br />
<br />
<span style="color:red"> TBW: Note on half-ring offsets, which differ by 1-2% from full ring offsets</span> <br />
<br />
[[Category:Mission products|001]]</div>
Fdesert
https://wiki.cosmos.esa.int/planckpla2015/index.php?title=Timelines&diff=9605
Timelines
2014-09-29T14:01:42Z
<p>Fdesert: /* HFI processing */</p>
<hr />
<div>==General description==<br />
<br />
The timelines, or TOIs for ''Time-Ordered Information'', are vectors of signal or of pointing or of some other quantity giving the value of that quantity as a function of time during the mission. The TOIs described here are sampled regularly at the (instrument dependent) detector sampling frequency and span the full science mission. They thus consists of a large amount of data. This section is concerned with the signal TOIs and their associated pointing TOIs. For each bolometer the DPCs provide a single timeline of calibrated signal and 3 timelines of coordinates, corresponding to the two angular coordinates and one position angle. <br />
<br />
Obviously these vectors are very long (~1.38E10 samples for HFI, nnnn for LFI) and thus need to be split into multiple files for export. Here the data are split by Operational Day (OD) as follows:<br />
* HFI: 12 files per OD, one for pointing and one for signal for each channel, covering ODs 91 -- 974, all sampled at F<sub>samp</sub> = 180.3737 Hz. <br />
* LFI: 6 files per OD, 3 each for signal and pointing, since the sampling frequency is channel dependent, and covering ODs 91--<span style="color:red">NNN</span>.<br />
<br />
The signal timelines are encoded as Real*4, but the pointing vectors had to be encoded as Real*8 to maintain the required accuracy. The result is that the total volume of the full dataset is ~30 TB. All files also contain the OBT.<br />
<br />
The signal timelines have been cleaned of all known instrumental systematic effects, they have been calibrated in flux, and the solar and earth motion dipole signals have been removed. But they have not been cleaned of the low-frequency, ''1/f'' noise that needs to be removed via a destriping tool. The methods consists of removing offsets or ''baselines'' determined by minimizing the differences in the signals at points on the sky where they intersect. For HFI, an offset per ring is determined; <span style="color:red"> For LFI ..... </span>. These offsets are determined using the full mission and all the valid detectors per channel, and they are then used for all the maps produced, i.e., for those maps using any fraction of the mission (year, survey) or any subset of the detectors (single detector, detector set), and it was shown that using offsets determined from a limited part of the mission and/or a subset of the detectors yields maps that are less consistent than otherwise. The offsets are delivered separately, as described below.<br />
<br />
<span style="color:red"> Maybe insert a figure for example? </span><br />
<br />
Furthermore, for HFI, there are three sets of offsets produced: the primary set using the full rings, and the secondary ones using the first and second half rings, only. The difference between the primary and the secondary sets are fairly minor, but they are necessary to rebuild the maps as they were built by the HFI-DPC. Overall the offsets are useful for users wishing to build a map of a small area of the sky. <br />
<br />
<span style="color:red"> TBW: baselines for LFI</span> <br />
<br />
The HFI delivers its offsets in a ROI, or "Ring-Ordered Information" file. That file contains a table of N<sub>rings</sub> = 26766 rows by N<sub>bolometers</sub> = 50 columns in which each cell contains a 3-elements vector containing the full-ring, first half-ring, and second half-ring offsets. The indices of the signal TOI delimiting the rings are given in a second ROI file with the same N<sub>rings</sub> rows containing global parameters.<br />
<br />
===Indexing===<br />
<br />
Each FITS file contains keywords giving the first and last index of the section of the full TOI contained in the file. These indices are those of the internal DPC representation; they begin at 0 corresponding to the switch-on of the instrument, and run to some very large number (~25 billion for HFI) representing the full mission. Only the ''science'' part of the mission is exported, which is about 60% of the total for HFI. Are left out the early mission phase (cool-down and validation phases) and the warm phase of HFI, during which LFI continued to collect data. The HFI keywords also indicated which rings are included in each OD, but be aware that normally an OD does not begin at the same time (with the same index) as a ring. <br />
The HFI offsets are determined on a per-ring basis, and must be applied ring-by-ring, and for this purpose the begin and end ring indices are given in <br />
<br />
<br />
<span style="color:red"> TBW: HFI: explain TOI indices and (BEG/ENDINDEX kwds) and same for ROI </span><br />
<br />
==Production process==<br />
<br />
=== HFI processing ===<br />
<br />
The input TOIs are in ADUs representing the voltage signal at the end of the electronics (see [[HFI_detection_chain | Detection chain]] for details). The processing applied to remove instrumental effects and to calibrate them is described in detail in the [[TOI processing | HFI TOI processing]] section, and we give a very brief summary here for convenience. That pipeline performs the following operations:<br />
<br />
; ADC correction: corrects for the uneven size of the ADC bins.<br />
; demodulation: this is performed around a variable level which is determined from the valid input data (using a validity flag from a previous version of the processing), and the data are converted to engineering units (V) using known conversion coefficients.<br />
; despiking: the redundancy within a ring is used to determine where glitches occur. Once identified, the glitches are fitted with templates. A glitch flag is produced that identifies the strongest part of the glitches, and a timeline of glitch tails. produced from the template fits, is subtracted from the demodulated timeline. Finally, the flagged regions are replaced with data from an average over the pointing period in order to not leave holes in the timeline that would perturb the usage of Fourier methods on them<br />
; dark template removal: the two dark bolometers are demodulated and despiked as above; the resulting timelines are then smoothed and used as an indicator of the overall temperature variations of the bolometer plate. Where the variations are consistent with each other, they are combined and removed from the bolometer signal timelines using appropriate coupling coefficients. The few percent of the data where they are not consistent are flagged as not valid on the timelines.<br />
; conversion to absorbed power: the timeline is converted to watts of absorbed power using the bolometer function. This includes a non-linearity correction; removal of the 4K cooler lines: the electromagnetic interference of the 4K cooler with the bolometer readout wires induces some sharp lines in the signal power spectra at frequencies of the 4K cooler's fundamental and its multiples, folded by the signal modulations. Fourier coefficients of the relevant lines are determined on a per-ring basis, and then removed from the data. The quality of the removal depends on the bolometer.<br />
; deconvolution by the time transfer function: this is done to correct for the non-instantaneous time response of the bolometers. The function itself is modeled using up to 8 time constants which are adjusted primarily on the planet data and also from comparisons of the northward and southward scans of the Galactic Plane. It is then removed using Fourier techniques, which has the side-effect of increasing the noise at high frequencies.<br />
; jump correction: removes some (relatively rare: 0.3 jumps per bolometer per pointing period, on average) jumps in the signal baseline. The jumps are detected characterized on smoothed TOIs, and corrected by adding a constant to part of the signal timeline. The origin of the jumps is not known.<br />
<br />
The results of this processing are a timeline of signal (in absorbed watts) and a ''valid data'' flag timeline for each of the 50 valid bolometers which are processed, out of the 52 HFI bolometers; these timelines contain the full sky signal, i.e., including the solar and orbital dipoles, the Zodiacal light, and contributions from the Far-Side lobes. The dipoles are necessary for the flux calibration and are removed at the mapmaking stage. The remaining two bolometers (143-8 and 545-3) show semi-random jumps in the signal level, typically jumping over 2-5 different ''pseudo-baseline'' levels, a behavior known as ''Random Telegraphic Signal'', so that these are commonly called the RTS bolometers. Finally, ring-level statistics of different types (mean, median, rms, kurtosis, etc.) are determined on a per-ring basis for all timelines, and a selection based on these statistics is used to discard anomalous rings, which are recorded in a ring-level flag for each valid bolometer timeline (see the [[TOI_processing#Discarded_rings | Discarded rings]] section). <br />
<br />
Throughout this processing, some planet signals are masked in the timeline in order to avoid ringing effects in the processing. It concerns only Jupiter at 100 and 143 GHZ, Jupiter and Saturn at 217 GHz, Jupiter, Saturn and Mars at 353, 545 and 857 GHz (thanks to various improvements in the despiking and in the transfer functions it is no longer necessary to flag these three planets in the CMB channels as was done in Release 1). Since they move on the sky, the portion of the sky masked during one survey is observed during one, and no hole is left in the final map. In parallel, the planet data are processed in a similar way, but with different parameters for the despiking step. These results are processed separately to determine the beam shapes and the focal plane geometry.<br />
<br />
Finally, the TOIs are calibrated in astrophysical units using the results of the calibration the calibration pipeline, described in the [[Map-making | Map-making and calibration]] section, which are stored in the IMO, and the best estimate of the zero-point offsets (a constant level for each bolometer) and of the far-side lobes (another signal TOI determined from the input TOI) are removed. These TOIs are ready for projection onto a map. That projection normally requires that the low-frequency noise be removed via a "destriping" (sometimes called "baseline removal") step. The HFI-DPC does its destriping at ring-level; should the user want to use the DPC's offsets, they are provided separately ([##REF##]). Due to a technical reason, for a ring with a duration larger than 72 minutes, only the first 72 minutes of the ring are used to make the HFI maps which are officially delivered (the loss of data only amounts to a total of 1/2 hour for the 29 months of HFI data).<br />
<br />
The pointing (see common [[Detector_pointing | Detector Pointing]] section for details) is determined from the AHF produced by MOC, which gives the direction and orientation of the LOS of a fiducial position in the focal plane at frequencies of 8Hz during stable pointing and 4 Hz during maneuvers. This is interpolated to the timestamps of data samples, corrected for the wobble and other time-dependent offsets determined from the observed positions of a large number of sources around the sky, and finally converted to the LOS of each detector using the quaternions in the IMO.<br />
<br />
==File Names==<br />
<br />
The file names are of the form:<br />
<br />
''{H,L}FI_TOI_{fff}-{SCI,PTG}_R2.nn_ODxxxx.fits''<br />
<br />
where<br />
* ''fff'' denotes the frequency<br />
* SCI or PTG denote signal or pointing TOIs<br />
* R2.nn is the version, and<br />
* ODxxxx indicates the OD.<br />
<br />
==FITS file structure==<br />
<br />
All FITS files begin with a primary extension containing no data and a minimal header, which is followed by one or more ''BINTABLE'' extensions with data, and with a description of the data in the header keywords. In what follows will usually ignore the primary extension, and count only the extensions containing data.<br />
<br />
===TOI files===<br />
<br />
The signal FITS files contain N ,'BINTABLE', data extensions, where N is stye number of detectors in that frequency channel. The first extension contains the OBT and its flags, called the ''global'' flags since they apply to all detectors, which is followed by one extension for each detector, containing the signal and its ''local'' flag. The flags columns are written as ''byte'' in which each of the 8 bits (max) encodes one flag timeline. The meaning of each bit is given in the header comments and is subject to change. For the global flag they include:<br />
* Unstable pointing: 1= pointing is not stable (e.g., during repointing maneuvers)<br />
* Dark correlation: 1 = darks are uncorrelated and data are flagged<br />
* First/Second half ring: which samples are in which half (only covers the stable pointing part of the ring)<br />
* HCM: in HCM mode (unstable pointing)<br />
* and more<br />
<br />
And for the local flag they include<br />
<br />
* Total Flag: a combination of the various flags that is the one finally used in the map-making<br />
* Data not valid: glitched samples<br />
* Despike Common: (for PSBs only) glitch on current or other of PSB pair<br />
* StrongSignal: on Galactic Plane<br />
* Strong Source: on point source<br />
* other<br />
<br />
<br />
{| border="1" cellpadding="3" cellspacing="0" align="center" style="text-align:left" width=800px<br />
|+ '''TOI file data structure'''<br />
<br />
|- bgcolor="ffdead" <br />
!colspan="4" | 1. EXTNAME = 'OBT' : Data columns<br />
|- bgcolor="ffdead" <br />
! Column Name || Data Type || Units || Description<br />
|-<br />
|OBT || Int*8 || 2<sup>-16</sup> sec || On-board time<br />
|-<br />
|FLAG || Byte || none || the various bit-level flags<br />
<br />
|- bgcolor="ffdead" <br />
! Keyword || Data Type || Value || Description<br />
|-<br />
|OD || Int || || OD covered (as in filename)<br />
|-<br />
|BEGIDX || Int || || first sample index of given OD<br />
|-<br />
|ENDIDX || Int || || last sample index of given OD<br />
|-<br />
|BEGRING || Int || || first ring in given OD<br />
|-<br />
|ENDRING || Int || || last ring in given OD<br />
|-<br />
|TIMEZERO || String || 1958-01-01z00:00 || Origin of OBT<br />
<br />
|- bgcolor="ffdead" <br />
!colspan="4" | n. EXTNAME = ''DETNAME'' : Data columns<br />
<br />
|- bgcolor="ffdead" <br />
! Column Name || Data Type || Units || Description<br />
|-<br />
|SIGNAL || Real*4 || K<sub>cmb</sub> or MJy/sr || Value of signal<br />
|-<br />
|FLAG || Byte || none || the various bit-level flags<br />
|- bgcolor="ffdead" <br />
<br />
! Keyword || Data Type || Value || Description<br />
|-<br />
|UNIT || String || || Units of signal<br />
|-<br />
|DESTRIPE || 1/0 || || whether timeline is destriped<br />
|-<br />
|OD || Int || || OD covered (as in filename)<br />
|-<br />
|BEGIDX || Int || || first sample index of given OD<br />
|-<br />
|ENDIDX || Int || || last sample index of given OD<br />
|-<br />
|BEGRING || Int || || first ring in given OD<br />
|-<br />
|ENDRING || Int || || last ring in given OD<br />
|}<br />
<br />
<br />
The pointing files have a similar structure, except that ''DETNAME'' extensions contain 3 columns of Real*8 with the phi, theta, and psi Galactic spherical coordinates of each sample in radians. There is no local flag for the coordinates. <br />
<br />
<br />
===ROI files===<br />
<br />
The files provided by HFI are<br />
* ''HFI_ROI_GlobalParams_RelNum_full.fits''<br />
* ''HFI_ROI_DestripeOffsets_RelNum_full.fits''<br />
which are described below.<br />
<br />
;Global parameters: this ROI file contains a single ''BINTABLE'' extension with 3 columns that give, for each ring, the index number of the first sample, the ESA pointing ID of the ring, and the ring start time. All three are encoded as Int*8, and the time is given in units of 2<sup>-16</sup> sec from ''TIMEZERO'', 1958-01-01z00:00, as in the TOI files. Rings are numbered from 240 to 27005, as given in the ''BEGRING'' and ''ENDRING'' keywords<br />
<br />
; Destriping offsets: this ROI file contains a single ''BINTABLE'' extension with 3 columns that give, for each ring, the offsets to subtract from each bolometer signal timeline or vector. There are three offsets available for each bolometer: full-ring, first half-ring, and second half-ring, for building the full-ring or the two half-ring maps, respectively. The offsets must be subtracted from Index(N) to Index(N+1)-1 of the corresponding signal timeline, where N is stye ring number, and the indices are given in the Global parameters file. The offsets are given in the same units as the signal vectors, that is K<sub>cmv</sub> for the 100-353 GHz channels, and MJy/sr for the 545 and 857 GHz channels. All values are encoded as Real*4. The rings for which the offsets are set to 0.000 are [[TOI_processing#Discarded_rings | ''badrings'' ]] which are rejected in the mapmaking process.<br />
<br />
<span style="color:red"> TBW: Note on half-ring offsets, which differ by 1-2% from full ring offsets</span> <br />
<br />
[[Category:Mission products|001]]</div>
Fdesert
https://wiki.cosmos.esa.int/planckpla2015/index.php?title=Timelines&diff=9604
Timelines
2014-09-29T14:00:24Z
<p>Fdesert: /* HFI processing */</p>
<hr />
<div>==General description==<br />
<br />
The timelines, or TOIs for ''Time-Ordered Information'', are vectors of signal or of pointing or of some other quantity giving the value of that quantity as a function of time during the mission. The TOIs described here are sampled regularly at the (instrument dependent) detector sampling frequency and span the full science mission. They thus consists of a large amount of data. This section is concerned with the signal TOIs and their associated pointing TOIs. For each bolometer the DPCs provide a single timeline of calibrated signal and 3 timelines of coordinates, corresponding to the two angular coordinates and one position angle. <br />
<br />
Obviously these vectors are very long (~1.38E10 samples for HFI, nnnn for LFI) and thus need to be split into multiple files for export. Here the data are split by Operational Day (OD) as follows:<br />
* HFI: 12 files per OD, one for pointing and one for signal for each channel, covering ODs 91 -- 974, all sampled at F<sub>samp</sub> = 180.3737 Hz. <br />
* LFI: 6 files per OD, 3 each for signal and pointing, since the sampling frequency is channel dependent, and covering ODs 91--<span style="color:red">NNN</span>.<br />
<br />
The signal timelines are encoded as Real*4, but the pointing vectors had to be encoded as Real*8 to maintain the required accuracy. The result is that the total volume of the full dataset is ~30 TB. All files also contain the OBT.<br />
<br />
The signal timelines have been cleaned of all known instrumental systematic effects, they have been calibrated in flux, and the solar and earth motion dipole signals have been removed. But they have not been cleaned of the low-frequency, ''1/f'' noise that needs to be removed via a destriping tool. The methods consists of removing offsets or ''baselines'' determined by minimizing the differences in the signals at points on the sky where they intersect. For HFI, an offset per ring is determined; <span style="color:red"> For LFI ..... </span>. These offsets are determined using the full mission and all the valid detectors per channel, and they are then used for all the maps produced, i.e., for those maps using any fraction of the mission (year, survey) or any subset of the detectors (single detector, detector set), and it was shown that using offsets determined from a limited part of the mission and/or a subset of the detectors yields maps that are less consistent than otherwise. The offsets are delivered separately, as described below.<br />
<br />
<span style="color:red"> Maybe insert a figure for example? </span><br />
<br />
Furthermore, for HFI, there are three sets of offsets produced: the primary set using the full rings, and the secondary ones using the first and second half rings, only. The difference between the primary and the secondary sets are fairly minor, but they are necessary to rebuild the maps as they were built by the HFI-DPC. Overall the offsets are useful for users wishing to build a map of a small area of the sky. <br />
<br />
<span style="color:red"> TBW: baselines for LFI</span> <br />
<br />
The HFI delivers its offsets in a ROI, or "Ring-Ordered Information" file. That file contains a table of N<sub>rings</sub> = 26766 rows by N<sub>bolometers</sub> = 50 columns in which each cell contains a 3-elements vector containing the full-ring, first half-ring, and second half-ring offsets. The indices of the signal TOI delimiting the rings are given in a second ROI file with the same N<sub>rings</sub> rows containing global parameters.<br />
<br />
===Indexing===<br />
<br />
Each FITS file contains keywords giving the first and last index of the section of the full TOI contained in the file. These indices are those of the internal DPC representation; they begin at 0 corresponding to the switch-on of the instrument, and run to some very large number (~25 billion for HFI) representing the full mission. Only the ''science'' part of the mission is exported, which is about 60% of the total for HFI. Are left out the early mission phase (cool-down and validation phases) and the warm phase of HFI, during which LFI continued to collect data. The HFI keywords also indicated which rings are included in each OD, but be aware that normally an OD does not begin at the same time (with the same index) as a ring. <br />
The HFI offsets are determined on a per-ring basis, and must be applied ring-by-ring, and for this purpose the begin and end ring indices are given in <br />
<br />
<br />
<span style="color:red"> TBW: HFI: explain TOI indices and (BEG/ENDINDEX kwds) and same for ROI </span><br />
<br />
==Production process==<br />
<br />
=== HFI processing ===<br />
<br />
The input TOIs are in ADUs representing the voltage signal at the end of the electronics (see [[HFI_detection_chain | Detection chain]] for details). The processing applied to remove instrumental effects and to calibrate them is described in detail in the [[TOI processing | HFI TOI processing]] section, and we give a very brief summary here for convenience. That pipeline performs the following operations:<br />
<br />
; ADC correction: corrects for the uneven size of the ADC bins.<br />
; demodulation: this is performed around a variable level which is determined from the valid input data (using a validity flag from a previous version of the processing), and the data are converted to engineering units (V) using known conversion coefficients.<br />
; despiking: the redundancy within a ring is used to determine where glitches occur. Once identified, the glitches are fitted with templates. A glitch flag is produced that identifies the strongest part of the glitches, and a timeline of glitch tails. produced from the template fits, is subtracted from the demodulated timeline. Finally, the flagged regions are replaced with data from an average over the pointing period in order to not leave holes in the timeline that would perturb the usage of Fourier methods on them<br />
; dark template removal: the two dark bolometers are demodulated and despiked as above; the resulting timelines are then smoothed and used as an indicator of the overall temperature variations of the bolometer plate. Where the variations are consistent with each other, they are combined and removed from the bolometer signal timelines using appropriate coupling coefficients. The few percent of the data where they are not consistent are flagged as not valid on the timelines.<br />
; conversion to absorbed power: the timeline is converted to watts of absorbed power using the bolometer function. This includes a non-linearity correction; removal of the 4K cooler lines: the electromagnetic interference of the 4K cooler with the bolometer readout wires induces some sharp lines in the signal power spectra at frequencies of the 4K cooler's fundamental and its multiples, folded by the signal modulations. Fourier coefficients of the relevant lines are determined on a per-ring basis, and then removed from the data. The quality of the removal depends on the bolometer.<br />
; deconvolution by the time transfer function: this is done to correct for the non-instantaneous time response of the bolometers. The function itself is modeled using up to 8 time constants which are adjusted primarily on the planet data and also from comparisons of the northward and southward scans of the Galactic Plane. It is then removed using Fourier techniques, which has the side-effect of increasing the noise at high frequencies.<br />
; jump correction: removes some (relatively rare: 0.3 jumps per bolometer per pointing period, on average) jumps in the signal baseline. The jumps are detected characterized on smoothed TOIs, and corrected by adding a constant to part of the signal timeline. The origin of the jumps is not known.<br />
<br />
The results of this processing are a timeline of signal (in absorbed watts) and a ''valid data'' flag timeline for each of the 50 valid bolometers which are processed, out of the 52 HFI bolometers; these timelines contain the full sky signal, i.e., including the solar and orbital dipoles, the Zodiacal light, and contributions from the Far-Side lobes. The dipoles are necessary for the flux calibration and are removed at the mapmaking stage. The remaining two bolometers (143-8 and 545-3) show semi-random jumps in the signal level, typically jumping over 2-5 different ''pseudo-baseline'' levels, a behavior known as ''Random Telegraphic Signal'', so that these are commonly called the RTS bolometers. Finally, ring-level statistics of different types (mean, median, rms, kurtosis, etc.) are determined on a per-ring basis for all timelines, and a selection based on these statistics is used to discard anomalous rings, which are recorded in a ring-level flag for each valid bolometer timeline (see the [[TOI_processing#Discarded_rings | Discarded rings]] section). <br />
<br />
Throughout this processing, some planet signals are masked in the timeline in order to avoid ringing effects in the processing. It concerns only Jupiter at 100 and 143 GHZ, Jupiter and Saturn at 217 GHz, Jupiter, Saturn and Mars at 353, 545 and 857 GHz (thanks to various improvements in the despiking and in the transfer functions it is no longer necessary to flag these three planets in the CMB channels as was done in Release 1). Since they move on the sky, the portion of the sky masked during one survey is observed during one, and no hole is left in the final map. In parallel, the planet data are processed in a similar way, but with different parameters for the despiking step. These results are processed separately to determine the beam shapes and the focal plane geometry.<br />
<br />
Finally, the TOIs are calibrated in astrophysical units using the results of the calibration the calibration pipeline, described in the [[Map-making | Map-making and calibration]] section, which are stored in the IMO, and the best estimate of the zero-point offsets (a constant level for each bolometer) and of the far-side lobes (another signal TOI determined from the input TOI) are removed. These TOIs are ready for projection onto a map. That projection normally requires that the low-frequency noise be removed via a "destriping" (sometimes called "baseline removal") step. The HFI-DPC does its destriping at ring-level; should the user want to use the DPC's offsets, they are provided separately ([##REF##]). Due to a technical reason, for rings with a duration larger than 72 minutes, only the first 72 minutes of such a ring are used to make the HFI maps which are officially delivered (the loss of data amount to a total of 1/2 hour for the 29 months of HFI data).<br />
<br />
The pointing (see common [[Detector_pointing | Detector Pointing]] section for details) is determined from the AHF produced by MOC, which gives the direction and orientation of the LOS of a fiducial position in the focal plane at frequencies of 8Hz during stable pointing and 4 Hz during maneuvers. This is interpolated to the timestamps of data samples, corrected for the wobble and other time-dependent offsets determined from the observed positions of a large number of sources around the sky, and finally converted to the LOS of each detector using the quaternions in the IMO.<br />
<br />
==File Names==<br />
<br />
The file names are of the form:<br />
<br />
''{H,L}FI_TOI_{fff}-{SCI,PTG}_R2.nn_ODxxxx.fits''<br />
<br />
where<br />
* ''fff'' denotes the frequency<br />
* SCI or PTG denote signal or pointing TOIs<br />
* R2.nn is the version, and<br />
* ODxxxx indicates the OD.<br />
<br />
==FITS file structure==<br />
<br />
All FITS files begin with a primary extension containing no data and a minimal header, which is followed by one or more ''BINTABLE'' extensions with data, and with a description of the data in the header keywords. In what follows will usually ignore the primary extension, and count only the extensions containing data.<br />
<br />
===TOI files===<br />
<br />
The signal FITS files contain N ,'BINTABLE', data extensions, where N is stye number of detectors in that frequency channel. The first extension contains the OBT and its flags, called the ''global'' flags since they apply to all detectors, which is followed by one extension for each detector, containing the signal and its ''local'' flag. The flags columns are written as ''byte'' in which each of the 8 bits (max) encodes one flag timeline. The meaning of each bit is given in the header comments and is subject to change. For the global flag they include:<br />
* Unstable pointing: 1= pointing is not stable (e.g., during repointing maneuvers)<br />
* Dark correlation: 1 = darks are uncorrelated and data are flagged<br />
* First/Second half ring: which samples are in which half (only covers the stable pointing part of the ring)<br />
* HCM: in HCM mode (unstable pointing)<br />
* and more<br />
<br />
And for the local flag they include<br />
<br />
* Total Flag: a combination of the various flags that is the one finally used in the map-making<br />
* Data not valid: glitched samples<br />
* Despike Common: (for PSBs only) glitch on current or other of PSB pair<br />
* StrongSignal: on Galactic Plane<br />
* Strong Source: on point source<br />
* other<br />
<br />
<br />
{| border="1" cellpadding="3" cellspacing="0" align="center" style="text-align:left" width=800px<br />
|+ '''TOI file data structure'''<br />
<br />
|- bgcolor="ffdead" <br />
!colspan="4" | 1. EXTNAME = 'OBT' : Data columns<br />
|- bgcolor="ffdead" <br />
! Column Name || Data Type || Units || Description<br />
|-<br />
|OBT || Int*8 || 2<sup>-16</sup> sec || On-board time<br />
|-<br />
|FLAG || Byte || none || the various bit-level flags<br />
<br />
|- bgcolor="ffdead" <br />
! Keyword || Data Type || Value || Description<br />
|-<br />
|OD || Int || || OD covered (as in filename)<br />
|-<br />
|BEGIDX || Int || || first sample index of given OD<br />
|-<br />
|ENDIDX || Int || || last sample index of given OD<br />
|-<br />
|BEGRING || Int || || first ring in given OD<br />
|-<br />
|ENDRING || Int || || last ring in given OD<br />
|-<br />
|TIMEZERO || String || 1958-01-01z00:00 || Origin of OBT<br />
<br />
|- bgcolor="ffdead" <br />
!colspan="4" | n. EXTNAME = ''DETNAME'' : Data columns<br />
<br />
|- bgcolor="ffdead" <br />
! Column Name || Data Type || Units || Description<br />
|-<br />
|SIGNAL || Real*4 || K<sub>cmb</sub> or MJy/sr || Value of signal<br />
|-<br />
|FLAG || Byte || none || the various bit-level flags<br />
|- bgcolor="ffdead" <br />
<br />
! Keyword || Data Type || Value || Description<br />
|-<br />
|UNIT || String || || Units of signal<br />
|-<br />
|DESTRIPE || 1/0 || || whether timeline is destriped<br />
|-<br />
|OD || Int || || OD covered (as in filename)<br />
|-<br />
|BEGIDX || Int || || first sample index of given OD<br />
|-<br />
|ENDIDX || Int || || last sample index of given OD<br />
|-<br />
|BEGRING || Int || || first ring in given OD<br />
|-<br />
|ENDRING || Int || || last ring in given OD<br />
|}<br />
<br />
<br />
The pointing files have a similar structure, except that ''DETNAME'' extensions contain 3 columns of Real*8 with the phi, theta, and psi Galactic spherical coordinates of each sample in radians. There is no local flag for the coordinates. <br />
<br />
<br />
===ROI files===<br />
<br />
The files provided by HFI are<br />
* ''HFI_ROI_GlobalParams_RelNum_full.fits''<br />
* ''HFI_ROI_DestripeOffsets_RelNum_full.fits''<br />
which are described below.<br />
<br />
;Global parameters: this ROI file contains a single ''BINTABLE'' extension with 3 columns that give, for each ring, the index number of the first sample, the ESA pointing ID of the ring, and the ring start time. All three are encoded as Int*8, and the time is given in units of 2<sup>-16</sup> sec from ''TIMEZERO'', 1958-01-01z00:00, as in the TOI files. Rings are numbered from 240 to 27005, as given in the ''BEGRING'' and ''ENDRING'' keywords<br />
<br />
; Destriping offsets: this ROI file contains a single ''BINTABLE'' extension with 3 columns that give, for each ring, the offsets to subtract from each bolometer signal timeline or vector. There are three offsets available for each bolometer: full-ring, first half-ring, and second half-ring, for building the full-ring or the two half-ring maps, respectively. The offsets must be subtracted from Index(N) to Index(N+1)-1 of the corresponding signal timeline, where N is stye ring number, and the indices are given in the Global parameters file. The offsets are given in the same units as the signal vectors, that is K<sub>cmv</sub> for the 100-353 GHz channels, and MJy/sr for the 545 and 857 GHz channels. All values are encoded as Real*4. The rings for which the offsets are set to 0.000 are [[TOI_processing#Discarded_rings | ''badrings'' ]] which are rejected in the mapmaking process.<br />
<br />
<span style="color:red"> TBW: Note on half-ring offsets, which differ by 1-2% from full ring offsets</span> <br />
<br />
[[Category:Mission products|001]]</div>
Fdesert
https://wiki.cosmos.esa.int/planckpla2015/index.php?title=Timelines&diff=9603
Timelines
2014-09-29T13:56:05Z
<p>Fdesert: /* HFI processing */</p>
<hr />
<div>==General description==<br />
<br />
The timelines, or TOIs for ''Time-Ordered Information'', are vectors of signal or of pointing or of some other quantity giving the value of that quantity as a function of time during the mission. The TOIs described here are sampled regularly at the (instrument dependent) detector sampling frequency and span the full science mission. They thus consists of a large amount of data. This section is concerned with the signal TOIs and their associated pointing TOIs. For each bolometer the DPCs provide a single timeline of calibrated signal and 3 timelines of coordinates, corresponding to the two angular coordinates and one position angle. <br />
<br />
Obviously these vectors are very long (~1.38E10 samples for HFI, nnnn for LFI) and thus need to be split into multiple files for export. Here the data are split by Operational Day (OD) as follows:<br />
* HFI: 12 files per OD, one for pointing and one for signal for each channel, covering ODs 91 -- 974, all sampled at F<sub>samp</sub> = 180.3737 Hz. <br />
* LFI: 6 files per OD, 3 each for signal and pointing, since the sampling frequency is channel dependent, and covering ODs 91--<span style="color:red">NNN</span>.<br />
<br />
The signal timelines are encoded as Real*4, but the pointing vectors had to be encoded as Real*8 to maintain the required accuracy. The result is that the total volume of the full dataset is ~30 TB. All files also contain the OBT.<br />
<br />
The signal timelines have been cleaned of all known instrumental systematic effects, they have been calibrated in flux, and the solar and earth motion dipole signals have been removed. But they have not been cleaned of the low-frequency, ''1/f'' noise that needs to be removed via a destriping tool. The methods consists of removing offsets or ''baselines'' determined by minimizing the differences in the signals at points on the sky where they intersect. For HFI, an offset per ring is determined; <span style="color:red"> For LFI ..... </span>. These offsets are determined using the full mission and all the valid detectors per channel, and they are then used for all the maps produced, i.e., for those maps using any fraction of the mission (year, survey) or any subset of the detectors (single detector, detector set), and it was shown that using offsets determined from a limited part of the mission and/or a subset of the detectors yields maps that are less consistent than otherwise. The offsets are delivered separately, as described below.<br />
<br />
<span style="color:red"> Maybe insert a figure for example? </span><br />
<br />
Furthermore, for HFI, there are three sets of offsets produced: the primary set using the full rings, and the secondary ones using the first and second half rings, only. The difference between the primary and the secondary sets are fairly minor, but they are necessary to rebuild the maps as they were built by the HFI-DPC. Overall the offsets are useful for users wishing to build a map of a small area of the sky. <br />
<br />
<span style="color:red"> TBW: baselines for LFI</span> <br />
<br />
The HFI delivers its offsets in a ROI, or "Ring-Ordered Information" file. That file contains a table of N<sub>rings</sub> = 26766 rows by N<sub>bolometers</sub> = 50 columns in which each cell contains a 3-elements vector containing the full-ring, first half-ring, and second half-ring offsets. The indices of the signal TOI delimiting the rings are given in a second ROI file with the same N<sub>rings</sub> rows containing global parameters.<br />
<br />
===Indexing===<br />
<br />
Each FITS file contains keywords giving the first and last index of the section of the full TOI contained in the file. These indices are those of the internal DPC representation; they begin at 0 corresponding to the switch-on of the instrument, and run to some very large number (~25 billion for HFI) representing the full mission. Only the ''science'' part of the mission is exported, which is about 60% of the total for HFI. Are left out the early mission phase (cool-down and validation phases) and the warm phase of HFI, during which LFI continued to collect data. The HFI keywords also indicated which rings are included in each OD, but be aware that normally an OD does not begin at the same time (with the same index) as a ring. <br />
The HFI offsets are determined on a per-ring basis, and must be applied ring-by-ring, and for this purpose the begin and end ring indices are given in <br />
<br />
<br />
<span style="color:red"> TBW: HFI: explain TOI indices and (BEG/ENDINDEX kwds) and same for ROI </span><br />
<br />
==Production process==<br />
<br />
=== HFI processing ===<br />
<br />
The input TOIs are in ADUs representing the voltage signal at the end of the electronics (see [[HFI_detection_chain | Detection chain]] for details). The processing applied to remove instrumental effects and to calibrate them is described in detail in the [[TOI processing | HFI TOI processing]] section, and we give a very brief summary here for convenience. That pipeline performs the following operations:<br />
<br />
; ADC correction: corrects for the uneven size of the ADC bins.<br />
; demodulation: this is performed around a variable level which is determined from the valid input data (using a validity flag from a previous version of the processing), and the data are converted to engineering units (V) using known conversion coefficients.<br />
; despiking: the redundancy within a ring is used to determine where glitches occur. Once identified, the glitches are fitted with templates. A glitch flag is produced that identifies the strongest part of the glitches, and a timeline of glitch tails. produced from the template fits, is subtracted from the demodulated timeline. Finally, the flagged regions are replaced with data from an average over the pointing period in order to not leave holes in the timeline that would perturb the usage of Fourier methods on them<br />
; dark template removal: the two dark bolometers are demodulated and despiked as above; the resulting timelines are then smoothed and used as an indicator of the overall temperature variations of the bolometer plate. Where the variations are consistent with each other, they are combined and removed from the bolometer signal timelines using appropriate coupling coefficients. The few percent of the data where they are not consistent are flagged as not valid on the timelines.<br />
; conversion to absorbed power: the timeline is converted to watts of absorbed power using the bolometer function. This includes a non-linearity correction; removal of the 4K cooler lines: the electromagnetic interference of the 4K cooler with the bolometer readout wires induces some sharp lines in the signal power spectra at frequencies of the 4K cooler's fundamental and its multiples, folded by the signal modulations. Fourier coefficients of the relevant lines are determined on a per-ring basis, and then removed from the data. The quality of the removal depends on the bolometer.<br />
; deconvolution by the time transfer function: this is done to correct for the non-instantaneous time response of the bolometers. The function itself is modeled using up to 8 time constants which are adjusted primarily on the planet data and also from comparisons of the northward and southward scans of the Galactic Plane. It is then removed using Fourier techniques, which has the side-effect of increasing the noise at high frequencies.<br />
; jump correction: removes some (relatively rare: 0.3 jumps per bolometer per pointing period, on average) jumps in the signal baseline. The jumps are detected characterized on smoothed TOIs, and corrected by adding a constant to part of the signal timeline. The origin of the jumps is not known.<br />
<br />
The results of this processing are a timeline of signal (in absorbed watts) and a ''valid data'' flag timeline for each of the 50 valid bolometers which are processed, out of the 52 HFI bolometers; these timelines contain the full sky signal, i.e., including the solar and orbital dipoles, the Zodiacal light, and contributions from the Far-Side lobes. The dipoles are necessary for the flux calibration and are removed at the mapmaking stage. The remaining two bolometers (143-8 and 545-3) show semi-random jumps in the signal level, typically jumping over 2-5 different ''pseudo-baseline'' levels, a behavior known as ''Random Telegraphic Signal'', so that these are commonly called the RTS bolometers. Finally, ring-level statistics of different types (mean, median, rms, kurtosis, etc.) are determined on a per-ring basis for all timelines, and a selection based on these statistics is used to discard anomalous rings, which are recorded in a ring-level flag for each valid bolometer timeline (see the [[TOI_processing#Discarded_rings | Discarded rings]] section). <br />
<br />
Throughout this processing, some planet signals are masked in the timeline in order to avoid ringing effects in the processing. It concerns only Jupiter at 100 and 143 GHZ, Jupiter and Saturn at 217 GHz, Jupiter, Saturn and Mars at 353, 545 and 857 GHz (thanks to various improvements in the despiking and in the transfer functions it is no longer necessary to flag these three planets in the CMB channels as was done in Release 1). Since they move on the sky, the portion of the sky masked during one survey is observed during one, and no hole is left in the final map. In parallel, the planet data are processed in a similar way, but with different parameters for the despiking step. These results are processed separately to determine the beam shapes and the focal plane geometry.<br />
<br />
Finally, the TOIs are calibrated in astrophysical units using the results of the calibration the calibration pipeline, described in the [[Map-making | Map-making and calibration]] section, which are stored in the IMO, and the best estimate of the zero-point offsets (a constant level for each bolometer) and of the far-side lobes (another signal TOI determined from the input TOI) are removed. These TOIs are ready for projection onto a map. That projection normally requires that the low-frequency noise be removed via a "destriping" (sometimes called "baseline removal") step. The HFI-DPC does its destriping at ring-level; should the user want to use the DPC's offsets, they are provided separately ([##REF##]).<br />
<br />
The pointing (see common [[Detector_pointing | Detector Pointing]] section for details) is determined from the AHF produced by MOC, which gives the direction and orientation of the LOS of a fiducial position in the focal plane at frequencies of 8Hz during stable pointing and 4 Hz during maneuvers. This is interpolated to the timestamps of data samples, corrected for the wobble and other time-dependent offsets determined from the observed positions of a large number of sources around the sky, and finally converted to the LOS of each detector using the quaternions in the IMO.<br />
<br />
==File Names==<br />
<br />
The file names are of the form:<br />
<br />
''{H,L}FI_TOI_{fff}-{SCI,PTG}_R2.nn_ODxxxx.fits''<br />
<br />
where<br />
* ''fff'' denotes the frequency<br />
* SCI or PTG denote signal or pointing TOIs<br />
* R2.nn is the version, and<br />
* ODxxxx indicates the OD.<br />
<br />
==FITS file structure==<br />
<br />
All FITS files begin with a primary extension containing no data and a minimal header, which is followed by one or more ''BINTABLE'' extensions with data, and with a description of the data in the header keywords. In what follows will usually ignore the primary extension, and count only the extensions containing data.<br />
<br />
===TOI files===<br />
<br />
The signal FITS files contain N ,'BINTABLE', data extensions, where N is stye number of detectors in that frequency channel. The first extension contains the OBT and its flags, called the ''global'' flags since they apply to all detectors, which is followed by one extension for each detector, containing the signal and its ''local'' flag. The flags columns are written as ''byte'' in which each of the 8 bits (max) encodes one flag timeline. The meaning of each bit is given in the header comments and is subject to change. For the global flag they include:<br />
* Unstable pointing: 1= pointing is not stable (e.g., during repointing maneuvers)<br />
* Dark correlation: 1 = darks are uncorrelated and data are flagged<br />
* First/Second half ring: which samples are in which half (only covers the stable pointing part of the ring)<br />
* HCM: in HCM mode (unstable pointing)<br />
* and more<br />
<br />
And for the local flag they include<br />
<br />
* Total Flag: a combination of the various flags that is the one finally used in the map-making<br />
* Data not valid: glitched samples<br />
* Despike Common: (for PSBs only) glitch on current or other of PSB pair<br />
* StrongSignal: on Galactic Plane<br />
* Strong Source: on point source<br />
* other<br />
<br />
<br />
{| border="1" cellpadding="3" cellspacing="0" align="center" style="text-align:left" width=800px<br />
|+ '''TOI file data structure'''<br />
<br />
|- bgcolor="ffdead" <br />
!colspan="4" | 1. EXTNAME = 'OBT' : Data columns<br />
|- bgcolor="ffdead" <br />
! Column Name || Data Type || Units || Description<br />
|-<br />
|OBT || Int*8 || 2<sup>-16</sup> sec || On-board time<br />
|-<br />
|FLAG || Byte || none || the various bit-level flags<br />
<br />
|- bgcolor="ffdead" <br />
! Keyword || Data Type || Value || Description<br />
|-<br />
|OD || Int || || OD covered (as in filename)<br />
|-<br />
|BEGIDX || Int || || first sample index of given OD<br />
|-<br />
|ENDIDX || Int || || last sample index of given OD<br />
|-<br />
|BEGRING || Int || || first ring in given OD<br />
|-<br />
|ENDRING || Int || || last ring in given OD<br />
|-<br />
|TIMEZERO || String || 1958-01-01z00:00 || Origin of OBT<br />
<br />
|- bgcolor="ffdead" <br />
!colspan="4" | n. EXTNAME = ''DETNAME'' : Data columns<br />
<br />
|- bgcolor="ffdead" <br />
! Column Name || Data Type || Units || Description<br />
|-<br />
|SIGNAL || Real*4 || K<sub>cmb</sub> or MJy/sr || Value of signal<br />
|-<br />
|FLAG || Byte || none || the various bit-level flags<br />
|- bgcolor="ffdead" <br />
<br />
! Keyword || Data Type || Value || Description<br />
|-<br />
|UNIT || String || || Units of signal<br />
|-<br />
|DESTRIPE || 1/0 || || whether timeline is destriped<br />
|-<br />
|OD || Int || || OD covered (as in filename)<br />
|-<br />
|BEGIDX || Int || || first sample index of given OD<br />
|-<br />
|ENDIDX || Int || || last sample index of given OD<br />
|-<br />
|BEGRING || Int || || first ring in given OD<br />
|-<br />
|ENDRING || Int || || last ring in given OD<br />
|}<br />
<br />
<br />
The pointing files have a similar structure, except that ''DETNAME'' extensions contain 3 columns of Real*8 with the phi, theta, and psi Galactic spherical coordinates of each sample in radians. There is no local flag for the coordinates. <br />
<br />
<br />
===ROI files===<br />
<br />
The files provided by HFI are<br />
* ''HFI_ROI_GlobalParams_RelNum_full.fits''<br />
* ''HFI_ROI_DestripeOffsets_RelNum_full.fits''<br />
which are described below.<br />
<br />
;Global parameters: this ROI file contains a single ''BINTABLE'' extension with 3 columns that give, for each ring, the index number of the first sample, the ESA pointing ID of the ring, and the ring start time. All three are encoded as Int*8, and the time is given in units of 2<sup>-16</sup> sec from ''TIMEZERO'', 1958-01-01z00:00, as in the TOI files. Rings are numbered from 240 to 27005, as given in the ''BEGRING'' and ''ENDRING'' keywords<br />
<br />
; Destriping offsets: this ROI file contains a single ''BINTABLE'' extension with 3 columns that give, for each ring, the offsets to subtract from each bolometer signal timeline or vector. There are three offsets available for each bolometer: full-ring, first half-ring, and second half-ring, for building the full-ring or the two half-ring maps, respectively. The offsets must be subtracted from Index(N) to Index(N+1)-1 of the corresponding signal timeline, where N is stye ring number, and the indices are given in the Global parameters file. The offsets are given in the same units as the signal vectors, that is K<sub>cmv</sub> for the 100-353 GHz channels, and MJy/sr for the 545 and 857 GHz channels. All values are encoded as Real*4. The rings for which the offsets are set to 0.000 are [[TOI_processing#Discarded_rings | ''badrings'' ]] which are rejected in the mapmaking process.<br />
<br />
<span style="color:red"> TBW: Note on half-ring offsets, which differ by 1-2% from full ring offsets</span> <br />
<br />
[[Category:Mission products|001]]</div>
Fdesert
https://wiki.cosmos.esa.int/planckpla2015/index.php?title=Timelines&diff=9602
Timelines
2014-09-29T13:54:07Z
<p>Fdesert: /* HFI processing */</p>
<hr />
<div>==General description==<br />
<br />
The timelines, or TOIs for ''Time-Ordered Information'', are vectors of signal or of pointing or of some other quantity giving the value of that quantity as a function of time during the mission. The TOIs described here are sampled regularly at the (instrument dependent) detector sampling frequency and span the full science mission. They thus consists of a large amount of data. This section is concerned with the signal TOIs and their associated pointing TOIs. For each bolometer the DPCs provide a single timeline of calibrated signal and 3 timelines of coordinates, corresponding to the two angular coordinates and one position angle. <br />
<br />
Obviously these vectors are very long (~1.38E10 samples for HFI, nnnn for LFI) and thus need to be split into multiple files for export. Here the data are split by Operational Day (OD) as follows:<br />
* HFI: 12 files per OD, one for pointing and one for signal for each channel, covering ODs 91 -- 974, all sampled at F<sub>samp</sub> = 180.3737 Hz. <br />
* LFI: 6 files per OD, 3 each for signal and pointing, since the sampling frequency is channel dependent, and covering ODs 91--<span style="color:red">NNN</span>.<br />
<br />
The signal timelines are encoded as Real*4, but the pointing vectors had to be encoded as Real*8 to maintain the required accuracy. The result is that the total volume of the full dataset is ~30 TB. All files also contain the OBT.<br />
<br />
The signal timelines have been cleaned of all known instrumental systematic effects, they have been calibrated in flux, and the solar and earth motion dipole signals have been removed. But they have not been cleaned of the low-frequency, ''1/f'' noise that needs to be removed via a destriping tool. The methods consists of removing offsets or ''baselines'' determined by minimizing the differences in the signals at points on the sky where they intersect. For HFI, an offset per ring is determined; <span style="color:red"> For LFI ..... </span>. These offsets are determined using the full mission and all the valid detectors per channel, and they are then used for all the maps produced, i.e., for those maps using any fraction of the mission (year, survey) or any subset of the detectors (single detector, detector set), and it was shown that using offsets determined from a limited part of the mission and/or a subset of the detectors yields maps that are less consistent than otherwise. The offsets are delivered separately, as described below.<br />
<br />
<span style="color:red"> Maybe insert a figure for example? </span><br />
<br />
Furthermore, for HFI, there are three sets of offsets produced: the primary set using the full rings, and the secondary ones using the first and second half rings, only. The difference between the primary and the secondary sets are fairly minor, but they are necessary to rebuild the maps as they were built by the HFI-DPC. Overall the offsets are useful for users wishing to build a map of a small area of the sky. <br />
<br />
<span style="color:red"> TBW: baselines for LFI</span> <br />
<br />
The HFI delivers its offsets in a ROI, or "Ring-Ordered Information" file. That file contains a table of N<sub>rings</sub> = 26766 rows by N<sub>bolometers</sub> = 50 columns in which each cell contains a 3-elements vector containing the full-ring, first half-ring, and second half-ring offsets. The indices of the signal TOI delimiting the rings are given in a second ROI file with the same N<sub>rings</sub> rows containing global parameters.<br />
<br />
===Indexing===<br />
<br />
Each FITS file contains keywords giving the first and last index of the section of the full TOI contained in the file. These indices are those of the internal DPC representation; they begin at 0 corresponding to the switch-on of the instrument, and run to some very large number (~25 billion for HFI) representing the full mission. Only the ''science'' part of the mission is exported, which is about 60% of the total for HFI. Are left out the early mission phase (cool-down and validation phases) and the warm phase of HFI, during which LFI continued to collect data. The HFI keywords also indicated which rings are included in each OD, but be aware that normally an OD does not begin at the same time (with the same index) as a ring. <br />
The HFI offsets are determined on a per-ring basis, and must be applied ring-by-ring, and for this purpose the begin and end ring indices are given in <br />
<br />
<br />
<span style="color:red"> TBW: HFI: explain TOI indices and (BEG/ENDINDEX kwds) and same for ROI </span><br />
<br />
==Production process==<br />
<br />
=== HFI processing ===<br />
<br />
The input TOIs are in ADUs representing the voltage signal at the end of the electronics (see [[HFI_detection_chain | Detection chain]] for details). The processing applied to remove instrumental effects and to calibrate them is described in detail in the [[TOI processing | HFI TOI processing]] section, and we give a very brief summary here for convenience. That pipeline performs the following operations:<br />
<br />
; ADC correction: corrects for the uneven size of the ADC bins.<br />
; demodulation: this is performed around a variable level which is determined from the valid input data (using a validity flag from a previous version of the processing), and the data are converted to engineering units (V) using known conversion coefficients.<br />
; despiking: the redundancy within a ring is used to determine where glitches occur. Once identified, the glitches are fitted with templates. A glitch flag is produced that identifies the strongest part of the glitches, and a timeline of glitch tails. produced from the template fits, is subtracted from the demodulated timeline. Finally, the flagged regions are replaced with data from an average over the pointing period in order to not leave holes in the timeline that would perturb the usage of Fourier methods on them<br />
; dark template removal: the two dark bolometers are demodulated and despiked as above; the resulting timelines are then smoothed and used as an indicator of the overall temperature variations of the bolometer plate. Where the variations are consistent with each other, they are combined and removed from the bolometer signal timelines using appropriate coupling coefficients. The few percent of the data where they are not consistent are flagged as not valid on the timelines.<br />
; conversion to absorbed power: the timeline is converted to watts of absorbed power using the bolometer function. This includes a non-linearity correction; removal of the 4K cooler lines: the electromagnetic interference of the 4K cooler with the bolometer readout wires induces some sharp lines in the signal power spectra at frequencies of the 4K cooler's fundamental and its multiples, folded by the signal modulations. Fourier coefficients of the relevant lines are determined on a per-ring basis, and then removed from the data. The quality of the removal depends on the bolometer.<br />
; deconvolution by the time transfer function: this is done to correct for the non-instantaneous time response of the bolometers. The function itself is modeled using up to 8 time constants which are adjusted primarily on the planet data and also from comparisons of the northward and southward scans of the Galactic Plane. It is then removed using Fourier techniques, which has the side-effect of increasing the noise at high frequencies.<br />
; jump correction: removes some (relatively rare: 0.3 jumps per bolometer per pointing period, on average) jumps in the signal baseline. The jumps are detected characterized on smoothed TOIs, and corrected by adding a constant to part of the signal timeline. The origin of the jumps is not known.<br />
<br />
The results of this processing are a timeline of signal (in absorbed watts) and a ''valid data'' flag timeline for each of the 50 valid bolometers which are processed, out of the 52 HFI bolometers; these timelines contain the full sky signal, i.e., including the solar and orbital dipoles, the Zodiacal light, and contributions from the Far-Side lobes. The dipoles are necessary for the flux calibration and are removed at the mapmaking stage. The remaining two bolometers (143-8 and 545-3) show semi-random jumps in the signal level, typically jumping over 2-5 different ''pseudo-baseline'' levels, a behavior known as ''Random Telegraphic Signal'', so that these are commonly called the RTS bolometers. Finally, ring-level statistics of different types (mean, median, rms, kurtosis, etc.) are determined on a per-ring basis for all timelines, and a selection based on these statistics is used to discard anomalous rings, which are recorded in a ring-level flag for each valid bolometer timeline (see the [[TOI_processing#Discarded_rings | Discarded rings]] section). <br />
<br />
Throughout this processing, some planet signals are masked in the timeline in order to avoid ringing effects in the processing. It concerns only Jupiter at 100 and 143 GHZ, Jupiter and Saturn at 217 GHz, Jupiter, Saturn and Mars at 353, 545 and 857 GHz (thanks to various improvements in the despiking and in the transfer functions it is no longer necessary to flag these three planets in the CMB channels as was done in Release 1). Since they move on the sky, the portion of the sky masked during one survey is observed during one, and no hole is left in the final map. In parallel, the planet data are processed in a similar way, but with different parameters for the despiking step. These results are processed separately to determine the beam shapes and the focal plane geometry.<br />
<br />
Finally, the TOIs are calibrated in astrophysical units using the results of the calibration the calibration pipeline, described in the [[Map-making | Map-making and calibration]] section, which are stored in the IMO, and the best estimate of the zero-point offsets (a constant level for each bolometer) and of the far-side lobes (another signal TOI determined from the input TOI) are removed. These TOIs are ready for projection onto a map. That projection normally requires that the low-frequency noise be removed via a "destriping" (sometimes called "baseline removal") step. The HFI-DPC does its destriping at ring-level; should the user want to use the DPC's offsets, they are provided separately ([##REF##]).<br />
<br />
The pointing (see common [[Detector_pointing | Detector Pointing]] section for details) is determined from the AHF produced by MOC, which gives the direction and orientation of the LOS of a fiducial position in the focal plane at frequencies of 8Hz during stable pointing and 4 Hz during maneuvers. This is interpolated to the times of data observations, corrected for the wobble and other time-dependent offsets determined from the observed positions of a large number of sources around the sky, and finally converted to the LOS of each detector using the quaternions in the IMO.<br />
<br />
==File Names==<br />
<br />
The file names are of the form:<br />
<br />
''{H,L}FI_TOI_{fff}-{SCI,PTG}_R2.nn_ODxxxx.fits''<br />
<br />
where<br />
* ''fff'' denotes the frequency<br />
* SCI or PTG denote signal or pointing TOIs<br />
* R2.nn is the version, and<br />
* ODxxxx indicates the OD.<br />
<br />
==FITS file structure==<br />
<br />
All FITS files begin with a primary extension containing no data and a minimal header, which is followed by one or more ''BINTABLE'' extensions with data, and with a description of the data in the header keywords. In what follows will usually ignore the primary extension, and count only the extensions containing data.<br />
<br />
===TOI files===<br />
<br />
The signal FITS files contain N ,'BINTABLE', data extensions, where N is stye number of detectors in that frequency channel. The first extension contains the OBT and its flags, called the ''global'' flags since they apply to all detectors, which is followed by one extension for each detector, containing the signal and its ''local'' flag. The flags columns are written as ''byte'' in which each of the 8 bits (max) encodes one flag timeline. The meaning of each bit is given in the header comments and is subject to change. For the global flag they include:<br />
* Unstable pointing: 1= pointing is not stable (e.g., during repointing maneuvers)<br />
* Dark correlation: 1 = darks are uncorrelated and data are flagged<br />
* First/Second half ring: which samples are in which half (only covers the stable pointing part of the ring)<br />
* HCM: in HCM mode (unstable pointing)<br />
* and more<br />
<br />
And for the local flag they include<br />
<br />
* Total Flag: a combination of the various flags that is the one finally used in the map-making<br />
* Data not valid: glitched samples<br />
* Despike Common: (for PSBs only) glitch on current or other of PSB pair<br />
* StrongSignal: on Galactic Plane<br />
* Strong Source: on point source<br />
* other<br />
<br />
<br />
{| border="1" cellpadding="3" cellspacing="0" align="center" style="text-align:left" width=800px<br />
|+ '''TOI file data structure'''<br />
<br />
|- bgcolor="ffdead" <br />
!colspan="4" | 1. EXTNAME = 'OBT' : Data columns<br />
|- bgcolor="ffdead" <br />
! Column Name || Data Type || Units || Description<br />
|-<br />
|OBT || Int*8 || 2<sup>-16</sup> sec || On-board time<br />
|-<br />
|FLAG || Byte || none || the various bit-level flags<br />
<br />
|- bgcolor="ffdead" <br />
! Keyword || Data Type || Value || Description<br />
|-<br />
|OD || Int || || OD covered (as in filename)<br />
|-<br />
|BEGIDX || Int || || first sample index of given OD<br />
|-<br />
|ENDIDX || Int || || last sample index of given OD<br />
|-<br />
|BEGRING || Int || || first ring in given OD<br />
|-<br />
|ENDRING || Int || || last ring in given OD<br />
|-<br />
|TIMEZERO || String || 1958-01-01z00:00 || Origin of OBT<br />
<br />
|- bgcolor="ffdead" <br />
!colspan="4" | n. EXTNAME = ''DETNAME'' : Data columns<br />
<br />
|- bgcolor="ffdead" <br />
! Column Name || Data Type || Units || Description<br />
|-<br />
|SIGNAL || Real*4 || K<sub>cmb</sub> or MJy/sr || Value of signal<br />
|-<br />
|FLAG || Byte || none || the various bit-level flags<br />
|- bgcolor="ffdead" <br />
<br />
! Keyword || Data Type || Value || Description<br />
|-<br />
|UNIT || String || || Units of signal<br />
|-<br />
|DESTRIPE || 1/0 || || whether timeline is destriped<br />
|-<br />
|OD || Int || || OD covered (as in filename)<br />
|-<br />
|BEGIDX || Int || || first sample index of given OD<br />
|-<br />
|ENDIDX || Int || || last sample index of given OD<br />
|-<br />
|BEGRING || Int || || first ring in given OD<br />
|-<br />
|ENDRING || Int || || last ring in given OD<br />
|}<br />
<br />
<br />
The pointing files have a similar structure, except that ''DETNAME'' extensions contain 3 columns of Real*8 with the phi, theta, and psi Galactic spherical coordinates of each sample in radians. There is no local flag for the coordinates. <br />
<br />
<br />
===ROI files===<br />
<br />
The files provided by HFI are<br />
* ''HFI_ROI_GlobalParams_RelNum_full.fits''<br />
* ''HFI_ROI_DestripeOffsets_RelNum_full.fits''<br />
which are described below.<br />
<br />
;Global parameters: this ROI file contains a single ''BINTABLE'' extension with 3 columns that give, for each ring, the index number of the first sample, the ESA pointing ID of the ring, and the ring start time. All three are encoded as Int*8, and the time is given in units of 2<sup>-16</sup> sec from ''TIMEZERO'', 1958-01-01z00:00, as in the TOI files. Rings are numbered from 240 to 27005, as given in the ''BEGRING'' and ''ENDRING'' keywords<br />
<br />
; Destriping offsets: this ROI file contains a single ''BINTABLE'' extension with 3 columns that give, for each ring, the offsets to subtract from each bolometer signal timeline or vector. There are three offsets available for each bolometer: full-ring, first half-ring, and second half-ring, for building the full-ring or the two half-ring maps, respectively. The offsets must be subtracted from Index(N) to Index(N+1)-1 of the corresponding signal timeline, where N is stye ring number, and the indices are given in the Global parameters file. The offsets are given in the same units as the signal vectors, that is K<sub>cmv</sub> for the 100-353 GHz channels, and MJy/sr for the 545 and 857 GHz channels. All values are encoded as Real*4. The rings for which the offsets are set to 0.000 are [[TOI_processing#Discarded_rings | ''badrings'' ]] which are rejected in the mapmaking process.<br />
<br />
<span style="color:red"> TBW: Note on half-ring offsets, which differ by 1-2% from full ring offsets</span> <br />
<br />
[[Category:Mission products|001]]</div>
Fdesert
https://wiki.cosmos.esa.int/planckpla2015/index.php?title=Timelines&diff=9601
Timelines
2014-09-29T13:51:34Z
<p>Fdesert: /* HFI processing */</p>
<hr />
<div>==General description==<br />
<br />
The timelines, or TOIs for ''Time-Ordered Information'', are vectors of signal or of pointing or of some other quantity giving the value of that quantity as a function of time during the mission. The TOIs described here are sampled regularly at the (instrument dependent) detector sampling frequency and span the full science mission. They thus consists of a large amount of data. This section is concerned with the signal TOIs and their associated pointing TOIs. For each bolometer the DPCs provide a single timeline of calibrated signal and 3 timelines of coordinates, corresponding to the two angular coordinates and one position angle. <br />
<br />
Obviously these vectors are very long (~1.38E10 samples for HFI, nnnn for LFI) and thus need to be split into multiple files for export. Here the data are split by Operational Day (OD) as follows:<br />
* HFI: 12 files per OD, one for pointing and one for signal for each channel, covering ODs 91 -- 974, all sampled at F<sub>samp</sub> = 180.3737 Hz. <br />
* LFI: 6 files per OD, 3 each for signal and pointing, since the sampling frequency is channel dependent, and covering ODs 91--<span style="color:red">NNN</span>.<br />
<br />
The signal timelines are encoded as Real*4, but the pointing vectors had to be encoded as Real*8 to maintain the required accuracy. The result is that the total volume of the full dataset is ~30 TB. All files also contain the OBT.<br />
<br />
The signal timelines have been cleaned of all known instrumental systematic effects, they have been calibrated in flux, and the solar and earth motion dipole signals have been removed. But they have not been cleaned of the low-frequency, ''1/f'' noise that needs to be removed via a destriping tool. The methods consists of removing offsets or ''baselines'' determined by minimizing the differences in the signals at points on the sky where they intersect. For HFI, an offset per ring is determined; <span style="color:red"> For LFI ..... </span>. These offsets are determined using the full mission and all the valid detectors per channel, and they are then used for all the maps produced, i.e., for those maps using any fraction of the mission (year, survey) or any subset of the detectors (single detector, detector set), and it was shown that using offsets determined from a limited part of the mission and/or a subset of the detectors yields maps that are less consistent than otherwise. The offsets are delivered separately, as described below.<br />
<br />
<span style="color:red"> Maybe insert a figure for example? </span><br />
<br />
Furthermore, for HFI, there are three sets of offsets produced: the primary set using the full rings, and the secondary ones using the first and second half rings, only. The difference between the primary and the secondary sets are fairly minor, but they are necessary to rebuild the maps as they were built by the HFI-DPC. Overall the offsets are useful for users wishing to build a map of a small area of the sky. <br />
<br />
<span style="color:red"> TBW: baselines for LFI</span> <br />
<br />
The HFI delivers its offsets in a ROI, or "Ring-Ordered Information" file. That file contains a table of N<sub>rings</sub> = 26766 rows by N<sub>bolometers</sub> = 50 columns in which each cell contains a 3-elements vector containing the full-ring, first half-ring, and second half-ring offsets. The indices of the signal TOI delimiting the rings are given in a second ROI file with the same N<sub>rings</sub> rows containing global parameters.<br />
<br />
===Indexing===<br />
<br />
Each FITS file contains keywords giving the first and last index of the section of the full TOI contained in the file. These indices are those of the internal DPC representation; they begin at 0 corresponding to the switch-on of the instrument, and run to some very large number (~25 billion for HFI) representing the full mission. Only the ''science'' part of the mission is exported, which is about 60% of the total for HFI. Are left out the early mission phase (cool-down and validation phases) and the warm phase of HFI, during which LFI continued to collect data. The HFI keywords also indicated which rings are included in each OD, but be aware that normally an OD does not begin at the same time (with the same index) as a ring. <br />
The HFI offsets are determined on a per-ring basis, and must be applied ring-by-ring, and for this purpose the begin and end ring indices are given in <br />
<br />
<br />
<span style="color:red"> TBW: HFI: explain TOI indices and (BEG/ENDINDEX kwds) and same for ROI </span><br />
<br />
==Production process==<br />
<br />
=== HFI processing ===<br />
<br />
The input TOIs are in ADUs representing the voltage signal at the end of the electronics (see [[HFI_detection_chain | Detection chain]] for details). The processing applied to remove instrumental effects and to calibrate them is described in detail in the [[TOI processing | HFI TOI processing]] section, and we give a very brief summary here for convenience. That pipeline performs the following operations:<br />
<br />
; ADC correction: corrects for the uneven size of the ADC bins.<br />
; demodulation: this is performed around a variable level which is determined from the valid input data (using a validity flag from a previous version of the processing), and the data are converted to engineering units (V) using known conversion coefficients.<br />
; despiking: the redundancy within a ring is used to determine where glitches occur. Once identified, the glitches are fitted with templates. A glitch flag is produced that identifies the strongest part of the glitches, and a timeline of glitch tails. produced from the template fits, is subtracted from the demodulated timeline. Finally, the flagged regions are replaced with data from an average over the pointing period in order to not leave holes in the timeline that would perturb the usage of Fourier methods on them<br />
; dark template removal: the two dark bolometers are demodulated and despiked as above; the resulting timelines are then smoothed and used as an indicator of the overall temperature variations of the bolometer plate. Where the variations are consistent with each other, they are combined and removed from the bolometer signal timelines using appropriate coupling coefficients. The few percent of the data where they are not consistent are flagged as not valid on the timelines.<br />
; conversion to absorbed power: the timeline is converted to watts of absorbed power using the bolometer function. This includes a non-linearity correction; removal of the 4K cooler lines: the electromagnetic interference of the 4K cooler with the bolometer readout wires induces some sharp lines in the signal power spectra at frequencies of the 4K cooler's fundamental and its multiples, folded by the signal modulations. Fourier coefficients of the relevant lines are determined on a per-ring basis, and then removed from the data. The quality of the removal depends on the bolometer.<br />
; deconvolution by the time transfer function: this is done to correct for the non-instantaneous time response of the bolometers. The function itself is modeled using up to 8 time constants which are adjusted primarily on the planet data and also from comparisons of the northward and southward scans of the Galactic Plane. It is then removed using Fourier techniques, which has the side-effect of increasing the noise at high frequencies.<br />
; jump correction: removes some (relatively rare: 0.3 jumps per bolometer per pointing period, on average) jumps in the signal baseline. The jumps are detected characterized on smoothed TOIs, and corrected by adding a constant to part of the signal timeline. The origin of the jumps is not known.<br />
<br />
The results of this processing are a timeline of signal (in absorbed watts) and a ''valid data'' flag timeline for each of the 50 valid bolometers processed; these timelines contain the full sky signal, i.e., including the solar and orbital dipoles, the Zodiacal light, and contributions from the Far-Side lobes. The dipoles are necessary for the flux calibration and are removed at the mapmaking stage. The remaining two bolometers (143-8 and 545-3) show semi-random jumps in the signal level, typically jumping over 2-5 different ''pseudo-baseline'' levels, a behavior known as ''Random Telegraphic Signal'', so that these are commonly called the RTS bolometers. Finally, ring-level statistics of different types (mean, median, rms, kurtosis, etc.) are determined on a per-ring basis for all timelines, and a selection based on these statistics is used to discard anomalous rings, which are recorded in a ring-level flag for each valid bolometer timeline (see the [[TOI_processing#Discarded_rings | Discarded rings]] section). <br />
<br />
Throughout this processing, some planet signals are masked in the timeline in order to avoid ringing effects in the processing. It concerns only Jupiter at 100 and 143 GHZ, Jupiter and Saturn at 217 GHz, Jupiter, Saturn and Mars at 353, 545 and 857 GHz (thanks to various improvements in the despiking and in the transfer functions it is no longer necessary to flag these three planets in the CMB channels as was done in Release 1). Since they move on the sky, the portion of the sky masked during one survey is observed during one, and no hole is left in the final map. In parallel, the planet data are processed in a similar way, but with different parameters for the despiking step. These results are processed separately to determine the beam shapes and the focal plane geometry.<br />
<br />
Finally, the TOIs are calibrated in astrophysical units using the results of the calibration the calibration pipeline, described in the [[Map-making | Map-making and calibration]] section, which are stored in the IMO, and the best estimate of the zero-point offsets (a constant level for each bolometer) and of the far-side lobes (another signal TOI determined from the input TOI) are removed. These TOIs are ready for projection onto a map. That projection normally requires that the low-frequency noise be removed via a "destriping" (sometimes called "baseline removal") step. The HFI-DPC does its destriping at ring-level; should the user want to use the DPC's offsets, they are provided separately ([##REF##]).<br />
<br />
The pointing (see common [[Detector_pointing | Detector Pointing]] section for details) is determined from the AHF produced by MOC, which gives the direction and orientation of the LOS of a fiducial position in the focal plane at frequencies of 8Hz during stable pointing and 4 Hz during maneuvers. This is interpolated to the times of data observations, corrected for the wobble and other time-dependent offsets determined from the observed positions of a large number of sources around the sky, and finally converted to the LOS of each detector using the quaternions in the IMO.<br />
<br />
==File Names==<br />
<br />
The file names are of the form:<br />
<br />
''{H,L}FI_TOI_{fff}-{SCI,PTG}_R2.nn_ODxxxx.fits''<br />
<br />
where<br />
* ''fff'' denotes the frequency<br />
* SCI or PTG denote signal or pointing TOIs<br />
* R2.nn is the version, and<br />
* ODxxxx indicates the OD.<br />
<br />
==FITS file structure==<br />
<br />
All FITS files begin with a primary extension containing no data and a minimal header, which is followed by one or more ''BINTABLE'' extensions with data, and with a description of the data in the header keywords. In what follows will usually ignore the primary extension, and count only the extensions containing data.<br />
<br />
===TOI files===<br />
<br />
The signal FITS files contain N ,'BINTABLE', data extensions, where N is stye number of detectors in that frequency channel. The first extension contains the OBT and its flags, called the ''global'' flags since they apply to all detectors, which is followed by one extension for each detector, containing the signal and its ''local'' flag. The flags columns are written as ''byte'' in which each of the 8 bits (max) encodes one flag timeline. The meaning of each bit is given in the header comments and is subject to change. For the global flag they include:<br />
* Unstable pointing: 1= pointing is not stable (e.g., during repointing maneuvers)<br />
* Dark correlation: 1 = darks are uncorrelated and data are flagged<br />
* First/Second half ring: which samples are in which half (only covers the stable pointing part of the ring)<br />
* HCM: in HCM mode (unstable pointing)<br />
* and more<br />
<br />
And for the local flag they include<br />
<br />
* Total Flag: a combination of the various flags that is the one finally used in the map-making<br />
* Data not valid: glitched samples<br />
* Despike Common: (for PSBs only) glitch on current or other of PSB pair<br />
* StrongSignal: on Galactic Plane<br />
* Strong Source: on point source<br />
* other<br />
<br />
<br />
{| border="1" cellpadding="3" cellspacing="0" align="center" style="text-align:left" width=800px<br />
|+ '''TOI file data structure'''<br />
<br />
|- bgcolor="ffdead" <br />
!colspan="4" | 1. EXTNAME = 'OBT' : Data columns<br />
|- bgcolor="ffdead" <br />
! Column Name || Data Type || Units || Description<br />
|-<br />
|OBT || Int*8 || 2<sup>-16</sup> sec || On-board time<br />
|-<br />
|FLAG || Byte || none || the various bit-level flags<br />
<br />
|- bgcolor="ffdead" <br />
! Keyword || Data Type || Value || Description<br />
|-<br />
|OD || Int || || OD covered (as in filename)<br />
|-<br />
|BEGIDX || Int || || first sample index of given OD<br />
|-<br />
|ENDIDX || Int || || last sample index of given OD<br />
|-<br />
|BEGRING || Int || || first ring in given OD<br />
|-<br />
|ENDRING || Int || || last ring in given OD<br />
|-<br />
|TIMEZERO || String || 1958-01-01z00:00 || Origin of OBT<br />
<br />
|- bgcolor="ffdead" <br />
!colspan="4" | n. EXTNAME = ''DETNAME'' : Data columns<br />
<br />
|- bgcolor="ffdead" <br />
! Column Name || Data Type || Units || Description<br />
|-<br />
|SIGNAL || Real*4 || K<sub>cmb</sub> or MJy/sr || Value of signal<br />
|-<br />
|FLAG || Byte || none || the various bit-level flags<br />
|- bgcolor="ffdead" <br />
<br />
! Keyword || Data Type || Value || Description<br />
|-<br />
|UNIT || String || || Units of signal<br />
|-<br />
|DESTRIPE || 1/0 || || whether timeline is destriped<br />
|-<br />
|OD || Int || || OD covered (as in filename)<br />
|-<br />
|BEGIDX || Int || || first sample index of given OD<br />
|-<br />
|ENDIDX || Int || || last sample index of given OD<br />
|-<br />
|BEGRING || Int || || first ring in given OD<br />
|-<br />
|ENDRING || Int || || last ring in given OD<br />
|}<br />
<br />
<br />
The pointing files have a similar structure, except that ''DETNAME'' extensions contain 3 columns of Real*8 with the phi, theta, and psi Galactic spherical coordinates of each sample in radians. There is no local flag for the coordinates. <br />
<br />
<br />
===ROI files===<br />
<br />
The files provided by HFI are<br />
* ''HFI_ROI_GlobalParams_RelNum_full.fits''<br />
* ''HFI_ROI_DestripeOffsets_RelNum_full.fits''<br />
which are described below.<br />
<br />
;Global parameters: this ROI file contains a single ''BINTABLE'' extension with 3 columns that give, for each ring, the index number of the first sample, the ESA pointing ID of the ring, and the ring start time. All three are encoded as Int*8, and the time is given in units of 2<sup>-16</sup> sec from ''TIMEZERO'', 1958-01-01z00:00, as in the TOI files. Rings are numbered from 240 to 27005, as given in the ''BEGRING'' and ''ENDRING'' keywords<br />
<br />
; Destriping offsets: this ROI file contains a single ''BINTABLE'' extension with 3 columns that give, for each ring, the offsets to subtract from each bolometer signal timeline or vector. There are three offsets available for each bolometer: full-ring, first half-ring, and second half-ring, for building the full-ring or the two half-ring maps, respectively. The offsets must be subtracted from Index(N) to Index(N+1)-1 of the corresponding signal timeline, where N is stye ring number, and the indices are given in the Global parameters file. The offsets are given in the same units as the signal vectors, that is K<sub>cmv</sub> for the 100-353 GHz channels, and MJy/sr for the 545 and 857 GHz channels. All values are encoded as Real*4. The rings for which the offsets are set to 0.000 are [[TOI_processing#Discarded_rings | ''badrings'' ]] which are rejected in the mapmaking process.<br />
<br />
<span style="color:red"> TBW: Note on half-ring offsets, which differ by 1-2% from full ring offsets</span> <br />
<br />
[[Category:Mission products|001]]</div>
Fdesert
https://wiki.cosmos.esa.int/planckpla2015/index.php?title=Timelines&diff=9600
Timelines
2014-09-29T13:36:08Z
<p>Fdesert: /* General description */</p>
<hr />
<div>==General description==<br />
<br />
The timelines, or TOIs for ''Time-Ordered Information'', are vectors of signal or of pointing or of some other quantity giving the value of that quantity as a function of time during the mission. The TOIs described here are sampled regularly at the (instrument dependent) detector sampling frequency and span the full science mission. They thus consists of a large amount of data. This section is concerned with the signal TOIs and their associated pointing TOIs. For each bolometer the DPCs provide a single timeline of calibrated signal and 3 timelines of coordinates, corresponding to the two angular coordinates and one position angle. <br />
<br />
Obviously these vectors are very long (~1.38E10 samples for HFI, nnnn for LFI) and thus need to be split into multiple files for export. Here the data are split by Operational Day (OD) as follows:<br />
* HFI: 12 files per OD, one for pointing and one for signal for each channel, covering ODs 91 -- 974, all sampled at F<sub>samp</sub> = 180.3737 Hz. <br />
* LFI: 6 files per OD, 3 each for signal and pointing, since the sampling frequency is channel dependent, and covering ODs 91--<span style="color:red">NNN</span>.<br />
<br />
The signal timelines are encoded as Real*4, but the pointing vectors had to be encoded as Real*8 to maintain the required accuracy. The result is that the total volume of the full dataset is ~30 TB. All files also contain the OBT.<br />
<br />
The signal timelines have been cleaned of all known instrumental systematic effects, they have been calibrated in flux, and the solar and earth motion dipole signals have been removed. But they have not been cleaned of the low-frequency, ''1/f'' noise that needs to be removed via a destriping tool. The methods consists of removing offsets or ''baselines'' determined by minimizing the differences in the signals at points on the sky where they intersect. For HFI, an offset per ring is determined; <span style="color:red"> For LFI ..... </span>. These offsets are determined using the full mission and all the valid detectors per channel, and they are then used for all the maps produced, i.e., for those maps using any fraction of the mission (year, survey) or any subset of the detectors (single detector, detector set), and it was shown that using offsets determined from a limited part of the mission and/or a subset of the detectors yields maps that are less consistent than otherwise. The offsets are delivered separately, as described below.<br />
<br />
<span style="color:red"> Maybe insert a figure for example? </span><br />
<br />
Furthermore, for HFI, there are three sets of offsets produced: the primary set using the full rings, and the secondary ones using the first and second half rings, only. The difference between the primary and the secondary sets are fairly minor, but they are necessary to rebuild the maps as they were built by the HFI-DPC. Overall the offsets are useful for users wishing to build a map of a small area of the sky. <br />
<br />
<span style="color:red"> TBW: baselines for LFI</span> <br />
<br />
The HFI delivers its offsets in a ROI, or "Ring-Ordered Information" file. That file contains a table of N<sub>rings</sub> = 26766 rows by N<sub>bolometers</sub> = 50 columns in which each cell contains a 3-elements vector containing the full-ring, first half-ring, and second half-ring offsets. The indices of the signal TOI delimiting the rings are given in a second ROI file with the same N<sub>rings</sub> rows containing global parameters.<br />
<br />
===Indexing===<br />
<br />
Each FITS file contains keywords giving the first and last index of the section of the full TOI contained in the file. These indices are those of the internal DPC representation; they begin at 0 corresponding to the switch-on of the instrument, and run to some very large number (~25 billion for HFI) representing the full mission. Only the ''science'' part of the mission is exported, which is about 60% of the total for HFI. Are left out the early mission phase (cool-down and validation phases) and the warm phase of HFI, during which LFI continued to collect data. The HFI keywords also indicated which rings are included in each OD, but be aware that normally an OD does not begin at the same time (with the same index) as a ring. <br />
The HFI offsets are determined on a per-ring basis, and must be applied ring-by-ring, and for this purpose the begin and end ring indices are given in <br />
<br />
<br />
<span style="color:red"> TBW: HFI: explain TOI indices and (BEG/ENDINDEX kwds) and same for ROI </span><br />
<br />
==Production process==<br />
<br />
=== HFI processing ===<br />
<br />
The input TOIs are in ADUs representing the voltage signal at the end of the electronics (see [[HFI_detection_chain | Detection chain]] for details). The processing applied to remove instrumental effects and to calibrate them is described in detail in the [[TOI processing | HFI TOI processing]] section, and we give a very brief summary here for convenience. That pipeline performs the following operations:<br />
<br />
; ADC correction: corrects for the uneven size of the ADC bins.<br />
; demodulation: this is performed around a variable level which is determined from the valid input data (using a validity flag from a previous version of the processing), and the data are converted to engineering units (V) using known conversion coefficients.<br />
; despiking: the redundancy within a ring is used to determine where glitches occur. Once identified, the glitches are fitted with templates. A glitch flag is produced that identifies the strongest part of the glitches, and a timeline of glitch tails. produced from the template fits, is subtracted from the demodulated timeline. Finally, the flagged regions are replaced with data from an average over the pointing period in order to not leave holes in the timeline that would perturb the usage of Fourier methods on them<br />
; dark template removal: the two dark bolometers are demodulated and despiked as above; the resulting timelines are then smoothed and used as an indicator of the overall temperature variations of the bolometer plate. Where the variations are consistent with each other, they are combined and removed from the bolometer signal timelines using appropriate coupling coefficients. The few percent of the data where they are not consistent are flagged as not valid on the timelines.<br />
; conversion to absorbed power: the timeline is converted to watts of absorbed power using the bolometer function. This includes a non-linearity correction; removal of the 4K cooler lines: the electromagnetic interference of the 4K cooler with the bolometer readout wires induces some sharp lines in the signal power spectra at frequencies of the 4K cooler's fundamental and its multiples, folded by the signal modulations. Fourier coefficients of the relevant lines are determined on a per-ring basis, and then removed from the data. The quality of the removal depends on the bolometer.<br />
; deconvolution by the time transfer function: this is done to correct for the non-instantaneous time response of the bolometers. The function itself is modeled using up to 8 parameters which are adjusted primarily on the planet data and also from comparisons of the northward and southward scans of the Galactic Plane. It is then removed using Fourier techniques, which has the side-effect of increasing the noise at high frequencies.<br />
; jump correction: removes some (relatively rare: 0.3 jumps per bolometer per pointing period, on average) jumps in the signal baseline. The jumps are detected characterized on smoothed TOIs, and corrected by adding a constant to part of the signal timeline. The origin of the jumps is not known.<br />
<br />
The results of this processing are a timeline of signal (in absorbed watts) and a ''valid data'' flag timeline for each of the 50 valid bolometers processed; these timelines contain the full sky signal, i.e., including the solar and orbital dipoles, the Zodiacal light, and contributions from the Far-Side lobes. The dipoles are necessary for the flux calibration and are removed at the mapmaking stage. The remaining two bolometers (143-8 and 545-3) show semi-random jumps in the signal level, typically jumping over 2-5 different ''pseudo-baseline'' levels, a behavior known as ''Random Telegraphic Signal'', so that these are commonly called the RTS bolometers. Finally, ring-level statistics of different types (mean, median, rms, kurtosis, etc.) are determined on a per-ring basis for all timelines, and a selection based on these statistics is used to discard anomalous rings, which are recorded in a ring-level flag for each valid bolometer timeline (see the [[TOI_processing#Discarded_rings | Discarded rings]] section). <br />
<br />
Throughout this processing, some planet signals are masked in the timeline in order to avoid ringing effects in the processing. It concerns only Jupiter at 100 and 143 GHZ, Jupietr and Saturn at 217 GHz, Jupietr, Saturn and Mars at 353, 545 and 857 GHz (thanks to various improvements in the despiking and in the transfer functions it is no longer necessary to flag these three planets in the CMB channels as was done in Release 1). Since they move on the sky, the portion of the sky masked during one survey is observed during one, and no hole is left in the final map. In parallel, the planet data are processed in a similar way, but with different parameters for the despiking step. These results are processed separately to determine the beam shapes and the focal plane geometry.<br />
<br />
Finally, the TOIs are calibrated in astrophysical units using the results of the calibration the calibration pipeline, described in the [[Map-making | Map-making and calibration]] section, which are stored in the IMO, and the best estimate of the zero-point offsets (a constant level for each bolometer) and of the far-side lobes (another signal TOI determined from the input TOI) are removed. These TOIs are ready for projection onto a map. That projection normally requires that the low-frequency noise be removed via a "destriping" (sometimes called "baseline removal") step. The HFI-DPC does its destriping at ring-level; should the user want to use the DPC's offsets, they are provided separately ([##REF##]).<br />
<br />
The pointing (see common [[Detector_pointing | Detector Pointing]] section for details) is determined from the AHF produced by MOC, which gives the direction and orientation of the LOS of a fiducial position in the focal plane at frequencies of 8Hz during stable pointing and 4 Hz during maneuvers. This is interpolated to the times of data observations, corrected for the wobble and other time-dependent offsets determined from the observed positions of a large number of sources around the sky, and finally converted to the LOS of each detector using the quaternions in the IMO. <br />
<br />
<br />
==File Names==<br />
<br />
The file names are of the form:<br />
<br />
''{H,L}FI_TOI_{fff}-{SCI,PTG}_R2.nn_ODxxxx.fits''<br />
<br />
where<br />
* ''fff'' denotes the frequency<br />
* SCI or PTG denote signal or pointing TOIs<br />
* R2.nn is the version, and<br />
* ODxxxx indicates the OD.<br />
<br />
==FITS file structure==<br />
<br />
All FITS files begin with a primary extension containing no data and a minimal header, which is followed by one or more ''BINTABLE'' extensions with data, and with a description of the data in the header keywords. In what follows will usually ignore the primary extension, and count only the extensions containing data.<br />
<br />
===TOI files===<br />
<br />
The signal FITS files contain N ,'BINTABLE', data extensions, where N is stye number of detectors in that frequency channel. The first extension contains the OBT and its flags, called the ''global'' flags since they apply to all detectors, which is followed by one extension for each detector, containing the signal and its ''local'' flag. The flags columns are written as ''byte'' in which each of the 8 bits (max) encodes one flag timeline. The meaning of each bit is given in the header comments and is subject to change. For the global flag they include:<br />
* Unstable pointing: 1= pointing is not stable (e.g., during repointing maneuvers)<br />
* Dark correlation: 1 = darks are uncorrelated and data are flagged<br />
* First/Second half ring: which samples are in which half (only covers the stable pointing part of the ring)<br />
* HCM: in HCM mode (unstable pointing)<br />
* and more<br />
<br />
And for the local flag they include<br />
<br />
* Total Flag: a combination of the various flags that is the one finally used in the map-making<br />
* Data not valid: glitched samples<br />
* Despike Common: (for PSBs only) glitch on current or other of PSB pair<br />
* StrongSignal: on Galactic Plane<br />
* Strong Source: on point source<br />
* other<br />
<br />
<br />
{| border="1" cellpadding="3" cellspacing="0" align="center" style="text-align:left" width=800px<br />
|+ '''TOI file data structure'''<br />
<br />
|- bgcolor="ffdead" <br />
!colspan="4" | 1. EXTNAME = 'OBT' : Data columns<br />
|- bgcolor="ffdead" <br />
! Column Name || Data Type || Units || Description<br />
|-<br />
|OBT || Int*8 || 2<sup>-16</sup> sec || On-board time<br />
|-<br />
|FLAG || Byte || none || the various bit-level flags<br />
<br />
|- bgcolor="ffdead" <br />
! Keyword || Data Type || Value || Description<br />
|-<br />
|OD || Int || || OD covered (as in filename)<br />
|-<br />
|BEGIDX || Int || || first sample index of given OD<br />
|-<br />
|ENDIDX || Int || || last sample index of given OD<br />
|-<br />
|BEGRING || Int || || first ring in given OD<br />
|-<br />
|ENDRING || Int || || last ring in given OD<br />
|-<br />
|TIMEZERO || String || 1958-01-01z00:00 || Origin of OBT<br />
<br />
|- bgcolor="ffdead" <br />
!colspan="4" | n. EXTNAME = ''DETNAME'' : Data columns<br />
<br />
|- bgcolor="ffdead" <br />
! Column Name || Data Type || Units || Description<br />
|-<br />
|SIGNAL || Real*4 || K<sub>cmb</sub> or MJy/sr || Value of signal<br />
|-<br />
|FLAG || Byte || none || the various bit-level flags<br />
|- bgcolor="ffdead" <br />
<br />
! Keyword || Data Type || Value || Description<br />
|-<br />
|UNIT || String || || Units of signal<br />
|-<br />
|DESTRIPE || 1/0 || || whether timeline is destriped<br />
|-<br />
|OD || Int || || OD covered (as in filename)<br />
|-<br />
|BEGIDX || Int || || first sample index of given OD<br />
|-<br />
|ENDIDX || Int || || last sample index of given OD<br />
|-<br />
|BEGRING || Int || || first ring in given OD<br />
|-<br />
|ENDRING || Int || || last ring in given OD<br />
|}<br />
<br />
<br />
The pointing files have a similar structure, except that ''DETNAME'' extensions contain 3 columns of Real*8 with the phi, theta, and psi Galactic spherical coordinates of each sample in radians. There is no local flag for the coordinates. <br />
<br />
<br />
===ROI files===<br />
<br />
The files provided by HFI are<br />
* ''HFI_ROI_GlobalParams_RelNum_full.fits''<br />
* ''HFI_ROI_DestripeOffsets_RelNum_full.fits''<br />
which are described below.<br />
<br />
;Global parameters: this ROI file contains a single ''BINTABLE'' extension with 3 columns that give, for each ring, the index number of the first sample, the ESA pointing ID of the ring, and the ring start time. All three are encoded as Int*8, and the time is given in units of 2<sup>-16</sup> sec from ''TIMEZERO'', 1958-01-01z00:00, as in the TOI files. Rings are numbered from 240 to 27005, as given in the ''BEGRING'' and ''ENDRING'' keywords<br />
<br />
; Destriping offsets: this ROI file contains a single ''BINTABLE'' extension with 3 columns that give, for each ring, the offsets to subtract from each bolometer signal timeline or vector. There are three offsets available for each bolometer: full-ring, first half-ring, and second half-ring, for building the full-ring or the two half-ring maps, respectively. The offsets must be subtracted from Index(N) to Index(N+1)-1 of the corresponding signal timeline, where N is stye ring number, and the indices are given in the Global parameters file. The offsets are given in the same units as the signal vectors, that is K<sub>cmv</sub> for the 100-353 GHz channels, and MJy/sr for the 545 and 857 GHz channels. All values are encoded as Real*4. The rings for which the offsets are set to 0.000 are [[TOI_processing#Discarded_rings | ''badrings'' ]] which are rejected in the mapmaking process.<br />
<br />
<span style="color:red"> TBW: Note on half-ring offsets, which differ by 1-2% from full ring offsets</span> <br />
<br />
[[Category:Mission products|001]]</div>
Fdesert
https://wiki.cosmos.esa.int/planckpla2015/index.php?title=TOI_processing&diff=5907
TOI processing
2013-03-13T16:48:08Z
<p>Fdesert: </p>
<hr />
<div>This Section is kept short as this Planck release does not contain TOIs. The main information in the HFI data processing paper <cite>#planck2013-p03</cite> is not duplicated here.<br />
<br />
==Overview==<br />
<br />
We describe here how the TOIs are processed in order to be used for map production. We do not repeat the general features of the pipeline which are given in the HFI Data Processing article <cite>#planck2013-p03</cite>. Here we give complementary explanations on some details. The TOI of each bolometer is processed independently of the other bolometers, so as to keep the noise properties as uncorrelated as possible. The processing involves modifying the TOI itself for what concerns the conversion to absorbed power and the correction of glitch tails. It also adds a flag TOI that masks the TOI samples that are not to be projected on maps for various reasons.<br />
<br />
<br />
=== Acronyms and definitions ===<br />
<br />
* 4K lines: EMI/EMC influence of the 4K cooler mechanical motion on the bolometer readout electronics.<br />
* ADC : analog to digital converter<br />
* IMO : instrument model<br />
* Jump : sudden change of the baseline level inside a ring<br />
* LFER : low frequency excess response<br />
* PBR : phase binned ring<br />
* RIMO : reduced IMO<br />
* Ring : pointing period<br />
* ROI : ring ordered information<br />
* RTS : random telegraphic signal<br />
* SSO : solar system object<br />
* TOI : time ordered information<br />
<br />
== Input TOI ==<br />
<br />
The input TOI consists in the AC modulated voltage output of the readout of each bolometer. The input has previously been decompressed, and converted from internal digital units to voltage via a constant factor. The TOI has a regular sampling at the acquisition frequency of facq=180.373700+-0.000050 Hz. There are almost no missing data in the TOIs, except for few hundred samples of 545 and 857GHz TOIs which are lost in the on-board compression due to saturation on the Galactic Center crossings. <br />
<br />
== General Pipeline Structure ==<br />
<br />
The figure on the right shows how the initial TOI is transformed and how flags are produced:<br />
<br />
[[File:HFI_4_4_2_ToiProc_ExplS.png|thumb|600px|A schematic of the TOI processing pipeline]]<br />
<br />
<br />
== Output TOIs and products ==<br />
<br />
A TOI of clean calibrated samples (ccTOI) and a combined flag TOI (fTOI) are the outputs of the processing. The ccTOI is calibrated so as to represent the instantaneous power absorbed by the detector up to a constant (which will be determined by the map-making destriper). It is worth mentioning how the ccTOI is changed with respect to the input TOI, beyond the harmless constant conversion factor from voltage to absorbed power. The demodulation stage allows to get the demodulated bolometer voltage. The non-linearity correction is a second-order polynomial correction based on the physical but static bolometer model. In order to avoid too much masking after glitches, a glitch tail is subtracted after an occurrence of a glitch in the TOI. The 4K cooler lines noise is substituted at a series of 9 single temporal frequencies. Finally, the temporal response of the bolometer is deconvolved. This affects mostly the high-temporal frequency part of the TOI, although a small but significant low frequency (the long time response) tail is corrected too. Although flagged samples are not projected, their value influences the valid samples somehow. Hence interpolation procedures introduce some indirect modifications of the TOI.<br />
The flag TOI is a combination a dozen flags with an OR logic. Only unflagged data are projected. The exhaustive list of flags is given here: CompressionError, NoData, SSO, UnstablePointing, Glitch, BoloPlateFluctuation, RTS, Jump, PSBab. A complete qualification of the data is obtained at the ring level. If the TOI shows an anomalous behaviour on a given ring, this ring is discarded from projection. A special production of TOIs is also made as an input to the beam analysis with Mars, Jupiter and Saturn.<br />
<br />
<br />
== Examples of clean TOIs ==<br />
<gallery widths=300px perrow=3><br />
File:10_143_5_PBR_10800RING_LFER4_JC.jpg|Phase-binned signal (PBR) for 6 consecutive rings from 4000 to 4005<br />
File:10_143_5_rcircles_LFER4_JC.jpg| Valid signal (up) and noise (bottom) for 2 consecutive circles of the ring 4000<br />
File:10_143_5_fft_LFER4_JC.jpg| Fourier power spectral density of the first 30 minutes of the ring 4005 with signal before (pink) and after (black) deconvolution (top) and noise (bottom)<br />
File:14_545_1_PBR_10800RING_LFER4_JC.jpg|Phase-binned signal (PBR) for 6 consecutive rings from 4000 to 4005<br />
File:14_545_1_rcircles_LFER4_JC.jpg| Valid signal (up) and noise (bottom) for 2 consecutive circles of the ring 4000<br />
File:14_545_1_fft_LFER4_JC.jpg| Fourier power spectral density of the first 30 minutes of the ring 4005 with signal before (pink) and after (black) deconvolution (top) and noise (bottom)<br />
</gallery><br />
<br />
[[Media:check_ring.pdf|Samples of PBR, TOIs, and PSDs of all detectors are shown in this file]]<br />
<br />
== Trends in the output processing variables ==<br />
<br />
Here we intend to show the trend of the systematic effects that are dealt with in the TOI processing. The full impact of each of them is analyzed in [[HFI-Validation]].<br />
<br />
=== ADC baseline ===<br />
The following figure shows the ADC baseline which is used prior to demodulation (a constant offset is removed for clarity). This baseline is obtained by smoothing on an hour block average the undemodulated TOI on unflagged samples.<br />
<br />
[[File:baseline.jpg|400px|ADC baseline for all bolometers]]<br />
<br />
=== Glitch statistics ===<br />
The glitch rate per channel is shown in this figure. For details, see copap. <br />
<br />
[[File:figIntermPaperGR.jpg|400px|Glitch rate evolution]]<br />
<br />
The percentage of flagged data (mostly due to Cosmic Rays) at the ring level is shown in these examples. No smoothing was applied. Only valid rings are shown.<br />
<gallery widths=450px perrow=2 caption="Percentage of flagged data"><br />
File:group10_143_5.jpg | 143 GHz bolometers<br />
File:group14_545_1.jpg | 545 GHz bolometers<br />
</gallery><br />
[[Media:PercentUnvalid2.pdf|The complete set of plots is here]]<br />
<br />
=== Thermal template for decorrelation ===<br />
[[File:T90.jpg|400px|Thermal template used for the decorrelation from bolometer plate temperature fluctuations]]<br />
<br />
A simple linear decorrelation is performed using the 2 dark bolometers as a proxy of the bolometer plate temperature. Coupling coefficients were measured during the CPV phase.<br />
<br />
=== 4K cooler lines variability ===<br />
The amplitude of the nine 4K cooler lines in aW at 10, 20, 30, 40, 50, 60, 70, 80 and 17 Hz is shown for 2 bolometers in the following figures. The trend is smoothed over 31 ring values after having discarded measurements done at a ring which is discarded for all bolometers.<br />
<gallery widths=450px perrow=2 caption="Amplitude of the nine 4K cooler lines"><br />
File:10_143_5_4Klines.jpg | 143_5 bolometer<br />
File:14_545_1_4Klines.jpg | 545_1 bolometer<br />
</gallery><br />
[[Media:Lines4K.pdf|The 4K cooler line coefficients of all bolometers are shown in this file]]<br />
<br />
The 4K cooler lines project onto the maps only for a limited fraction of rings, the so-called resonant rings. This is graphically shown in the following figure. For each ring (stable pointing period), the spin rate is very stable at about 1 rpm. From one ring to another, the spin frequency (shown as diamonds) changes around that value. The sky signal is imprinted at the corresponding spin frequency and its 5400 (60x90) harmonics. If one of the nine 4K cooler lines happens to coincide with one of the spin frequency harmonics (a resonant ring), it will project a sine-wave systematic on the maps. The horizontal coloured bars show the zone of influence of a particular 4K line (labeled on the left side of the plot), when folded around 16.666 mHz. When the spin frequency hits one of these zones, we have a resonant ring. The 4K line coefficient is interpolated for this ring and an estimate of the systematic effect is subtracted from the TOI. Resonant rings are different for different 4K lines. Note the two-level oscillation pattern of the spin frequency is due to the satellite attitude control system.<br />
<br />
[[File:spinfreq1.jpg|600px|Spin frequency and 4K line zones of influence]]<br />
<br />
=== jump correction ===<br />
A piecewise constant value is removed to the TOI if a jump is detected. See a jump example in this figure:<br />
<br />
[[File:jump_exe.png|200px|An example of a jump seen on the rmsigTOI]]<br />
<br />
The number of jumps per day (all bolometers included) is shown in this figure:<br />
<br />
[[File:jumps_per_day.png|200px|Evolution of jump number during the mission]]<br />
<br />
The jumps are uncorrelated from bolometer to bolometer. The total number of jumps detected in the nominal and full mission is shown here:<br />
<br />
[[File:jumps_per_bolometer.png|200px|Number of jumps per bolometer]]<br />
<br />
<br />
=== Trends in noise and signal ===<br />
<gallery widths=450px perrow=2 caption="Signal (top) and noise (bottom) smoothed at 1 minute. All values falling in a discarded ring are not plotted."><br />
File:10_143_5_smooth_Watt.jpg | 143_5 bolometer<br />
File:14_545_1_smooth_Watt.jpg | 545_1 bolometer<br />
</gallery><br />
<br />
[[Media:check_smooth.pdf|The smooth TOIs of all detectors are shown in this file]]<br />
<br />
=== Noise stationarity ===<br />
This is not the final version but gives a good idea of power spectra at the detector level of rmsigTOIs.<br />
[[Media:v53_meanSpectra_bySurvey.pdf|All PSDs can be seen in this file]]<br />
<br />
The standard deviation per ring corrected by ring duration bias is given here, one per bolometer using only valid rings. No smoothing is applied (except a 31-point smoothing for the 545 and 857 GHz channels) but values for rings discarded for all bolometers (see below) are not used. The standard deviation is computed on samples valid for map-making which are also not affected by the Galaxy or the point-sources using the usual flags. Two examples are given here.<br />
<gallery widths=450px perrow=2 caption="Standard deviation per ring"><br />
File:stddev_group10_143_5.jpg | 143_5 bolometer<br />
File:stddev_group_14_545_1.jpg | 545_1 bolometer<br />
</gallery><br />
<br />
<br />
The full series of plots is here:<br />
[[Media:StationarityInROIoutput2.pdf|Standard deviation of rmsig TOIs at the ring level]]<br />
<br />
Note the presence for 3 bolometers of a two-level noise system. No correction can be done for that effect. See one example here:<br />
<br />
[[File:23_353_TwoLevel.jpg|300px|An example of two-level noise system is seen in bolometer 23_353_3a]]<br />
<br />
<br />
An example of the higher order statistics which are used to unveil rings affected by RTS problems.<br />
<br />
[[File:HFI_4_4_2_RTSexample13.jpg|400px|Example of RTS detection]]<br />
<br />
=== Discarded rings ===<br />
<br />
Some rings are discarded (flagged) from further use (beam making, map making) by using ring statistics (see above and <cite>#planck2013-p03</cite>). For each statistic, we compare each ring value to the ring values averaged (RVA) over a large selection of rings (between 3000 and 21700). We also define the modified standard deviation (MSD) of a ring quantity as the standard deviation of that quantity over the rings that deviate by less than five nominal standard deviations. This truncation is necessary to be robust against extreme deviant rings.<br />
<br />
More specifically, a ring is discarded if it matches one of the following criteria:<br />
<br />
* the | mean-median | deviates from the RVA by more than fifteen times the MSD.<br />
* the standard deviation deviates from the RVA by more than -5 times the MSD (all cases corresponding to almost empty rings) or +15 times the MSD.<br />
* the Kolmogorov-Smirnov test deviates from the RVA by more than 15 times the MSD.<br />
* the ring duration is more than 90 min.<br />
* the ring is contaminated by RTS with an amplitude of more than one standard deviation of the noise. It concerns a few hundreds of rings for 3 bolometers (44_217_6b, 71_217_8a, 74_857_4). Notice that two bolometers are completely discarded for maps: 55_545_3 and 70_143_8, which present RTS at all time.<br />
<br />
For the three first criteria, a visual inspection of the rmsigTOI at each of the incriminated rings has shown that all these anomalous rings are due to either a drift, a small jump in the TOI trend or a sudden change of noise level, the origin of which is unknown at present.<br />
<br />
Once the list of discarded rings per bolometer is produced, a common list of discarded rings can be extracted for all bolometers (by using discarded rings for at least half the bolometers). Such rings correspond to identified phenomena, as can be seen on the following table.<br />
<br />
Furthermore, an isolated valid ring stuck between two common discarded rings becomes discarded as well.<br />
<br />
Table of '''common discarded rings''' of the nominal mission (rings 240-14723).<br />
<br />
{|table border=1 align=center cellpadding="3" cellspacing="0" style=text-align:center<br />
! '''Cause''' !! '''ring number'''<br />
|-<br />
!align="left"| manoeuvre<br />
|align="center"|304 1312 3590 3611 3642 3922 4949 6379 8456 11328<br />
|-<br />
! align="left"| Sorption Cooler Switchover <br />
|align="center"| 11149 11150 11151 11152 <br />
|-<br />
! align="left"| Rings too long<br />
| align="center"| 440 474 509 544 897 898 3589 13333 14627 14653 <br />
|-<br />
! align="left"| Star tracker switchover <br />
| align="center"| 14628 14654 <br />
|-<br />
! align="left"| Massive Glitch Event<br />
| align="center"| 7665 <br />
|-<br />
! align="left"| Solar Flare<br />
| align="center"| 11235 <br />
|}<br />
<br />
<br />
<br />
The following figure is a summary of the impact of the discarding process '''for each bolometer''' (the solid black line is the common discarded ring percentage). The outlier bolometers have some RTS problems as mentionned above.<br />
<br />
[[File:Fraction_discarded_rings_v53.png|300px|Fraction of discarded rings]]<br />
[[File:Fraction_discarded_time_v53.png|300px|Fraction of time due to discarded rings]]<br />
<br />
=== Effective integration time ===<br />
The following figure summarizes the effective integration time per bolometer. For that purpose the number of unflagged samples in non-discarded pointing periods have been used within the nominal mission. The average value is of about 335 days of effective integration time.<br />
<br />
[[File:HFInominal_IntegrationTime.jpg|600px|Effective integration time per bolometer]]<br />
<br />
== Flag description ==<br />
<br />
==== Input flags ====<br />
<br />
These flags are used as inputs to the TOI processing<br />
<br />
; The point-source flag (PSflag) <br />
: An earlier version of HFI point-source catalog is read back into a flag TOIs, at a given frequency. In practice, 5 sigma sources are masked within a radius of 1.3 FWHM (9, 7, 5, 5, 5, 5 arcmin at 100,143,217,353,545,857 GHz) TBC.<br />
<br />
; the galactic flag (Galflag<br />
: An earlier version of HFI maps is thresholded and apodized. The produced masks are read into flag TOIs. The retained threshold corresponds to a sky coverage of respectively 70, 70, 80, 90, 90, 90% at 100,143,217,353,545,857 GHz.<br />
<br />
; Solar System Object flag <br />
: For the TOI flag, Mars, Jupiter, Saturn are flagged up to a radius of NbBeam= 2,3,3,4,4,4 times the fiducial SSO_FWHM with SSO_FWHM= 9, 7, 5, 5, 5, 5 arcmin at 100,143,217,353,545,857 GHz.<br />
<br />
: As an input to planet mask for maps, Mars, Jupiter, Saturn are flagged with a radius computed as a coefficient depending on the planet (Factor_per_source) times NbBeam times SSO_FWHM, with Factor_per_source = 1.1, 2.25, 1.25 for Mars, Jupiter, Saturn respectively and NbBeam = 2.25, 4.25, 4.0, 5.0,.6.0, 8.0 at 100, 143, 217, 353, 545, 857 GHz. This flag is called SSOflag4map.<br />
<br />
: A small trailing tail is added to the mask to take into account the non-deconvolution of the planet signal which has been replaced by background values. The width of that tail is 10 % of the main flag diameter. The number of samples which are additionnaly flagged are the Factor_per_Source times AddSNafter with (trail) times (Factor_per_source)^3 samples where trail = 10, 30, 20, 20, 30, 40 at 100, 143, 217, 353, 545, 857 GHz.<br />
<br />
: Uranus and Neptune together with detected asteroids are masked by HFI. They are masked at the TOI level using an exclusion radius of 1.5 SSO_FWHM. At 857 GHz, 24 asteroids have been detected with HFI : 1Ceres, 2Pallas, 3Juno, 4Vesta, 7Iris, 8Flora, 9Metis, 10Hygiea, 11Parthenope, 12Victoria, 13Egeria, 14Irene, 15Eunomia, 16Psyche, 18Melpomene, 19Fortuna, 20Massalia, 29Amphitrite, 41Daphne, 45Eugenia, 52Europa, 88Thisbe, 704Interamnia, 324Bamberga.<br />
<br />
<gallery widths=200px perrow=4 caption="Local maps showing the SSO flag. The colors correspond to the surveys involved in the nominal mission (green = Survey I, yellow = Survey II, red = Survey III)."><br />
File:SSOflag_10_143_5J.png | 143 SWB row, around Jupiter 1st crossing<br />
File:SSOflag_10_143_5S.png | 143 SWB row, around Saturn<br />
File:SSOflag_10_143_5M.png | 143 SWB row, around Mars<br />
File:SSOflag_10_143_5a.png | 143 SWB row, random field in the ecliptic plane<br />
File:SSOflag_25_857_1J.png | 545/857 row, around Jupiter 1st crossing<br />
File:SSOflag_25_857_1S.png | 545/857 row, around Saturn<br />
File:SSOflag_25_857_1M.png | 545/857 row, around Mars<br />
File:SSOflag_25_857_1a.png | 545/857 row, random field in the ecliptic plane<br />
</gallery><br />
<br />
<br />
=== Output flags ===<br />
<br />
A FlagTOIproc is produced by the TOIprocessing. It marks measurements which are not reliable for any of the following reasons:<br />
* gap (no valid input data), enlarged by one sample on each side. It flags less than 0.00044% (resp. 0.00062%) of the nominal (resp. complete) mission. It is equivalent to less than 3 (resp. 8) minutes of data.<br />
* "glitch" on dark bolometers: as the thermal template used for decorrelation is computed from these bolometer data, chunks of one minute-length data are discarded for all bolometers if at least 50\% of the data for at least one dark bolometer are flagged during this time. It is efficient to flag the data around the maximum of thermal events.<br />
* "glitch" on individual bolometers : samples where the signal from a cosmic ray hit dominates the sky signal at more than 3 $\sigma$ are discarded.<br />
* jump as 100 samples are flagged around the computed position of the jump to take into account the error on this reconstructed position.<br />
<br />
<br />
So the flag produced for the map making, called Total_flag, is defined by:<br />
* Total_flag = UnstablePointing Flag OR FlagTOIproc OR SSOflag4map OR SSOflag seen<br />
where FlagTOIproc = gap OR flag thermal template OR glitch OR jump<br />
and for PSB bolometers, FlagTOIproc_AB = FlagTOIproc_A OR FlagTOIproc_B.<br />
Note that the Total_flag is then identical for the A and B bolometers of a PSB pair.<br />
<br />
At the destriping stage, a more restricted flag, called Total_flag_PS, is used. It is defined by<br />
Total_flag_PS = Total_flag OR PS_flag.<br />
<br />
<br />
== References ==<br />
<biblio force=false><br />
#[[References]] <br />
</biblio><br />
<br />
<br />
[[Category:Data processing|0042]]</div>
Fdesert
https://wiki.cosmos.esa.int/planckpla2015/index.php?title=TOI_processing&diff=4698
TOI processing
2013-02-25T08:53:16Z
<p>Fdesert: /* References */</p>
<hr />
<div>This Section is kept short as this Planck release does not contain TOIs. The main information in the HFI processing paper is not duplicated here.<br />
<br />
==Overview==<br />
<br />
We describe here the how the TOIs are processed in order to be used for map production. We do not repeat the general features of the pipeline which are given in the HFI Data Processing article <cite>#planck2013-p03</cite>. Here we give complementary explanations on some details. The TOI of each bolometer is processed independently of the other bolometers, so as to keep the noise properties as uncorrelated as possible. The processing involves modifying the TOI itself for what concerns the conversion to absorbed power and the correction of glitch tails. It also adds a flag TOI that masks the TOI samples that are not to be projected on maps for various reasons.<br />
<br />
<br />
=== Acronyms and definitions ===<br />
<br />
* 4K lines: EMI/EMC influence of the 4K cooler mechanical motion on the bolometer readout electronics.<br />
* ADC : analog to digital converter<br />
* IMO : instrument model<br />
* Jump : sudden change of the baseline level inside a ring<br />
* LFER : low frequency excess response<br />
* PBR : phase binned ring<br />
* RIMO : reduced IMO<br />
* Ring : pointing period<br />
* ROI : ring ordered information<br />
* RTS : random telegraphic signal<br />
* SSO : solar system object<br />
* TOI : time ordered information<br />
<br />
== Input TOI ==<br />
<br />
The input TOI consists in the AC modulated voltage output of the readout of each bolometer. The input has previously been decompressed, and converted from internal digital units to voltage via a constant factor. The TOI has a regular sampling at the acquisition frequency of facq=180.373700+-0.000050 Hz. There are almost no missing data in the TOIs, except for few hundred samples of 545 and 857GHz TOIs which are lost in the on-board compression due to saturation on the Galactic Center crossings. <br />
<br />
== General Pipeline Structure ==<br />
<br />
The figure on the right shows how the initial TOI is transformed and how flags are produced:<br />
<br />
[[File:HFI_4_4_2_ToiProc_ExplS.png|thumb|600px|A schematic of the TOI processing pipeline]]<br />
<br />
<br />
== Output TOIs and products ==<br />
<br />
A TOI of clean calibrated samples (ccTOI) and a combined flag TOI (fTOI) are the outputs of the processing. The ccTOI is calibrated so as to represent the instantaneous power absorbed by the detector up to a constant (which will be determined by the map-making destriper). It is worth mentioning how the ccTOI is changed with respect to the input TOI, beyond the harmless constant conversion factor from voltage to absorbed power. The demodulation stage allows to get the demodulated bolometer voltage. The non-linearity correction is a second-order polynomial correction based on the physical but static bolometer model. In order to avoid too much masking after glitches, a glitch tail is subtracted after an occurrence of a glitch in the TOI. The 4K cooler lines noise is substituted at a series of 9 single temporal frequencies. Finally, the temporal response of the bolometer is deconvolved. This affects mostly the high-temporal frequency part of the TOI, although a small but significant low frequency (the long time response) tail is corrected too. Although flagged samples are not projected, their value influences the valid samples somehow. Hence interpolation procedures introduce some indirect modifications of the TOI.<br />
The flag TOI is a combination a dozen flags with an OR logic. Only unflagged data are projected. The exhaustive list of flags is given here: CompressionError, NoData, SSO, UnstablePointing, Glitch, BoloPlateFluctuation, RTS, Jump, PSBab. A complete qualification of the data is obtained at the ring level. If the TOI shows an anomalous behaviour on a given ring, this ring is discarded from projection. A special production of TOIs is also made as an input to the beam analysis with Mars, Jupiter and Saturn.<br />
<br />
<br />
== Examples of clean TOIs ==<br />
<gallery widths=300px perrow=3><br />
File:10_143_5_PBR_10800RING_LFER4_JC.jpg|Phase-binned signal (PBR) for 6 consecutive rings from 4000 to 4005<br />
File:10_143_5_rcircles_LFER4_JC.jpg| Valid signal (up) and noise (bottom) for 2 consecutive circles of the ring 4000<br />
File:10_143_5_fft_LFER4_JC.jpg| Fourier power spectral density of the first 30 minutes of the ring 4005 with signal before (pink) and after (black) deconvolution (top) and noise (bottom)<br />
File:14_545_1_PBR_10800RING_LFER4_JC.jpg|Phase-binned signal (PBR) for 6 consecutive rings from 4000 to 4005<br />
File:14_545_1_rcircles_LFER4_JC.jpg| Valid signal (up) and noise (bottom) for 2 consecutive circles of the ring 4000<br />
File:14_545_1_fft_LFER4_JC.jpg| Fourier power spectral density of the first 30 minutes of the ring 4005 with signal before (pink) and after (black) deconvolution (top) and noise (bottom)<br />
</gallery><br />
<br />
[[Media:check_ring.pdf|Samples of PBR, TOIs, and PSDs of all detectors are shown in this file]]<br />
<br />
== Trends in the output processing variables ==<br />
<br />
Here we intend to show the trend of the systematic effects that are dealt with in the TOI processing. The full impact of each of them is analyzed in [[HFI-Validation]].<br />
<br />
=== ADC baseline ===<br />
The following figure shows the ADC baseline which is used prior to demodulation (a constant offset is removed for clarity). This baseline is obtained by smoothing on an hour block average the undemodulated TOI on unflagged samples.<br />
<br />
[[File:baseline.jpg|400px|ADC baseline for all bolometers]]<br />
<br />
=== Glitch statistics ===<br />
The glitch rate per channel is shown in this figure. For details, see copap. <br />
<br />
[[File:figIntermPaperGR.jpg|400px|Glitch rate evolution]]<br />
<br />
The percentage of flagged data (mostly due to Cosmic Rays) at the ring level is shown in these examples. No smoothing was applied. Only valid rings are shown.<br />
<gallery widths=450px perrow=2 caption="Percentage of flagged data"><br />
File:group10_143_5.jpg | 143 GHz bolometers<br />
File:group14_545_1.jpg | 545 GHz bolometers<br />
</gallery><br />
[[Media:PercentUnvalid2.pdf|The complete set of plots is here]]<br />
<br />
=== Thermal template for decorrelation ===<br />
[[File:T90.jpg|400px|Thermal template used for the decorrelation from bolometer plate temperature fluctuations]]<br />
<br />
A simple linear decorrelation is performed using the 2 dark bolometers as a proxy of the bolometer plate temperature. Coupling coefficients were measured during the CPV phase.<br />
<br />
=== 4K cooler lines variability ===<br />
The amplitude of the nine 4K cooler lines in aW at 10, 20, 30, 40, 50, 60, 70, 80 and 17 Hz is shown for 2 bolometers in the following figures. The trend is smoothed over 31 ring values after having discarded measurements done at a ring which is discarded for all bolometers.<br />
<gallery widths=450px perrow=2 caption="Amplitude of the nine 4K cooler lines"><br />
File:10_143_5_4Klines.jpg | 143_5 bolometer<br />
File:14_545_1_4Klines.jpg | 545_1 bolometer<br />
</gallery><br />
[[Media:Lines4K.pdf|The 4K cooler line coefficients of all bolometers are shown in this file]]<br />
<br />
The 4K cooler lines project onto the maps only for a limited fraction of rings, the so-called resonant rings. This is graphically shown in the following figure. For each ring (stable pointing period), the spin rate is very stable at about 1 rpm. From one ring to another, the spin frequency (shown as diamonds) changes around that value. The sky signal is imprinted at the corresponding spin frequency and its 5400 (60x90) harmonics. If one of the nine 4K cooler lines happens to coincide with one of the spin frequency harmonics (a resonant ring), it will project a sine-wave systematic on the maps. The horizontal coloured bars show the zone of influence of a particular 4K line (labeled on the left side of the plot), when folded around 16.666 mHz. When the spin frequency hits one of these zones, we have a resonant ring. The 4K line coefficient is interpolated for this ring and an estimate of the systematic effect is subtracted from the TOI. Resonant rings are different for different 4K lines. Note the two-level oscillation pattern of the spin frequency is due to the satellite attitude control system.<br />
<br />
[[File:spinfreq1.jpg|600px|Spin frequency and 4K line zones of influence]]<br />
<br />
=== jump correction ===<br />
A piecewise constant value is removed to the TOI if a jump is detected. See a jump example in this figure:<br />
<br />
[[File:jump_exe.png|200px|An example of a jump seen on the rmsigTOI]]<br />
<br />
The number of jumps per day (all bolometers included) is shown in this figure:<br />
<br />
[[File:jumps_per_day.png|200px|Evolution of jump number during the mission]]<br />
<br />
The jumps are uncorrelated from bolometer to bolometer. The total number of jumps detected in the nominal and full mission is shown here:<br />
<br />
[[File:jumps_per_bolometer.png|200px|Number of jumps per bolometer]]<br />
<br />
<br />
=== Trends in noise and signal ===<br />
<gallery widths=450px perrow=2 caption="Signal (top) and noise (bottom) smoothed at 1 minute. All values falling in a discarded ring are not plotted."><br />
File:10_143_5_smooth_Watt.jpg | 143_5 bolometer<br />
File:14_545_1_smooth_Watt.jpg | 545_1 bolometer<br />
</gallery><br />
<br />
[[Media:check_smooth.pdf|The smooth TOIs of all detectors are shown in this file]]<br />
<br />
=== Noise stationarity ===<br />
This is not the final version but gives a good idea of power spectra at the detector level of rmsigTOIs.<br />
[[Media:v53_meanSpectra_bySurvey.pdf|All PSDs can be seen in this file]]<br />
<br />
The standard deviation per ring corrected by ring duration bias is given here, one per bolometer using only valid rings. No smoothing is applied (except a 31-point smoothing for the 545 and 857 GHz channels) but values for rings discarded for all bolometers (see below) are not used. The standard deviation is computed on samples valid for map-making which are also not affected by the Galaxy or the point-sources using the usual flags. Two examples are given here.<br />
<gallery widths=450px perrow=2 caption="Standard deviation per ring"><br />
File:stddev_group10_143_5.jpg | 143_5 bolometer<br />
File:stddev_group_14_545_1.jpg | 545_1 bolometer<br />
</gallery><br />
<br />
<br />
The full series of plots is here:<br />
[[Media:StationarityInROIoutput2.pdf|Standard deviation of rmsig TOIs at the ring level]]<br />
<br />
Note the presence for 3 bolometers of a two-level noise system. No correction can be done for that effect. See one example here:<br />
<br />
[[File:23_353_TwoLevel.jpg|300px|An example of two-level noise system is seen in bolometer 23_353_3a]]<br />
<br />
<br />
An example of the higher order statistics which are used to unveil rings affected by RTS problems.<br />
<br />
[[File:HFI_4_4_2_RTSexample13.jpg|400px|Example of RTS detection]]<br />
<br />
=== Discarded rings ===<br />
<br />
Some rings are discarded (flagged) from further use (beam making, map making) by using ring statistics (see above and <cite>#planck2013-p03</cite>). For each statistic, we compare each ring value to the ring values averaged (RVA) over a large selection of rings (between 3000 and 21700). We also define the modified standard deviation (MSD) of a ring quantity as the standard deviation of that quantity over the rings that deviate by less than five nominal standard deviations. This truncation is necessary to be robust against extreme deviant rings.<br />
<br />
More specifically, a ring is discarded if it matches one of the following criteria:<br />
<br />
* the | mean-median | deviates from the RVA by more than fifteen times the MSD.<br />
* the standard deviation deviates from the RVA by more than -5 times the MSD (all cases corresponding to almost empty rings) or +15 times the MSD.<br />
* the Kolmogorov-Smirnov test deviates from the RVA by more than 15 times the MSD.<br />
* the ring duration is more than 90 min.<br />
* the ring is contaminated by RTS with an amplitude of more than one standard deviation of the noise. It concerns a few hundreds of rings for 3 bolometers (44_217_6b, 71_217_8a, 74_857_4). Notice that two bolometers are completely discarded for maps: 55_545_3 and 70_143_8, which present RTS at all time.<br />
<br />
For the three first criteria, a visual inspection of the rmsigTOI at each of the incriminated rings has shown that all these anomalous rings are due to either a drift, a small jump in the TOI trend or a sudden change of noise level, the origin of which is unknown at present.<br />
<br />
Once the list of discarded rings per bolometer is produced, a common list of discarded rings can be extracted for all bolometers (by using discarded rings for at least half the bolometers). Such rings correspond to identified phenomena, as can be seen on the following table.<br />
<br />
Furthermore, an isolated valid ring stuck between two common discarded rings becomes discarded as well.<br />
<br />
Table of '''common discarded rings''' of the nominal mission (rings 240-14723).<br />
<br />
{|table border=1 align=center cellpadding="3" cellspacing="0" style=text-align:center<br />
! '''Cause''' !! '''ring number'''<br />
|-<br />
!align="left"| manoeuvre<br />
|align="center"|304 1312 3590 3611 3642 3922 4949 6379 8456 11328<br />
|-<br />
! align="left"| Sorption Cooler Switchover <br />
|align="center"| 11149 11150 11151 11152 <br />
|-<br />
! align="left"| Rings too long<br />
| align="center"| 440 474 509 544 897 898 3589 13333 14627 14653 <br />
|-<br />
! align="left"| Star tracker switchover <br />
| align="center"| 14628 14654 <br />
|-<br />
! align="left"| Massive Glitch Event<br />
| align="center"| 7665 <br />
|-<br />
! align="left"| Solar Flare<br />
| align="center"| 11235 <br />
|}<br />
<br />
<br />
<br />
The following figure is a summary of the impact of the discarding process '''for each bolometer''' (the solid black line is the common discarded ring percentage). The outlier bolometers have some RTS problems as mentionned above.<br />
<br />
[[File:Fraction_discarded_rings_v53.png|300px|Fraction of discarded rings]]<br />
[[File:Fraction_discarded_time_v53.png|300px|Fraction of time due to discarded rings]]<br />
<br />
=== Effective integration time ===<br />
The following figure summarizes the effective integration time per bolometer. For that purpose the number of unflagged samples in non-discarded pointing periods have been used within the nominal mission. The average value is of about 335 days of effective integration time.<br />
<br />
[[File:HFInominal_IntegrationTime.jpg|600px|Effective integration time per bolometer]]<br />
<br />
== Flag description ==<br />
<br />
==== Input flags ====<br />
<br />
These flags are used as inputs to the TOI processing<br />
<br />
; The point-source flag (PSflag) <br />
: An earlier version of HFI point-source catalog is read back into a flag TOIs, at a given frequency. In practice, 5 sigma sources are masked within a radius of 1.3 FWHM (9, 7, 5, 5, 5, 5 arcmin at 100,143,217,353,545,857 GHz) TBC.<br />
<br />
; the galactic flag (Galflag<br />
: An earlier version of HFI maps is thresholded and apodized. The produced masks are read into flag TOIs. The retained threshold corresponds to a sky coverage of respectively 70, 70, 80, 90, 90, 90% at 100,143,217,353,545,857 GHz.<br />
<br />
; Solar System Object flag <br />
: For the TOI flag, Mars, Jupiter, Saturn are flagged up to a radius of NbBeam= 2,3,3,4,4,4 times the fiducial SSO_FWHM with SSO_FWHM= 9, 7, 5, 5, 5, 5 arcmin at 100,143,217,353,545,857 GHz.<br />
<br />
: As an input to planet mask for maps, Mars, Jupiter, Saturn are flagged with a radius computed as a coefficient depending on the planet (Factor_per_source) times NbBeam times SSO_FWHM, with Factor_per_source = 1.1, 2.25, 1.25 for Mars, Jupiter, Saturn respectively and NbBeam = 2.25, 4.25, 4.0, 5.0,.6.0, 8.0 at 100, 143, 217, 353, 545, 857 GHz. This flag is called SSOflag4map.<br />
<br />
: A small trailing tail is added to the mask to take into account the non-deconvolution of the planet signal which has been replaced by background values. The width of that tail is 10 % of the main flag diameter. The number of samples which are additionnaly flagged are the Factor_per_Source times AddSNafter with (trail) times (Factor_per_source)^3 samples where trail = 10, 30, 20, 20, 30, 40 at 100, 143, 217, 353, 545, 857 GHz.<br />
<br />
: Uranus and Neptune together with detected asteroids are masked by HFI. They are masked at the TOI level using an exclusion radius of 1.5 SSO_FWHM. At 857 GHz, 24 asteroids have been detected with HFI : 1Ceres, 2Pallas, 3Juno, 4Vesta, 7Iris, 8Flora, 9Metis, 10Hygiea, 11Parthenope, 12Victoria, 13Egeria, 14Irene, 15Eunomia, 16Psyche, 18Melpomene, 19Fortuna, 20Massalia, 29Amphitrite, 41Daphne, 45Eugenia, 52Europa, 88Thisbe, 704Interamnia, 324Bamberga.<br />
<br />
<gallery widths=200px perrow=4 caption="Local maps showing the SSO flag. The colors correspond to the surveys involved in the nominal mission (green = Survey I, yellow = Survey II, red = Survey III)."><br />
File:SSOflag_10_143_5J.png | 143 SWB row, around Jupiter 1st crossing<br />
File:SSOflag_10_143_5S.png | 143 SWB row, around Saturn<br />
File:SSOflag_10_143_5M.png | 143 SWB row, around Mars<br />
File:SSOflag_10_143_5a.png | 143 SWB row, random field in the ecliptic plane<br />
File:SSOflag_25_857_1J.png | 545/857 row, around Jupiter 1st crossing<br />
File:SSOflag_25_857_1S.png | 545/857 row, around Saturn<br />
File:SSOflag_25_857_1M.png | 545/857 row, around Mars<br />
File:SSOflag_25_857_1a.png | 545/857 row, random field in the ecliptic plane<br />
</gallery><br />
<br />
<br />
=== Output flags ===<br />
<br />
A FlagTOIproc is produced by the TOIprocessing. It marks measurements which are not reliable for any of the following reasons:<br />
* gap (no valid input data), enlarged by one sample on each side. It flags less than 0.00044% (resp. 0.00062%) of the nominal (resp. complete) mission. It is equivalent to less than 3 (resp. 8) minutes of data.<br />
* "glitch" on dark bolometers: as the thermal template used for decorrelation is computed from these bolometer data, chunks of one minute-length data are discarded for all bolometers if at least 50\% of the data for at least one dark bolometer are flagged during this time. It is efficient to flag the data around the maximum of thermal events.<br />
* "glitch" on individual bolometers : samples where the signal from a cosmic ray hit dominates the sky signal at more than 3 $\sigma$ are discarded.<br />
* jump as 100 samples are flagged around the computed position of the jump to take into account the error on this reconstructed position.<br />
<br />
<br />
So the flag produced for the map making, called Total_flag, is defined by:<br />
* Total_flag = UnstablePointing Flag OR FlagTOIproc OR SSOflag4map OR SSOflag seen<br />
where FlagTOIproc = gap OR flag thermal template OR glitch OR jump<br />
and for PSB bolometers, FlagTOIproc_AB = FlagTOIproc_A OR FlagTOIproc_B.<br />
Note that the Total_flag is then identical for the A and B bolometers of a PSB pair.<br />
<br />
At the destriping stage, a more restricted flag, called Total_flag_PS, is used. It is defined by<br />
Total_flag_PS = Total_flag OR PS_flag.<br />
<br />
<br />
== References ==<br />
<biblio force=false><br />
#[[References]] <br />
</biblio><br />
<br />
<br />
[[Category:Data processing|0042]]</div>
Fdesert
https://wiki.cosmos.esa.int/planckpla2015/index.php?title=TOI_processing&diff=4697
TOI processing
2013-02-25T08:52:28Z
<p>Fdesert: /* Discarded rings */</p>
<hr />
<div>This Section is kept short as this Planck release does not contain TOIs. The main information in the HFI processing paper is not duplicated here.<br />
<br />
==Overview==<br />
<br />
We describe here the how the TOIs are processed in order to be used for map production. We do not repeat the general features of the pipeline which are given in the HFI Data Processing article <cite>#planck2013-p03</cite>. Here we give complementary explanations on some details. The TOI of each bolometer is processed independently of the other bolometers, so as to keep the noise properties as uncorrelated as possible. The processing involves modifying the TOI itself for what concerns the conversion to absorbed power and the correction of glitch tails. It also adds a flag TOI that masks the TOI samples that are not to be projected on maps for various reasons.<br />
<br />
<br />
=== Acronyms and definitions ===<br />
<br />
* 4K lines: EMI/EMC influence of the 4K cooler mechanical motion on the bolometer readout electronics.<br />
* ADC : analog to digital converter<br />
* IMO : instrument model<br />
* Jump : sudden change of the baseline level inside a ring<br />
* LFER : low frequency excess response<br />
* PBR : phase binned ring<br />
* RIMO : reduced IMO<br />
* Ring : pointing period<br />
* ROI : ring ordered information<br />
* RTS : random telegraphic signal<br />
* SSO : solar system object<br />
* TOI : time ordered information<br />
<br />
== Input TOI ==<br />
<br />
The input TOI consists in the AC modulated voltage output of the readout of each bolometer. The input has previously been decompressed, and converted from internal digital units to voltage via a constant factor. The TOI has a regular sampling at the acquisition frequency of facq=180.373700+-0.000050 Hz. There are almost no missing data in the TOIs, except for few hundred samples of 545 and 857GHz TOIs which are lost in the on-board compression due to saturation on the Galactic Center crossings. <br />
<br />
== General Pipeline Structure ==<br />
<br />
The figure on the right shows how the initial TOI is transformed and how flags are produced:<br />
<br />
[[File:HFI_4_4_2_ToiProc_ExplS.png|thumb|600px|A schematic of the TOI processing pipeline]]<br />
<br />
<br />
== Output TOIs and products ==<br />
<br />
A TOI of clean calibrated samples (ccTOI) and a combined flag TOI (fTOI) are the outputs of the processing. The ccTOI is calibrated so as to represent the instantaneous power absorbed by the detector up to a constant (which will be determined by the map-making destriper). It is worth mentioning how the ccTOI is changed with respect to the input TOI, beyond the harmless constant conversion factor from voltage to absorbed power. The demodulation stage allows to get the demodulated bolometer voltage. The non-linearity correction is a second-order polynomial correction based on the physical but static bolometer model. In order to avoid too much masking after glitches, a glitch tail is subtracted after an occurrence of a glitch in the TOI. The 4K cooler lines noise is substituted at a series of 9 single temporal frequencies. Finally, the temporal response of the bolometer is deconvolved. This affects mostly the high-temporal frequency part of the TOI, although a small but significant low frequency (the long time response) tail is corrected too. Although flagged samples are not projected, their value influences the valid samples somehow. Hence interpolation procedures introduce some indirect modifications of the TOI.<br />
The flag TOI is a combination a dozen flags with an OR logic. Only unflagged data are projected. The exhaustive list of flags is given here: CompressionError, NoData, SSO, UnstablePointing, Glitch, BoloPlateFluctuation, RTS, Jump, PSBab. A complete qualification of the data is obtained at the ring level. If the TOI shows an anomalous behaviour on a given ring, this ring is discarded from projection. A special production of TOIs is also made as an input to the beam analysis with Mars, Jupiter and Saturn.<br />
<br />
<br />
== Examples of clean TOIs ==<br />
<gallery widths=300px perrow=3><br />
File:10_143_5_PBR_10800RING_LFER4_JC.jpg|Phase-binned signal (PBR) for 6 consecutive rings from 4000 to 4005<br />
File:10_143_5_rcircles_LFER4_JC.jpg| Valid signal (up) and noise (bottom) for 2 consecutive circles of the ring 4000<br />
File:10_143_5_fft_LFER4_JC.jpg| Fourier power spectral density of the first 30 minutes of the ring 4005 with signal before (pink) and after (black) deconvolution (top) and noise (bottom)<br />
File:14_545_1_PBR_10800RING_LFER4_JC.jpg|Phase-binned signal (PBR) for 6 consecutive rings from 4000 to 4005<br />
File:14_545_1_rcircles_LFER4_JC.jpg| Valid signal (up) and noise (bottom) for 2 consecutive circles of the ring 4000<br />
File:14_545_1_fft_LFER4_JC.jpg| Fourier power spectral density of the first 30 minutes of the ring 4005 with signal before (pink) and after (black) deconvolution (top) and noise (bottom)<br />
</gallery><br />
<br />
[[Media:check_ring.pdf|Samples of PBR, TOIs, and PSDs of all detectors are shown in this file]]<br />
<br />
== Trends in the output processing variables ==<br />
<br />
Here we intend to show the trend of the systematic effects that are dealt with in the TOI processing. The full impact of each of them is analyzed in [[HFI-Validation]].<br />
<br />
=== ADC baseline ===<br />
The following figure shows the ADC baseline which is used prior to demodulation (a constant offset is removed for clarity). This baseline is obtained by smoothing on an hour block average the undemodulated TOI on unflagged samples.<br />
<br />
[[File:baseline.jpg|400px|ADC baseline for all bolometers]]<br />
<br />
=== Glitch statistics ===<br />
The glitch rate per channel is shown in this figure. For details, see copap. <br />
<br />
[[File:figIntermPaperGR.jpg|400px|Glitch rate evolution]]<br />
<br />
The percentage of flagged data (mostly due to Cosmic Rays) at the ring level is shown in these examples. No smoothing was applied. Only valid rings are shown.<br />
<gallery widths=450px perrow=2 caption="Percentage of flagged data"><br />
File:group10_143_5.jpg | 143 GHz bolometers<br />
File:group14_545_1.jpg | 545 GHz bolometers<br />
</gallery><br />
[[Media:PercentUnvalid2.pdf|The complete set of plots is here]]<br />
<br />
=== Thermal template for decorrelation ===<br />
[[File:T90.jpg|400px|Thermal template used for the decorrelation from bolometer plate temperature fluctuations]]<br />
<br />
A simple linear decorrelation is performed using the 2 dark bolometers as a proxy of the bolometer plate temperature. Coupling coefficients were measured during the CPV phase.<br />
<br />
=== 4K cooler lines variability ===<br />
The amplitude of the nine 4K cooler lines in aW at 10, 20, 30, 40, 50, 60, 70, 80 and 17 Hz is shown for 2 bolometers in the following figures. The trend is smoothed over 31 ring values after having discarded measurements done at a ring which is discarded for all bolometers.<br />
<gallery widths=450px perrow=2 caption="Amplitude of the nine 4K cooler lines"><br />
File:10_143_5_4Klines.jpg | 143_5 bolometer<br />
File:14_545_1_4Klines.jpg | 545_1 bolometer<br />
</gallery><br />
[[Media:Lines4K.pdf|The 4K cooler line coefficients of all bolometers are shown in this file]]<br />
<br />
The 4K cooler lines project onto the maps only for a limited fraction of rings, the so-called resonant rings. This is graphically shown in the following figure. For each ring (stable pointing period), the spin rate is very stable at about 1 rpm. From one ring to another, the spin frequency (shown as diamonds) changes around that value. The sky signal is imprinted at the corresponding spin frequency and its 5400 (60x90) harmonics. If one of the nine 4K cooler lines happens to coincide with one of the spin frequency harmonics (a resonant ring), it will project a sine-wave systematic on the maps. The horizontal coloured bars show the zone of influence of a particular 4K line (labeled on the left side of the plot), when folded around 16.666 mHz. When the spin frequency hits one of these zones, we have a resonant ring. The 4K line coefficient is interpolated for this ring and an estimate of the systematic effect is subtracted from the TOI. Resonant rings are different for different 4K lines. Note the two-level oscillation pattern of the spin frequency is due to the satellite attitude control system.<br />
<br />
[[File:spinfreq1.jpg|600px|Spin frequency and 4K line zones of influence]]<br />
<br />
=== jump correction ===<br />
A piecewise constant value is removed to the TOI if a jump is detected. See a jump example in this figure:<br />
<br />
[[File:jump_exe.png|200px|An example of a jump seen on the rmsigTOI]]<br />
<br />
The number of jumps per day (all bolometers included) is shown in this figure:<br />
<br />
[[File:jumps_per_day.png|200px|Evolution of jump number during the mission]]<br />
<br />
The jumps are uncorrelated from bolometer to bolometer. The total number of jumps detected in the nominal and full mission is shown here:<br />
<br />
[[File:jumps_per_bolometer.png|200px|Number of jumps per bolometer]]<br />
<br />
<br />
=== Trends in noise and signal ===<br />
<gallery widths=450px perrow=2 caption="Signal (top) and noise (bottom) smoothed at 1 minute. All values falling in a discarded ring are not plotted."><br />
File:10_143_5_smooth_Watt.jpg | 143_5 bolometer<br />
File:14_545_1_smooth_Watt.jpg | 545_1 bolometer<br />
</gallery><br />
<br />
[[Media:check_smooth.pdf|The smooth TOIs of all detectors are shown in this file]]<br />
<br />
=== Noise stationarity ===<br />
This is not the final version but gives a good idea of power spectra at the detector level of rmsigTOIs.<br />
[[Media:v53_meanSpectra_bySurvey.pdf|All PSDs can be seen in this file]]<br />
<br />
The standard deviation per ring corrected by ring duration bias is given here, one per bolometer using only valid rings. No smoothing is applied (except a 31-point smoothing for the 545 and 857 GHz channels) but values for rings discarded for all bolometers (see below) are not used. The standard deviation is computed on samples valid for map-making which are also not affected by the Galaxy or the point-sources using the usual flags. Two examples are given here.<br />
<gallery widths=450px perrow=2 caption="Standard deviation per ring"><br />
File:stddev_group10_143_5.jpg | 143_5 bolometer<br />
File:stddev_group_14_545_1.jpg | 545_1 bolometer<br />
</gallery><br />
<br />
<br />
The full series of plots is here:<br />
[[Media:StationarityInROIoutput2.pdf|Standard deviation of rmsig TOIs at the ring level]]<br />
<br />
Note the presence for 3 bolometers of a two-level noise system. No correction can be done for that effect. See one example here:<br />
<br />
[[File:23_353_TwoLevel.jpg|300px|An example of two-level noise system is seen in bolometer 23_353_3a]]<br />
<br />
<br />
An example of the higher order statistics which are used to unveil rings affected by RTS problems.<br />
<br />
[[File:HFI_4_4_2_RTSexample13.jpg|400px|Example of RTS detection]]<br />
<br />
=== Discarded rings ===<br />
<br />
Some rings are discarded (flagged) from further use (beam making, map making) by using ring statistics (see above and <cite>#planck2013-p03</cite>). For each statistic, we compare each ring value to the ring values averaged (RVA) over a large selection of rings (between 3000 and 21700). We also define the modified standard deviation (MSD) of a ring quantity as the standard deviation of that quantity over the rings that deviate by less than five nominal standard deviations. This truncation is necessary to be robust against extreme deviant rings.<br />
<br />
More specifically, a ring is discarded if it matches one of the following criteria:<br />
<br />
* the | mean-median | deviates from the RVA by more than fifteen times the MSD.<br />
* the standard deviation deviates from the RVA by more than -5 times the MSD (all cases corresponding to almost empty rings) or +15 times the MSD.<br />
* the Kolmogorov-Smirnov test deviates from the RVA by more than 15 times the MSD.<br />
* the ring duration is more than 90 min.<br />
* the ring is contaminated by RTS with an amplitude of more than one standard deviation of the noise. It concerns a few hundreds of rings for 3 bolometers (44_217_6b, 71_217_8a, 74_857_4). Notice that two bolometers are completely discarded for maps: 55_545_3 and 70_143_8, which present RTS at all time.<br />
<br />
For the three first criteria, a visual inspection of the rmsigTOI at each of the incriminated rings has shown that all these anomalous rings are due to either a drift, a small jump in the TOI trend or a sudden change of noise level, the origin of which is unknown at present.<br />
<br />
Once the list of discarded rings per bolometer is produced, a common list of discarded rings can be extracted for all bolometers (by using discarded rings for at least half the bolometers). Such rings correspond to identified phenomena, as can be seen on the following table.<br />
<br />
Furthermore, an isolated valid ring stuck between two common discarded rings becomes discarded as well.<br />
<br />
Table of '''common discarded rings''' of the nominal mission (rings 240-14723).<br />
<br />
{|table border=1 align=center cellpadding="3" cellspacing="0" style=text-align:center<br />
! '''Cause''' !! '''ring number'''<br />
|-<br />
!align="left"| manoeuvre<br />
|align="center"|304 1312 3590 3611 3642 3922 4949 6379 8456 11328<br />
|-<br />
! align="left"| Sorption Cooler Switchover <br />
|align="center"| 11149 11150 11151 11152 <br />
|-<br />
! align="left"| Rings too long<br />
| align="center"| 440 474 509 544 897 898 3589 13333 14627 14653 <br />
|-<br />
! align="left"| Star tracker switchover <br />
| align="center"| 14628 14654 <br />
|-<br />
! align="left"| Massive Glitch Event<br />
| align="center"| 7665 <br />
|-<br />
! align="left"| Solar Flare<br />
| align="center"| 11235 <br />
|}<br />
<br />
<br />
<br />
The following figure is a summary of the impact of the discarding process '''for each bolometer''' (the solid black line is the common discarded ring percentage). The outlier bolometers have some RTS problems as mentionned above.<br />
<br />
[[File:Fraction_discarded_rings_v53.png|300px|Fraction of discarded rings]]<br />
[[File:Fraction_discarded_time_v53.png|300px|Fraction of time due to discarded rings]]<br />
<br />
=== Effective integration time ===<br />
The following figure summarizes the effective integration time per bolometer. For that purpose the number of unflagged samples in non-discarded pointing periods have been used within the nominal mission. The average value is of about 335 days of effective integration time.<br />
<br />
[[File:HFInominal_IntegrationTime.jpg|600px|Effective integration time per bolometer]]<br />
<br />
== Flag description ==<br />
<br />
==== Input flags ====<br />
<br />
These flags are used as inputs to the TOI processing<br />
<br />
; The point-source flag (PSflag) <br />
: An earlier version of HFI point-source catalog is read back into a flag TOIs, at a given frequency. In practice, 5 sigma sources are masked within a radius of 1.3 FWHM (9, 7, 5, 5, 5, 5 arcmin at 100,143,217,353,545,857 GHz) TBC.<br />
<br />
; the galactic flag (Galflag<br />
: An earlier version of HFI maps is thresholded and apodized. The produced masks are read into flag TOIs. The retained threshold corresponds to a sky coverage of respectively 70, 70, 80, 90, 90, 90% at 100,143,217,353,545,857 GHz.<br />
<br />
; Solar System Object flag <br />
: For the TOI flag, Mars, Jupiter, Saturn are flagged up to a radius of NbBeam= 2,3,3,4,4,4 times the fiducial SSO_FWHM with SSO_FWHM= 9, 7, 5, 5, 5, 5 arcmin at 100,143,217,353,545,857 GHz.<br />
<br />
: As an input to planet mask for maps, Mars, Jupiter, Saturn are flagged with a radius computed as a coefficient depending on the planet (Factor_per_source) times NbBeam times SSO_FWHM, with Factor_per_source = 1.1, 2.25, 1.25 for Mars, Jupiter, Saturn respectively and NbBeam = 2.25, 4.25, 4.0, 5.0,.6.0, 8.0 at 100, 143, 217, 353, 545, 857 GHz. This flag is called SSOflag4map.<br />
<br />
: A small trailing tail is added to the mask to take into account the non-deconvolution of the planet signal which has been replaced by background values. The width of that tail is 10 % of the main flag diameter. The number of samples which are additionnaly flagged are the Factor_per_Source times AddSNafter with (trail) times (Factor_per_source)^3 samples where trail = 10, 30, 20, 20, 30, 40 at 100, 143, 217, 353, 545, 857 GHz.<br />
<br />
: Uranus and Neptune together with detected asteroids are masked by HFI. They are masked at the TOI level using an exclusion radius of 1.5 SSO_FWHM. At 857 GHz, 24 asteroids have been detected with HFI : 1Ceres, 2Pallas, 3Juno, 4Vesta, 7Iris, 8Flora, 9Metis, 10Hygiea, 11Parthenope, 12Victoria, 13Egeria, 14Irene, 15Eunomia, 16Psyche, 18Melpomene, 19Fortuna, 20Massalia, 29Amphitrite, 41Daphne, 45Eugenia, 52Europa, 88Thisbe, 704Interamnia, 324Bamberga.<br />
<br />
<gallery widths=200px perrow=4 caption="Local maps showing the SSO flag. The colors correspond to the surveys involved in the nominal mission (green = Survey I, yellow = Survey II, red = Survey III)."><br />
File:SSOflag_10_143_5J.png | 143 SWB row, around Jupiter 1st crossing<br />
File:SSOflag_10_143_5S.png | 143 SWB row, around Saturn<br />
File:SSOflag_10_143_5M.png | 143 SWB row, around Mars<br />
File:SSOflag_10_143_5a.png | 143 SWB row, random field in the ecliptic plane<br />
File:SSOflag_25_857_1J.png | 545/857 row, around Jupiter 1st crossing<br />
File:SSOflag_25_857_1S.png | 545/857 row, around Saturn<br />
File:SSOflag_25_857_1M.png | 545/857 row, around Mars<br />
File:SSOflag_25_857_1a.png | 545/857 row, random field in the ecliptic plane<br />
</gallery><br />
<br />
<br />
=== Output flags ===<br />
<br />
A FlagTOIproc is produced by the TOIprocessing. It marks measurements which are not reliable for any of the following reasons:<br />
* gap (no valid input data), enlarged by one sample on each side. It flags less than 0.00044% (resp. 0.00062%) of the nominal (resp. complete) mission. It is equivalent to less than 3 (resp. 8) minutes of data.<br />
* "glitch" on dark bolometers: as the thermal template used for decorrelation is computed from these bolometer data, chunks of one minute-length data are discarded for all bolometers if at least 50\% of the data for at least one dark bolometer are flagged during this time. It is efficient to flag the data around the maximum of thermal events.<br />
* "glitch" on individual bolometers : samples where the signal from a cosmic ray hit dominates the sky signal at more than 3 $\sigma$ are discarded.<br />
* jump as 100 samples are flagged around the computed position of the jump to take into account the error on this reconstructed position.<br />
<br />
<br />
So the flag produced for the map making, called Total_flag, is defined by:<br />
* Total_flag = UnstablePointing Flag OR FlagTOIproc OR SSOflag4map OR SSOflag seen<br />
where FlagTOIproc = gap OR flag thermal template OR glitch OR jump<br />
and for PSB bolometers, FlagTOIproc_AB = FlagTOIproc_A OR FlagTOIproc_B.<br />
Note that the Total_flag is then identical for the A and B bolometers of a PSB pair.<br />
<br />
At the destriping stage, a more restricted flag, called Total_flag_PS, is used. It is defined by<br />
Total_flag_PS = Total_flag OR PS_flag.<br />
<br />
<br />
=== References ===<br />
<biblio force=false><br />
#[[References]] <br />
</biblio><br />
<br />
<br />
[[Category:Data processing|0042]]</div>
Fdesert
https://wiki.cosmos.esa.int/planckpla2015/index.php?title=TOI_processing&diff=4696
TOI processing
2013-02-25T08:50:19Z
<p>Fdesert: /* Output flags */</p>
<hr />
<div>This Section is kept short as this Planck release does not contain TOIs. The main information in the HFI processing paper is not duplicated here.<br />
<br />
==Overview==<br />
<br />
We describe here the how the TOIs are processed in order to be used for map production. We do not repeat the general features of the pipeline which are given in the HFI Data Processing article <cite>#planck2013-p03</cite>. Here we give complementary explanations on some details. The TOI of each bolometer is processed independently of the other bolometers, so as to keep the noise properties as uncorrelated as possible. The processing involves modifying the TOI itself for what concerns the conversion to absorbed power and the correction of glitch tails. It also adds a flag TOI that masks the TOI samples that are not to be projected on maps for various reasons.<br />
<br />
<br />
=== Acronyms and definitions ===<br />
<br />
* 4K lines: EMI/EMC influence of the 4K cooler mechanical motion on the bolometer readout electronics.<br />
* ADC : analog to digital converter<br />
* IMO : instrument model<br />
* Jump : sudden change of the baseline level inside a ring<br />
* LFER : low frequency excess response<br />
* PBR : phase binned ring<br />
* RIMO : reduced IMO<br />
* Ring : pointing period<br />
* ROI : ring ordered information<br />
* RTS : random telegraphic signal<br />
* SSO : solar system object<br />
* TOI : time ordered information<br />
<br />
== Input TOI ==<br />
<br />
The input TOI consists in the AC modulated voltage output of the readout of each bolometer. The input has previously been decompressed, and converted from internal digital units to voltage via a constant factor. The TOI has a regular sampling at the acquisition frequency of facq=180.373700+-0.000050 Hz. There are almost no missing data in the TOIs, except for few hundred samples of 545 and 857GHz TOIs which are lost in the on-board compression due to saturation on the Galactic Center crossings. <br />
<br />
== General Pipeline Structure ==<br />
<br />
The figure on the right shows how the initial TOI is transformed and how flags are produced:<br />
<br />
[[File:HFI_4_4_2_ToiProc_ExplS.png|thumb|600px|A schematic of the TOI processing pipeline]]<br />
<br />
<br />
== Output TOIs and products ==<br />
<br />
A TOI of clean calibrated samples (ccTOI) and a combined flag TOI (fTOI) are the outputs of the processing. The ccTOI is calibrated so as to represent the instantaneous power absorbed by the detector up to a constant (which will be determined by the map-making destriper). It is worth mentioning how the ccTOI is changed with respect to the input TOI, beyond the harmless constant conversion factor from voltage to absorbed power. The demodulation stage allows to get the demodulated bolometer voltage. The non-linearity correction is a second-order polynomial correction based on the physical but static bolometer model. In order to avoid too much masking after glitches, a glitch tail is subtracted after an occurrence of a glitch in the TOI. The 4K cooler lines noise is substituted at a series of 9 single temporal frequencies. Finally, the temporal response of the bolometer is deconvolved. This affects mostly the high-temporal frequency part of the TOI, although a small but significant low frequency (the long time response) tail is corrected too. Although flagged samples are not projected, their value influences the valid samples somehow. Hence interpolation procedures introduce some indirect modifications of the TOI.<br />
The flag TOI is a combination a dozen flags with an OR logic. Only unflagged data are projected. The exhaustive list of flags is given here: CompressionError, NoData, SSO, UnstablePointing, Glitch, BoloPlateFluctuation, RTS, Jump, PSBab. A complete qualification of the data is obtained at the ring level. If the TOI shows an anomalous behaviour on a given ring, this ring is discarded from projection. A special production of TOIs is also made as an input to the beam analysis with Mars, Jupiter and Saturn.<br />
<br />
<br />
== Examples of clean TOIs ==<br />
<gallery widths=300px perrow=3><br />
File:10_143_5_PBR_10800RING_LFER4_JC.jpg|Phase-binned signal (PBR) for 6 consecutive rings from 4000 to 4005<br />
File:10_143_5_rcircles_LFER4_JC.jpg| Valid signal (up) and noise (bottom) for 2 consecutive circles of the ring 4000<br />
File:10_143_5_fft_LFER4_JC.jpg| Fourier power spectral density of the first 30 minutes of the ring 4005 with signal before (pink) and after (black) deconvolution (top) and noise (bottom)<br />
File:14_545_1_PBR_10800RING_LFER4_JC.jpg|Phase-binned signal (PBR) for 6 consecutive rings from 4000 to 4005<br />
File:14_545_1_rcircles_LFER4_JC.jpg| Valid signal (up) and noise (bottom) for 2 consecutive circles of the ring 4000<br />
File:14_545_1_fft_LFER4_JC.jpg| Fourier power spectral density of the first 30 minutes of the ring 4005 with signal before (pink) and after (black) deconvolution (top) and noise (bottom)<br />
</gallery><br />
<br />
[[Media:check_ring.pdf|Samples of PBR, TOIs, and PSDs of all detectors are shown in this file]]<br />
<br />
== Trends in the output processing variables ==<br />
<br />
Here we intend to show the trend of the systematic effects that are dealt with in the TOI processing. The full impact of each of them is analyzed in [[HFI-Validation]].<br />
<br />
=== ADC baseline ===<br />
The following figure shows the ADC baseline which is used prior to demodulation (a constant offset is removed for clarity). This baseline is obtained by smoothing on an hour block average the undemodulated TOI on unflagged samples.<br />
<br />
[[File:baseline.jpg|400px|ADC baseline for all bolometers]]<br />
<br />
=== Glitch statistics ===<br />
The glitch rate per channel is shown in this figure. For details, see copap. <br />
<br />
[[File:figIntermPaperGR.jpg|400px|Glitch rate evolution]]<br />
<br />
The percentage of flagged data (mostly due to Cosmic Rays) at the ring level is shown in these examples. No smoothing was applied. Only valid rings are shown.<br />
<gallery widths=450px perrow=2 caption="Percentage of flagged data"><br />
File:group10_143_5.jpg | 143 GHz bolometers<br />
File:group14_545_1.jpg | 545 GHz bolometers<br />
</gallery><br />
[[Media:PercentUnvalid2.pdf|The complete set of plots is here]]<br />
<br />
=== Thermal template for decorrelation ===<br />
[[File:T90.jpg|400px|Thermal template used for the decorrelation from bolometer plate temperature fluctuations]]<br />
<br />
A simple linear decorrelation is performed using the 2 dark bolometers as a proxy of the bolometer plate temperature. Coupling coefficients were measured during the CPV phase.<br />
<br />
=== 4K cooler lines variability ===<br />
The amplitude of the nine 4K cooler lines in aW at 10, 20, 30, 40, 50, 60, 70, 80 and 17 Hz is shown for 2 bolometers in the following figures. The trend is smoothed over 31 ring values after having discarded measurements done at a ring which is discarded for all bolometers.<br />
<gallery widths=450px perrow=2 caption="Amplitude of the nine 4K cooler lines"><br />
File:10_143_5_4Klines.jpg | 143_5 bolometer<br />
File:14_545_1_4Klines.jpg | 545_1 bolometer<br />
</gallery><br />
[[Media:Lines4K.pdf|The 4K cooler line coefficients of all bolometers are shown in this file]]<br />
<br />
The 4K cooler lines project onto the maps only for a limited fraction of rings, the so-called resonant rings. This is graphically shown in the following figure. For each ring (stable pointing period), the spin rate is very stable at about 1 rpm. From one ring to another, the spin frequency (shown as diamonds) changes around that value. The sky signal is imprinted at the corresponding spin frequency and its 5400 (60x90) harmonics. If one of the nine 4K cooler lines happens to coincide with one of the spin frequency harmonics (a resonant ring), it will project a sine-wave systematic on the maps. The horizontal coloured bars show the zone of influence of a particular 4K line (labeled on the left side of the plot), when folded around 16.666 mHz. When the spin frequency hits one of these zones, we have a resonant ring. The 4K line coefficient is interpolated for this ring and an estimate of the systematic effect is subtracted from the TOI. Resonant rings are different for different 4K lines. Note the two-level oscillation pattern of the spin frequency is due to the satellite attitude control system.<br />
<br />
[[File:spinfreq1.jpg|600px|Spin frequency and 4K line zones of influence]]<br />
<br />
=== jump correction ===<br />
A piecewise constant value is removed to the TOI if a jump is detected. See a jump example in this figure:<br />
<br />
[[File:jump_exe.png|200px|An example of a jump seen on the rmsigTOI]]<br />
<br />
The number of jumps per day (all bolometers included) is shown in this figure:<br />
<br />
[[File:jumps_per_day.png|200px|Evolution of jump number during the mission]]<br />
<br />
The jumps are uncorrelated from bolometer to bolometer. The total number of jumps detected in the nominal and full mission is shown here:<br />
<br />
[[File:jumps_per_bolometer.png|200px|Number of jumps per bolometer]]<br />
<br />
<br />
=== Trends in noise and signal ===<br />
<gallery widths=450px perrow=2 caption="Signal (top) and noise (bottom) smoothed at 1 minute. All values falling in a discarded ring are not plotted."><br />
File:10_143_5_smooth_Watt.jpg | 143_5 bolometer<br />
File:14_545_1_smooth_Watt.jpg | 545_1 bolometer<br />
</gallery><br />
<br />
[[Media:check_smooth.pdf|The smooth TOIs of all detectors are shown in this file]]<br />
<br />
=== Noise stationarity ===<br />
This is not the final version but gives a good idea of power spectra at the detector level of rmsigTOIs.<br />
[[Media:v53_meanSpectra_bySurvey.pdf|All PSDs can be seen in this file]]<br />
<br />
The standard deviation per ring corrected by ring duration bias is given here, one per bolometer using only valid rings. No smoothing is applied (except a 31-point smoothing for the 545 and 857 GHz channels) but values for rings discarded for all bolometers (see below) are not used. The standard deviation is computed on samples valid for map-making which are also not affected by the Galaxy or the point-sources using the usual flags. Two examples are given here.<br />
<gallery widths=450px perrow=2 caption="Standard deviation per ring"><br />
File:stddev_group10_143_5.jpg | 143_5 bolometer<br />
File:stddev_group_14_545_1.jpg | 545_1 bolometer<br />
</gallery><br />
<br />
<br />
The full series of plots is here:<br />
[[Media:StationarityInROIoutput2.pdf|Standard deviation of rmsig TOIs at the ring level]]<br />
<br />
Note the presence for 3 bolometers of a two-level noise system. No correction can be done for that effect. See one example here:<br />
<br />
[[File:23_353_TwoLevel.jpg|300px|An example of two-level noise system is seen in bolometer 23_353_3a]]<br />
<br />
<br />
An example of the higher order statistics which are used to unveil rings affected by RTS problems.<br />
<br />
[[File:HFI_4_4_2_RTSexample13.jpg|400px|Example of RTS detection]]<br />
<br />
=== Discarded rings ===<br />
<br />
Some rings are discarded (flagged) from further use (beam making, map making) by using ring statistics (see above and the HFI data processing paper REF). For each statistic, we compare each ring value to the ring values averaged (RVA) over a large selection of rings (between 3000 and 21700). We also define the modified standard deviation (MSD) of a ring quantity as the standard deviation of that quantity over the rings that deviate by less than five nominal standard deviations. This truncation is necessary to be robust against extreme deviant rings.<br />
<br />
More specifically, a ring is discarded if it matches one of the following criteria:<br />
<br />
* the | mean-median | deviates from the RVA by more than fifteen times the MSD.<br />
* the standard deviation deviates from the RVA by more than -5 times the MSD (all cases corresponding to almost empty rings) or +15 times the MSD.<br />
* the Kolmogorov-Smirnov test deviates from the RVA by more than 15 times the MSD.<br />
* the ring duration is more than 90 min.<br />
* the ring is contaminated by RTS with an amplitude of more than one standard deviation of the noise. It concerns a few hundreds of rings for 3 bolometers (44_217_6b, 71_217_8a, 74_857_4). Notice that two bolometers are completely discarded for maps: 55_545_3 and 70_143_8, which present RTS at all time.<br />
<br />
For the three first criteria, a visual inspection of the rmsigTOI at each of the incriminated rings has shown that all these anomalous rings are due to either a drift, a small jump in the TOI trend or a sudden change of noise level, the origin of which is unknown at present.<br />
<br />
Once the list of discarded rings per bolometer is produced, a common list of discarded rings can be extracted for all bolometers (by using discarded rings for at least half the bolometers). Such rings correspond to identified phenomena, as can be seen on the following table.<br />
<br />
Furthermore, an isolated valid ring stuck between two common discarded rings becomes discarded as well.<br />
<br />
Table of '''common discarded rings''' of the nominal mission (rings 240-14723).<br />
<br />
{|table border=1 align=center cellpadding="3" cellspacing="0" style=text-align:center<br />
! '''Cause''' !! '''ring number'''<br />
|-<br />
!align="left"| manoeuvre<br />
|align="center"|304 1312 3590 3611 3642 3922 4949 6379 8456 11328<br />
|-<br />
! align="left"| Sorption Cooler Switchover <br />
|align="center"| 11149 11150 11151 11152 <br />
|-<br />
! align="left"| Rings too long<br />
| align="center"| 440 474 509 544 897 898 3589 13333 14627 14653 <br />
|-<br />
! align="left"| Star tracker switchover <br />
| align="center"| 14628 14654 <br />
|-<br />
! align="left"| Massive Glitch Event<br />
| align="center"| 7665 <br />
|-<br />
! align="left"| Solar Flare<br />
| align="center"| 11235 <br />
|}<br />
<br />
<br />
<br />
The following figure is a summary of the impact of the discarding process '''for each bolometer''' (the solid black line is the common discarded ring percentage). The outlier bolometers have some RTS problems as mentionned above.<br />
<br />
[[File:Fraction_discarded_rings_v53.png|300px|Fraction of discarded rings]]<br />
[[File:Fraction_discarded_time_v53.png|300px|Fraction of time due to discarded rings]]<br />
<br />
=== Effective integration time ===<br />
The following figure summarizes the effective integration time per bolometer. For that purpose the number of unflagged samples in non-discarded pointing periods have been used within the nominal mission. The average value is of about 335 days of effective integration time.<br />
<br />
[[File:HFInominal_IntegrationTime.jpg|600px|Effective integration time per bolometer]]<br />
<br />
== Flag description ==<br />
<br />
==== Input flags ====<br />
<br />
These flags are used as inputs to the TOI processing<br />
<br />
; The point-source flag (PSflag) <br />
: An earlier version of HFI point-source catalog is read back into a flag TOIs, at a given frequency. In practice, 5 sigma sources are masked within a radius of 1.3 FWHM (9, 7, 5, 5, 5, 5 arcmin at 100,143,217,353,545,857 GHz) TBC.<br />
<br />
; the galactic flag (Galflag<br />
: An earlier version of HFI maps is thresholded and apodized. The produced masks are read into flag TOIs. The retained threshold corresponds to a sky coverage of respectively 70, 70, 80, 90, 90, 90% at 100,143,217,353,545,857 GHz.<br />
<br />
; Solar System Object flag <br />
: For the TOI flag, Mars, Jupiter, Saturn are flagged up to a radius of NbBeam= 2,3,3,4,4,4 times the fiducial SSO_FWHM with SSO_FWHM= 9, 7, 5, 5, 5, 5 arcmin at 100,143,217,353,545,857 GHz.<br />
<br />
: As an input to planet mask for maps, Mars, Jupiter, Saturn are flagged with a radius computed as a coefficient depending on the planet (Factor_per_source) times NbBeam times SSO_FWHM, with Factor_per_source = 1.1, 2.25, 1.25 for Mars, Jupiter, Saturn respectively and NbBeam = 2.25, 4.25, 4.0, 5.0,.6.0, 8.0 at 100, 143, 217, 353, 545, 857 GHz. This flag is called SSOflag4map.<br />
<br />
: A small trailing tail is added to the mask to take into account the non-deconvolution of the planet signal which has been replaced by background values. The width of that tail is 10 % of the main flag diameter. The number of samples which are additionnaly flagged are the Factor_per_Source times AddSNafter with (trail) times (Factor_per_source)^3 samples where trail = 10, 30, 20, 20, 30, 40 at 100, 143, 217, 353, 545, 857 GHz.<br />
<br />
: Uranus and Neptune together with detected asteroids are masked by HFI. They are masked at the TOI level using an exclusion radius of 1.5 SSO_FWHM. At 857 GHz, 24 asteroids have been detected with HFI : 1Ceres, 2Pallas, 3Juno, 4Vesta, 7Iris, 8Flora, 9Metis, 10Hygiea, 11Parthenope, 12Victoria, 13Egeria, 14Irene, 15Eunomia, 16Psyche, 18Melpomene, 19Fortuna, 20Massalia, 29Amphitrite, 41Daphne, 45Eugenia, 52Europa, 88Thisbe, 704Interamnia, 324Bamberga.<br />
<br />
<gallery widths=200px perrow=4 caption="Local maps showing the SSO flag. The colors correspond to the surveys involved in the nominal mission (green = Survey I, yellow = Survey II, red = Survey III)."><br />
File:SSOflag_10_143_5J.png | 143 SWB row, around Jupiter 1st crossing<br />
File:SSOflag_10_143_5S.png | 143 SWB row, around Saturn<br />
File:SSOflag_10_143_5M.png | 143 SWB row, around Mars<br />
File:SSOflag_10_143_5a.png | 143 SWB row, random field in the ecliptic plane<br />
File:SSOflag_25_857_1J.png | 545/857 row, around Jupiter 1st crossing<br />
File:SSOflag_25_857_1S.png | 545/857 row, around Saturn<br />
File:SSOflag_25_857_1M.png | 545/857 row, around Mars<br />
File:SSOflag_25_857_1a.png | 545/857 row, random field in the ecliptic plane<br />
</gallery><br />
<br />
<br />
=== Output flags ===<br />
<br />
A FlagTOIproc is produced by the TOIprocessing. It marks measurements which are not reliable for any of the following reasons:<br />
* gap (no valid input data), enlarged by one sample on each side. It flags less than 0.00044% (resp. 0.00062%) of the nominal (resp. complete) mission. It is equivalent to less than 3 (resp. 8) minutes of data.<br />
* "glitch" on dark bolometers: as the thermal template used for decorrelation is computed from these bolometer data, chunks of one minute-length data are discarded for all bolometers if at least 50\% of the data for at least one dark bolometer are flagged during this time. It is efficient to flag the data around the maximum of thermal events.<br />
* "glitch" on individual bolometers : samples where the signal from a cosmic ray hit dominates the sky signal at more than 3 $\sigma$ are discarded.<br />
* jump as 100 samples are flagged around the computed position of the jump to take into account the error on this reconstructed position.<br />
<br />
<br />
So the flag produced for the map making, called Total_flag, is defined by:<br />
* Total_flag = UnstablePointing Flag OR FlagTOIproc OR SSOflag4map OR SSOflag seen<br />
where FlagTOIproc = gap OR flag thermal template OR glitch OR jump<br />
and for PSB bolometers, FlagTOIproc_AB = FlagTOIproc_A OR FlagTOIproc_B.<br />
Note that the Total_flag is then identical for the A and B bolometers of a PSB pair.<br />
<br />
At the destriping stage, a more restricted flag, called Total_flag_PS, is used. It is defined by<br />
Total_flag_PS = Total_flag OR PS_flag.<br />
<br />
<br />
=== References ===<br />
<biblio force=false><br />
#[[References]] <br />
</biblio><br />
<br />
<br />
[[Category:Data processing|0042]]</div>
Fdesert
https://wiki.cosmos.esa.int/planckpla2015/index.php?title=TOI_processing&diff=4695
TOI processing
2013-02-25T08:48:44Z
<p>Fdesert: /* Output flags */</p>
<hr />
<div>This Section is kept short as this Planck release does not contain TOIs. The main information in the HFI processing paper is not duplicated here.<br />
<br />
==Overview==<br />
<br />
We describe here the how the TOIs are processed in order to be used for map production. We do not repeat the general features of the pipeline which are given in the HFI Data Processing article <cite>#planck2013-p03</cite>. Here we give complementary explanations on some details. The TOI of each bolometer is processed independently of the other bolometers, so as to keep the noise properties as uncorrelated as possible. The processing involves modifying the TOI itself for what concerns the conversion to absorbed power and the correction of glitch tails. It also adds a flag TOI that masks the TOI samples that are not to be projected on maps for various reasons.<br />
<br />
<br />
=== Acronyms and definitions ===<br />
<br />
* 4K lines: EMI/EMC influence of the 4K cooler mechanical motion on the bolometer readout electronics.<br />
* ADC : analog to digital converter<br />
* IMO : instrument model<br />
* Jump : sudden change of the baseline level inside a ring<br />
* LFER : low frequency excess response<br />
* PBR : phase binned ring<br />
* RIMO : reduced IMO<br />
* Ring : pointing period<br />
* ROI : ring ordered information<br />
* RTS : random telegraphic signal<br />
* SSO : solar system object<br />
* TOI : time ordered information<br />
<br />
== Input TOI ==<br />
<br />
The input TOI consists in the AC modulated voltage output of the readout of each bolometer. The input has previously been decompressed, and converted from internal digital units to voltage via a constant factor. The TOI has a regular sampling at the acquisition frequency of facq=180.373700+-0.000050 Hz. There are almost no missing data in the TOIs, except for few hundred samples of 545 and 857GHz TOIs which are lost in the on-board compression due to saturation on the Galactic Center crossings. <br />
<br />
== General Pipeline Structure ==<br />
<br />
The figure on the right shows how the initial TOI is transformed and how flags are produced:<br />
<br />
[[File:HFI_4_4_2_ToiProc_ExplS.png|thumb|600px|A schematic of the TOI processing pipeline]]<br />
<br />
<br />
== Output TOIs and products ==<br />
<br />
A TOI of clean calibrated samples (ccTOI) and a combined flag TOI (fTOI) are the outputs of the processing. The ccTOI is calibrated so as to represent the instantaneous power absorbed by the detector up to a constant (which will be determined by the map-making destriper). It is worth mentioning how the ccTOI is changed with respect to the input TOI, beyond the harmless constant conversion factor from voltage to absorbed power. The demodulation stage allows to get the demodulated bolometer voltage. The non-linearity correction is a second-order polynomial correction based on the physical but static bolometer model. In order to avoid too much masking after glitches, a glitch tail is subtracted after an occurrence of a glitch in the TOI. The 4K cooler lines noise is substituted at a series of 9 single temporal frequencies. Finally, the temporal response of the bolometer is deconvolved. This affects mostly the high-temporal frequency part of the TOI, although a small but significant low frequency (the long time response) tail is corrected too. Although flagged samples are not projected, their value influences the valid samples somehow. Hence interpolation procedures introduce some indirect modifications of the TOI.<br />
The flag TOI is a combination a dozen flags with an OR logic. Only unflagged data are projected. The exhaustive list of flags is given here: CompressionError, NoData, SSO, UnstablePointing, Glitch, BoloPlateFluctuation, RTS, Jump, PSBab. A complete qualification of the data is obtained at the ring level. If the TOI shows an anomalous behaviour on a given ring, this ring is discarded from projection. A special production of TOIs is also made as an input to the beam analysis with Mars, Jupiter and Saturn.<br />
<br />
<br />
== Examples of clean TOIs ==<br />
<gallery widths=300px perrow=3><br />
File:10_143_5_PBR_10800RING_LFER4_JC.jpg|Phase-binned signal (PBR) for 6 consecutive rings from 4000 to 4005<br />
File:10_143_5_rcircles_LFER4_JC.jpg| Valid signal (up) and noise (bottom) for 2 consecutive circles of the ring 4000<br />
File:10_143_5_fft_LFER4_JC.jpg| Fourier power spectral density of the first 30 minutes of the ring 4005 with signal before (pink) and after (black) deconvolution (top) and noise (bottom)<br />
File:14_545_1_PBR_10800RING_LFER4_JC.jpg|Phase-binned signal (PBR) for 6 consecutive rings from 4000 to 4005<br />
File:14_545_1_rcircles_LFER4_JC.jpg| Valid signal (up) and noise (bottom) for 2 consecutive circles of the ring 4000<br />
File:14_545_1_fft_LFER4_JC.jpg| Fourier power spectral density of the first 30 minutes of the ring 4005 with signal before (pink) and after (black) deconvolution (top) and noise (bottom)<br />
</gallery><br />
<br />
[[Media:check_ring.pdf|Samples of PBR, TOIs, and PSDs of all detectors are shown in this file]]<br />
<br />
== Trends in the output processing variables ==<br />
<br />
Here we intend to show the trend of the systematic effects that are dealt with in the TOI processing. The full impact of each of them is analyzed in [[HFI-Validation]].<br />
<br />
=== ADC baseline ===<br />
The following figure shows the ADC baseline which is used prior to demodulation (a constant offset is removed for clarity). This baseline is obtained by smoothing on an hour block average the undemodulated TOI on unflagged samples.<br />
<br />
[[File:baseline.jpg|400px|ADC baseline for all bolometers]]<br />
<br />
=== Glitch statistics ===<br />
The glitch rate per channel is shown in this figure. For details, see copap. <br />
<br />
[[File:figIntermPaperGR.jpg|400px|Glitch rate evolution]]<br />
<br />
The percentage of flagged data (mostly due to Cosmic Rays) at the ring level is shown in these examples. No smoothing was applied. Only valid rings are shown.<br />
<gallery widths=450px perrow=2 caption="Percentage of flagged data"><br />
File:group10_143_5.jpg | 143 GHz bolometers<br />
File:group14_545_1.jpg | 545 GHz bolometers<br />
</gallery><br />
[[Media:PercentUnvalid2.pdf|The complete set of plots is here]]<br />
<br />
=== Thermal template for decorrelation ===<br />
[[File:T90.jpg|400px|Thermal template used for the decorrelation from bolometer plate temperature fluctuations]]<br />
<br />
A simple linear decorrelation is performed using the 2 dark bolometers as a proxy of the bolometer plate temperature. Coupling coefficients were measured during the CPV phase.<br />
<br />
=== 4K cooler lines variability ===<br />
The amplitude of the nine 4K cooler lines in aW at 10, 20, 30, 40, 50, 60, 70, 80 and 17 Hz is shown for 2 bolometers in the following figures. The trend is smoothed over 31 ring values after having discarded measurements done at a ring which is discarded for all bolometers.<br />
<gallery widths=450px perrow=2 caption="Amplitude of the nine 4K cooler lines"><br />
File:10_143_5_4Klines.jpg | 143_5 bolometer<br />
File:14_545_1_4Klines.jpg | 545_1 bolometer<br />
</gallery><br />
[[Media:Lines4K.pdf|The 4K cooler line coefficients of all bolometers are shown in this file]]<br />
<br />
The 4K cooler lines project onto the maps only for a limited fraction of rings, the so-called resonant rings. This is graphically shown in the following figure. For each ring (stable pointing period), the spin rate is very stable at about 1 rpm. From one ring to another, the spin frequency (shown as diamonds) changes around that value. The sky signal is imprinted at the corresponding spin frequency and its 5400 (60x90) harmonics. If one of the nine 4K cooler lines happens to coincide with one of the spin frequency harmonics (a resonant ring), it will project a sine-wave systematic on the maps. The horizontal coloured bars show the zone of influence of a particular 4K line (labeled on the left side of the plot), when folded around 16.666 mHz. When the spin frequency hits one of these zones, we have a resonant ring. The 4K line coefficient is interpolated for this ring and an estimate of the systematic effect is subtracted from the TOI. Resonant rings are different for different 4K lines. Note the two-level oscillation pattern of the spin frequency is due to the satellite attitude control system.<br />
<br />
[[File:spinfreq1.jpg|600px|Spin frequency and 4K line zones of influence]]<br />
<br />
=== jump correction ===<br />
A piecewise constant value is removed to the TOI if a jump is detected. See a jump example in this figure:<br />
<br />
[[File:jump_exe.png|200px|An example of a jump seen on the rmsigTOI]]<br />
<br />
The number of jumps per day (all bolometers included) is shown in this figure:<br />
<br />
[[File:jumps_per_day.png|200px|Evolution of jump number during the mission]]<br />
<br />
The jumps are uncorrelated from bolometer to bolometer. The total number of jumps detected in the nominal and full mission is shown here:<br />
<br />
[[File:jumps_per_bolometer.png|200px|Number of jumps per bolometer]]<br />
<br />
<br />
=== Trends in noise and signal ===<br />
<gallery widths=450px perrow=2 caption="Signal (top) and noise (bottom) smoothed at 1 minute. All values falling in a discarded ring are not plotted."><br />
File:10_143_5_smooth_Watt.jpg | 143_5 bolometer<br />
File:14_545_1_smooth_Watt.jpg | 545_1 bolometer<br />
</gallery><br />
<br />
[[Media:check_smooth.pdf|The smooth TOIs of all detectors are shown in this file]]<br />
<br />
=== Noise stationarity ===<br />
This is not the final version but gives a good idea of power spectra at the detector level of rmsigTOIs.<br />
[[Media:v53_meanSpectra_bySurvey.pdf|All PSDs can be seen in this file]]<br />
<br />
The standard deviation per ring corrected by ring duration bias is given here, one per bolometer using only valid rings. No smoothing is applied (except a 31-point smoothing for the 545 and 857 GHz channels) but values for rings discarded for all bolometers (see below) are not used. The standard deviation is computed on samples valid for map-making which are also not affected by the Galaxy or the point-sources using the usual flags. Two examples are given here.<br />
<gallery widths=450px perrow=2 caption="Standard deviation per ring"><br />
File:stddev_group10_143_5.jpg | 143_5 bolometer<br />
File:stddev_group_14_545_1.jpg | 545_1 bolometer<br />
</gallery><br />
<br />
<br />
The full series of plots is here:<br />
[[Media:StationarityInROIoutput2.pdf|Standard deviation of rmsig TOIs at the ring level]]<br />
<br />
Note the presence for 3 bolometers of a two-level noise system. No correction can be done for that effect. See one example here:<br />
<br />
[[File:23_353_TwoLevel.jpg|300px|An example of two-level noise system is seen in bolometer 23_353_3a]]<br />
<br />
<br />
An example of the higher order statistics which are used to unveil rings affected by RTS problems.<br />
<br />
[[File:HFI_4_4_2_RTSexample13.jpg|400px|Example of RTS detection]]<br />
<br />
=== Discarded rings ===<br />
<br />
Some rings are discarded (flagged) from further use (beam making, map making) by using ring statistics (see above and the HFI data processing paper REF). For each statistic, we compare each ring value to the ring values averaged (RVA) over a large selection of rings (between 3000 and 21700). We also define the modified standard deviation (MSD) of a ring quantity as the standard deviation of that quantity over the rings that deviate by less than five nominal standard deviations. This truncation is necessary to be robust against extreme deviant rings.<br />
<br />
More specifically, a ring is discarded if it matches one of the following criteria:<br />
<br />
* the | mean-median | deviates from the RVA by more than fifteen times the MSD.<br />
* the standard deviation deviates from the RVA by more than -5 times the MSD (all cases corresponding to almost empty rings) or +15 times the MSD.<br />
* the Kolmogorov-Smirnov test deviates from the RVA by more than 15 times the MSD.<br />
* the ring duration is more than 90 min.<br />
* the ring is contaminated by RTS with an amplitude of more than one standard deviation of the noise. It concerns a few hundreds of rings for 3 bolometers (44_217_6b, 71_217_8a, 74_857_4). Notice that two bolometers are completely discarded for maps: 55_545_3 and 70_143_8, which present RTS at all time.<br />
<br />
For the three first criteria, a visual inspection of the rmsigTOI at each of the incriminated rings has shown that all these anomalous rings are due to either a drift, a small jump in the TOI trend or a sudden change of noise level, the origin of which is unknown at present.<br />
<br />
Once the list of discarded rings per bolometer is produced, a common list of discarded rings can be extracted for all bolometers (by using discarded rings for at least half the bolometers). Such rings correspond to identified phenomena, as can be seen on the following table.<br />
<br />
Furthermore, an isolated valid ring stuck between two common discarded rings becomes discarded as well.<br />
<br />
Table of '''common discarded rings''' of the nominal mission (rings 240-14723).<br />
<br />
{|table border=1 align=center cellpadding="3" cellspacing="0" style=text-align:center<br />
! '''Cause''' !! '''ring number'''<br />
|-<br />
!align="left"| manoeuvre<br />
|align="center"|304 1312 3590 3611 3642 3922 4949 6379 8456 11328<br />
|-<br />
! align="left"| Sorption Cooler Switchover <br />
|align="center"| 11149 11150 11151 11152 <br />
|-<br />
! align="left"| Rings too long<br />
| align="center"| 440 474 509 544 897 898 3589 13333 14627 14653 <br />
|-<br />
! align="left"| Star tracker switchover <br />
| align="center"| 14628 14654 <br />
|-<br />
! align="left"| Massive Glitch Event<br />
| align="center"| 7665 <br />
|-<br />
! align="left"| Solar Flare<br />
| align="center"| 11235 <br />
|}<br />
<br />
<br />
<br />
The following figure is a summary of the impact of the discarding process '''for each bolometer''' (the solid black line is the common discarded ring percentage). The outlier bolometers have some RTS problems as mentionned above.<br />
<br />
[[File:Fraction_discarded_rings_v53.png|300px|Fraction of discarded rings]]<br />
[[File:Fraction_discarded_time_v53.png|300px|Fraction of time due to discarded rings]]<br />
<br />
=== Effective integration time ===<br />
The following figure summarizes the effective integration time per bolometer. For that purpose the number of unflagged samples in non-discarded pointing periods have been used within the nominal mission. The average value is of about 335 days of effective integration time.<br />
<br />
[[File:HFInominal_IntegrationTime.jpg|600px|Effective integration time per bolometer]]<br />
<br />
== Flag description ==<br />
<br />
==== Input flags ====<br />
<br />
These flags are used as inputs to the TOI processing<br />
<br />
; The point-source flag (PSflag) <br />
: An earlier version of HFI point-source catalog is read back into a flag TOIs, at a given frequency. In practice, 5 sigma sources are masked within a radius of 1.3 FWHM (9, 7, 5, 5, 5, 5 arcmin at 100,143,217,353,545,857 GHz) TBC.<br />
<br />
; the galactic flag (Galflag<br />
: An earlier version of HFI maps is thresholded and apodized. The produced masks are read into flag TOIs. The retained threshold corresponds to a sky coverage of respectively 70, 70, 80, 90, 90, 90% at 100,143,217,353,545,857 GHz.<br />
<br />
; Solar System Object flag <br />
: For the TOI flag, Mars, Jupiter, Saturn are flagged up to a radius of NbBeam= 2,3,3,4,4,4 times the fiducial SSO_FWHM with SSO_FWHM= 9, 7, 5, 5, 5, 5 arcmin at 100,143,217,353,545,857 GHz.<br />
<br />
: As an input to planet mask for maps, Mars, Jupiter, Saturn are flagged with a radius computed as a coefficient depending on the planet (Factor_per_source) times NbBeam times SSO_FWHM, with Factor_per_source = 1.1, 2.25, 1.25 for Mars, Jupiter, Saturn respectively and NbBeam = 2.25, 4.25, 4.0, 5.0,.6.0, 8.0 at 100, 143, 217, 353, 545, 857 GHz. This flag is called SSOflag4map.<br />
<br />
: A small trailing tail is added to the mask to take into account the non-deconvolution of the planet signal which has been replaced by background values. The width of that tail is 10 % of the main flag diameter. The number of samples which are additionnaly flagged are the Factor_per_Source times AddSNafter with (trail) times (Factor_per_source)^3 samples where trail = 10, 30, 20, 20, 30, 40 at 100, 143, 217, 353, 545, 857 GHz.<br />
<br />
: Uranus and Neptune together with detected asteroids are masked by HFI. They are masked at the TOI level using an exclusion radius of 1.5 SSO_FWHM. At 857 GHz, 24 asteroids have been detected with HFI : 1Ceres, 2Pallas, 3Juno, 4Vesta, 7Iris, 8Flora, 9Metis, 10Hygiea, 11Parthenope, 12Victoria, 13Egeria, 14Irene, 15Eunomia, 16Psyche, 18Melpomene, 19Fortuna, 20Massalia, 29Amphitrite, 41Daphne, 45Eugenia, 52Europa, 88Thisbe, 704Interamnia, 324Bamberga.<br />
<br />
<gallery widths=200px perrow=4 caption="Local maps showing the SSO flag. The colors correspond to the surveys involved in the nominal mission (green = Survey I, yellow = Survey II, red = Survey III)."><br />
File:SSOflag_10_143_5J.png | 143 SWB row, around Jupiter 1st crossing<br />
File:SSOflag_10_143_5S.png | 143 SWB row, around Saturn<br />
File:SSOflag_10_143_5M.png | 143 SWB row, around Mars<br />
File:SSOflag_10_143_5a.png | 143 SWB row, random field in the ecliptic plane<br />
File:SSOflag_25_857_1J.png | 545/857 row, around Jupiter 1st crossing<br />
File:SSOflag_25_857_1S.png | 545/857 row, around Saturn<br />
File:SSOflag_25_857_1M.png | 545/857 row, around Mars<br />
File:SSOflag_25_857_1a.png | 545/857 row, random field in the ecliptic plane<br />
</gallery><br />
<br />
<br />
=== Output flags ===<br />
<br />
A FlagTOIproc is produced by the TOIprocessing. It marks measurements which are not reliable for any of the following reasons:<br />
* gap (no valid input data), enlarged by one sample on each side. It flags less than 0.00044% (resp. 0.00062%) of the nominal (resp. complete) mission. It is equivalent to less than 3 (resp. 8) minutes of data.<br />
* "glitch" on dark bolometers: as the thermal template used for decorrelation is computed from these bolometer data, chunks of one minute-length data are discarded for all bolometers if at least 50\% of the data for at least one dark bolometer are flagged during this time. It is efficient to flag the data around the maximum of thermal events.<br />
* "glitch" on individual bolometers : samples where the signal from a cosmic ray hit dominates the sky signal at more than 3 $\sigma$ are discarded.<br />
* jump as 100 samples are flagged around the computed position of the jump to take into account the error on this reconstructed position.<br />
<br />
<br />
So the flag produced for the map making, called Total_flag, is defined by:<br />
* Total_flag = UnstablePointing Flag OR FlagTOIproc OR SSOflag4map OR SSOflag seen<br />
where FlagTOIproc = gap OR flag thermal template OR glitch OR jump<br />
and for PSB bolometers, FlagTOIproc_AB = FlagTOIproc_A OR FlagTOIproc_B.<br />
Note that the Total_flag is then identical for the A and B bolometers of a PSB pair.<br />
<br />
At the destriping stage, a more restricted flag, called Total_flag_PS, is used. It is defined by<br />
Total_flag_PS = Total_flag OR PS_flag.<br />
<br />
<br />
<biblio force=false><br />
#[[References]] <br />
</biblio><br />
<br />
<br />
[[Category:Data processing|0042]]</div>
Fdesert
https://wiki.cosmos.esa.int/planckpla2015/index.php?title=TOI_processing&diff=4694
TOI processing
2013-02-25T08:47:01Z
<p>Fdesert: /* Overview */</p>
<hr />
<div>This Section is kept short as this Planck release does not contain TOIs. The main information in the HFI processing paper is not duplicated here.<br />
<br />
==Overview==<br />
<br />
We describe here the how the TOIs are processed in order to be used for map production. We do not repeat the general features of the pipeline which are given in the HFI Data Processing article <cite>#planck2013-p03</cite>. Here we give complementary explanations on some details. The TOI of each bolometer is processed independently of the other bolometers, so as to keep the noise properties as uncorrelated as possible. The processing involves modifying the TOI itself for what concerns the conversion to absorbed power and the correction of glitch tails. It also adds a flag TOI that masks the TOI samples that are not to be projected on maps for various reasons.<br />
<br />
<br />
=== Acronyms and definitions ===<br />
<br />
* 4K lines: EMI/EMC influence of the 4K cooler mechanical motion on the bolometer readout electronics.<br />
* ADC : analog to digital converter<br />
* IMO : instrument model<br />
* Jump : sudden change of the baseline level inside a ring<br />
* LFER : low frequency excess response<br />
* PBR : phase binned ring<br />
* RIMO : reduced IMO<br />
* Ring : pointing period<br />
* ROI : ring ordered information<br />
* RTS : random telegraphic signal<br />
* SSO : solar system object<br />
* TOI : time ordered information<br />
<br />
== Input TOI ==<br />
<br />
The input TOI consists in the AC modulated voltage output of the readout of each bolometer. The input has previously been decompressed, and converted from internal digital units to voltage via a constant factor. The TOI has a regular sampling at the acquisition frequency of facq=180.373700+-0.000050 Hz. There are almost no missing data in the TOIs, except for few hundred samples of 545 and 857GHz TOIs which are lost in the on-board compression due to saturation on the Galactic Center crossings. <br />
<br />
== General Pipeline Structure ==<br />
<br />
The figure on the right shows how the initial TOI is transformed and how flags are produced:<br />
<br />
[[File:HFI_4_4_2_ToiProc_ExplS.png|thumb|600px|A schematic of the TOI processing pipeline]]<br />
<br />
<br />
== Output TOIs and products ==<br />
<br />
A TOI of clean calibrated samples (ccTOI) and a combined flag TOI (fTOI) are the outputs of the processing. The ccTOI is calibrated so as to represent the instantaneous power absorbed by the detector up to a constant (which will be determined by the map-making destriper). It is worth mentioning how the ccTOI is changed with respect to the input TOI, beyond the harmless constant conversion factor from voltage to absorbed power. The demodulation stage allows to get the demodulated bolometer voltage. The non-linearity correction is a second-order polynomial correction based on the physical but static bolometer model. In order to avoid too much masking after glitches, a glitch tail is subtracted after an occurrence of a glitch in the TOI. The 4K cooler lines noise is substituted at a series of 9 single temporal frequencies. Finally, the temporal response of the bolometer is deconvolved. This affects mostly the high-temporal frequency part of the TOI, although a small but significant low frequency (the long time response) tail is corrected too. Although flagged samples are not projected, their value influences the valid samples somehow. Hence interpolation procedures introduce some indirect modifications of the TOI.<br />
The flag TOI is a combination a dozen flags with an OR logic. Only unflagged data are projected. The exhaustive list of flags is given here: CompressionError, NoData, SSO, UnstablePointing, Glitch, BoloPlateFluctuation, RTS, Jump, PSBab. A complete qualification of the data is obtained at the ring level. If the TOI shows an anomalous behaviour on a given ring, this ring is discarded from projection. A special production of TOIs is also made as an input to the beam analysis with Mars, Jupiter and Saturn.<br />
<br />
<br />
== Examples of clean TOIs ==<br />
<gallery widths=300px perrow=3><br />
File:10_143_5_PBR_10800RING_LFER4_JC.jpg|Phase-binned signal (PBR) for 6 consecutive rings from 4000 to 4005<br />
File:10_143_5_rcircles_LFER4_JC.jpg| Valid signal (up) and noise (bottom) for 2 consecutive circles of the ring 4000<br />
File:10_143_5_fft_LFER4_JC.jpg| Fourier power spectral density of the first 30 minutes of the ring 4005 with signal before (pink) and after (black) deconvolution (top) and noise (bottom)<br />
File:14_545_1_PBR_10800RING_LFER4_JC.jpg|Phase-binned signal (PBR) for 6 consecutive rings from 4000 to 4005<br />
File:14_545_1_rcircles_LFER4_JC.jpg| Valid signal (up) and noise (bottom) for 2 consecutive circles of the ring 4000<br />
File:14_545_1_fft_LFER4_JC.jpg| Fourier power spectral density of the first 30 minutes of the ring 4005 with signal before (pink) and after (black) deconvolution (top) and noise (bottom)<br />
</gallery><br />
<br />
[[Media:check_ring.pdf|Samples of PBR, TOIs, and PSDs of all detectors are shown in this file]]<br />
<br />
== Trends in the output processing variables ==<br />
<br />
Here we intend to show the trend of the systematic effects that are dealt with in the TOI processing. The full impact of each of them is analyzed in [[HFI-Validation]].<br />
<br />
=== ADC baseline ===<br />
The following figure shows the ADC baseline which is used prior to demodulation (a constant offset is removed for clarity). This baseline is obtained by smoothing on an hour block average the undemodulated TOI on unflagged samples.<br />
<br />
[[File:baseline.jpg|400px|ADC baseline for all bolometers]]<br />
<br />
=== Glitch statistics ===<br />
The glitch rate per channel is shown in this figure. For details, see copap. <br />
<br />
[[File:figIntermPaperGR.jpg|400px|Glitch rate evolution]]<br />
<br />
The percentage of flagged data (mostly due to Cosmic Rays) at the ring level is shown in these examples. No smoothing was applied. Only valid rings are shown.<br />
<gallery widths=450px perrow=2 caption="Percentage of flagged data"><br />
File:group10_143_5.jpg | 143 GHz bolometers<br />
File:group14_545_1.jpg | 545 GHz bolometers<br />
</gallery><br />
[[Media:PercentUnvalid2.pdf|The complete set of plots is here]]<br />
<br />
=== Thermal template for decorrelation ===<br />
[[File:T90.jpg|400px|Thermal template used for the decorrelation from bolometer plate temperature fluctuations]]<br />
<br />
A simple linear decorrelation is performed using the 2 dark bolometers as a proxy of the bolometer plate temperature. Coupling coefficients were measured during the CPV phase.<br />
<br />
=== 4K cooler lines variability ===<br />
The amplitude of the nine 4K cooler lines in aW at 10, 20, 30, 40, 50, 60, 70, 80 and 17 Hz is shown for 2 bolometers in the following figures. The trend is smoothed over 31 ring values after having discarded measurements done at a ring which is discarded for all bolometers.<br />
<gallery widths=450px perrow=2 caption="Amplitude of the nine 4K cooler lines"><br />
File:10_143_5_4Klines.jpg | 143_5 bolometer<br />
File:14_545_1_4Klines.jpg | 545_1 bolometer<br />
</gallery><br />
[[Media:Lines4K.pdf|The 4K cooler line coefficients of all bolometers are shown in this file]]<br />
<br />
The 4K cooler lines project onto the maps only for a limited fraction of rings, the so-called resonant rings. This is graphically shown in the following figure. For each ring (stable pointing period), the spin rate is very stable at about 1 rpm. From one ring to another, the spin frequency (shown as diamonds) changes around that value. The sky signal is imprinted at the corresponding spin frequency and its 5400 (60x90) harmonics. If one of the nine 4K cooler lines happens to coincide with one of the spin frequency harmonics (a resonant ring), it will project a sine-wave systematic on the maps. The horizontal coloured bars show the zone of influence of a particular 4K line (labeled on the left side of the plot), when folded around 16.666 mHz. When the spin frequency hits one of these zones, we have a resonant ring. The 4K line coefficient is interpolated for this ring and an estimate of the systematic effect is subtracted from the TOI. Resonant rings are different for different 4K lines. Note the two-level oscillation pattern of the spin frequency is due to the satellite attitude control system.<br />
<br />
[[File:spinfreq1.jpg|600px|Spin frequency and 4K line zones of influence]]<br />
<br />
=== jump correction ===<br />
A piecewise constant value is removed to the TOI if a jump is detected. See a jump example in this figure:<br />
<br />
[[File:jump_exe.png|200px|An example of a jump seen on the rmsigTOI]]<br />
<br />
The number of jumps per day (all bolometers included) is shown in this figure:<br />
<br />
[[File:jumps_per_day.png|200px|Evolution of jump number during the mission]]<br />
<br />
The jumps are uncorrelated from bolometer to bolometer. The total number of jumps detected in the nominal and full mission is shown here:<br />
<br />
[[File:jumps_per_bolometer.png|200px|Number of jumps per bolometer]]<br />
<br />
<br />
=== Trends in noise and signal ===<br />
<gallery widths=450px perrow=2 caption="Signal (top) and noise (bottom) smoothed at 1 minute. All values falling in a discarded ring are not plotted."><br />
File:10_143_5_smooth_Watt.jpg | 143_5 bolometer<br />
File:14_545_1_smooth_Watt.jpg | 545_1 bolometer<br />
</gallery><br />
<br />
[[Media:check_smooth.pdf|The smooth TOIs of all detectors are shown in this file]]<br />
<br />
=== Noise stationarity ===<br />
This is not the final version but gives a good idea of power spectra at the detector level of rmsigTOIs.<br />
[[Media:v53_meanSpectra_bySurvey.pdf|All PSDs can be seen in this file]]<br />
<br />
The standard deviation per ring corrected by ring duration bias is given here, one per bolometer using only valid rings. No smoothing is applied (except a 31-point smoothing for the 545 and 857 GHz channels) but values for rings discarded for all bolometers (see below) are not used. The standard deviation is computed on samples valid for map-making which are also not affected by the Galaxy or the point-sources using the usual flags. Two examples are given here.<br />
<gallery widths=450px perrow=2 caption="Standard deviation per ring"><br />
File:stddev_group10_143_5.jpg | 143_5 bolometer<br />
File:stddev_group_14_545_1.jpg | 545_1 bolometer<br />
</gallery><br />
<br />
<br />
The full series of plots is here:<br />
[[Media:StationarityInROIoutput2.pdf|Standard deviation of rmsig TOIs at the ring level]]<br />
<br />
Note the presence for 3 bolometers of a two-level noise system. No correction can be done for that effect. See one example here:<br />
<br />
[[File:23_353_TwoLevel.jpg|300px|An example of two-level noise system is seen in bolometer 23_353_3a]]<br />
<br />
<br />
An example of the higher order statistics which are used to unveil rings affected by RTS problems.<br />
<br />
[[File:HFI_4_4_2_RTSexample13.jpg|400px|Example of RTS detection]]<br />
<br />
=== Discarded rings ===<br />
<br />
Some rings are discarded (flagged) from further use (beam making, map making) by using ring statistics (see above and the HFI data processing paper REF). For each statistic, we compare each ring value to the ring values averaged (RVA) over a large selection of rings (between 3000 and 21700). We also define the modified standard deviation (MSD) of a ring quantity as the standard deviation of that quantity over the rings that deviate by less than five nominal standard deviations. This truncation is necessary to be robust against extreme deviant rings.<br />
<br />
More specifically, a ring is discarded if it matches one of the following criteria:<br />
<br />
* the | mean-median | deviates from the RVA by more than fifteen times the MSD.<br />
* the standard deviation deviates from the RVA by more than -5 times the MSD (all cases corresponding to almost empty rings) or +15 times the MSD.<br />
* the Kolmogorov-Smirnov test deviates from the RVA by more than 15 times the MSD.<br />
* the ring duration is more than 90 min.<br />
* the ring is contaminated by RTS with an amplitude of more than one standard deviation of the noise. It concerns a few hundreds of rings for 3 bolometers (44_217_6b, 71_217_8a, 74_857_4). Notice that two bolometers are completely discarded for maps: 55_545_3 and 70_143_8, which present RTS at all time.<br />
<br />
For the three first criteria, a visual inspection of the rmsigTOI at each of the incriminated rings has shown that all these anomalous rings are due to either a drift, a small jump in the TOI trend or a sudden change of noise level, the origin of which is unknown at present.<br />
<br />
Once the list of discarded rings per bolometer is produced, a common list of discarded rings can be extracted for all bolometers (by using discarded rings for at least half the bolometers). Such rings correspond to identified phenomena, as can be seen on the following table.<br />
<br />
Furthermore, an isolated valid ring stuck between two common discarded rings becomes discarded as well.<br />
<br />
Table of '''common discarded rings''' of the nominal mission (rings 240-14723).<br />
<br />
{|table border=1 align=center cellpadding="3" cellspacing="0" style=text-align:center<br />
! '''Cause''' !! '''ring number'''<br />
|-<br />
!align="left"| manoeuvre<br />
|align="center"|304 1312 3590 3611 3642 3922 4949 6379 8456 11328<br />
|-<br />
! align="left"| Sorption Cooler Switchover <br />
|align="center"| 11149 11150 11151 11152 <br />
|-<br />
! align="left"| Rings too long<br />
| align="center"| 440 474 509 544 897 898 3589 13333 14627 14653 <br />
|-<br />
! align="left"| Star tracker switchover <br />
| align="center"| 14628 14654 <br />
|-<br />
! align="left"| Massive Glitch Event<br />
| align="center"| 7665 <br />
|-<br />
! align="left"| Solar Flare<br />
| align="center"| 11235 <br />
|}<br />
<br />
<br />
<br />
The following figure is a summary of the impact of the discarding process '''for each bolometer''' (the solid black line is the common discarded ring percentage). The outlier bolometers have some RTS problems as mentionned above.<br />
<br />
[[File:Fraction_discarded_rings_v53.png|300px|Fraction of discarded rings]]<br />
[[File:Fraction_discarded_time_v53.png|300px|Fraction of time due to discarded rings]]<br />
<br />
=== Effective integration time ===<br />
The following figure summarizes the effective integration time per bolometer. For that purpose the number of unflagged samples in non-discarded pointing periods have been used within the nominal mission. The average value is of about 335 days of effective integration time.<br />
<br />
[[File:HFInominal_IntegrationTime.jpg|600px|Effective integration time per bolometer]]<br />
<br />
== Flag description ==<br />
<br />
==== Input flags ====<br />
<br />
These flags are used as inputs to the TOI processing<br />
<br />
; The point-source flag (PSflag) <br />
: An earlier version of HFI point-source catalog is read back into a flag TOIs, at a given frequency. In practice, 5 sigma sources are masked within a radius of 1.3 FWHM (9, 7, 5, 5, 5, 5 arcmin at 100,143,217,353,545,857 GHz) TBC.<br />
<br />
; the galactic flag (Galflag<br />
: An earlier version of HFI maps is thresholded and apodized. The produced masks are read into flag TOIs. The retained threshold corresponds to a sky coverage of respectively 70, 70, 80, 90, 90, 90% at 100,143,217,353,545,857 GHz.<br />
<br />
; Solar System Object flag <br />
: For the TOI flag, Mars, Jupiter, Saturn are flagged up to a radius of NbBeam= 2,3,3,4,4,4 times the fiducial SSO_FWHM with SSO_FWHM= 9, 7, 5, 5, 5, 5 arcmin at 100,143,217,353,545,857 GHz.<br />
<br />
: As an input to planet mask for maps, Mars, Jupiter, Saturn are flagged with a radius computed as a coefficient depending on the planet (Factor_per_source) times NbBeam times SSO_FWHM, with Factor_per_source = 1.1, 2.25, 1.25 for Mars, Jupiter, Saturn respectively and NbBeam = 2.25, 4.25, 4.0, 5.0,.6.0, 8.0 at 100, 143, 217, 353, 545, 857 GHz. This flag is called SSOflag4map.<br />
<br />
: A small trailing tail is added to the mask to take into account the non-deconvolution of the planet signal which has been replaced by background values. The width of that tail is 10 % of the main flag diameter. The number of samples which are additionnaly flagged are the Factor_per_Source times AddSNafter with (trail) times (Factor_per_source)^3 samples where trail = 10, 30, 20, 20, 30, 40 at 100, 143, 217, 353, 545, 857 GHz.<br />
<br />
: Uranus and Neptune together with detected asteroids are masked by HFI. They are masked at the TOI level using an exclusion radius of 1.5 SSO_FWHM. At 857 GHz, 24 asteroids have been detected with HFI : 1Ceres, 2Pallas, 3Juno, 4Vesta, 7Iris, 8Flora, 9Metis, 10Hygiea, 11Parthenope, 12Victoria, 13Egeria, 14Irene, 15Eunomia, 16Psyche, 18Melpomene, 19Fortuna, 20Massalia, 29Amphitrite, 41Daphne, 45Eugenia, 52Europa, 88Thisbe, 704Interamnia, 324Bamberga.<br />
<br />
<gallery widths=200px perrow=4 caption="Local maps showing the SSO flag. The colors correspond to the surveys involved in the nominal mission (green = Survey I, yellow = Survey II, red = Survey III)."><br />
File:SSOflag_10_143_5J.png | 143 SWB row, around Jupiter 1st crossing<br />
File:SSOflag_10_143_5S.png | 143 SWB row, around Saturn<br />
File:SSOflag_10_143_5M.png | 143 SWB row, around Mars<br />
File:SSOflag_10_143_5a.png | 143 SWB row, random field in the ecliptic plane<br />
File:SSOflag_25_857_1J.png | 545/857 row, around Jupiter 1st crossing<br />
File:SSOflag_25_857_1S.png | 545/857 row, around Saturn<br />
File:SSOflag_25_857_1M.png | 545/857 row, around Mars<br />
File:SSOflag_25_857_1a.png | 545/857 row, random field in the ecliptic plane<br />
</gallery><br />
<br />
<br />
=== Output flags ===<br />
<br />
A FlagTOIproc is produced by the TOIprocessing. It marks measurements which are not reliable for any of the following reasons:<br />
* gap (no valid input data), enlarged by one sample on each side. It flags less than 0.00044% (resp. 0.00062%) of the nominal (resp. complete) mission. It is equivalent to less than 3 (resp. 8) minutes of data.<br />
* "glitch" on dark bolometers: as the thermal template used for decorrelation is computed from these bolometer data, chunks of one minute-length data are discarded for all bolometers if at least 50\% of the data for at least one dark bolometer are flagged during this time. It is efficient to flag the data around the maximum of thermal events.<br />
* "glitch" on individual bolometers : samples where the signal from a cosmic ray hit dominates the sky signal at more than 3 $\sigma$ are discarded.<br />
* jump as 100 samples are flagged around the computed position of the jump to take into account the error on this reconstructed position.<br />
<br />
<br />
So the flag produced for the map making, called Total_flag, is defined by:<br />
* Total_flag = UnstablePointing Flag OR FlagTOIproc OR SSOflag4map OR SSOflag seen<br />
where FlagTOIproc = gap OR flag thermal template OR glitch OR jump<br />
and for PSB bolometers, FlagTOIproc_AB = FlagTOIproc_A OR FlagTOIproc_B.<br />
Note that the Total_flag is then identical for the A and B bolometers of a PSB pair.<br />
<br />
At the destriping stage, a more restricted flag, called Total_flag_PS, is used. It is defined by<br />
Total_flag_PS = Total_flag OR PS_flag.<br />
<br />
<br />
<br />
[[Category:Data processing|0042]]</div>
Fdesert
https://wiki.cosmos.esa.int/planckpla2015/index.php?title=Map-making&diff=4060
Map-making
2013-02-11T14:11:31Z
<p>Fdesert: </p>
<hr />
<div>=Map-Making and photometric calibration=<br />
<br />
== Introduction ==<br />
<br />
This page will give an overview of the map-making and photometric calibration procedures used by the HFI DPC to build detector and frequency maps. <br />
This processing and its performances are described in the <span style="color:red">[[XXXXXX|the HFI DPC Paper]]</span> and the <span style="color:red">[[XXXXXX|the HFI DPC Calibration co-Paper]]</span> <br />
[which will be completed prior to this page].<br />
<br />
To build HFI maps, we use the destriping approximation, in which noise is assumed to decompose into two components : white noise plus low frequency drifts. Using the sky redundancy, the low frequency drifts are modelled as one constant, or offset, per pointing period. To speed up the ulterior processing we first build intermediate products, by taking advantage of redundancies : we average signal and detector orientation on healpix pixels visited during <br />
each fixed pointing period, which we call hereafter 'ring'. This intermediate product is called HPR for healpix pixel ring. They have been constructed using the same map resolution as the final HFI products (correspondontg to nside=2048). <br />
This new dataset is used as input in the following steps. <br />
<br />
== Photometric calibration ==<br />
<br />
=== dipole calibration (100 to 353 GHz) ===<br />
<br />
For the 2013 data release, the calibrator for the CMB frequency was the Solar dipole, as measured by the WMAP team (G. Hinshaw et al., Astrophys.J.Suppl.180:225-245,2009). <br />
We use a two components template fitting procedure, performed for each detector independently, to determine ring by ring an estimation of the dipole gain. <br />
The two fitted components are the Solar dipole and a sky template. We used the PSM for thermal dust emission at the detector's frequency as a first approximation of the sky template in pur early release. Using the HFI channel map as a template brings negligible change in the averaged gain, but reduces the systematic ring-to-ring dispersion of our estimation. We average these estimations over a subset of rings in the first survey (2000 to 6000) in which the dipole's amplitude is high enough with respect to that of the sky template, to get a single dipole gain per detector.<br />
<br />
Several pieces of evidence led us to the conclusion that out bolometers presented apparent gain variation with time, after comparing the 3rd scan of the sky with the first one. This was later (mid-2012) explained by inequalities in the steps of the analog-to-digital converters (ADC) used in each bolometer's electronic chain. These devices had to be characterized using warm data after the end of the HFI observations. This process is still on-going (01/2012). <br />
<br />
In the mean time we used an empiric correction, looking for a gain estimation and an offset per ring. This amounts to solve the non-linear equation : <br />
<math> \displaystyle{d\ =\ g_i.S + O_i + n}<br />
\label{nlequat}</math><br />
<br />
where d is s the detector measurement, both S the sky signal, g the detector gain, O the offset (for ring no i) are the unknowns to be determined, and n the noise. <br />
We linearized this equation starting from the constant gain approximation, to get a measurement of the apparent time-varying gains for each bolometer independently. The limitations of this process are intrinsic signal variability from one observation to the other, like polarization or intra-pixel gradient. This procedure was thus only used for the 100 to 217 GHz detectors, for which the dipole signal is brighter and galactic signal (and polarization). A mask was used to removed the inner part of the Galactic plane.<br />
<br />
<br />
=== Higher frequency calibration (545 and 857 GHz) ===<br />
<br />
We therefore finally derived the sub-mm channels' calibration for the 2013 Planck data release from the comparison of measurements of the Neptune and Uranus fluxes (with aperture photometry) with their expectations from the Moreno et al model of their atmospheres' emission. This procedure is justified, since for both planets, at the lower frequencies (100-353 Ghz), the fluxes we recover are in agreement within ~ +/-5\% with what is expected from the planet spectral model, and the HFI detector's band-passes.<br />
<br />
We determined zero-level for the released maps, by imposing that their average in regions where, according to HI template, dust emission is zero, be equal to that of the CIB model of Bethermin et al.<br />
<br />
<br />
== Building of Maps == <br />
<br />
Using the photometric calibration parameters, we build maps in two steps : <br />
<br />
* we determine the destriping offsets using the full mission data<br />
* we build the maps, using these offsets, by inverting the photometric equation :<br />
<br />
<math> \displaystyle{d_i = g(I^p+\eta [Q^p cos(2\psi_i) + U^p sin(2\psi_i)]) + n}<br />
\label{photeq}</math><br />
where d is the destriped and calibrated signal at the HPR level. Detector's data are combined with an inverse noise weights derived from each detector's NEP. Q and U maps are build whenever possible. We propagate the white noise by building the 3x3 (or 1x1 if only I is reconstructed) covariance matrices in each pixel. <br />
At each frequency we build maps combining all detectors and independent detector sets. <br />
We use the offsets build for the full mission for building maps for each scan survey and for the nominal mission duration. <br />
We also build maps from the two independent halves of each rings. Altogether, more than 6000 maps are built at each release. <br />
<br />
<br />
== Noise properties ==<br />
<br />
Map noise properties can be evaluated using several methods, thanks to the high level of observation redundancies. <br />
We can use the maps built from the difference between the first and second half of each rings, or compare individual sky scans, of detector sets with each other. <br />
<br />
Low resolution (nside = 8, 16, 32 ?) pixel-to-pixel noise covariance matrices are build using an analytic approach from the measured noise power spectra.<br />
<br />
== Zodi correction ==<br />
<br />
At the highest Planck frequencies, Zodiacal emission is visible in a survey difference map:<br />
<br />
[[File:857GHz_I_S2mS1.png|600px|857 GHz Survey 2 - Survey 1 Difference]]<br />
<br />
This map is a difference between the 857 GHz Survey 2 map and the 857 GHz Survey 1 map. This difference effectively removes Galactic and other emissions which originate far from Planck. As the Solar elongation is different for measurements of the same point on the sky for the two surveys, we see Zodiacal emission, while all emission from further sources is removed. The zodiacal emission follows the Ecliptic plane, which starts at the lower left of the image, then crosses the center of the plot towards the upper right. Note that the "arcs" at the top and bottom of the image are images of the Galactic center in the Far Sidelobes, which are discussed in the section below. Note also that the image above is from an early processing and will be updated. <br />
<br />
Zodiacal Emission is removed from the 353, 545 and 857 GHz channels. It is described in <span style="color:red">[[XXXXXX|the HFI DPC Paper]]</span>, but a synopsis of the procedure is as follows:<br />
* During each survey, a large fraction of the sky has observations which all fall within a week of each other. That is, during a single survey, most pixels are observed during a short, well-defined period. The contribution from Zodiacal Emission to the total brightness seen, then, is well defined. <br />
* We use the the COBE model of the Zodiacal Light to make predictions for this Zodiacal emission for those pixels observed over a span of one week or less, and use GRASP models of the beams to predict the emission from the Galaxy given our sidelobes. <br />
* We fit the survey difference maps with these model templates to estimate the emissivity of each Zodi component and sidelobe at the Planck wavelengths. The results of these fits at each frequency are shown [[zodiFreqFits|here]]. <br />
* We reconstruct each ring of the the full mission using the combination of the COBE geometric model with the emissivities determined above and the sidelobe models. <br />
* We remove the reconstruction above from each ring of data. <br />
* We then make maps as described previously in this section. Maps with and without, as well as the differences between the two, are shown [[withAndWithoutRemoval|here]]. The survey differences before and after this removal are shown with and without Zodical emission removal [[beforeAndAfterSurveyDifferences|here]]. The power spectra of what is removed from each map is shown [[zodiCorrectionSpectra|here]].<br />
<br />
== Far Sidelobes ==<br />
<br />
The far sidelobe correction for the highest frequency HFI channels is described in [[Map-making#Zodi correction|the section above]]. Note that this correction is done only for the 857 and 545 GHz channels, as it is not seen at longer wavelengths. As for the Zodical emission correction, it is not important for the CMB. <br />
<br />
We have made estimates of the contamination of the far sidelobes at 143 GHz by taking the 143 GHz map, adding the dipole, and passing it through our simulator, using a GRASP calculation of the far sidelobes for the 143-1a detector as the beam. The resulting maps is shown here:<br />
[[File:FSLMap.png|600px|Estimated Spurious Far Sidelobe Signal]]<br />
<br />
While there is one small region that might reach 20 micro-K (this happens when the secondary spillover overlaps with the Galactic center), most of the map is quite quiet. This is evidenced by the power spectrum of the above map, which is quite small. <br />
[[File:FSLCl.png|300px|Estimated Spurious Far Sidelobe Power Spectrum]]<br />
<br />
== CO Correction ==<br />
<br />
The extraction of CO maps from HFI maps is described in detail in the <span style="color:red">[[XXXXXX|the CO Paper]]</span>. The CO maps are produced by a combination of bolometer maps or frequency maps. The method is summarized [[http://www.sciops.esa.int/wikiSI/planckpla/index.php?title=Science&instance=Planck_PLA_ES#CO_maps |here]]<br />
<br />
== Map validation ==<br />
<br />
<br />
[[Category:Data processing]]</div>
Fdesert
https://wiki.cosmos.esa.int/planckpla2015/index.php?title=Map-making&diff=4059
Map-making
2013-02-11T14:09:15Z
<p>Fdesert: </p>
<hr />
<div>=Map-Making and photometric calibration=<br />
<br />
== Introduction ==<br />
<br />
This page will give an overview of the map-making and photometric calibration procedures used by the HFI DPC to build detector and frequency maps. <br />
This processing and its performances are described in the <span style="color:red">[[XXXXXX|the HFI DPC Paper]]</span> and the <span style="color:red">[[XXXXXX|the HFI DPC Calibration co-Paper]]</span> <br />
[which will be completed prior to this page].<br />
<br />
To build HFI maps, we use the destriping approximation, in which noise is assumed to decompose into two components : white noise plus low frequency drifts. Using the sky redundancy, the low frequency drifts are modelled as one constant, or offset, per pointing period. To speed up the ulterior processing we first build intermediate products, by taking advantage of redundancies : we average signal and detector orientation on healpix pixels visited during <br />
each fixed pointing period, which we call hereafter 'ring'. This intermediate product is called HPR for healpix pixel ring. They have been constructed using the same map resolution as the final HFI products (correspondontg to nside=2048). <br />
This new dataset is used as input in the following steps. <br />
<br />
== Photometric calibration ==<br />
<br />
=== dipole calibration (100 to 353 GHz) ===<br />
<br />
For the 2013 data release, the calibrator for the CMB frequency was the Solar dipole, as measured by the WMAP team (G. Hinshaw et al., Astrophys.J.Suppl.180:225-245,2009). <br />
We use a two components template fitting procedure, performed for each detector independently, to determine ring by ring an estimation of the dipole gain. <br />
The two fitted components are the Solar dipole and a sky template. We used the PSM for thermal dust emission at the detector's frequency as a first approximation of the sky template in pur early release. Using the HFI channel map as a template brings negligible change in the averaged gain, but reduces the systematic ring-to-ring dispersion of our estimation. We average these estimations over a subset of rings in the first survey (2000 to 6000) in which the dipole's amplitude is high enough with respect to that of the sky template, to get a single dipole gain per detector.<br />
<br />
Several pieces of evidence led us to the conclusion that out bolometers presented apparent gain variation with time, after comparing the 3rd scan of the sky with the first one. This was later (mid-2012) explained by inequalities in the steps of the analog-to-digital converters (ADC) used in each bolometer's electronic chain. These devices had to be characterized using warm data after the end of the HFI observations. This process is still on-going (01/2012). <br />
<br />
In the mean time we used an empiric correction, looking for a gain estimation and an offset per ring. This amounts to solve the non-linear equation : <br />
<math> \displaystyle{d\ =\ g_i.S + O_i + n}<br />
\label{nlequat}</math><br />
<br />
where d is s the detector measurement, both S the sky signal, g the detector gain, O the offset (for ring no i) are the unknowns to be determined, and n the noise. <br />
We linearized this equation starting from the constant gain approximation, to get a measurement of the apparent time-varying gains for each bolometer independently. The limitations of this process are intrinsic signal variability from one observation to the other, like polarization or intra-pixel gradient. This procedure was thus only used for the 100 to 217 GHz detectors, for which the dipole signal is brighter and galactic signal (and polarization). A mask was used to removed the inner part of the Galactic plane.<br />
<br />
<br />
=== Higher frequency calibration (545 and 857 GHz) ===<br />
<br />
We therefore finally derived the sub-mm channels' calibration for the 2013 Planck data release from the comparison of measurements of the Neptune and Uranus fluxes (with aperture photometry) with their expectations from the Moreno et al model of their atmospheres' emission. This procedure is justified, since for both planets, at the lower frequencies (100-353 Ghz), the fluxes we recover are in agreement within ~ +/-5\% with what is expected from the planet spectral model, and the HFI detector's band-passes.<br />
<br />
We determined zero-level for the released maps, by imposing that their average in regions where, according to HI template, dust emission is zero, be equal to that of the CIB model of Bethermin et al.<br />
<br />
<br />
== Building of Maps == <br />
<br />
Using the photometric calibration parameters, we build maps in two steps : <br />
<br />
* we determine the destriping offsets using the full mission data<br />
* we build the maps, using these offsets, by inverting the photometric equation :<br />
<br />
<math> \displaystyle{d_i = g(I^p+\eta [Q^p cos(2\psi_i) + U^p sin(2\psi_i)]) + n}<br />
\label{photeq}</math><br />
where d is the destriped and calibrated signal at the HPR level. Detector's data are combined with an inverse noise weights derived from each detector's NEP. Q and U maps are build whenever possible. We propagate the white noise by building the 3x3 (or 1x1 if only I is reconstructed) covariance matrices in each pixel. <br />
At each frequency we build maps combining all detectors and independent detector sets. <br />
We use the offsets build for the full mission for building maps for each scan survey and for the nominal mission duration. <br />
We also build maps from the two independent halves of each rings. Altogether, more than 6000 maps are built at each release. <br />
<br />
<br />
== Noise properties ==<br />
<br />
Map noise properties can be evaluated using several methods, thanks to the high level of observation redundancies. <br />
We can use the maps built from the difference between the first and second half of each rings, or compare individual sky scans, of detector sets with each other. <br />
<br />
Low resolution (nside = 8, 16, 32 ?) pixel-to-pixel noise covariance matrices are build using an analytic approach from the measured noise power spectra.<br />
<br />
== Zodi correction ==<br />
<br />
At the highest Planck frequencies, Zodiacal emission is visible in a survey difference map:<br />
<br />
[[File:857GHz_I_S2mS1.png|600px|857 GHz Survey 2 - Survey 1 Difference]]<br />
<br />
This map is a difference between the 857 GHz Survey 2 map and the 857 GHz Survey 1 map. This difference effectively removes Galactic and other emissions which originate far from Planck. As the Solar elongation is different for measurements of the same point on the sky for the two surveys, we see Zodiacal emission, while all emission from further sources is removed. The zodiacal emission follows the Ecliptic plane, which starts at the lower left of the image, then crosses the center of the plot towards the upper right. Note that the "arcs" at the top and bottom of the image are images of the Galactic center in the Far Sidelobes, which are discussed in the section below. Note also that the image above is from an early processing and will be updated. <br />
<br />
Zodiacal Emission is removed from the 353, 545 and 857 GHz channels. It is described in <span style="color:red">[[XXXXXX|the HFI DPC Paper]]</span>, but a synopsis of the procedure is as follows:<br />
* During each survey, a large fraction of the sky has observations which all fall within a week of each other. That is, during a single survey, most pixels are observed during a short, well-defined period. The contribution from Zodiacal Emission to the total brightness seen, then, is well defined. <br />
* We use the the COBE model of the Zodiacal Light to make predictions for this Zodiacal emission for those pixels observed over a span of one week or less, and use GRASP models of the beams to predict the emission from the Galaxy given our sidelobes. <br />
* We fit the survey difference maps with these model templates to estimate the emissivity of each Zodi component and sidelobe at the Planck wavelengths. The results of these fits at each frequency are shown [[zodiFreqFits|here]]. <br />
* We reconstruct each ring of the the full mission using the combination of the COBE geometric model with the emissivities determined above and the sidelobe models. <br />
* We remove the reconstruction above from each ring of data. <br />
* We then make maps as described previously in this section. Maps with and without, as well as the differences between the two, are shown [[withAndWithoutRemoval|here]]. The survey differences before and after this removal are shown with and without Zodical emission removal [[beforeAndAfterSurveyDifferences|here]]. The power spectra of what is removed from each map is shown [[zodiCorrectionSpectra|here]].<br />
<br />
== Far Sidelobes ==<br />
<br />
The far sidelobe correction for the highest frequency HFI channels is described in [[Map-making#Zodi correction|the section above]]. Note that this correction is done only for the 857 and 545 GHz channels, as it is not seen at longer wavelengths. As for the Zodical emission correction, it is not important for the CMB. <br />
<br />
We have made estimates of the contamination of the far sidelobes at 143 GHz by taking the 143 GHz map, adding the dipole, and passing it through our simulator, using a GRASP calculation of the far sidelobes for the 143-1a detector as the beam. The resulting maps is shown here:<br />
[[File:FSLMap.png|600px|Estimated Spurious Far Sidelobe Signal]]<br />
<br />
While there is one small region that might reach 20 micro-K (this happens when the secondary spillover overlaps with the Galactic center), most of the map is quite quiet. This is evidenced by the power spectrum of the above map, which is quite small. <br />
[[File:FSLCl.png|300px|Estimated Spurious Far Sidelobe Power Spectrum]]<br />
<br />
== CO Correction ==<br />
<br />
This is described in the <span style="color:red">[[XXXXXX|the CO Paper]]</span>. The CO maps are produced by a combination of bolometer maps or frequency maps. This is summarized [[http://www.sciops.esa.int/wikiSI/planckpla/index.php?title=Science&instance=Planck_PLA_ES#CO_maps |here]]<br />
<br />
== Map validation ==<br />
<br />
<br />
[[Category:Data processing]]</div>
Fdesert
https://wiki.cosmos.esa.int/planckpla2015/index.php?title=Map-making&diff=4058
Map-making
2013-02-11T14:08:41Z
<p>Fdesert: </p>
<hr />
<div>=Map-Making and photometric calibration=<br />
<br />
== Introduction ==<br />
<br />
This page will give an overview of the map-making and photometric calibration procedures used by the HFI DPC to build detector and frequency maps. <br />
This processing and its performances are described in the <span style="color:red">[[XXXXXX|the HFI DPC Paper]]</span> and the <span style="color:red">[[XXXXXX|the HFI DPC Calibration co-Paper]]</span> <br />
[which will be completed prior to this page].<br />
<br />
To build HFI maps, we use the destriping approximation, in which noise is assumed to decompose into two components : white noise plus low frequency drifts. Using the sky redundancy, the low frequency drifts are modelled as one constant, or offset, per pointing period. To speed up the ulterior processing we first build intermediate products, by taking advantage of redundancies : we average signal and detector orientation on healpix pixels visited during <br />
each fixed pointing period, which we call hereafter 'ring'. This intermediate product is called HPR for healpix pixel ring. They have been constructed using the same map resolution as the final HFI products (correspondontg to nside=2048). <br />
This new dataset is used as input in the following steps. <br />
<br />
== Photometric calibration ==<br />
<br />
=== dipole calibration (100 to 353 GHz) ===<br />
<br />
For the 2013 data release, the calibrator for the CMB frequency was the Solar dipole, as measured by the WMAP team (G. Hinshaw et al., Astrophys.J.Suppl.180:225-245,2009). <br />
We use a two components template fitting procedure, performed for each detector independently, to determine ring by ring an estimation of the dipole gain. <br />
The two fitted components are the Solar dipole and a sky template. We used the PSM for thermal dust emission at the detector's frequency as a first approximation of the sky template in pur early release. Using the HFI channel map as a template brings negligible change in the averaged gain, but reduces the systematic ring-to-ring dispersion of our estimation. We average these estimations over a subset of rings in the first survey (2000 to 6000) in which the dipole's amplitude is high enough with respect to that of the sky template, to get a single dipole gain per detector.<br />
<br />
Several pieces of evidence led us to the conclusion that out bolometers presented apparent gain variation with time, after comparing the 3rd scan of the sky with the first one. This was later (mid-2012) explained by inequalities in the steps of the analog-to-digital converters (ADC) used in each bolometer's electronic chain. These devices had to be characterized using warm data after the end of the HFI observations. This process is still on-going (01/2012). <br />
<br />
In the mean time we used an empiric correction, looking for a gain estimation and an offset per ring. This amounts to solve the non-linear equation : <br />
<math> \displaystyle{d\ =\ g_i.S + O_i + n}<br />
\label{nlequat}</math><br />
<br />
where d is s the detector measurement, both S the sky signal, g the detector gain, O the offset (for ring no i) are the unknowns to be determined, and n the noise. <br />
We linearized this equation starting from the constant gain approximation, to get a measurement of the apparent time-varying gains for each bolometer independently. The limitations of this process are intrinsic signal variability from one observation to the other, like polarization or intra-pixel gradient. This procedure was thus only used for the 100 to 217 GHz detectors, for which the dipole signal is brighter and galactic signal (and polarization). A mask was used to removed the inner part of the Galactic plane.<br />
<br />
<br />
=== Higher frequency calibration (545 and 857 GHz) ===<br />
<br />
We therefore finally derived the sub-mm channels' calibration for the 2013 Planck data release from the comparison of measurements of the Neptune and Uranus fluxes (with aperture photometry) with their expectations from the Moreno et al model of their atmospheres' emission. This procedure is justified, since for both planets, at the lower frequencies (100-353 Ghz), the fluxes we recover are in agreement within ~ +/-5\% with what is expected from the planet spectral model, and the HFI detector's band-passes.<br />
<br />
We determined zero-level for the released maps, by imposing that their average in regions where, according to HI template, dust emission is zero, be equal to that of the CIB model of Bethermin et al.<br />
<br />
<br />
== Building of Maps == <br />
<br />
Using the photometric calibration parameters, we build maps in two steps : <br />
<br />
* we determine the destriping offsets using the full mission data<br />
* we build the maps, using these offsets, by inverting the photometric equation :<br />
<br />
<math> \displaystyle{d_i = g(I^p+\eta [Q^p cos(2\psi_i) + U^p sin(2\psi_i)]) + n}<br />
\label{photeq}</math><br />
where d is the destriped and calibrated signal at the HPR level. Detector's data are combined with an inverse noise weights derived from each detector's NEP. Q and U maps are build whenever possible. We propagate the white noise by building the 3x3 (or 1x1 if only I is reconstructed) covariance matrices in each pixel. <br />
At each frequency we build maps combining all detectors and independent detector sets. <br />
We use the offsets build for the full mission for building maps for each scan survey and for the nominal mission duration. <br />
We also build maps from the two independent halves of each rings. Altogether, more than 6000 maps are built at each release. <br />
<br />
<br />
== Noise properties ==<br />
<br />
Map noise properties can be evaluated using several methods, thanks to the high level of observation redundancies. <br />
We can use the maps built from the difference between the first and second half of each rings, or compare individual sky scans, of detector sets with each other. <br />
<br />
Low resolution (nside = 8, 16, 32 ?) pixel-to-pixel noise covariance matrices are build using an analytic approach from the measured noise power spectra.<br />
<br />
== Zodi correction ==<br />
<br />
At the highest Planck frequencies, Zodiacal emission is visible in a survey difference map:<br />
<br />
[[File:857GHz_I_S2mS1.png|600px|857 GHz Survey 2 - Survey 1 Difference]]<br />
<br />
This map is a difference between the 857 GHz Survey 2 map and the 857 GHz Survey 1 map. This difference effectively removes Galactic and other emissions which originate far from Planck. As the Solar elongation is different for measurements of the same point on the sky for the two surveys, we see Zodiacal emission, while all emission from further sources is removed. The zodiacal emission follows the Ecliptic plane, which starts at the lower left of the image, then crosses the center of the plot towards the upper right. Note that the "arcs" at the top and bottom of the image are images of the Galactic center in the Far Sidelobes, which are discussed in the section below. Note also that the image above is from an early processing and will be updated. <br />
<br />
Zodiacal Emission is removed from the 353, 545 and 857 GHz channels. It is described in <span style="color:red">[[XXXXXX|the HFI DPC Paper]]</span>, but a synopsis of the procedure is as follows:<br />
* During each survey, a large fraction of the sky has observations which all fall within a week of each other. That is, during a single survey, most pixels are observed during a short, well-defined period. The contribution from Zodiacal Emission to the total brightness seen, then, is well defined. <br />
* We use the the COBE model of the Zodiacal Light to make predictions for this Zodiacal emission for those pixels observed over a span of one week or less, and use GRASP models of the beams to predict the emission from the Galaxy given our sidelobes. <br />
* We fit the survey difference maps with these model templates to estimate the emissivity of each Zodi component and sidelobe at the Planck wavelengths. The results of these fits at each frequency are shown [[zodiFreqFits|here]]. <br />
* We reconstruct each ring of the the full mission using the combination of the COBE geometric model with the emissivities determined above and the sidelobe models. <br />
* We remove the reconstruction above from each ring of data. <br />
* We then make maps as described previously in this section. Maps with and without, as well as the differences between the two, are shown [[withAndWithoutRemoval|here]]. The survey differences before and after this removal are shown with and without Zodical emission removal [[beforeAndAfterSurveyDifferences|here]]. The power spectra of what is removed from each map is shown [[zodiCorrectionSpectra|here]].<br />
<br />
== Far Sidelobes ==<br />
<br />
The far sidelobe correction for the highest frequency HFI channels is described in [[Map-making#Zodi correction|the section above]]. Note that this correction is done only for the 857 and 545 GHz channels, as it is not seen at longer wavelengths. As for the Zodical emission correction, it is not important for the CMB. <br />
<br />
We have made estimates of the contamination of the far sidelobes at 143 GHz by taking the 143 GHz map, adding the dipole, and passing it through our simulator, using a GRASP calculation of the far sidelobes for the 143-1a detector as the beam. The resulting maps is shown here:<br />
[[File:FSLMap.png|600px|Estimated Spurious Far Sidelobe Signal]]<br />
<br />
While there is one small region that might reach 20 micro-K (this happens when the secondary spillover overlaps with the Galactic center), most of the map is quite quiet. This is evidenced by the power spectrum of the above map, which is quite small. <br />
[[File:FSLCl.png|300px|Estimated Spurious Far Sidelobe Power Spectrum]]<br />
<br />
== CO Correction ==<br />
<br />
This is described in the <span style="color:red">[[XXXXXX|the CO Paper]]</span>. The CO maps are produced by a combination of bolometer maps or frequency maps. This is summarized [[http://www.sciops.esa.int/wikiSI/planckpla/index.php?title=Science&instance=Planck_PLA_ES#CO_maps|here]]<br />
<br />
== Map validation ==<br />
<br />
<br />
[[Category:Data processing]]</div>
Fdesert
https://wiki.cosmos.esa.int/planckpla2015/index.php?title=TOI_processing&diff=3863
TOI processing
2013-02-05T13:51:07Z
<p>Fdesert: </p>
<hr />
<div>This Section is kept short as this Planck release does not contain TOIs. The main information in the HFI processing paper is not duplicated here.<br />
<br />
==Overview==<br />
<br />
We describe here the how the TOIs are processed in order to be used for map production. We do not repeat the general features of the pipeline which are given in the HFI Data Processing article (REF). Here we give complementary explanations on some details. The TOI of each bolometer is processed independently of the other bolometers, so as to keep the noise properties as uncorrelated as possible. The processing involves modifying the TOI itself for what concerns the conversion to absorbed power and the correction of glitch tails. It also adds a flag TOI that masks the TOI samples that are not to be projected on maps for various reasons.<br />
<br />
<br />
=== Acronyms and definitions ===<br />
<br />
* 4K lines: EMI/EMC influence of the 4K cooler mechanical motion on the bolometer readout electronics.<br />
* ADC : analog to digital converter<br />
* IMO : instrument model<br />
* Jump : sudden change of the baseline level inside a ring<br />
* LFER : low frequency excess response<br />
* PBR : phase binned ring<br />
* RIMO : reduced IMO<br />
* Ring : pointing period<br />
* ROI : ring ordered information<br />
* RTS : random telegraphic signal<br />
* SSO : solar system object<br />
* TOI : time ordered information<br />
<br />
== Input TOI ==<br />
<br />
The input TOI consists in the AC modulated voltage output of the readout of each bolometer. The input has previously been decompressed, and converted from internal digital units to voltage via a constant factor. The TOI has a regular sampling at the acquisition frequency of facq=180.373700+-0.000050 Hz. There are almost no missing data in the TOIs, except for few hundred samples of 545 and 857GHz TOIs which are lost in the on-board compression due to saturation on the Galactic Center crossings. <br />
<br />
== General Pipeline Structure ==<br />
<br />
The figure on the right shows how the initial TOI is transformed and how flags are produced:<br />
<br />
[[File:HFI_4_4_2_ToiProc_ExplS.png|thumb|600px|A schematic of the TOI processing pipeline]]<br />
<br />
<br />
== Output TOIs and products ==<br />
<br />
A TOI of clean calibrated samples (ccTOI) and a combined flag TOI (fTOI) are the outputs of the processing. The ccTOI is calibrated so as to represent the instantaneous power absorbed by the detector up to a constant (which will be determined by the map-making destriper). It is worth mentioning how the ccTOI is changed with respect to the input TOI, beyond the harmless constant conversion factor from voltage to absorbed power. The demodulation stage allows to get the demodulated bolometer voltage. The non-linearity correction is a second-order polynomial correction based on the physical but static bolometer model. In order to avoid too much masking after glitches, a glitch tail is subtracted after an occurrence of a glitch in the TOI. The 4K cooler lines noise is substituted at a series of 9 single temporal frequencies. Finally, the temporal response of the bolometer is deconvolved. This affects mostly the high-temporal frequency part of the TOI, although a small but significant low frequency (the long time response) tail is corrected too. Although flagged samples are not projected, their value influences the valid samples somehow. Hence interpolation procedures introduce some indirect modifications of the TOI.<br />
The flag TOI is a combination a dozen flags with an OR logic. Only unflagged data are projected. The exhaustive list of flags is given here: CompressionError, NoData, SSO, UnstablePointing, Glitch, BoloPlateFluctuation, RTS, Jump, PSBab. A complete qualification of the data is obtained at the ring level. If the TOI shows an anomalous behaviour on a given ring, this ring is discarded from projection. A special production of TOIs is also made as an input to the beam analysis with Mars, Jupiter and Saturn.<br />
<br />
<br />
== Examples of clean TOIs ==<br />
<gallery widths=300px perrow=3><br />
File:10_143_5_PBR_10800RING_LFER4_JC.jpg|Phase-binned signal (PBR) for 6 consecutive rings from 4000 to 4005<br />
File:10_143_5_rcircles_LFER4_JC.jpg| Valid signal (up) and noise (bottom) for 2 consecutive circles of the ring 4000<br />
File:10_143_5_fft_LFER4_JC.jpg| Fourier power spectral density of the first 30 minutes of the ring 4005 with signal before (pink) and after (black) deconvolution (top) and noise (bottom)<br />
File:14_545_1_PBR_10800RING_LFER4_JC.jpg|Phase-binned signal (PBR) for 6 consecutive rings from 4000 to 4005<br />
File:14_545_1_rcircles_LFER4_JC.jpg| Valid signal (up) and noise (bottom) for 2 consecutive circles of the ring 4000<br />
File:14_545_1_fft_LFER4_JC.jpg| Fourier power spectral density of the first 30 minutes of the ring 4005 with signal before (pink) and after (black) deconvolution (top) and noise (bottom)<br />
</gallery><br />
<br />
[[Media:check_ring.pdf|Samples of PBR, TOIs, and PSDs of all detectors are shown in this file]]<br />
<br />
== Trends in the output processing variables ==<br />
<br />
Here we intend to show the trend of the systematic effects that are dealt with in the TOI processing. The full impact of each of them is analyzed in [[HFI-Validation]].<br />
<br />
=== ADC baseline ===<br />
The following figure shows the ADC baseline which is used prior to demodulation (a constant offset is removed for clarity). This baseline is obtained by smoothing on an hour block average the undemodulated TOI on unflagged samples.<br />
<br />
[[File:baseline.jpg|400px|ADC baseline for all bolometers]]<br />
<br />
=== Glitch statistics ===<br />
The glitch rate per channel is shown in this figure. For details, see copap. <br />
<br />
[[File:figIntermPaperGR.jpg|400px|Glitch rate evolution]]<br />
<br />
The percentage of flagged data (mostly due to Cosmic Rays) at the ring level is shown in these examples. No smoothing was applied. Only valid rings are shown.<br />
<gallery widths=450px perrow=2 caption="Percentage of flagged data"><br />
File:group10_143_5.jpg | 143 GHz bolometers<br />
File:group14_545_1.jpg | 545 GHz bolometers<br />
</gallery><br />
[[Media:PercentUnvalid2.pdf|The complete set of plots is here]]<br />
<br />
=== Thermal template for decorrelation ===<br />
[[File:T90.jpg|400px|Thermal template used for the decorrelation from bolometer plate temperature fluctuations]]<br />
<br />
A simple linear decorrelation is performed using the 2 dark bolometers as a proxy of the bolometer plate temperature. Coupling coefficients were measured during the CPV phase.<br />
<br />
=== 4K cooler lines variability ===<br />
The amplitude of the nine 4K cooler lines in aW at 10, 20, 30, 40, 50, 60, 70, 80 and 17 Hz is shown for 2 bolometers in the following figures. The trend is smoothed over 31 ring values after having discarded measurements done at a ring which is discarded for all bolometers.<br />
<gallery widths=450px perrow=2 caption="Amplitude of the nine 4K cooler lines"><br />
File:10_143_5_4Klines.jpg | 143_5 bolometer<br />
File:14_545_1_4Klines.jpg | 545_1 bolometer<br />
</gallery><br />
[[Media:Lines4K.pdf|The 4K cooler line coefficients of all bolometers are shown in this file]]<br />
<br />
The 4K cooler lines project onto the maps only for a limited fraction of rings, the so-called resonant rings. This is graphically shown in the following figure. For each ring (stable pointing period), the spin rate is very stable at about 1 rpm. From one ring to another, the spin frequency (shown as diamonds) changes around that value. The sky signal is imprinted at the corresponding spin frequency and its 5400 (60x90) harmonics. If one of the nine 4K cooler lines happens to coincide with one of the spin frequency harmonics (a resonant ring), it will project a sine-wave systematic on the maps. The horizontal coloured bars show the zone of influence of a particular 4K line (labeled on the left side of the plot), when folded around 16.666 mHz. When the spin frequency hits one of these zones, we have a resonant ring. The 4K line coefficient is interpolated for this ring and an estimate of the systematic effect is subtracted from the TOI. Resonant rings are different for different 4K lines. Note the two-level oscillation pattern of the spin frequency is due to the satellite attitude control system.<br />
<br />
[[File:spinfreq1.jpg|600px|Spin frequency and 4K line zones of influence]]<br />
<br />
=== jump correction ===<br />
A piecewise constant value is removed to the TOI if a jump is detected. See a jump example in this figure:<br />
<br />
[[File:jump_exe.png|200px|An example of a jump seen on the rmsigTOI]]<br />
<br />
The number of jumps per day (all bolometers included) is shown in this figure:<br />
<br />
[[File:jumps_per_day.png|200px|Evolution of jump number during the mission]]<br />
<br />
The jumps are uncorrelated from bolometer to bolometer. The total number of jumps detected in the nominal and full mission is shown here:<br />
<br />
[[File:jumps_per_bolometer.png|200px|Number of jumps per bolometer]]<br />
<br />
<br />
=== Trends in noise and signal ===<br />
<gallery widths=450px perrow=2 caption="Signal (top) and noise (bottom) smoothed at 1 minute. All values falling in a discarded ring are not plotted."><br />
File:10_143_5_smooth_Watt.jpg | 143_5 bolometer<br />
File:14_545_1_smooth_Watt.jpg | 545_1 bolometer<br />
</gallery><br />
<br />
[[Media:check_smooth.pdf|The smooth TOIs of all detectors are shown in this file]]<br />
<br />
=== Noise stationarity ===<br />
This is not the final version but gives a good idea of power spectra at the detector level of rmsigTOIs.<br />
[[Media:v53_meanSpectra_bySurvey.pdf|All PSDs can be seen in this file]]<br />
<br />
The standard deviation per ring corrected by ring duration bias is given here, one per bolometer using only valid rings. No smoothing is applied (except a 31-point smoothing for the 545 and 857 GHz channels) but values for rings discarded for all bolometers (see below) are not used. The standard deviation is computed on samples valid for map-making which are also not affected by the Galaxy or the point-sources using the usual flags. Two examples are given here.<br />
<gallery widths=450px perrow=2 caption="Standard deviation per ring"><br />
File:stddev_group10_143_5.jpg | 143_5 bolometer<br />
File:stddev_group_14_545_1.jpg | 545_1 bolometer<br />
</gallery><br />
<br />
<br />
The full series of plots is here:<br />
[[Media:StationarityInROIoutput2.pdf|Standard deviation of rmsig TOIs at the ring level]]<br />
<br />
Note the presence for 3 bolometers of a two-level noise system. No correction can be done for that effect. See one example here:<br />
<br />
[[File:23_353_TwoLevel.jpg|300px|An example of two-level noise system is seen in bolometer 23_353_3a]]<br />
<br />
<br />
An example of the higher order statistics which are used to unveil rings affected by RTS problems.<br />
<br />
[[File:HFI_4_4_2_RTSexample13.jpg|400px|Example of RTS detection]]<br />
<br />
=== Discarded rings ===<br />
<br />
Some rings are discarded (flagged) from further use (beam making, map making) by using ring statistics (see above and the HFI data processing paper REF). For each statistic, we compare each ring value to the ring values averaged (RVA) over a large selection of rings (between 3000 and 21700). We also define the modified standard deviation (MSD) of a ring quantity as the standard deviation of that quantity over the rings that deviate by less than five nominal standard deviations. This truncation is necessary to be robust against extreme deviant rings.<br />
<br />
More specifically, a ring is discarded if it matches one of the following criteria:<br />
<br />
* the | mean-median | deviates from the RVA by more than fifteen times the MSD.<br />
* the standard deviation deviates from the RVA by more than -5 times the MSD (all cases corresponding to almost empty rings) or +15 times the MSD.<br />
* the Kolmogorov-Smirnov test deviates from the RVA by more than 15 times the MSD.<br />
* the ring duration is more than 90 min.<br />
* the ring is contaminated by RTS with an amplitude of more than one standard deviation of the noise. It concerns a few hundreds of rings for 3 bolometers (44_217_6b, 71_217_8a, 74_857_4). Notice that two bolometers are completely discarded for maps: 55_545_3 and 70_143_8, which present RTS at all time.<br />
<br />
For the three first criteria, a visual inspection of the rmsigTOI at each of the incriminated rings has shown that all these anomalous rings are due to either a drift, a small jump in the TOI trend or a sudden change of noise level, the origin of which is unknown at present.<br />
<br />
Once the list of discarded rings per bolometer is produced, a common list of discarded rings can be extracted for all bolometers (by using discarded rings for at least half the bolometers). Such rings correspond to identified phenomena, as can be seen on the following table.<br />
<br />
Furthermore, an isolated valid ring stuck between two common discarded rings becomes discarded as well.<br />
<br />
Table of '''common discarded rings''' of the nominal mission (rings 240-14723).<br />
<br />
{|table border=1 align=center cellpadding="3" cellspacing="0" style=text-align:center<br />
! '''Cause''' !! '''ring number'''<br />
|-<br />
!align="left"| manoeuvre<br />
|align="center"|304 1312 3590 3611 3642 3922 4949 6379 8456 11328<br />
|-<br />
! align="left"| Sorption Cooler Switchover <br />
|align="center"| 11149 11150 11151 11152 <br />
|-<br />
! align="left"| Rings too long<br />
| align="center"| 440 474 509 544 897 898 3589 13333 14627 14653 <br />
|-<br />
! align="left"| Star tracker switchover <br />
| align="center"| 14628 14654 <br />
|-<br />
! align="left"| Massive Glitch Event<br />
| align="center"| 7665 <br />
|-<br />
! align="left"| Solar Flare<br />
| align="center"| 11235 <br />
|}<br />
<br />
<br />
<br />
The following figure is a summary of the impact of the discarding process '''for each bolometer''' (the solid black line is the common discarded ring percentage). The outlier bolometers have some RTS problems as mentionned above.<br />
<br />
[[File:Fraction_discarded_rings_v53.png|300px|Fraction of discarded rings]]<br />
[[File:Fraction_discarded_time_v53.png|300px|Fraction of time due to discarded rings]]<br />
<br />
=== Effective integration time ===<br />
The following figure summarizes the effective integration time per bolometer. For that purpose the number of unflagged samples in non-discarded pointing periods have been used within the nominal mission. The average value is of about 335 days of effective integration time.<br />
<br />
[[File:HFInominal_IntegrationTime.jpg|600px|Effective integration time per bolometer]]<br />
<br />
== Flag description ==<br />
<br />
==== Input flags ====<br />
<br />
These flags are used as inputs to the TOI processing<br />
<br />
; The point-source flag (PSflag) <br />
: An earlier version of HFI point-source catalog is read back into a flag TOIs, at a given frequency. In practice, 5 sigma sources are masked within a radius of 1.3 FWHM (9, 7, 5, 5, 5, 5 arcmin at 100,143,217,353,545,857 GHz) TBC.<br />
<br />
; the galactic flag (Galflag<br />
: An earlier version of HFI maps is thresholded and apodized. The produced masks are read into flag TOIs. The retained threshold corresponds to a sky coverage of respectively 70, 70, 80, 90, 90, 90% at 100,143,217,353,545,857 GHz.<br />
<br />
; Solar System Object flag <br />
: For the TOI flag, Mars, Jupiter, Saturn are flagged up to a radius of NbBeam= 2,3,3,4,4,4 times the fiducial SSO_FWHM with SSO_FWHM= 9, 7, 5, 5, 5, 5 arcmin at 100,143,217,353,545,857 GHz.<br />
<br />
: As an input to planet mask for maps, Mars, Jupiter, Saturn are flagged with a radius computed as a coefficient depending on the planet (Factor_per_source) times NbBeam times SSO_FWHM, with Factor_per_source = 1.1, 2.25, 1.25 for Mars, Jupiter, Saturn respectively and NbBeam = 2.25, 4.25, 4.0, 5.0,.6.0, 8.0 at 100, 143, 217, 353, 545, 857 GHz. This flag is called SSOflag4map.<br />
<br />
: A small trailing tail is added to the mask to take into account the non-deconvolution of the planet signal which has been replaced by background values. The width of that tail is 10 % of the main flag diameter. The number of samples which are additionnaly flagged are the Factor_per_Source times AddSNafter with (trail) times (Factor_per_source)^3 samples where trail = 10, 30, 20, 20, 30, 40 at 100, 143, 217, 353, 545, 857 GHz.<br />
<br />
: Uranus and Neptune together with detected asteroids are masked by HFI. They are masked at the TOI level using an exclusion radius of 1.5 SSO_FWHM. At 857 GHz, 24 asteroids have been detected with HFI : 1Ceres, 2Pallas, 3Juno, 4Vesta, 7Iris, 8Flora, 9Metis, 10Hygiea, 11Parthenope, 12Victoria, 13Egeria, 14Irene, 15Eunomia, 16Psyche, 18Melpomene, 19Fortuna, 20Massalia, 29Amphitrite, 41Daphne, 45Eugenia, 52Europa, 88Thisbe, 704Interamnia, 324Bamberga.<br />
<br />
<gallery widths=200px perrow=4 caption="Local maps showing the SSO flag. The colors correspond to the surveys involved in the nominal mission (green = Survey I, yellow = Survey II, red = Survey III)."><br />
File:SSOflag_10_143_5J.png | 143 SWB row, around Jupiter 1st crossing<br />
File:SSOflag_10_143_5S.png | 143 SWB row, around Saturn<br />
File:SSOflag_10_143_5M.png | 143 SWB row, around Mars<br />
File:SSOflag_10_143_5a.png | 143 SWB row, random field in the ecliptic plane<br />
File:SSOflag_25_857_1J.png | 545/857 row, around Jupiter 1st crossing<br />
File:SSOflag_25_857_1S.png | 545/857 row, around Saturn<br />
File:SSOflag_25_857_1M.png | 545/857 row, around Mars<br />
File:SSOflag_25_857_1a.png | 545/857 row, random field in the ecliptic plane<br />
</gallery><br />
<br />
<br />
=== Output flags ===<br />
<br />
A FlagTOIproc is produced by the TOIprocessing. It marks measurements which are not reliable for any of the following reasons:<br />
* gap (no valid input data), enlarged by one sample on each side. It flags less than 0.00044% (resp. 0.00062%) of the nominal (resp. complete) mission. It is equivalent to less than 3 (resp. 8) minutes of data.<br />
* "glitch" on dark bolometers: as the thermal template used for decorrelation is computed from these bolometer data, chunks of one minute-length data are discarded for all bolometers if at least 50\% of the data for at least one dark bolometer are flagged during this time. It is efficient to flag the data around the maximum of thermal events.<br />
* "glitch" on individual bolometers : samples where the signal from a cosmic ray hit dominates the sky signal at more than 3 $\sigma$ are discarded.<br />
* jump as 100 samples are flagged around the computed position of the jump to take into account the error on this reconstructed position.<br />
<br />
<br />
So the flag produced for the map making, called Total_flag, is defined by:<br />
* Total_flag = UnstablePointing Flag OR FlagTOIproc OR SSOflag4map OR SSOflag seen<br />
where FlagTOIproc = gap OR flag thermal template OR glitch OR jump<br />
and for PSB bolometers, FlagTOIproc_AB = FlagTOIproc_A OR FlagTOIproc_B.<br />
Note that the Total_flag is then identical for the A and B bolometers of a PSB pair.<br />
<br />
At the destriping stage, a more restricted flag, called Total_flag_PS, is used. It is defined by<br />
Total_flag_PS = Total_flag OR PS_flag.<br />
<br />
<br />
<br />
[[Category:Data processing]]</div>
Fdesert
https://wiki.cosmos.esa.int/planckpla2015/index.php?title=File:HFInominal_IntegrationTime.jpg&diff=3862
File:HFInominal IntegrationTime.jpg
2013-02-05T12:36:49Z
<p>Fdesert: </p>
<hr />
<div></div>
Fdesert
https://wiki.cosmos.esa.int/planckpla2015/index.php?title=TOI_processing&diff=3861
TOI processing
2013-02-05T12:36:08Z
<p>Fdesert: </p>
<hr />
<div>This Section is kept short as this Planck release does not contain TOIs. The main information in the HFI processing paper is not duplicated here.<br />
<br />
==Overview==<br />
<br />
We describe here the how the TOIs are processed in order to be used for map production. We do not repeat the general features of the pipeline which are given in the HFI Data Processing article (REF). Here we give complementary explanations on some details. The TOI of each bolometer is processed independently of the other bolometers, so as to keep the noise properties as uncorrelated as possible. The processing involves modifying the TOI itself for what concerns the conversion to absorbed power and the correction of glitch tails. It also adds a flag TOI that masks the TOI samples that are not to be projected on maps for various reasons.<br />
<br />
<br />
=== Acronyms and definitions ===<br />
<br />
* 4K lines: EMI/EMC influence of the 4K cooler mechanical motion on the bolometer readout electronics.<br />
* ADC : analog to digital converter<br />
* IMO : instrument model<br />
* Jump : sudden change of the baseline level inside a ring<br />
* LFER : low frequency excess response<br />
* PBR : phase binned ring<br />
* RIMO : reduced IMO<br />
* Ring : pointing period<br />
* ROI : ring ordered information<br />
* RTS : random telegraphic signal<br />
* SSO : solar system object<br />
* TOI : time ordered information<br />
<br />
== Input TOI ==<br />
<br />
The input TOI consists in the AC modulated voltage output of the readout of each bolometer. The input has previously been decompressed, and converted from internal digital units to voltage via a constant factor. The TOI has a regular sampling at the acquisition frequency of facq=180.373700+-0.000050 Hz. There are almost no missing data in the TOIs, except for few hundred samples of 545 and 857GHz TOIs which are lost in the on-board compression due to saturation on the Galactic Center crossings. <br />
<br />
== General Pipeline Structure ==<br />
<br />
The figure on the right shows how the initial TOI is transformed and how flags are produced:<br />
<br />
[[File:HFI_4_4_2_ToiProc_ExplS.png|thumb|600px|A schematic of the TOI processing pipeline]]<br />
<br />
<br />
== Output TOIs and products ==<br />
<br />
A TOI of clean calibrated samples (ccTOI) and a combined flag TOI (fTOI) are the outputs of the processing. The ccTOI is calibrated so as to represent the instantaneous power absorbed by the detector up to a constant (which will be determined by the map-making destriper). It is worth mentioning how the ccTOI is changed with respect to the input TOI, beyond the harmless constant conversion factor from voltage to absorbed power. The demodulation stage allows to get the demodulated bolometer voltage. The non-linearity correction is a second-order polynomial correction based on the physical but static bolometer model. In order to avoid too much masking after glitches, a glitch tail is subtracted after an occurrence of a glitch in the TOI. The 4K cooler lines noise is substituted at a series of 9 single temporal frequencies. Finally, the temporal response of the bolometer is deconvolved. This affects mostly the high-temporal frequency part of the TOI, although a small but significant low frequency (the long time response) tail is corrected too. Although flagged samples are not projected, their value influences the valid samples somehow. Hence interpolation procedures introduce some indirect modifications of the TOI.<br />
The flag TOI is a combination a dozen flags with an OR logic. Only unflagged data are projected. The exhaustive list of flags is given here: CompressionError, NoData, SSO, UnstablePointing, Glitch, BoloPlateFluctuation, RTS, Jump, PSBab. A complete qualification of the data is obtained at the ring level. If the TOI shows an anomalous behaviour on a given ring, this ring is discarded from projection. A special production of TOIs is also made as an input to the beam analysis with Mars, Jupiter and Saturn.<br />
<br />
<br />
== Examples of clean TOIs ==<br />
<gallery widths=300px perrow=3><br />
File:10_143_5_PBR_10800RING_LFER4_JC.jpg|Phase-binned signal (PBR) for 6 consecutive rings from 4000 to 4005<br />
File:10_143_5_rcircles_LFER4_JC.jpg| Valid signal (up) and noise (bottom) for 2 consecutive circles of the ring 4000<br />
File:10_143_5_fft_LFER4_JC.jpg| Fourier power spectral density of the first 30 minutes of the ring 4005 with signal before (pink) and after (black) deconvolution (top) and noise (bottom)<br />
File:14_545_1_PBR_10800RING_LFER4_JC.jpg|Phase-binned signal (PBR) for 6 consecutive rings from 4000 to 4005<br />
File:14_545_1_rcircles_LFER4_JC.jpg| Valid signal (up) and noise (bottom) for 2 consecutive circles of the ring 4000<br />
File:14_545_1_fft_LFER4_JC.jpg| Fourier power spectral density of the first 30 minutes of the ring 4005 with signal before (pink) and after (black) deconvolution (top) and noise (bottom)<br />
</gallery><br />
<br />
[[Media:check_ring.pdf|Samples of PBR, TOIs, and PSDs of all detectors are shown in this file]]<br />
<br />
== Trends in the output processing variables ==<br />
<br />
Here we intend to show the trend of the systematic effects that are dealt with in the TOI processing. The full impact of each of them is analyzed in [[HFI-Validation]].<br />
<br />
=== ADC baseline ===<br />
The following figure shows the ADC baseline which is used prior to demodulation (a constant offset is removed for clarity). This baseline is obtained by smoothing on an hour block average the undemodulated TOI on unflagged samples.<br />
<br />
[[File:baseline.jpg|400px|ADC baseline for all bolometers]]<br />
<br />
=== Glitch statistics ===<br />
The glitch rate per channel is shown in this figure. For details, see copap. <br />
<br />
[[File:figIntermPaperGR.jpg|400px|Glitch rate evolution]]<br />
<br />
The percentage of flagged data (mostly due to Cosmic Rays) at the ring level is shown in these examples. No smoothing was applied. Only valid rings are shown.<br />
<gallery widths=450px perrow=2 caption="Percentage of flagged data"><br />
File:group10_143_5.jpg | 143 GHz bolometers<br />
File:group14_545_1.jpg | 545 GHz bolometers<br />
</gallery><br />
[[Media:PercentUnvalid2.pdf|The complete set of plots is here]]<br />
<br />
=== Thermal template for decorrelation ===<br />
[[File:T90.jpg|400px|Thermal template used for the decorrelation from bolometer plate temperature fluctuations]]<br />
<br />
A simple linear decorrelation is performed using the 2 dark bolometers as a proxy of the bolometer plate temperature. Coupling coefficients were measured during the CPV phase.<br />
<br />
=== 4K cooler lines variability ===<br />
The amplitude of the nine 4K cooler lines in aW at 10, 20, 30, 40, 50, 60, 70, 80 and 17 Hz is shown for 2 bolometers in the following figures. The trend is smoothed over 31 ring values after having discarded measurements done at a ring which is discarded for all bolometers.<br />
<gallery widths=450px perrow=2 caption="Amplitude of the nine 4K cooler lines"><br />
File:10_143_5_4Klines.jpg | 143_5 bolometer<br />
File:14_545_1_4Klines.jpg | 545_1 bolometer<br />
</gallery><br />
[[Media:Lines4K.pdf|The 4K cooler line coefficients of all bolometers are shown in this file]]<br />
<br />
The 4K cooler lines project onto the maps only for a limited fraction of rings, the so-called resonant rings. This is graphically shown in the following figure. For each ring (stable pointing period), the spin rate is very stable at about 1 rpm. From one ring to another, the spin frequency (shown as diamonds) changes around that value. The sky signal is imprinted at the corresponding spin frequency and its 5400 (60x90) harmonics. If one of the nine 4K cooler lines happens to coincide with one of the spin frequency harmonics (a resonant ring), it will project a sine-wave systematic on the maps. The horizontal coloured bars show the zone of influence of a particular 4K line (labeled on the left side of the plot), when folded around 16.666 mHz. When the spin frequency hits one of these zones, we have a resonant ring. The 4K line coefficient is interpolated for this ring and an estimate of the systematic effect is subtracted from the TOI. Resonant rings are different for different 4K lines. Note the two-level oscillation pattern of the spin frequency is due to the satellite attitude control system.<br />
<br />
[[File:spinfreq1.jpg|600px|Spin frequency and 4K line zones of influence]]<br />
<br />
=== jump correction ===<br />
A piecewise constant value is removed to the TOI if a jump is detected. See a jump example in this figure:<br />
<br />
[[File:jump_exe.png|200px|An example of a jump seen on the rmsigTOI]]<br />
<br />
The number of jumps per day (all bolometers included) is shown in this figure:<br />
<br />
[[File:jumps_per_day.png|200px|Evolution of jump number during the mission]]<br />
<br />
The jumps are uncorrelated from bolometer to bolometer. The total number of jumps detected in the nominal and full mission is shown here:<br />
<br />
[[File:jumps_per_bolometer.png|200px|Number of jumps per bolometer]]<br />
<br />
<br />
=== Trends in noise and signal ===<br />
<gallery widths=450px perrow=2 caption="Signal (top) and noise (bottom) smoothed at 1 minute. All values falling in a discarded ring are not plotted."><br />
File:10_143_5_smooth_Watt.jpg | 143_5 bolometer<br />
File:14_545_1_smooth_Watt.jpg | 545_1 bolometer<br />
</gallery><br />
<br />
[[Media:check_smooth.pdf|The smooth TOIs of all detectors are shown in this file]]<br />
<br />
=== Noise stationarity ===<br />
This is not the final version but gives a good idea of power spectra at the detector level of rmsigTOIs.<br />
[[Media:v53_meanSpectra_bySurvey.pdf|All PSDs can be seen in this file]]<br />
<br />
The standard deviation per ring corrected by ring duration bias is given here, one per bolometer using only valid rings. No smoothing is applied (except a 31-point smoothing for the 545 and 857 GHz channels) but values for rings discarded for all bolometers (see below) are not used. The standard deviation is computed on samples valid for map-making which are also not affected by the Galaxy or the point-sources using the usual flags. Two examples are given here.<br />
<gallery widths=450px perrow=2 caption="Standard deviation per ring"><br />
File:stddev_group10_143_5.jpg | 143_5 bolometer<br />
File:stddev_group_14_545_1.jpg | 545_1 bolometer<br />
</gallery><br />
<br />
<br />
The full series of plots is here:<br />
[[Media:StationarityInROIoutput2.pdf|Standard deviation of rmsig TOIs at the ring level]]<br />
<br />
Note the presence for 3 bolometers of a two-level noise system. No correction can be done for that effect. See one example here:<br />
<br />
[[File:23_353_TwoLevel.jpg|300px|An example of two-level noise system is seen in bolometer 23_353_3a]]<br />
<br />
<br />
An example of the higher order statistics which are used to unveil rings affected by RTS problems.<br />
<br />
[[File:HFI_4_4_2_RTSexample13.jpg|400px|Example of RTS detection]]<br />
<br />
=== Discarded rings ===<br />
<br />
Some rings are discarded (flagged) from further use (beam making, map making) by using ring statistics (see above and the HFI data processing paper REF). For each statistic, we compare each ring value to the ring values averaged (RVA) over a large selection of rings (between 3000 and 21700). We also define the modified standard deviation (MSD) of a ring quantity as the standard deviation of that quantity over the rings that deviate by less than five nominal standard deviations. This truncation is necessary to be robust against extreme deviant rings.<br />
<br />
More specifically, a ring is discarded if it matches one of the following criteria:<br />
<br />
* the | mean-median | deviates from the RVA by more than fifteen times the MSD.<br />
* the standard deviation deviates from the RVA by more than -5 times the MSD (all cases corresponding to almost empty rings) or +15 times the MSD.<br />
* the Kolmogorov-Smirnov test deviates from the RVA by more than 15 times the MSD.<br />
* the ring duration is more than 90 min.<br />
* the ring is contaminated by RTS with an amplitude of more than one standard deviation of the noise. It concerns a few hundreds of rings for 3 bolometers (44_217_6b, 71_217_8a, 74_857_4). Notice that two bolometers are completely discarded for maps: 55_545_3 and 70_143_8, which present RTS at all time.<br />
<br />
For the three first criteria, a visual inspection of the rmsigTOI at each of the incriminated rings has shown that all these anomalous rings are due to either a drift, a small jump in the TOI trend or a sudden change of noise level, the origin of which is unknown at present.<br />
<br />
Once the list of discarded rings per bolometer is produced, a common list of discarded rings can be extracted for all bolometers (by using discarded rings for at least half the bolometers). Such rings correspond to identified phenomena, as can be seen on the following table.<br />
<br />
Furthermore, an isolated valid ring stuck between two common discarded rings becomes discarded as well.<br />
<br />
Table of '''common discarded rings''' of the nominal mission (rings 240-14723).<br />
<br />
{|table border=1 align=center cellpadding="3" cellspacing="0" style=text-align:center<br />
! '''Cause''' !! '''ring number'''<br />
|-<br />
!align="left"| manoeuvre<br />
|align="center"|304 1312 3590 3611 3642 3922 4949 6379 8456 11328<br />
|-<br />
! align="left"| Sorption Cooler Switchover <br />
|align="center"| 11149 11150 11151 11152 <br />
|-<br />
! align="left"| Rings too long<br />
| align="center"| 440 474 509 544 897 898 3589 13333 14627 14653 <br />
|-<br />
! align="left"| Star tracker switchover <br />
| align="center"| 14628 14654 <br />
|-<br />
! align="left"| Massive Glitch Event<br />
| align="center"| 7665 <br />
|-<br />
! align="left"| Solar Flare<br />
| align="center"| 11235 <br />
|}<br />
<br />
<br />
<br />
The following figure is a summary of the impact of the discarding process '''for each bolometer''' (the solid black line is the common discarded ring percentage). The outlier bolometers have some RTS problems as mentionned above.<br />
<br />
[[File:Fraction_discarded_rings_v53.png|300px|Fraction of discarded rings]]<br />
[[File:Fraction_discarded_time_v53.png|300px|Fraction of time due to discarded rings]]<br />
<br />
=== Effective integration time ===<br />
The following figure summarizes the effective integration time per bolometer. For that purpose the number of unflagged samples in non-discarded pointing periods have been used within the nominal mission. The average value is of about 324 days of effective integration time.<br />
<br />
[[File:HFInominal_IntegrationTime.jpg|600px|Effective integration time per bolometer]]<br />
<br />
== Flag description ==<br />
<br />
==== Input flags ====<br />
<br />
These flags are used as inputs to the TOI processing<br />
<br />
; The point-source flag (PSflag) <br />
: An earlier version of HFI point-source catalog is read back into a flag TOIs, at a given frequency. In practice, 5 sigma sources are masked within a radius of 1.3 FWHM (9, 7, 5, 5, 5, 5 arcmin at 100,143,217,353,545,857 GHz) TBC.<br />
<br />
; the galactic flag (Galflag<br />
: An earlier version of HFI maps is thresholded and apodized. The produced masks are read into flag TOIs. The retained threshold corresponds to a sky coverage of respectively 70, 70, 80, 90, 90, 90% at 100,143,217,353,545,857 GHz.<br />
<br />
; Solar System Object flag <br />
: For the TOI flag, Mars, Jupiter, Saturn are flagged up to a radius of NbBeam= 2,3,3,4,4,4 times the fiducial SSO_FWHM with SSO_FWHM= 9, 7, 5, 5, 5, 5 arcmin at 100,143,217,353,545,857 GHz.<br />
<br />
: As an input to planet mask for maps, Mars, Jupiter, Saturn are flagged with a radius computed as a coefficient depending on the planet (Factor_per_source) times NbBeam times SSO_FWHM, with Factor_per_source = 1.1, 2.25, 1.25 for Mars, Jupiter, Saturn respectively and NbBeam = 2.25, 4.25, 4.0, 5.0,.6.0, 8.0 at 100, 143, 217, 353, 545, 857 GHz. This flag is called SSOflag4map.<br />
<br />
: A small trailing tail is added to the mask to take into account the non-deconvolution of the planet signal which has been replaced by background values. The width of that tail is 10 % of the main flag diameter. The number of samples which are additionnaly flagged are the Factor_per_Source times AddSNafter with (trail) times (Factor_per_source)^3 samples where trail = 10, 30, 20, 20, 30, 40 at 100, 143, 217, 353, 545, 857 GHz.<br />
<br />
: Uranus and Neptune together with detected asteroids are masked by HFI. They are masked at the TOI level using an exclusion radius of 1.5 SSO_FWHM. At 857 GHz, 24 asteroids have been detected with HFI : 1Ceres, 2Pallas, 3Juno, 4Vesta, 7Iris, 8Flora, 9Metis, 10Hygiea, 11Parthenope, 12Victoria, 13Egeria, 14Irene, 15Eunomia, 16Psyche, 18Melpomene, 19Fortuna, 20Massalia, 29Amphitrite, 41Daphne, 45Eugenia, 52Europa, 88Thisbe, 704Interamnia, 324Bamberga.<br />
<br />
<gallery widths=200px perrow=4 caption="Local maps showing the SSO flag. The colors correspond to the surveys involved in the nominal mission (green = Survey I, yellow = Survey II, red = Survey III)."><br />
File:SSOflag_10_143_5J.png | 143 SWB row, around Jupiter 1st crossing<br />
File:SSOflag_10_143_5S.png | 143 SWB row, around Saturn<br />
File:SSOflag_10_143_5M.png | 143 SWB row, around Mars<br />
File:SSOflag_10_143_5a.png | 143 SWB row, random field in the ecliptic plane<br />
File:SSOflag_25_857_1J.png | 545/857 row, around Jupiter 1st crossing<br />
File:SSOflag_25_857_1S.png | 545/857 row, around Saturn<br />
File:SSOflag_25_857_1M.png | 545/857 row, around Mars<br />
File:SSOflag_25_857_1a.png | 545/857 row, random field in the ecliptic plane<br />
</gallery><br />
<br />
<br />
=== Output flags ===<br />
<br />
A FlagTOIproc is produced by the TOIprocessing. It marks measurements which are not reliable for any of the following reasons:<br />
* gap (no valid input data), enlarged by one sample on each side. It flags less than 0.00044% (resp. 0.00062%) of the nominal (resp. complete) mission. It is equivalent to less than 3 (resp. 8) minutes of data.<br />
* "glitch" on dark bolometers: as the thermal template used for decorrelation is computed from these bolometer data, chunks of one minute-length data are discarded for all bolometers if at least 50\% of the data for at least one dark bolometer are flagged during this time. It is efficient to flag the data around the maximum of thermal events.<br />
* "glitch" on individual bolometers : samples where the signal from a cosmic ray hit dominates the sky signal at more than 3 $\sigma$ are discarded.<br />
* jump as 100 samples are flagged around the computed position of the jump to take into account the error on this reconstructed position.<br />
<br />
<br />
So the flag produced for the map making, called Total_flag, is defined by:<br />
* Total_flag = UnstablePointing Flag OR FlagTOIproc OR SSOflag4map OR SSOflag seen<br />
where FlagTOIproc = gap OR flag thermal template OR glitch OR jump<br />
and for PSB bolometers, FlagTOIproc_AB = FlagTOIproc_A OR FlagTOIproc_B.<br />
Note that the Total_flag is then identical for the A and B bolometers of a PSB pair.<br />
<br />
At the destriping stage, a more restricted flag, called Total_flag_PS, is used. It is defined by<br />
Total_flag_PS = Total_flag OR PS_flag.<br />
<br />
<br />
<br />
[[Category:Data processing]]</div>
Fdesert
https://wiki.cosmos.esa.int/planckpla2015/index.php?title=TOI_processing&diff=3655
TOI processing
2013-01-29T09:20:45Z
<p>Fdesert: </p>
<hr />
<div>This Section is kept short as this Planck release does not contain TOIs. The main information in the HFI processing paper is not duplicated here.<br />
<br />
==Overview==<br />
<br />
We describe here the how the TOIs are processed in order to be used for map production. We do not repeat the general features of the pipeline which are given in the HFI Data Processing article (REF). Here we give complementary explanations on some details. The TOI of each bolometer is processed independently of the other bolometers, so as to keep the noise properties as uncorrelated as possible. The processing involves modifying the TOI itself for what concerns the conversion to absorbed power and the correction of glitch tails. It also adds a flag TOI that masks the TOI samples that are not to be projected on maps for various reasons.<br />
<br />
<br />
=== Acronyms and definitions ===<br />
<br />
* 4K lines: EMI/EMC influence of the 4K cooler mechanical motion on the bolometer readout electronics.<br />
* ADC : analog to digital converter<br />
* IMO : instrument model<br />
* Jump : sudden change of the baseline level inside a ring<br />
* LFER : low frequency excess response<br />
* PBR : phase binned ring<br />
* RIMO : reduced IMO<br />
* Ring : pointing period<br />
* ROI : ring ordered information<br />
* RTS : random telegraphic signal<br />
* SSO : solar system object<br />
* TOI : time ordered information<br />
<br />
== Input TOI ==<br />
<br />
The input TOI consists in the AC modulated voltage output of the readout of each bolometer. The input has previously been decompressed, and converted from internal digital units to voltage via a constant factor. The TOI has a regular sampling at the acquisition frequency of facq=180.373700+-0.000050 Hz. There are almost no missing data in the TOIs, except for few hundred samples of 545 and 857GHz TOIs which are lost in the on-board compression due to saturation on the Galactic Center crossings. <br />
<br />
== General Pipeline Structure ==<br />
<br />
The figure on the right shows how the initial TOI is transformed and how flags are produced:<br />
<br />
[[File:HFI_4_4_2_ToiProc_ExplS.png|thumb|600px|A schematic of the TOI processing pipeline]]<br />
<br />
<br />
== Output TOIs and products ==<br />
<br />
A TOI of clean calibrated samples (ccTOI) and a combined flag TOI (fTOI) are the outputs of the processing. The ccTOI is calibrated so as to represent the instantaneous power absorbed by the detector up to a constant (which will be determined by the map-making destriper). It is worth mentioning how the ccTOI is changed with respect to the input TOI, beyond the harmless constant conversion factor from voltage to absorbed power. The demodulation stage allows to get the demodulated bolometer voltage. The non-linearity correction is a second-order polynomial correction based on the physical but static bolometer model. In order to avoid too much masking after glitches, a glitch tail is subtracted after an occurrence of a glitch in the TOI. The 4K cooler lines noise is substituted at a series of 9 single temporal frequencies. Finally, the temporal response of the bolometer is deconvolved. This affects mostly the high-temporal frequency part of the TOI, although a small but significant low frequency (the long time response) tail is corrected too. Although flagged samples are not projected, their value influences the valid samples somehow. Hence interpolation procedures introduce some indirect modifications of the TOI.<br />
The flag TOI is a combination a dozen flags with an OR logic. Only unflagged data are projected. The exhaustive list of flags is given here: CompressionError, NoData, SSO, UnstablePointing, Glitch, BoloPlateFluctuation, RTS, Jump, PSBab. A complete qualification of the data is obtained at the ring level. If the TOI shows an anomalous behaviour on a given ring, this ring is discarded from projection. A special production of TOIs is also made as an input to the beam analysis with Mars, Jupiter and Saturn.<br />
<br />
<br />
== Examples of clean TOIs ==<br />
<gallery widths=300px perrow=3><br />
File:10_143_5_PBR_10800RING_LFER4_JC.jpg|Phase-binned signal (PBR) for 6 consecutive rings from 4000 to 4005<br />
File:10_143_5_rcircles_LFER4_JC.jpg| Valid signal (up) and noise (bottom) for 2 consecutive circles of the ring 4000<br />
File:10_143_5_fft_LFER4_JC.jpg| Fourier power spectral density of the first 30 minutes of the ring 4005 with signal before (pink) and after (black) deconvolution (top) and noise (bottom)<br />
File:14_545_1_PBR_10800RING_LFER4_JC.jpg|Phase-binned signal (PBR) for 6 consecutive rings from 4000 to 4005<br />
File:14_545_1_rcircles_LFER4_JC.jpg| Valid signal (up) and noise (bottom) for 2 consecutive circles of the ring 4000<br />
File:14_545_1_fft_LFER4_JC.jpg| Fourier power spectral density of the first 30 minutes of the ring 4005 with signal before (pink) and after (black) deconvolution (top) and noise (bottom)<br />
</gallery><br />
<br />
[[Media:check_ring.pdf|Samples of PBR, TOIs, and PSDs of all detectors are shown in this file]]<br />
<br />
== Trends in the output processing variables ==<br />
<br />
Here we intend to show the trend of the systematic effects that are dealt with in the TOI processing. The full impact of each of them is analyzed in [[HFI-Validation]].<br />
<br />
=== ADC baseline ===<br />
The following figure shows the ADC baseline which is used prior to demodulation (a constant offset is removed for clarity). This baseline is obtained by smoothing on an hour block average the undemodulated TOI on unflagged samples.<br />
<br />
[[File:baseline.jpg|400px|ADC baseline for all bolometers]]<br />
<br />
=== Glitch statistics ===<br />
The glitch rate per channel is shown in this figure. For details, see copap. <br />
<br />
[[File:figIntermPaperGR.jpg|400px|Glitch rate evolution]]<br />
<br />
The percentage of flagged data (mostly due to Cosmic Rays) at the ring level is shown in these examples. No smoothing was applied. Only valid rings are shown.<br />
<gallery widths=450px perrow=2 caption="Percentage of flagged data"><br />
File:group10_143_5.jpg | 143 GHz bolometers<br />
File:group14_545_1.jpg | 545 GHz bolometers<br />
</gallery><br />
[[Media:PercentUnvalid2.pdf|The complete set of plots is here]]<br />
<br />
=== Thermal template for decorrelation ===<br />
[[File:T90.jpg|400px|Thermal template used for the decorrelation from bolometer plate temperature fluctuations]]<br />
<br />
A simple linear decorrelation is performed using the 2 dark bolometers as a proxy of the bolometer plate temperature. Coupling coefficients were measured during the CPV phase.<br />
<br />
=== 4K cooler lines variability ===<br />
The amplitude of the nine 4K cooler lines in aW at 10, 20, 30, 40, 50, 60, 70, 80 and 17 Hz is shown for 2 bolometers in the following figures. The trend is smoothed over 31 ring values after having discarded measurements done at a ring which is discarded for all bolometers.<br />
<gallery widths=450px perrow=2 caption="Amplitude of the nine 4K cooler lines"><br />
File:10_143_5_4Klines.jpg | 143_5 bolometer<br />
File:14_545_1_4Klines.jpg | 545_1 bolometer<br />
</gallery><br />
[[Media:Lines4K.pdf|The 4K cooler line coefficients of all bolometers are shown in this file]]<br />
<br />
The 4K cooler lines project onto the maps only for a limited fraction of rings, the so-called resonant rings. This is graphically shown in the following figure. For each ring (stable pointing period), the spin rate is very stable at about 1 rpm. From one ring to another, the spin frequency (shown as diamonds) changes around that value. The sky signal is imprinted at the corresponding spin frequency and its 5400 (60x90) harmonics. If one of the nine 4K cooler lines happens to coincide with one of the spin frequency harmonics (a resonant ring), it will project a sine-wave systematic on the maps. The horizontal coloured bars show the zone of influence of a particular 4K line (labeled on the left side of the plot), when folded around 16.666 mHz. When the spin frequency hits one of these zones, we have a resonant ring. The 4K line coefficient is interpolated for this ring and an estimate of the systematic effect is subtracted from the TOI. Resonant rings are different for different 4K lines. Note the two-level oscillation pattern of the spin frequency is due to the satellite attitude control system.<br />
<br />
[[File:spinfreq1.jpg|600px|Spin frequency and 4K line zones of influence]]<br />
<br />
=== jump correction ===<br />
A piecewise constant value is removed to the TOI if a jump is detected. See a jump example in this figure:<br />
<br />
[[File:jump_exe.png|200px|An example of a jump seen on the rmsigTOI]]<br />
<br />
The number of jumps per day (all bolometers included) is shown in this figure:<br />
<br />
[[File:jumps_per_day.png|200px|Evolution of jump number during the mission]]<br />
<br />
The jumps are uncorrelated from bolometer to bolometer. The total number of jumps detected in the nominal and full mission is shown here:<br />
<br />
[[File:jumps_per_bolometer.png|200px|Number of jumps per bolometer]]<br />
<br />
<br />
=== Trends in noise and signal ===<br />
<gallery widths=450px perrow=2 caption="Signal (top) and noise (bottom) smoothed at 1 minute. All values falling in a discarded ring are not plotted."><br />
File:10_143_5_smooth_Watt.jpg | 143_5 bolometer<br />
File:14_545_1_smooth_Watt.jpg | 545_1 bolometer<br />
</gallery><br />
<br />
[[Media:check_smooth.pdf|The smooth TOIs of all detectors are shown in this file]]<br />
<br />
=== Noise stationarity ===<br />
This is not the final version but gives a good idea of power spectra at the detector level of rmsigTOIs.<br />
[[Media:v53_meanSpectra_bySurvey.pdf|All PSDs can be seen in this file]]<br />
<br />
The standard deviation per ring corrected by ring duration bias is given here, one per bolometer using only valid rings. No smoothing is applied (except a 31-point smoothing for the 545<br />
and 857 GHz channels) but values for rings discarded for all bolometers (see below) are not used. The standard deviation is computed on samples valid for map-making which are also not affected by the Galaxy or the point-sources using the usual flags. Two examples are given here.<br />
<gallery widths=450px perrow=2 caption="Standard deviation per ring"><br />
File:stddev_group10_143_5.jpg | 143_5 bolometer<br />
File:stddev_group_14_545_1.jpg | 545_1 bolometer<br />
</gallery><br />
<br />
<br />
The full series of plots is here:<br />
[[Media:StationarityInROIoutput2.pdf|Standard deviation of rmsig TOIs at the ring level]]<br />
<br />
Note the presence for 3 bolometers of a two-level noise system. No correction can be done for that effect. See one example here:<br />
<br />
[[File:23_353_TwoLevel.jpg|300px|An example of two-level noise system is seen in bolometer 23_353_3a]]<br />
<br />
<br />
An example of the higher order statistics which are used to unveil rings affected by RTS problems.<br />
<br />
[[File:HFI_4_4_2_RTSexample13.jpg|400px|Example of RTS detection]]<br />
<br />
=== Discarded rings ===<br />
<br />
Some rings are discarded (flagged) from further use (beam making, map making) by using ring statistics (see above and the HFI data processing paper REF). For each statistic, we compare each ring value to the ring values averaged (RVA) over a large selection of rings (between 3000 and 21700). We also define the modified standard deviation (MSD) of a ring quantity as the standard deviation of that quantity over the rings that deviate by less than five nominal standard deviations. This truncation is necessary to be robust against extreme deviant rings.<br />
<br />
More specifically, a ring is discarded if it matches one of the following criteria:<br />
<br />
* the | mean-median | deviates from the RVA by more than fifteen times the MSD.<br />
* the standard deviation deviates from the RVA by more than -5 times the MSD (all cases corresponding to almost empty rings) or +15 times the MSD.<br />
* the Kolmogorov-Smirnov test deviates from the RVA by more than 15 times the MSD.<br />
* the ring duration is more than 90 min.<br />
* the ring is contaminated by RTS with an amplitude of more than one standard deviation of the noise. It concerns a few hundreds of rings for 3 bolometers (44_217_6b, 71_217_8a, 74_857_4). Notice that two bolometers are completely discarded for maps: 55_545_3 and 70_143_8, which present RTS at all time.<br />
<br />
For the three first criteria, a visual inspection of the rmsigTOI at each of the incriminated rings has shown that all these anomalous rings are due to either a drift, a small jump in the TOI trend or a sudden change of noise level, the origin of which is unknown at present.<br />
<br />
Once the list of discarded rings per bolometer is produced, a common list of discarded rings can be extracted for all bolometers (by using discarded rings for at least half the bolometers). Such rings correspond to identified phenomena, as can be seen on the following table.<br />
<br />
Furthermore, an isolated valid ring stuck between two common discarded rings becomes discarded as well.<br />
<br />
Table of '''common discarded rings''' of the nominal mission (rings 240-14723).<br />
<br />
{|table border=1 align=center cellpadding="3" cellspacing="0" style=text-align:center<br />
! '''Cause''' !! '''ring number'''<br />
|-<br />
!align="left"| manoeuvre<br />
|align="center"|304 1312 3590 3611 3642 3922 4949 6379 8456 11328<br />
|-<br />
! align="left"| Sorption Cooler Switchover <br />
|align="center"| 11149 11150 11151 11152 <br />
|-<br />
! align="left"| Rings too long<br />
| align="center"| 440 474 509 544 897 898 3589 13333 14627 14653 <br />
|-<br />
! align="left"| Star tracker switchover <br />
| align="center"| 14628 14654 <br />
|-<br />
! align="left"| Massive Glitch Event<br />
| align="center"| 7665 <br />
|-<br />
! align="left"| Solar Flare<br />
| align="center"| 11235 <br />
|}<br />
<br />
<br />
<br />
The following figure is a summary of the impact of the discarding process '''for each bolometer''' (the solid black line is the common discarded ring percentage). The outlier bolometers have some RTS problems as mentionned above.<br />
<br />
[[File:Fraction_discarded_rings_v53.png|300px|Fraction of discarded rings]]<br />
[[File:Fraction_discarded_time_v53.png|300px|Fraction of time due to discarded rings]]<br />
<br />
== Flag description ==<br />
<br />
==== Input flags ====<br />
<br />
These flags are used as inputs to the TOI processing<br />
<br />
* The point-source flag (PSflag):<br />
<br />
An earlier version of HFI point-source catalog is read back into a flag TOIs, at a given frequency. In practice, 5 sigma sources are masked within a radius of 1.3 FWHM (9, 7, 5, 5, 5, 5 arcmin at 100,143,217,353,545,857 GHz) TBC.<br />
<br />
* the galactic flag (Galflag):<br />
<br />
An earlier version of HFI maps is thresholded and apodized. The produced masks are read into flag TOIs. The retained threshold corresponds to a sky coverage of respectively 70, 70, 80, 90, 90, 90% at 100,143,217,353,545,857 GHz.<br />
<br />
* Solar System Object flag <br />
<br />
For the TOI flag, Mars, Jupiter, Saturn are flagged up to a radius of NbBeam= 2,3,3,4,4,4 times the fiducial SSO_FWHM with SSO_FWHM= 9, 7, 5, 5, 5, 5 arcmin at 100,143,217,353,545,857 GHz.<br />
<br />
As an input to planet mask for maps, Mars, Jupiter, Saturn are flagged with a radius computed as a coefficient depending on the planet (Factor_per_source) times NbBeam times SSO_FWHM, with Factor_per_source = 1.1, 2.25, 1.25 for Mars, Jupiter, Saturn respectively and NbBeam = 2.25, 4.25, 4.0, 5.0,.6.0, 8.0 at 100, 143, 217, 353, 545, 857 GHz. This flag is called SSOflag4map.<br />
<br />
A small trailing tail is added to the mask to take into account the non-deconvolution of the planet signal which has been replaced by background values. The width of that tail is 10 % of the main flag diameter. The number of samples which are additionnaly flagged are the Factor_per_Source times AddSNafter with 10, 30, 20, 20, 30, 40 samples at 100, 143, 217, 353, 545, 857 GHz.<br />
<br />
Uranus and Neptune together with detected asteroids are masked by HFI. They are masked at the TOI level using an exclusion radius of 1.5 SSO_FWHM. At 857 GHz, 24 asteroids have been detected with HFI : 1Ceres, 2Pallas, 3Juno, 4Vesta, 7Iris, 8Flora, 9Metis, 10Hygiea, 11Parthenope, 12Victoria, 13Egeria, 14Irene, 15Eunomia, 16Psyche, 18Melpomene, 19Fortuna, 20Massalia, 29Amphitrite, 41Daphne, 45Eugenia, 52Europa, 88Thisbe, 704Interamnia, 324Bamberga.<br />
<br />
<br />
=== Output flags ===<br />
<br />
A FlagTOIproc is produced by the TOIprocessing. It marks measurements which are not reliable for ny of the following reasons :<br />
* gap (no valid input data), enlarged by one sample on each side. It flags less than 0.00044% (resp. 0.00062%) of the nominal (resp. complete) mission. It is equivallen to less thena 3 (resp. 8) minutes of data.<br />
* glitch : samples where the signal from a cosmic ray hit dominates the sky signal at more than 3 $\sigma$ are discarded.<br />
* jump as 100 samples are flagged around the computed position of the jump to take into account the error on this position.<br />
<br />
With and without point-sources, identical to a and b PSBs.<br />
<br />
<br />
Flags produced for the map making :<br />
* UnstablePointing Flag OR FlagTOIproc OR SSOflag4map OR SSOflag seen<br />
* FlagTOIproc = glitch OR jump OR flag thermal template <br />
* PSB flag glitch = flag glitch A OR flag glitch B<br />
<br />
Flag used in the destriping: same + point-source flag<br />
<br />
<br />
<br />
<br />
[[Category:Data processing]]</div>
Fdesert
https://wiki.cosmos.esa.int/planckpla2015/index.php?title=TOI_processing&diff=3654
TOI processing
2013-01-29T09:12:27Z
<p>Fdesert: </p>
<hr />
<div>This Section is kept short as this Planck release does not contain TOIs. The main information in the HFI processing paper is not duplicated here.<br />
<br />
==Overview==<br />
<br />
We describe here the how the TOIs are processed in order to be used for map production. We do not repeat the general features of the pipeline which are given in the HFI Data Processing article (REF). Here we give complementary explanations on some details. The TOI of each bolometer is processed independently of the other bolometers, so as to keep the noise properties as uncorrelated as possible. The processing involves modifying the TOI itself for what concerns the conversion to absorbed power and the correction of glitch tails. It also adds a flag TOI that masks the TOI samples that are not to be projected on maps for various reasons.<br />
<br />
<br />
=== Acronyms and definitions ===<br />
<br />
* 4K lines: EMI/EMC influence of the 4K cooler mechanical motion on the bolometer readout electronics.<br />
* ADC : analog to digital converter<br />
* IMO : instrument model<br />
* Jump : sudden change of the baseline level inside a ring<br />
* LFER : low frequency excess response<br />
* PBR : phase binned ring<br />
* RIMO : reduced IMO<br />
* Ring : pointing period<br />
* ROI : ring ordered information<br />
* RTS : randomn telegraphic signal<br />
* SSO : solar system object<br />
* TOI : time ordered information<br />
<br />
== Input TOI ==<br />
<br />
The input TOI consists in the AC modulated voltage output of the readout of each bolometer. The input has previously been decompressed, and converted from internal digital units to voltage via a constant factor. The TOI has a regular sampling at the acquisition frequency of facq=180.373700+-0.000050 Hz. There are almost no missing data in the TOIs, except for few hundred samples of 545 and 857GHz TOIs which are lost in the on-board compression due to saturation on the Galactic Center crossings. <br />
<br />
== General Pipeline Structure ==<br />
<br />
The figure on the right shows how the initial TOI is transformed and how flags are produced:<br />
<br />
[[File:HFI_4_4_2_ToiProc_ExplS.png|thumb|600px|A schematic of the TOI processing pipeline]]<br />
<br />
<br />
== Output TOIs and products ==<br />
<br />
A TOI of clean calibrated samples (ccTOI) and a combined flag TOI (fTOI) are the outputs of the processing. The ccTOI is calibrated so as to represent the instantaneous power absorbed by the detector up to a constant (which will be determined by the map-making destriper). It is worth mentioning how the ccTOI is changed with respect to the input TOI, beyond the harmless constant conversion factor from voltage to absorbed power. The demodulation stage allows to get the demodulated bolometer voltage. The non-linearity correction is a second-order polynomial correction based on the physical but static bolometer model. In order to avoid too much masking after glitches, a glitch tail is subtracted after an occurrence of a glitch in the TOI. The 4K cooler lines noise is substituted at a series of 9 single temporal frequencies. Finally, the temporal response of the bolometer is deconvolved. This affects mostly the high-temporal frequency part of the TOI, although a small but significant low frequency (the long time response) tail is corrected too. Although flagged samples are not projected, their value influences the valid samples somehow. Hence interpolation procedures introduce some indirect modifications of the TOI.<br />
The flag TOI is a combination a dozen flags with an OR logic. Only unflagged data are projected. The exhaustive list of flags is given here: CompressionError, NoData, SSO, UnstablePointing, Glitch, BoloPlateFluctuation, RTS, Jump, PSBab. A complete qualification of the data is obtained at the ring level. If the TOI shows an anomalous behaviour on a given ring, this ring is discarded from projection. A special production of TOIs is also made as an input to the beam analysis with Mars, Jupiter and Saturn.<br />
<br />
<br />
== Examples of clean TOIs ==<br />
<gallery widths=300px perrow=3><br />
File:10_143_5_PBR_10800RING_LFER4_JC.jpg|Phase-binned signal (PBR) for 6 consecutive rings from 4000 to 4005<br />
File:10_143_5_rcircles_LFER4_JC.jpg| Valid signal (up) and noise (bottom) for 2 consecutive circles of the ring 4000<br />
File:10_143_5_fft_LFER4_JC.jpg| Fourier power spectral density of the first 30 minutes of the ring 4005 with signal before (pink) and after (black) deconvolution (top) and noise (bottom)<br />
File:14_545_1_PBR_10800RING_LFER4_JC.jpg|Phase-binned signal (PBR) for 6 consecutive rings from 4000 to 4005<br />
File:14_545_1_rcircles_LFER4_JC.jpg| Valid signal (up) and noise (bottom) for 2 consecutive circles of the ring 4000<br />
File:14_545_1_fft_LFER4_JC.jpg| Fourier power spectral density of the first 30 minutes of the ring 4005 with signal before (pink) and after (black) deconvolution (top) and noise (bottom)<br />
</gallery><br />
<br />
[[Media:check_ring.pdf|Samples of PBR, TOIs, and PSDs of all detectors are shown in this file]]<br />
<br />
== Trends in the output processing variables ==<br />
<br />
Here we intend to show the trend of the systematic effects that are dealt with in the TOI processing. The full impact of each of them is analyzed in [[HFI-Validation]].<br />
<br />
=== ADC baseline ===<br />
The following figure shows the ADC baseline which is used prior to demodulation (a constant offset is removed for clarity). This baseline is obtained by smoothing on an hour block average the undemodulated TOI on unflagged samples.<br />
<br />
[[File:baseline.jpg|400px|ADC baseline for all bolometers]]<br />
<br />
=== Glitch statistics ===<br />
The glitch rate per channel is shown in this figure. For details, see copap. <br />
<br />
[[File:figIntermPaperGR.jpg|400px|Glitch rate evolution]]<br />
<br />
The percentage of flagged data (mostly due to Cosmic Rays) at the ring level is shown in these examples. No smoothing was applied. Only valid rings are shown.<br />
<gallery widths=450px perrow=2 caption="Percentage of flagged data"><br />
File:group10_143_5.jpg | 143 GHz bolometers<br />
File:group14_545_1.jpg | 545 GHz bolometers<br />
</gallery><br />
[[Media:PercentUnvalid2.pdf|The complete set of plots is here]]<br />
<br />
=== Thermal template for decorrelation ===<br />
[[File:T90.jpg|400px|Thermal template used for the decorrelation from bolometer plate temperature fluctuations]]<br />
<br />
A simple linear decorrelation is performed using the 2 dark bolometers as a proxy of the bolometer plate temperature. Coupling coefficients were measured during the CPV phase.<br />
<br />
=== 4K cooler lines variability ===<br />
The amplitude of the nine 4K cooler lines in aW at 10, 20, 30, 40, 50, 60, 70, 80 and 17 Hz is shown for 2 bolometers in the following figures. The trend is smoothed over 31 ring values after having discarded measurements done at a ring which is discarded for all bolometers.<br />
<gallery widths=450px perrow=2 caption="Amplitude of the nine 4K cooler lines"><br />
File:10_143_5_4Klines.jpg | 143_5 bolometer<br />
File:14_545_1_4Klines.jpg | 545_1 bolometer<br />
</gallery><br />
[[Media:Lines4K.pdf|The 4K cooler line coefficients of all bolometers are shown in this file]]<br />
<br />
The 4K cooler lines project onto the maps only for a limited fraction of rings, the so-called resonant rings. This is graphically shown in the following figure. For each ring (stable pointing period), the spin rate is very stable at about 1 rpm. From one ring to another, the spin frequency (shown as diamonds) changes around that value. The sky signal is imprinted at the corresponding spin frequency and its 5400 (60x90) harmonics. If one of the nine 4K cooler lines happens to coincide with one of the spin frequency harmonics, it will project a sine-wave systematic on the maps. This is what we call a resonant ring. The horizontal coloured bars show the zone of influence of a particular 4K line, when folded around 16.666 mHz. When the spin frequency hits one of these zones, we have a resonant ring. The 4K line coefficient is interpolated for this ring and an estimate of the systematic effect is subtracted from the TOI. Resonant rings are different for different 4K lines. Note the two-level oscillation pattern of the spin frequency due to the satellite attitude control system.<br />
<br />
[[File:spinfreq1.jpg|600px|Spin frequency and 4K line zones of influence]]<br />
<br />
=== jump correction ===<br />
A piecewise constant value is removed to the TOI if a jump is detected. See a jump example in this figure:<br />
<br />
[[File:jump_exe.png|200px|An example of a jump seen on the rmsigTOI]]<br />
<br />
The number of jumps per day (all bolometers included) is shown in this figure:<br />
<br />
[[File:jumps_per_day.png|200px|Evolution of jump number during the mission]]<br />
<br />
The jumps are uncorrelated from bolometer to bolometer. The total number of jumps detected in the nominal and full mission is shown here:<br />
<br />
[[File:jumps_per_bolometer.png|200px|Number of jumps per bolometer]]<br />
<br />
<br />
=== Trends in noise and signal ===<br />
<gallery widths=450px perrow=2 caption="Signal (top) and noise (bottom) smoothed at 1 minute. All values falling in a discarded ring are not plotted."><br />
File:10_143_5_smooth_Watt.jpg | 143_5 bolometer<br />
File:14_545_1_smooth_Watt.jpg | 545_1 bolometer<br />
</gallery><br />
<br />
[[Media:check_smooth.pdf|The smooth TOIs of all detectors are shown in this file]]<br />
<br />
=== Noise stationarity ===<br />
This is not the final version but gives a good idea of power spectra at the detector level of rmsigTOIs.<br />
[[Media:v53_meanSpectra_bySurvey.pdf|All PSDs can be seen in this file]]<br />
<br />
The standard deviation per ring corrected by ring duration bias is given here, one per bolometer using only valid rings. No smoothing is applied (except a 31-point smoothing for the 545<br />
and 857 GHz channels) but values for rings discarded for all bolometers (see below) are not used. The standard deviation is computed on samples valid for map-making which are also not affected by the Galaxy or the point-sources using the usual flags. Two examples are given here.<br />
<gallery widths=450px perrow=2 caption="Standard deviation per ring"><br />
File:stddev_group10_143_5.jpg | 143_5 bolometer<br />
File:stddev_group_14_545_1.jpg | 545_1 bolometer<br />
</gallery><br />
<br />
<br />
The full series of plots is here:<br />
[[Media:StationarityInROIoutput2.pdf|Standard deviation of rmsig TOIs at the ring level]]<br />
<br />
Note the presence for 3 bolometers of a two-level noise system. No correction can be done for that effect. See one example here:<br />
<br />
[[File:23_353_TwoLevel.jpg|300px|An example of two-level noise system is seen in bolometer 23_353_3a]]<br />
<br />
<br />
An example of the higher order statistics which are used to unveil rings affected by RTS problems.<br />
<br />
[[File:HFI_4_4_2_RTSexample13.jpg|400px|Example of RTS detection]]<br />
<br />
=== Discarded rings ===<br />
<br />
Some rings are discarded (flagged) from further use (beam making, map making) by using ring statistics (see above and the HFI data processing paper REF). For each statistic, we compare each ring value to the ring values averaged (RVA) over a large selection of rings (between 3000 and 21700). We also define the modified standard deviation (MSD) of a ring quantity as the standard deviation of that quantity over the rings that deviate by less than five nominal standard deviations. This truncation is necessary to be robust against extreme deviant rings.<br />
<br />
More specifically, a ring is discarded if it matches one of the following criteria:<br />
<br />
* the | mean-median | deviates from the RVA by more than fifteen times the MSD.<br />
* the standard deviation deviates from the RVA by more than -5 times the MSD (all cases corresponding to almost empty rings) or +15 times the MSD.<br />
* the Kolmogorov-Smirnov test deviates from the RVA by more than 15 times the MSD.<br />
* the ring duration is more than 90 min.<br />
* the ring is contaminated by RTS with an amplitude of more than one standard deviation of the noise. It concerns a few hundreds of rings for 3 bolometers (44_217_6b, 71_217_8a, 74_857_4). Notice that two bolometers are completely discarded for maps: 55_545_3 and 70_143_8, which present RTS at all time.<br />
<br />
For the three first criteria, a visual inspection of the rmsigTOI at each of the incriminated rings has shown that all these anomalous rings are due to either a drift, a small jump in the TOI trend or a sudden change of noise level, the origin of which is unknown at present.<br />
<br />
Once the list of discarded rings per bolometer is produced, a common list of discarded rings can be extracted for all bolometers (by using discarded rings for at least half the bolometers). Such rings correspond to identified phenomena, as can be seen on the following table.<br />
<br />
Furthermore, an isolated valid ring stuck between two common discarded rings becomes discarded as well.<br />
<br />
Table of '''common discarded rings''' of the nominal mission (rings 240-14723).<br />
<br />
{|table border=1 align=center cellpadding="3" cellspacing="0" style=text-align:center<br />
! '''Cause''' !! '''ring number'''<br />
|-<br />
!align="left"| manoeuvre<br />
|align="center"|304 1312 3590 3611 3642 3922 4949 6379 8456 11328<br />
|-<br />
! align="left"| Sorption Cooler Switchover <br />
|align="center"| 11149 11150 11151 11152 <br />
|-<br />
! align="left"| Rings too long<br />
| align="center"| 440 474 509 544 897 898 3589 13333 14627 14653 <br />
|-<br />
! align="left"| Star tracker switchover <br />
| align="center"| 14628 14654 <br />
|-<br />
! align="left"| Massive Glitch Event<br />
| align="center"| 7665 <br />
|-<br />
! align="left"| Solar Flare<br />
| align="center"| 11235 <br />
|}<br />
<br />
<br />
<br />
The following figure is a summary of the impact of the discarding process '''for each bolometer''' (the solid black line is the common discarded ring percentage). The outlier bolometers have some RTS problems as mentionned above.<br />
<br />
[[File:Fraction_discarded_rings_v53.png|300px|Fraction of discarded rings]]<br />
[[File:Fraction_discarded_time_v53.png|300px|Fraction of time due to discarded rings]]<br />
<br />
== Flag description ==<br />
<br />
==== Input flags ====<br />
<br />
These flags are used as inputs to the TOI processing<br />
<br />
* The point-source flag (PSflag):<br />
<br />
An earlier version of HFI point-source catalog is read back into a flag TOIs, at a given frequency. In practice, 5 sigma sources are masked within a radius of 1.3 FWHM (9, 7, 5, 5, 5, 5 arcmin at 100,143,217,353,545,857 GHz) TBC.<br />
<br />
* the galactic flag (Galflag):<br />
<br />
An earlier version of HFI maps is thresholded and apodized. The produced masks are read into flag TOIs. The retained threshold corresponds to a sky coverage of respectively 70, 70, 80, 90, 90, 90% at 100,143,217,353,545,857 GHz.<br />
<br />
* Solar System Object flag <br />
<br />
For the TOI flag, Mars, Jupiter, Saturn are flagged up to a radius of NbBeam= 2,3,3,4,4,4 times the fiducial SSO_FWHM with SSO_FWHM= 9, 7, 5, 5, 5, 5 arcmin at 100,143,217,353,545,857 GHz.<br />
<br />
As an input to planet mask for maps, Mars, Jupiter, Saturn are flagged with a radius computed as a coefficient depending on the planet (Factor_per_source) times NbBeam times SSO_FWHM, with Factor_per_source = 1.1, 2.25, 1.25 for Mars, Jupiter, Saturn respectively and NbBeam = 2.25, 4.25, 4.0, 5.0,.6.0, 8.0 at 100, 143, 217, 353, 545, 857 GHz. This flag is called SSOflag4map.<br />
<br />
A small trailing tail is added to the mask to take into account the non-deconvolution of the planet signal which has been replaced by background values. The width of that tail is 10 % of the main flag diameter. The number of samples which are additionnaly flagged are the Factor_per_Source times AddSNafter with 10, 30, 20, 20, 30, 40 samples at 100, 143, 217, 353, 545, 857 GHz.<br />
<br />
Uranus and Neptune together with detected asteroids are masked by HFI. They are masked at the TOI level using an exclusion radius of 1.5 SSO_FWHM. At 857 GHz, 24 asteroids have been detected with HFI : 1Ceres, 2Pallas, 3Juno, 4Vesta, 7Iris, 8Flora, 9Metis, 10Hygiea, 11Parthenope, 12Victoria, 13Egeria, 14Irene, 15Eunomia, 16Psyche, 18Melpomene, 19Fortuna, 20Massalia, 29Amphitrite, 41Daphne, 45Eugenia, 52Europa, 88Thisbe, 704Interamnia, 324Bamberga.<br />
<br />
<br />
=== Output flags ===<br />
<br />
A FlagTOIproc is produced by the TOIprocessing. It marks measurements which are not reliable for ny of the following reasons :<br />
* gap (no valid input data), enlarged by one sample on each side. It flags less than 0.00044% (resp. 0.00062%) of the nominal (resp. complete) mission. It is equivallen to less thena 3 (resp. 8) minutes of data.<br />
* glitch : samples where the signal from a cosmic ray hit dominates the sky signal at more than 3 $\sigma$ are discarded.<br />
* jump as 100 samples are flagged around the computed position of the jump to take into account the error on this position.<br />
<br />
With and without point-sources, identical to a and b PSBs.<br />
<br />
<br />
Flags produced for the map making :<br />
* UnstablePointing Flag OR FlagTOIproc OR SSOflag4map OR SSOflag seen<br />
* FlagTOIproc = glitch OR jump OR flag thermal template <br />
* PSB flag glitch = flag glitch A OR flag glitch B<br />
<br />
Flag used in the destriping: same + point-source flag<br />
<br />
<br />
<br />
<br />
[[Category:Data processing]]</div>
Fdesert
https://wiki.cosmos.esa.int/planckpla2015/index.php?title=TOI_processing&diff=3652
TOI processing
2013-01-29T09:09:04Z
<p>Fdesert: </p>
<hr />
<div>This Section is kept short as this Planck release does not contain TOIs. The main information in the HFI processing paper is not duplicated here.<br />
<br />
==Overview==<br />
<br />
We describe here the how the TOIs are processed in order to be used for map production. We do not repeat the general features of the pipeline which are given in the HFI Data Processing article (REF). Here we give complementary explanations on some details. The TOI of each bolometer is processed independently of the other bolometers, so as to keep the noise properties as uncorrelated as possible. The processing involves modifying the TOI itself for what concerns the conversion to absorbed power and the correction of glitch tails. It also adds a flag TOI that masks the TOI samples that are not to be projected on maps for various reasons.<br />
<br />
<br />
=== Acronyms and definitions ===<br />
<br />
* 4K lines: EMI/EMC influence of the 4K cooler mechanical motion on the bolometer readout electronics.<br />
* ADC : analog to digital converter<br />
* IMO : instrument model<br />
* Jump : sudden change of the baseline level inside a ring<br />
* LFER : low frequency excess response<br />
* PBR : phase binned ring<br />
* RIMO : reduced IMO<br />
* Ring : pointing period<br />
* ROI : ring ordered information<br />
* RTS : randomn telegraphic signal<br />
* SSO : solar system object<br />
* TOI : time ordered information<br />
<br />
== Input TOI ==<br />
<br />
The input TOI consists in the AC modulated voltage output of the readout of each bolometer. The input has previously been decompressed, and converted from internal digital units to voltage via a constant factor. The TOI has a regular sampling at the acquisition frequency of facq=180.373700+-0.000050 Hz. There are almost no missing data in the TOIs, except for few hundred samples of 545 and 857GHz TOIs which are lost in the on-board compression due to saturation on the Galactic Center crossings. <br />
<br />
== General Pipeline Structure ==<br />
<br />
The figure on the right shows how the initial TOI is transformed and how flags are produced:<br />
<br />
[[File:HFI_4_4_2_ToiProc_ExplS.png|thumb|600px|A schematic of the TOI processing pipeline]]<br />
<br />
<br />
== Output TOIs and products ==<br />
<br />
A TOI of clean calibrated samples (ccTOI) and a combined flag TOI (fTOI) are the outputs of the processing. The ccTOI is calibrated so as to represent the instantaneous power absorbed by the detector up to a constant (which will be determined by the map-making destriper). It is worth mentioning how the ccTOI is changed with respect to the input TOI, beyond the harmless constant conversion factor from voltage to absorbed power. The demodulation stage allows to get the demodulated bolometer voltage. The non-linearity correction is a second-order polynomial correction based on the physical but static bolometer model. In order to avoid too much masking after glitches, a glitch tail is subtracted after an occurrence of a glitch in the TOI. The 4K cooler lines noise is substituted at a series of 9 single temporal frequencies. Finally, the temporal response of the bolometer is deconvolved. This affects mostly the high-temporal frequency part of the TOI, although a small but significant low frequency (the long time response) tail is corrected too. Although flagged samples are not projected, their value influences the valid samples somehow. Hence interpolation procedures introduce some indirect modifications of the TOI.<br />
The flag TOI is a combination a dozen flags with an OR logic. Only unflagged data are projected. The exhaustive list of flags is given here: CompressionError, NoData, SSO, UnstablePointing, Glitch, BoloPlateFluctuation, RTS, Jump, PSBab. A complete qualification of the data is obtained at the ring level. If the TOI shows an anomalous behaviour on a given ring, this ring is discarded from projection. A special production of TOIs is also made as an input to the beam analysis with Mars, Jupiter and Saturn.<br />
<br />
<br />
== Examples of clean TOIs ==<br />
<gallery widths=300px perrow=3><br />
File:10_143_5_PBR_10800RING_LFER4_JC.jpg|Phase-binned signal (PBR) for 6 consecutive rings from 4000 to 4005<br />
File:10_143_5_rcircles_LFER4_JC.jpg| Valid signal (up) and noise (bottom) for 2 consecutive circles of the ring 4000<br />
File:10_143_5_fft_LFER4_JC.jpg| Fourier power spectral density of the first 30 minutes of the ring 4005 with signal before (pink) and after (black) deconvolution (top) and noise (bottom)<br />
File:14_545_1_PBR_10800RING_LFER4_JC.jpg|Phase-binned signal (PBR) for 6 consecutive rings from 4000 to 4005<br />
File:14_545_1_rcircles_LFER4_JC.jpg| Valid signal (up) and noise (bottom) for 2 consecutive circles of the ring 4000<br />
File:14_545_1_fft_LFER4_JC.jpg| Fourier power spectral density of the first 30 minutes of the ring 4005 with signal before (pink) and after (black) deconvolution (top) and noise (bottom)<br />
</gallery><br />
<br />
[[Media:check_ring.pdf|Samples of PBR, TOIs, and PSDs of all detectors are shown in this file]]<br />
<br />
== Trends in the output processing variables ==<br />
<br />
Here we intend to show the trend of the systematic effects that are dealt with in the TOI processing. The full impact of each of them is analyzed in [[HFI-Validation]].<br />
<br />
=== ADC baseline ===<br />
The following figure shows the ADC baseline which is used prior to demodulation (a constant offset is removed for clarity). This baseline is obtained by smoothing on an hour block average the undemodulated TOI on unflagged samples.<br />
<br />
[[File:baseline.jpg|400px|ADC baseline for all bolometers]]<br />
<br />
=== Glitch statistics ===<br />
The glitch rate per channel is shown in this figure. For details, see copap. <br />
<br />
[[File:figIntermPaperGR.jpg|400px|Glitch rate evolution]]<br />
<br />
The percentage of flagged data (mostly due to Cosmic Rays) at the ring level is shown in these examples. No smoothing was applied. Only valid rings are shown.<br />
<gallery widths=450px perrow=2 caption="Percentage of flagged data"><br />
File:group10_143_5.jpg | 143 GHz bolometers<br />
File:group14_545_1.jpg | 545 GHz bolometers<br />
</gallery><br />
[[Media:PercentUnvalid2.pdf|The complete set of plots is here]]<br />
<br />
=== Thermal template for decorrelation ===<br />
[[File:T90.jpg|400px|Thermal template used for the decorrelation from bolometer plate temperature fluctuations]]<br />
<br />
A simple linear decorrelation is performed using the 2 dark bolometers as a proxy of the bolometer plate temperature. Coupling coefficients were measured during the CPV phase.<br />
<br />
=== 4K cooler lines variability ===<br />
The amplitude of the nine 4K cooler lines in aW at 10, 20, 30, 40, 50, 60, 70, 80 and 17 Hz is shown for 2 bolometers in the following figures. The trend is smoothed over 31 ring values after having discarded measurements done at a ring which is discarded for all bolometers.<br />
<gallery widths=450px perrow=2 caption="Amplitude of the nine 4K cooler lines"><br />
File:10_143_5_4Klines.jpg | 143_5 bolometer<br />
File:14_545_1_4Klines.jpg | 545_1 bolometer<br />
</gallery><br />
[[Media:Lines4K.pdf|The 4K cooler line coefficients of all bolometers are shown in this file]]<br />
<br />
The 4K cooler lines project onto the maps only for a limited fraction of rings, the so-called resonant rings. This is graphically shown in the following figure. For each ring (stable pointing period), the spin rate is very stable at about 1 rpm. From one ring to another, the spin frequency (shown as diamonds) changes around that value. The sky signal is imprinted at the corresponding spin frequency and its 5400 (60x90) harmonics. If one of the nine 4K cooler lines happens to coincide with one of the spin frequency harmonics, it will project a sine-wave systematic on the maps. This is what we call a resonant ring. The horizontal coloured bars show the zone of influence of a particular 4K line, when folded around 16.666 mHz. When the spin frequency hits one of these zones, we have a so-called resonant ring. The 4K line coefficient is interpolated for this ring and an estimate of the systematic effect is subtracted from the TOI.<br />
<br />
[[File:spinfreq1.jpg|600px|Spin frequency and 4K line zones of influence]]<br />
<br />
=== jump correction ===<br />
A piecewise constant value is removed to the TOI if a jump is detected. See a jump example in this figure:<br />
<br />
[[File:jump_exe.png|200px|An example of a jump seen on the rmsigTOI]]<br />
<br />
The number of jumps per day (all bolometers included) is shown in this figure:<br />
<br />
[[File:jumps_per_day.png|200px|Evolution of jump number during the mission]]<br />
<br />
The jumps are uncorrelated from bolometer to bolometer. The total number of jumps detected in the nominal and full mission is shown here:<br />
<br />
[[File:jumps_per_bolometer.png|200px|Number of jumps per bolometer]]<br />
<br />
<br />
=== Trends in noise and signal ===<br />
<gallery widths=450px perrow=2 caption="Signal (top) and noise (bottom) smoothed at 1 minute. All values falling in a discarded ring are not plotted."><br />
File:10_143_5_smooth_Watt.jpg | 143_5 bolometer<br />
File:14_545_1_smooth_Watt.jpg | 545_1 bolometer<br />
</gallery><br />
<br />
[[Media:check_smooth.pdf|The smooth TOIs of all detectors are shown in this file]]<br />
<br />
=== Noise stationarity ===<br />
This is not the final version but gives a good idea of power spectra at the detector level of rmsigTOIs.<br />
[[Media:v53_meanSpectra_bySurvey.pdf|All PSDs can be seen in this file]]<br />
<br />
The standard deviation per ring corrected by ring duration bias is given here, one per bolometer using only valid rings. No smoothing is applied (except a 31-point smoothing for the 545<br />
and 857 GHz channels) but values for rings discarded for all bolometers (see below) are not used. The standard deviation is computed on samples valid for map-making which are also not affected by the Galaxy or the point-sources using the usual flags. Two examples are given here.<br />
<gallery widths=450px perrow=2 caption="Standard deviation per ring"><br />
File:stddev_group10_143_5.jpg | 143_5 bolometer<br />
File:stddev_group_14_545_1.jpg | 545_1 bolometer<br />
</gallery><br />
<br />
<br />
The full series of plots is here:<br />
[[Media:StationarityInROIoutput2.pdf|Standard deviation of rmsig TOIs at the ring level]]<br />
<br />
Note the presence for 3 bolometers of a two-level noise system. No correction can be done for that effect. See one example here:<br />
<br />
[[File:23_353_TwoLevel.jpg|300px|An example of two-level noise system is seen in bolometer 23_353_3a]]<br />
<br />
<br />
An example of the higher order statistics which are used to unveil rings affected by RTS problems.<br />
<br />
[[File:HFI_4_4_2_RTSexample13.jpg|400px|Example of RTS detection]]<br />
<br />
=== Discarded rings ===<br />
<br />
Some rings are discarded (flagged) from further use (beam making, map making) by using ring statistics (see above and the HFI data processing paper REF). For each statistic, we compare each ring value to the ring values averaged (RVA) over a large selection of rings (between 3000 and 21700). We also define the modified standard deviation (MSD) of a ring quantity as the standard deviation of that quantity over the rings that deviate by less than five nominal standard deviations. This truncation is necessary to be robust against extreme deviant rings.<br />
<br />
More specifically, a ring is discarded if it matches one of the following criteria:<br />
<br />
* the | mean-median | deviates from the RVA by more than fifteen times the MSD.<br />
* the standard deviation deviates from the RVA by more than -5 times the MSD (all cases corresponding to almost empty rings) or +15 times the MSD.<br />
* the Kolmogorov-Smirnov test deviates from the RVA by more than 15 times the MSD.<br />
* the ring duration is more than 90 min.<br />
* the ring is contaminated by RTS with an amplitude of more than one standard deviation of the noise. It concerns a few hundreds of rings for 3 bolometers (44_217_6b, 71_217_8a, 74_857_4). Notice that two bolometers are completely discarded for maps: 55_545_3 and 70_143_8, which present RTS at all time.<br />
<br />
For the three first criteria, a visual inspection of the rmsigTOI at each of the incriminated rings has shown that all these anomalous rings are due to either a drift, a small jump in the TOI trend or a sudden change of noise level, the origin of which is unknown at present.<br />
<br />
Once the list of discarded rings per bolometer is produced, a common list of discarded rings can be extracted for all bolometers (by using discarded rings for at least half the bolometers). Such rings correspond to identified phenomena, as can be seen on the following table.<br />
<br />
Furthermore, an isolated valid ring stuck between two common discarded rings becomes discarded as well.<br />
<br />
Table of '''common discarded rings''' of the nominal mission (rings 240-14723).<br />
<br />
{|table border=1 align=center cellpadding="3" cellspacing="0" style=text-align:center<br />
! '''Cause''' !! '''ring number'''<br />
|-<br />
!align="left"| manoeuvre<br />
|align="center"|304 1312 3590 3611 3642 3922 4949 6379 8456 11328<br />
|-<br />
! align="left"| Sorption Cooler Switchover <br />
|align="center"| 11149 11150 11151 11152 <br />
|-<br />
! align="left"| Rings too long<br />
| align="center"| 440 474 509 544 897 898 3589 13333 14627 14653 <br />
|-<br />
! align="left"| Star tracker switchover <br />
| align="center"| 14628 14654 <br />
|-<br />
! align="left"| Massive Glitch Event<br />
| align="center"| 7665 <br />
|-<br />
! align="left"| Solar Flare<br />
| align="center"| 11235 <br />
|}<br />
<br />
<br />
<br />
The following figure is a summary of the impact of the discarding process '''for each bolometer''' (the solid black line is the common discarded ring percentage). The outlier bolometers have some RTS problems as mentionned above.<br />
<br />
[[File:Fraction_discarded_rings_v53.png|300px|Fraction of discarded rings]]<br />
[[File:Fraction_discarded_time_v53.png|300px|Fraction of time due to discarded rings]]<br />
<br />
== Flag description ==<br />
<br />
==== Input flags ====<br />
<br />
These flags are used as inputs to the TOI processing<br />
<br />
* The point-source flag (PSflag):<br />
<br />
An earlier version of HFI point-source catalog is read back into a flag TOIs, at a given frequency. In practice, 5 sigma sources are masked within a radius of 1.3 FWHM (9, 7, 5, 5, 5, 5 arcmin at 100,143,217,353,545,857 GHz) TBC.<br />
<br />
* the galactic flag (Galflag):<br />
<br />
An earlier version of HFI maps is thresholded and apodized. The produced masks are read into flag TOIs. The retained threshold corresponds to a sky coverage of respectively 70, 70, 80, 90, 90, 90% at 100,143,217,353,545,857 GHz.<br />
<br />
* Solar System Object flag <br />
<br />
For the TOI flag, Mars, Jupiter, Saturn are flagged up to a radius of NbBeam= 2,3,3,4,4,4 times the fiducial SSO_FWHM with SSO_FWHM= 9, 7, 5, 5, 5, 5 arcmin at 100,143,217,353,545,857 GHz.<br />
<br />
As an input to planet mask for maps, Mars, Jupiter, Saturn are flagged with a radius computed as a coefficient depending on the planet (Factor_per_source) times NbBeam times SSO_FWHM, with Factor_per_source = 1.1, 2.25, 1.25 for Mars, Jupiter, Saturn respectively and NbBeam = 2.25, 4.25, 4.0, 5.0,.6.0, 8.0 at 100, 143, 217, 353, 545, 857 GHz. This flag is called SSOflag4map.<br />
<br />
A small trailing tail is added to the mask to take into account the non-deconvolution of the planet signal which has been replaced by background values. The width of that tail is 10 % of the main flag diameter. The number of samples which are additionnaly flagged are the Factor_per_Source times AddSNafter with 10, 30, 20, 20, 30, 40 samples at 100, 143, 217, 353, 545, 857 GHz.<br />
<br />
Uranus and Neptune together with detected asteroids are masked by HFI. They are masked at the TOI level using an exclusion radius of 1.5 SSO_FWHM. At 857 GHz, 24 asteroids have been detected with HFI : 1Ceres, 2Pallas, 3Juno, 4Vesta, 7Iris, 8Flora, 9Metis, 10Hygiea, 11Parthenope, 12Victoria, 13Egeria, 14Irene, 15Eunomia, 16Psyche, 18Melpomene, 19Fortuna, 20Massalia, 29Amphitrite, 41Daphne, 45Eugenia, 52Europa, 88Thisbe, 704Interamnia, 324Bamberga.<br />
<br />
<br />
=== Output flags ===<br />
<br />
A FlagTOIproc is produced by the TOIprocessing. It marks measurements which are not reliable for ny of the following reasons :<br />
* gap (no valid input data), enlarged by one sample on each side. It flags less than 0.00044% (resp. 0.00062%) of the nominal (resp. complete) mission. It is equivallen to less thena 3 (resp. 8) minutes of data.<br />
* glitch : samples where the signal from a cosmic ray hit dominates the sky signal at more than 3 $\sigma$ are discarded.<br />
* jump as 100 samples are flagged around the computed position of the jump to take into account the error on this position.<br />
<br />
With and without point-sources, identical to a and b PSBs.<br />
<br />
<br />
Flags produced for the map making :<br />
* UnstablePointing Flag OR FlagTOIproc OR SSOflag4map OR SSOflag seen<br />
* FlagTOIproc = glitch OR jump OR flag thermal template <br />
* PSB flag glitch = flag glitch A OR flag glitch B<br />
<br />
Flag used in the destriping: same + point-source flag<br />
<br />
<br />
<br />
<br />
[[Category:Data processing]]</div>
Fdesert
https://wiki.cosmos.esa.int/planckpla2015/index.php?title=File:Spinfreq1.jpg&diff=3651
File:Spinfreq1.jpg
2013-01-29T09:08:00Z
<p>Fdesert: </p>
<hr />
<div></div>
Fdesert
https://wiki.cosmos.esa.int/planckpla2015/index.php?title=TOI_processing&diff=3650
TOI processing
2013-01-29T09:07:27Z
<p>Fdesert: </p>
<hr />
<div>This Section is kept short as this Planck release does not contain TOIs. The main information in the HFI processing paper is not duplicated here.<br />
<br />
==Overview==<br />
<br />
We describe here the how the TOIs are processed in order to be used for map production. We do not repeat the general features of the pipeline which are given in the HFI Data Processing article (REF). Here we give complementary explanations on some details. The TOI of each bolometer is processed independently of the other bolometers, so as to keep the noise properties as uncorrelated as possible. The processing involves modifying the TOI itself for what concerns the conversion to absorbed power and the correction of glitch tails. It also adds a flag TOI that masks the TOI samples that are not to be projected on maps for various reasons.<br />
<br />
<br />
=== Acronyms and definitions ===<br />
<br />
* 4K lines: EMI/EMC influence of the 4K cooler mechanical motion on the bolometer readout electronics.<br />
* ADC : analog to digital converter<br />
* IMO : instrument model<br />
* Jump : sudden change of the baseline level inside a ring<br />
* LFER : low frequency excess response<br />
* PBR : phase binned ring<br />
* RIMO : reduced IMO<br />
* Ring : pointing period<br />
* ROI : ring ordered information<br />
* RTS : randomn telegraphic signal<br />
* SSO : solar system object<br />
* TOI : time ordered information<br />
<br />
== Input TOI ==<br />
<br />
The input TOI consists in the AC modulated voltage output of the readout of each bolometer. The input has previously been decompressed, and converted from internal digital units to voltage via a constant factor. The TOI has a regular sampling at the acquisition frequency of facq=180.373700+-0.000050 Hz. There are almost no missing data in the TOIs, except for few hundred samples of 545 and 857GHz TOIs which are lost in the on-board compression due to saturation on the Galactic Center crossings. <br />
<br />
== General Pipeline Structure ==<br />
<br />
The figure on the right shows how the initial TOI is transformed and how flags are produced:<br />
<br />
[[File:HFI_4_4_2_ToiProc_ExplS.png|thumb|600px|A schematic of the TOI processing pipeline]]<br />
<br />
<br />
== Output TOIs and products ==<br />
<br />
A TOI of clean calibrated samples (ccTOI) and a combined flag TOI (fTOI) are the outputs of the processing. The ccTOI is calibrated so as to represent the instantaneous power absorbed by the detector up to a constant (which will be determined by the map-making destriper). It is worth mentioning how the ccTOI is changed with respect to the input TOI, beyond the harmless constant conversion factor from voltage to absorbed power. The demodulation stage allows to get the demodulated bolometer voltage. The non-linearity correction is a second-order polynomial correction based on the physical but static bolometer model. In order to avoid too much masking after glitches, a glitch tail is subtracted after an occurrence of a glitch in the TOI. The 4K cooler lines noise is substituted at a series of 9 single temporal frequencies. Finally, the temporal response of the bolometer is deconvolved. This affects mostly the high-temporal frequency part of the TOI, although a small but significant low frequency (the long time response) tail is corrected too. Although flagged samples are not projected, their value influences the valid samples somehow. Hence interpolation procedures introduce some indirect modifications of the TOI.<br />
The flag TOI is a combination a dozen flags with an OR logic. Only unflagged data are projected. The exhaustive list of flags is given here: CompressionError, NoData, SSO, UnstablePointing, Glitch, BoloPlateFluctuation, RTS, Jump, PSBab. A complete qualification of the data is obtained at the ring level. If the TOI shows an anomalous behaviour on a given ring, this ring is discarded from projection. A special production of TOIs is also made as an input to the beam analysis with Mars, Jupiter and Saturn.<br />
<br />
<br />
== Examples of clean TOIs ==<br />
<gallery widths=300px perrow=3><br />
File:10_143_5_PBR_10800RING_LFER4_JC.jpg|Phase-binned signal (PBR) for 6 consecutive rings from 4000 to 4005<br />
File:10_143_5_rcircles_LFER4_JC.jpg| Valid signal (up) and noise (bottom) for 2 consecutive circles of the ring 4000<br />
File:10_143_5_fft_LFER4_JC.jpg| Fourier power spectral density of the first 30 minutes of the ring 4005 with signal before (pink) and after (black) deconvolution (top) and noise (bottom)<br />
File:14_545_1_PBR_10800RING_LFER4_JC.jpg|Phase-binned signal (PBR) for 6 consecutive rings from 4000 to 4005<br />
File:14_545_1_rcircles_LFER4_JC.jpg| Valid signal (up) and noise (bottom) for 2 consecutive circles of the ring 4000<br />
File:14_545_1_fft_LFER4_JC.jpg| Fourier power spectral density of the first 30 minutes of the ring 4005 with signal before (pink) and after (black) deconvolution (top) and noise (bottom)<br />
</gallery><br />
<br />
[[Media:check_ring.pdf|Samples of PBR, TOIs, and PSDs of all detectors are shown in this file]]<br />
<br />
== Trends in the output processing variables ==<br />
<br />
Here we intend to show the trend of the systematic effects that are dealt with in the TOI processing. The full impact of each of them is analyzed in [[HFI-Validation]].<br />
<br />
=== ADC baseline ===<br />
The following figure shows the ADC baseline which is used prior to demodulation (a constant offset is removed for clarity). This baseline is obtained by smoothing on an hour block average the undemodulated TOI on unflagged samples.<br />
<br />
[[File:baseline.jpg|400px|ADC baseline for all bolometers]]<br />
<br />
=== Glitch statistics ===<br />
The glitch rate per channel is shown in this figure. For details, see copap. <br />
<br />
[[File:figIntermPaperGR.jpg|400px|Glitch rate evolution]]<br />
<br />
The percentage of flagged data (mostly due to Cosmic Rays) at the ring level is shown in these examples. No smoothing was applied. Only valid rings are shown.<br />
<gallery widths=450px perrow=2 caption="Percentage of flagged data"><br />
File:group10_143_5.jpg | 143 GHz bolometers<br />
File:group14_545_1.jpg | 545 GHz bolometers<br />
</gallery><br />
[[Media:PercentUnvalid2.pdf|The complete set of plots is here]]<br />
<br />
=== Thermal template for decorrelation ===<br />
[[File:T90.jpg|400px|Thermal template used for the decorrelation from bolometer plate temperature fluctuations]]<br />
<br />
A simple linear decorrelation is performed using the 2 dark bolometers as a proxy of the bolometer plate temperature. Coupling coefficients were measured during the CPV phase.<br />
<br />
=== 4K cooler lines variability ===<br />
The amplitude of the nine 4K cooler lines in aW at 10, 20, 30, 40, 50, 60, 70, 80 and 17 Hz is shown for 2 bolometers in the following figures. The trend is smoothed over 31 ring values after having discarded measurements done at a ring which is discarded for all bolometers.<br />
<gallery widths=450px perrow=2 caption="Amplitude of the nine 4K cooler lines"><br />
File:10_143_5_4Klines.jpg | 143_5 bolometer<br />
File:14_545_1_4Klines.jpg | 545_1 bolometer<br />
</gallery><br />
[[Media:Lines4K.pdf|The 4K cooler line coefficients of all bolometers are shown in this file]]<br />
<br />
The 4K cooler lines project onto the maps only for a limited fraction of rings, the so-called resonant rings. This is graphically shown in the following figure. For each ring (stable pointing period), the spin rate is very stable at about 1 rpm. From one ring to another, the spin frequency (shown as diamonds) changes around that value. The sky signal is imprinted at the corresponding spin frequency and its 5400 (60x90) harmonics. If one of the nine 4K cooler lines happens to coincide with one of the spin frequency harmonics, it will project a sine-wave systematic on the maps. This is what we call a resonant ring. The horizontal coloured bars show the zone of influence of a particular 4K line, when folded around 16.666 mHz. When the spin frequency hits one of these zones, we have a so-called resonant ring. The 4K line coefficient is interpolated for this ring and an estimate of the systematic effect is subtracted from the TOI.<br />
[[File:spinfreq1.jpg|600px|Spin frequency and 4K line zones of influence]]<br />
<br />
=== jump correction ===<br />
A piecewise constant value is removed to the TOI if a jump is detected. See a jump example in this figure:<br />
<br />
[[File:jump_exe.png|200px|An example of a jump seen on the rmsigTOI]]<br />
<br />
The number of jumps per day (all bolometers included) is shown in this figure:<br />
<br />
[[File:jumps_per_day.png|200px|Evolution of jump number during the mission]]<br />
<br />
The jumps are uncorrelated from bolometer to bolometer. The total number of jumps detected in the nominal and full mission is shown here:<br />
<br />
[[File:jumps_per_bolometer.png|200px|Number of jumps per bolometer]]<br />
<br />
<br />
=== Trends in noise and signal ===<br />
<gallery widths=450px perrow=2 caption="Signal (top) and noise (bottom) smoothed at 1 minute. All values falling in a discarded ring are not plotted."><br />
File:10_143_5_smooth_Watt.jpg | 143_5 bolometer<br />
File:14_545_1_smooth_Watt.jpg | 545_1 bolometer<br />
</gallery><br />
<br />
[[Media:check_smooth.pdf|The smooth TOIs of all detectors are shown in this file]]<br />
<br />
=== Noise stationarity ===<br />
This is not the final version but gives a good idea of power spectra at the detector level of rmsigTOIs.<br />
[[Media:v53_meanSpectra_bySurvey.pdf|All PSDs can be seen in this file]]<br />
<br />
The standard deviation per ring corrected by ring duration bias is given here, one per bolometer using only valid rings. No smoothing is applied (except a 31-point smoothing for the 545<br />
and 857 GHz channels) but values for rings discarded for all bolometers (see below) are not used. The standard deviation is computed on samples valid for map-making which are also not affected by the Galaxy or the point-sources using the usual flags. Two examples are given here.<br />
<gallery widths=450px perrow=2 caption="Standard deviation per ring"><br />
File:stddev_group10_143_5.jpg | 143_5 bolometer<br />
File:stddev_group_14_545_1.jpg | 545_1 bolometer<br />
</gallery><br />
<br />
<br />
The full series of plots is here:<br />
[[Media:StationarityInROIoutput2.pdf|Standard deviation of rmsig TOIs at the ring level]]<br />
<br />
Note the presence for 3 bolometers of a two-level noise system. No correction can be done for that effect. See one example here:<br />
<br />
[[File:23_353_TwoLevel.jpg|300px|An example of two-level noise system is seen in bolometer 23_353_3a]]<br />
<br />
<br />
An example of the higher order statistics which are used to unveil rings affected by RTS problems.<br />
<br />
[[File:HFI_4_4_2_RTSexample13.jpg|400px|Example of RTS detection]]<br />
<br />
=== Discarded rings ===<br />
<br />
Some rings are discarded (flagged) from further use (beam making, map making) by using ring statistics (see above and the HFI data processing paper REF). For each statistic, we compare each ring value to the ring values averaged (RVA) over a large selection of rings (between 3000 and 21700). We also define the modified standard deviation (MSD) of a ring quantity as the standard deviation of that quantity over the rings that deviate by less than five nominal standard deviations. This truncation is necessary to be robust against extreme deviant rings.<br />
<br />
More specifically, a ring is discarded if it matches one of the following criteria:<br />
<br />
* the | mean-median | deviates from the RVA by more than fifteen times the MSD.<br />
* the standard deviation deviates from the RVA by more than -5 times the MSD (all cases corresponding to almost empty rings) or +15 times the MSD.<br />
* the Kolmogorov-Smirnov test deviates from the RVA by more than 15 times the MSD.<br />
* the ring duration is more than 90 min.<br />
* the ring is contaminated by RTS with an amplitude of more than one standard deviation of the noise. It concerns a few hundreds of rings for 3 bolometers (44_217_6b, 71_217_8a, 74_857_4). Notice that two bolometers are completely discarded for maps: 55_545_3 and 70_143_8, which present RTS at all time.<br />
<br />
For the three first criteria, a visual inspection of the rmsigTOI at each of the incriminated rings has shown that all these anomalous rings are due to either a drift, a small jump in the TOI trend or a sudden change of noise level, the origin of which is unknown at present.<br />
<br />
Once the list of discarded rings per bolometer is produced, a common list of discarded rings can be extracted for all bolometers (by using discarded rings for at least half the bolometers). Such rings correspond to identified phenomena, as can be seen on the following table.<br />
<br />
Furthermore, an isolated valid ring stuck between two common discarded rings becomes discarded as well.<br />
<br />
Table of '''common discarded rings''' of the nominal mission (rings 240-14723).<br />
<br />
{|table border=1 align=center cellpadding="3" cellspacing="0" style=text-align:center<br />
! '''Cause''' !! '''ring number'''<br />
|-<br />
!align="left"| manoeuvre<br />
|align="center"|304 1312 3590 3611 3642 3922 4949 6379 8456 11328<br />
|-<br />
! align="left"| Sorption Cooler Switchover <br />
|align="center"| 11149 11150 11151 11152 <br />
|-<br />
! align="left"| Rings too long<br />
| align="center"| 440 474 509 544 897 898 3589 13333 14627 14653 <br />
|-<br />
! align="left"| Star tracker switchover <br />
| align="center"| 14628 14654 <br />
|-<br />
! align="left"| Massive Glitch Event<br />
| align="center"| 7665 <br />
|-<br />
! align="left"| Solar Flare<br />
| align="center"| 11235 <br />
|}<br />
<br />
<br />
<br />
The following figure is a summary of the impact of the discarding process '''for each bolometer''' (the solid black line is the common discarded ring percentage). The outlier bolometers have some RTS problems as mentionned above.<br />
<br />
[[File:Fraction_discarded_rings_v53.png|300px|Fraction of discarded rings]]<br />
[[File:Fraction_discarded_time_v53.png|300px|Fraction of time due to discarded rings]]<br />
<br />
== Flag description ==<br />
<br />
==== Input flags ====<br />
<br />
These flags are used as inputs to the TOI processing<br />
<br />
* The point-source flag (PSflag):<br />
<br />
An earlier version of HFI point-source catalog is read back into a flag TOIs, at a given frequency. In practice, 5 sigma sources are masked within a radius of 1.3 FWHM (9, 7, 5, 5, 5, 5 arcmin at 100,143,217,353,545,857 GHz) TBC.<br />
<br />
* the galactic flag (Galflag):<br />
<br />
An earlier version of HFI maps is thresholded and apodized. The produced masks are read into flag TOIs. The retained threshold corresponds to a sky coverage of respectively 70, 70, 80, 90, 90, 90% at 100,143,217,353,545,857 GHz.<br />
<br />
* Solar System Object flag <br />
<br />
For the TOI flag, Mars, Jupiter, Saturn are flagged up to a radius of NbBeam= 2,3,3,4,4,4 times the fiducial SSO_FWHM with SSO_FWHM= 9, 7, 5, 5, 5, 5 arcmin at 100,143,217,353,545,857 GHz.<br />
<br />
As an input to planet mask for maps, Mars, Jupiter, Saturn are flagged with a radius computed as a coefficient depending on the planet (Factor_per_source) times NbBeam times SSO_FWHM, with Factor_per_source = 1.1, 2.25, 1.25 for Mars, Jupiter, Saturn respectively and NbBeam = 2.25, 4.25, 4.0, 5.0,.6.0, 8.0 at 100, 143, 217, 353, 545, 857 GHz. This flag is called SSOflag4map.<br />
<br />
A small trailing tail is added to the mask to take into account the non-deconvolution of the planet signal which has been replaced by background values. The width of that tail is 10 % of the main flag diameter. The number of samples which are additionnaly flagged are the Factor_per_Source times AddSNafter with 10, 30, 20, 20, 30, 40 samples at 100, 143, 217, 353, 545, 857 GHz.<br />
<br />
Uranus and Neptune together with detected asteroids are masked by HFI. They are masked at the TOI level using an exclusion radius of 1.5 SSO_FWHM. At 857 GHz, 24 asteroids have been detected with HFI : 1Ceres, 2Pallas, 3Juno, 4Vesta, 7Iris, 8Flora, 9Metis, 10Hygiea, 11Parthenope, 12Victoria, 13Egeria, 14Irene, 15Eunomia, 16Psyche, 18Melpomene, 19Fortuna, 20Massalia, 29Amphitrite, 41Daphne, 45Eugenia, 52Europa, 88Thisbe, 704Interamnia, 324Bamberga.<br />
<br />
<br />
=== Output flags ===<br />
<br />
A FlagTOIproc is produced by the TOIprocessing. It marks measurements which are not reliable for ny of the following reasons :<br />
* gap (no valid input data), enlarged by one sample on each side. It flags less than 0.00044% (resp. 0.00062%) of the nominal (resp. complete) mission. It is equivallen to less thena 3 (resp. 8) minutes of data.<br />
* glitch : samples where the signal from a cosmic ray hit dominates the sky signal at more than 3 $\sigma$ are discarded.<br />
* jump as 100 samples are flagged around the computed position of the jump to take into account the error on this position.<br />
<br />
With and without point-sources, identical to a and b PSBs.<br />
<br />
<br />
Flags produced for the map making :<br />
* UnstablePointing Flag OR FlagTOIproc OR SSOflag4map OR SSOflag seen<br />
* FlagTOIproc = glitch OR jump OR flag thermal template <br />
* PSB flag glitch = flag glitch A OR flag glitch B<br />
<br />
Flag used in the destriping: same + point-source flag<br />
<br />
<br />
<br />
<br />
[[Category:Data processing]]</div>
Fdesert
https://wiki.cosmos.esa.int/planckpla2015/index.php?title=TOI_processing&diff=3533
TOI processing
2013-01-25T09:41:23Z
<p>Fdesert: </p>
<hr />
<div>This Section is kept short as this Planck release does not contain TOIs. The main information in the HFI processing paper is not duplicated here.<br />
<br />
==Overview==<br />
<br />
We describe here the how the TOIs are processed in order to be used for map production. We do not repeat the general features of the pipeline which are given in the HFI Data Processing article (REF). Here we give complementary explanations on some details. The TOI of each bolometer is processed independently of the other bolometers, so as to keep the noise properties as uncorrelated as possible. The processing involves modifying the TOI itself for what concerns the conversion to absorbed power and the correction of glitch tails. It also adds a flag TOI that masks the TOI samples that are not to be projected on maps for various reasons.<br />
<br />
<br />
=== Acronyms and definitions ===<br />
<br />
* 4K lines: EMI/EMC influence of the 4K cooler mechanical motion on the bolometer readout electronics.<br />
* ADC : analog to digital converter<br />
* IMO : instrument model<br />
* Jump : sudden change of the baseline level inside a ring<br />
* LFER : low frequency excess response<br />
* PBR : phase binned ring<br />
* RIMO : reduced IMO<br />
* Ring : pointing period<br />
* ROI : ring ordered information<br />
* RTS : randomn telegraphic signal<br />
* SSO : solar system object<br />
* TOI : time ordered information<br />
<br />
== Input TOI ==<br />
<br />
The input TOI consists in the AC modulated voltage output of the readout of each bolometer. The input has previously been decompressed, and converted from internal digital units to voltage via a constant factor. The TOI has a regular sampling at the acquisition frequency of facq=180.373700+-0.000050 Hz. There are almost no missing data in the TOIs, except for few hundred samples of 545 and 857GHz TOIs which are lost in the on-board compression due to saturation on the Galactic Center crossings. <br />
<br />
== General Pipeline Structure ==<br />
<br />
The figure on the right shows how the initial TOI is transformed and how flags are produced:<br />
<br />
[[File:HFI_4_4_2_ToiProc_ExplS.png|thumb|600px|A schematic of the TOI processing pipeline]]<br />
<br />
<br />
== Output TOIs and products ==<br />
<br />
A TOI of clean calibrated samples (ccTOI) and a combined flag TOI (fTOI) are the outputs of the processing. The ccTOI is calibrated so as to represent the instantaneous power absorbed by the detector up to a constant (which will be determined by the map-making destriper). It is worth mentioning how the ccTOI is changed with respect to the input TOI, beyond the harmless constant conversion factor from voltage to absorbed power. The demodulation stage allows to get the demodulated bolometer voltage. The non-linearity correction is a second-order polynomial correction based on the physical but static bolometer model. In order to avoid too much masking after glitches, a glitch tail is subtracted after an occurrence of a glitch in the TOI. The 4K cooler lines noise is substituted at a series of 9 single temporal frequencies. Finally, the temporal response of the bolometer is deconvolved. This affects mostly the high-temporal frequency part of the TOI, although a small but significant low frequency (the long time response) tail is corrected too. Although flagged samples are not projected, their value influences the valid samples somehow. Hence interpolation procedures introduce some indirect modifications of the TOI.<br />
The flag TOI is a combination a dozen flags with an OR logic. Only unflagged data are projected. The exhaustive list of flags is given here: CompressionError, NoData, SSO, UnstablePointing, Glitch, BoloPlateFluctuation, RTS, Jump, PSBab. A complete qualification of the data is obtained at the ring level. If the TOI shows an anomalous behaviour on a given ring, this ring is discarded from projection. A special production of TOIs is also made as an input to the beam analysis with Mars, Jupiter and Saturn.<br />
<br />
<br />
== Examples of clean TOIs ==<br />
<gallery widths=300px perrow=3><br />
File:10_143_5_PBR_10800RING_LFER4_JC.jpg|Phase-binned signal (PBR) for 6 consecutive rings from 4000 to 4005<br />
File:10_143_5_rcircles_LFER4_JC.jpg| Valid signal (up) and noise (bottom) for 2 consecutive circles of the ring 4000<br />
File:10_143_5_fft_LFER4_JC.jpg| Fourier power spectral density of the first 30 minutes of the ring 4005 with signal before (pink) and after (black) deconvolution (top) and noise (bottom)<br />
File:14_545_1_PBR_10800RING_LFER4_JC.jpg|Phase-binned signal (PBR) for 6 consecutive rings from 4000 to 4005<br />
File:14_545_1_rcircles_LFER4_JC.jpg| Valid signal (up) and noise (bottom) for 2 consecutive circles of the ring 4000<br />
File:14_545_1_fft_LFER4_JC.jpg| Fourier power spectral density of the first 30 minutes of the ring 4005 with signal before (pink) and after (black) deconvolution (top) and noise (bottom)<br />
</gallery><br />
<br />
[[Media:check_ring.pdf|Samples of PBR, TOIs, and PSDs of all detectors are shown in this file]]<br />
<br />
== Trends in the output processing variables ==<br />
<br />
Here we intend to show the trend of the systematic effects that are dealt with in the TOI processing. The full impact of each of them is analyzed in [[HFI-Validation]].<br />
<br />
=== ADC baseline ===<br />
The following figure shows the ADC baseline which is used prior to demodulation (a constant offset is removed for clarity). This baseline is obtained by smoothing on an hour block average the undemodulated TOI on unflagged samples.<br />
<br />
[[File:baseline.jpg|400px|ADC baseline for all bolometers]]<br />
<br />
=== Glitch statistics ===<br />
The glitch rate per channel is shown in this figure. For details, see copap. <br />
<br />
[[File:figIntermPaperGR.jpg|400px|Glitch rate evolution]]<br />
<br />
The percentage of flagged data (mostly due to Cosmic Rays) at the ring level is shown in these examples. No smoothing was applied. Only valid rings are shown.<br />
<gallery widths=450px perrow=2 caption="Percentage of flagged data"><br />
File:group10_143_5.jpg | 143 GHz bolometers<br />
File:group14_545_1.jpg | 545 GHz bolometers<br />
</gallery><br />
[[Media:PercentUnvalid2.pdf|The complete set of plots is here]]<br />
<br />
=== Thermal template for decorrelation ===<br />
[[File:T90.jpg|400px|Thermal template used for the decorrelation from bolometer plate temperature fluctuations]]<br />
<br />
A simple linear decorrelation is performed using the 2 dark bolometers as a proxy of the bolometer plate temperature. Coupling coefficients were measured during the CPV phase.<br />
<br />
=== 4K cooler lines variability ===<br />
The amplitude of the nine 4K cooler lines in aW at 10, 20, 30, 40, 50, 60, 70, 80 and 17 Hz is shown for 2 bolometers in the following figures. The trend is smoothed over 31 ring values after having discarded measurements done at a ring which is discarded for all bolometers.<br />
<gallery widths=450px perrow=2 caption="Amplitude of the nine 4K cooler lines"><br />
File:10_143_5_4Klines.jpg | 143_5 bolometer<br />
File:14_545_1_4Klines.jpg | 545_1 bolometer<br />
</gallery><br />
[[Media:Lines4K.pdf|The 4K cooler line coefficients of all bolometers are shown in this file]]<br />
<br />
=== jump correction ===<br />
A piecewise constant value is removed to the TOI if a jump is detected. See a jump example in this figure:<br />
<br />
[[File:jump_exe.png|200px|An example of a jump seen on the rmsigTOI]]<br />
<br />
The number of jumps per day (all bolometers included) is shown in this figure:<br />
<br />
[[File:jumps_per_day.png|200px|Evolution of jump number during the mission]]<br />
<br />
The jumps are uncorrelated from bolometer to bolometer. The total number of jumps detected in the nominal and full mission is shown here:<br />
<br />
[[File:jumps_per_bolometer.png|200px|Number of jumps per bolometer]]<br />
<br />
<br />
=== Trends in noise and signal ===<br />
<gallery widths=450px perrow=2 caption="Signal (top) and noise (bottom) smoothed at 1 minute. All values falling in a discarded ring are not plotted."><br />
File:10_143_5_smooth_Watt.jpg | 143_5 bolometer<br />
File:14_545_1_smooth_Watt.jpg | 545_1 bolometer<br />
</gallery><br />
<br />
[[Media:check_smooth.pdf|The smooth TOIs of all detectors are shown in this file]]<br />
<br />
=== Noise stationarity ===<br />
This is not the final version but gives a good idea of power spectra at the detector level of rmsigTOIs.<br />
[[Media:v53_meanSpectra_bySurvey.pdf|All PSDs can be seen in this file]]<br />
<br />
The standard deviation per ring corrected by ring duration bias is given here, one per bolometer using only valid rings. No smoothing is applied (except a 31-point smoothing for the 545<br />
and 857 GHz channels) but values for rings discarded for all bolometers (see below) are not used. The standard deviation is computed on samples valid for map-making which are also not affected by the Galaxy or the point-sources using the usual flags. Two examples are given here.<br />
<gallery widths=450px perrow=2 caption="Standard deviation per ring"><br />
File:stddev_group10_143_5.jpg | 143_5 bolometer<br />
File:stddev_group_14_545_1.jpg | 545_1 bolometer<br />
</gallery><br />
<br />
<br />
The full series of plots is here:<br />
[[Media:StationarityInROIoutput2.pdf|Standard deviation of rmsig TOIs at the ring level]]<br />
<br />
Note the presence for 3 bolometers of a two-level noise system. No correction can be done for that effect. See one example here:<br />
<br />
[[File:23_353_TwoLevel.jpg|300px|An example of two-level noise system is seen in bolometer 23_353_3a]]<br />
<br />
<br />
An example of the higher order statistics which are used to unveil rings affected by RTS problems.<br />
<br />
[[File:HFI_4_4_2_RTSexample13.jpg|400px|Example of RTS detection]]<br />
<br />
=== Discarded rings ===<br />
<br />
Some rings are discarded (flagged) from further use (beam making, map making) by using ring statistics (see above and the HFI data processing paper REF). For each statistic, we compare each ring value to the ring values averaged (RVA) over a large selection of rings (between 3000 and 21700). We also define the modified standard deviation (MSD) of a ring quantity as the standard deviation of that quantity over the rings that deviate by less than five nominal standard deviations. This truncation is necessary to be robust against extreme deviant rings.<br />
<br />
More specifically, a ring is discarded if it matches one of the following criteria:<br />
<br />
* the | mean-median | deviates from the RVA by more than fifteen times the MSD.<br />
* the standard deviation deviates from the RVA by more than -5 times the MSD (all cases corresponding to almost empty rings) or +15 times the MSD.<br />
* the Kolmogorov-Smirnov test deviates from the RVA by more than 15 times the MSD.<br />
* the ring duration is more than 90 min.<br />
* the ring is contaminated by RTS with an amplitude of more than one standard deviation of the noise. It concerns a few hundreds of rings for 3 bolometers (44_217_6b, 71_217_8a, 74_857_4). Notice that two bolometers are completely discarded for maps: 55_545_3 and 70_143_8, which present RTS at all time.<br />
<br />
For the three first criteria, a visual inspection of the rmsigTOI at each of the incriminated rings has shown that all these anomalous rings are due to either a drift, a small jump in the TOI trend or a sudden change of noise level, the origin of which is unknown at present.<br />
<br />
Once the list of discarded rings per bolometer is produced, a common list of discarded rings can be extracted for all bolometers (by using discarded rings for at least half the bolometers). Such rings correspond to identified phenomena, as can be seen on the following table.<br />
<br />
Furthermore, an isolated valid ring stuck between two common discarded rings becomes discarded as well.<br />
<br />
Table of '''common discarded rings''' of the nominal mission (rings 240-14723).<br />
<br />
{|table border=1 align=center cellpadding="3" cellspacing="0" style=text-align:center<br />
! '''Cause''' !! '''ring number'''<br />
|-<br />
!align="left"| manoeuvre<br />
|align="center"|304 1312 3590 3611 3642 3922 4949 6379 8456 11328<br />
|-<br />
! align="left"| Sorption Cooler Switchover <br />
|align="center"| 11149 11150 11151 11152 <br />
|-<br />
! align="left"| Rings too long<br />
| align="center"| 440 474 509 544 897 898 3589 13333 14627 14653 <br />
|-<br />
! align="left"| Star tracker switchover <br />
| align="center"| 14628 14654 <br />
|-<br />
! align="left"| Massive Glitch Event<br />
| align="center"| 7665 <br />
|-<br />
! align="left"| Solar Flare<br />
| align="center"| 11235 <br />
|}<br />
<br />
<br />
<br />
The following figure is a summary of the impact of the discarding process '''for each bolometer''' (the solid black line is the common discarded ring percentage). The outlier bolometers have some RTS problems as mentionned above.<br />
<br />
[[File:Fraction_discarded_rings_v53.png|300px|Fraction of discarded rings]]<br />
[[File:Fraction_discarded_time_v53.png|300px|Fraction of time due to discarded rings]]<br />
<br />
== Flag description ==<br />
<br />
==== Input flags ====<br />
<br />
These flags are used as inputs to the TOI processing<br />
<br />
* The point-source flag (PSflag):<br />
<br />
An earlier version of HFI point-source catalog is read back into a flag TOIs, at a given frequency. In practice, 5 sigma sources are masked within a radius of 1.3 FWHM (9, 7, 5, 5, 5, 5 arcmin at 100,143,217,353,545,857 GHz) TBC.<br />
<br />
* the galactic flag (Galflag):<br />
<br />
An earlier version of HFI maps is thresholded and apodized. The produced masks are read into flag TOIs. The retained threshold corresponds to a sky coverage of respectively 70, 70, 80, 90, 90, 90% at 100,143,217,353,545,857 GHz.<br />
<br />
* Solar System Object flag <br />
<br />
For the TOI flag, Mars, Jupiter, Saturn are flagged up to a radius of NbBeam= 2,3,3,4,4,4 times the fiducial SSO_FWHM with SSO_FWHM= 9, 7, 5, 5, 5, 5 arcmin at 100,143,217,353,545,857 GHz.<br />
<br />
As an input to planet mask for maps, Mars, Jupiter, Saturn are flagged with a radius computed as a coefficient depending on the planet (Factor_per_source) times NbBeam times SSO_FWHM, with Factor_per_source = 1.1, 2.25, 1.25 for Mars, Jupiter, Saturn respectively and NbBeam = 2.25, 4.25, 4.0, 5.0,.6.0, 8.0 at 100, 143, 217, 353, 545, 857 GHz. This flag is called SSOflag4map.<br />
<br />
A small trailing tail is added to the mask to take into account the non-deconvolution of the planet signal which has been replaced by background values. The width of that tail is 10 % of the main flag diameter. The number of samples which are additionnaly flagged are the Factor_per_Source times AddSNafter with 10, 30, 20, 20, 30, 40 samples at 100, 143, 217, 353, 545, 857 GHz.<br />
<br />
Uranus and Neptune together with detected asteroids are masked by HFI. They are masked at the TOI level using an exclusion radius of 1.5 SSO_FWHM. At 857 GHz, 24 asteroids have been detected with HFI : 1Ceres, 2Pallas, 3Juno, 4Vesta, 7Iris, 8Flora, 9Metis, 10Hygiea, 11Parthenope, 12Victoria, 13Egeria, 14Irene, 15Eunomia, 16Psyche, 18Melpomene, 19Fortuna, 20Massalia, 29Amphitrite, 41Daphne, 45Eugenia, 52Europa, 88Thisbe, 704Interamnia, 324Bamberga.<br />
<br />
<br />
=== Output flags ===<br />
<br />
A FlagTOIproc is produced by the TOIprocessing. It marks measurements which are not reliable for ny of the following reasons :<br />
* gap (no valid input data), enlarged by one sample on each side. It flags less than 0.00044% (resp. 0.00062%) of the nominal (resp. complete) mission. It is equivallen to less thena 3 (resp. 8) minutes of data.<br />
* glitch : samples where the signal from a cosmic ray hit dominates the sky signal at more than 3 $\sigma$ are discarded.<br />
* jump as 100 samples are flagged around the computed position of the jump to take into account the error on this position.<br />
<br />
With and without point-sources, identical to a and b PSBs.<br />
<br />
<br />
Flags produced for the map making :<br />
* UnstablePointing Flag OR FlagTOIproc OR SSOflag4map OR SSOflag seen<br />
* FlagTOIproc = glitch OR jump OR flag thermal template <br />
* PSB flag glitch = flag glitch A OR flag glitch B<br />
<br />
Flag used in the destriping: same + point-source flag<br />
<br />
<br />
<br />
<br />
[[Category:Data processing]]</div>
Fdesert
https://wiki.cosmos.esa.int/planckpla2015/index.php?title=TOI_processing&diff=3532
TOI processing
2013-01-25T09:39:41Z
<p>Fdesert: </p>
<hr />
<div>This Section is kept short as this Planck release does not contain TOIs. The main information in the HFI processing paper is not duplicated here.<br />
<br />
==Overview==<br />
<br />
We describe here the how the TOIs are processed in order to be used for map production. We do not repeat the general features of the pipeline which are given in the HFI Data Processing article (REF). Here we give complementary explanations on some details. The TOI of each bolometer is processed independently of the other bolometers, so as to keep the noise properties as uncorrelated as possible. The processing involves modifying the TOI itself for what concerns the conversion to absorbed power and the correction of glitch tails. It also adds a flag TOI that masks the TOI samples that are not to be projected on maps for various reasons.<br />
<br />
<br />
=== Acronyms and definitions ===<br />
<br />
* 4K lines: EMI/EMC influence of the 4K cooler mechanical motion on the bolometer readout electronics.<br />
* ADC : analog to digital converter<br />
* IMO : instrument model<br />
* Jump : sudden change of the baseline level inside a ring<br />
* LFER : low frequency excess response<br />
* PBR : phase binned ring<br />
* RIMO : reduced IMO<br />
* Ring : pointing period<br />
* ROI : ring ordered information<br />
* RTS : randomn telegraphic signal<br />
* SSO : solar system object<br />
* TOI : time ordered information<br />
<br />
== Input TOI ==<br />
<br />
The input TOI consists in the AC modulated voltage output of the readout of each bolometer. The input has previously been decompressed, and converted from internal digital units to voltage via a constant factor. The TOI has a regular sampling at the acquisition frequency of facq=180.373700+-0.000050 Hz. There are almost no missing data in the TOIs, except for few hundred samples of 545 and 857GHz TOIs which are lost in the on-board compression due to saturation on the Galactic Center crossings. <br />
<br />
== General Pipeline Structure ==<br />
<br />
The figure on the right shows how the initial ccTOI is transformed and how flags are produced:<br />
<br />
[[File:HFI_4_4_2_ToiProc_ExplS.png|thumb|600px|A schematic of the TOI processing pipeline]]<br />
<br />
<br />
== Output TOIs and products ==<br />
<br />
A TOI of clean calibrated samples (ccTOI) and a combined flag TOI (fTOI) are the outputs of the processing. The ccTOI is calibrated so as to represent the instantaneous power absorbed by the detector up to a constant (which will be determined by the map-making destriper). It is worth mentioning how the ccTOI is changed with respect to the input TOI, beyond the harmless constant conversion factor from voltage to absorbed power. The demodulation stage allows to get the demodulated bolometer voltage. The non-linearity correction is a second-order polynomial correction based on the physical but static bolometer model. In order to avoid too much masking after glitches, a glitch tail is subtracted after an occurrence of a glitch in the TOI. The 4K cooler lines noise is substituted at a series of 9 single temporal frequencies. Finally, the temporal response of the bolometer is deconvolved. This affects mostly the high-temporal frequency part of the TOI, although a small but significant low frequency (the long time response) tail is corrected too. Although flagged samples are not projected, their value influences the valid samples somehow. Hence interpolation procedures introduce some indirect modifications of the TOI.<br />
The flag TOI is a combination a dozen flags with an OR logic. Only unflagged data are projected. The exhaustive list of flags is given here: CompressionError, NoData, SSO, UnstablePointing, Glitch, BoloPlateFluctuation, RTS, Jump, PSBab. A complete qualification of the data is obtained at the ring level. If the TOI shows an anomalous behaviour on a given ring, this ring is discarded from projection. A special production of TOIs is also made as an input to the beam analysis with Mars, Jupiter and Saturn.<br />
<br />
<br />
== Examples of clean TOIs ==<br />
<gallery widths=300px perrow=3><br />
File:10_143_5_PBR_10800RING_LFER4_JC.jpg|Phase-binned signal (PBR) for 6 consecutive rings from 4000 to 4005<br />
File:10_143_5_rcircles_LFER4_JC.jpg| Valid signal (up) and noise (bottom) for 2 consecutive circles of the ring 4000<br />
File:10_143_5_fft_LFER4_JC.jpg| Fourier power spectral density of the first 30 minutes of the ring 4005 with signal before (pink) and after (black) deconvolution (top) and noise (bottom)<br />
File:14_545_1_PBR_10800RING_LFER4_JC.jpg|Phase-binned signal (PBR) for 6 consecutive rings from 4000 to 4005<br />
File:14_545_1_rcircles_LFER4_JC.jpg| Valid signal (up) and noise (bottom) for 2 consecutive circles of the ring 4000<br />
File:14_545_1_fft_LFER4_JC.jpg| Fourier power spectral density of the first 30 minutes of the ring 4005 with signal before (pink) and after (black) deconvolution (top) and noise (bottom)<br />
</gallery><br />
<br />
[[Media:check_ring.pdf|Samples of PBR, TOIs, and PSDs of all detectors are shown in this file]]<br />
<br />
== Trends in the output processing variables ==<br />
<br />
Here we intend to show the trend of the systematic effects that are dealt with in the TOI processing. The full impact of each of them is analyzed in [[HFI-Validation]].<br />
<br />
=== ADC baseline ===<br />
The following figure shows the ADC baseline which is used prior to demodulation (a constant offset is removed for clarity). This baseline is obtained by smoothing on an hour block average the undemodulated TOI on unflagged samples.<br />
<br />
[[File:baseline.jpg|400px|ADC baseline for all bolometers]]<br />
<br />
=== Glitch statistics ===<br />
The glitch rate per channel is shown in this figure. For details, see copap. <br />
<br />
[[File:figIntermPaperGR.jpg|400px|Glitch rate evolution]]<br />
<br />
The percentage of flagged data (mostly due to Cosmic Rays) at the ring level is shown in these examples. No smoothing was applied. Only valid rings are shown.<br />
<gallery widths=450px perrow=2 caption="Percentage of flagged data"><br />
File:group10_143_5.jpg | 143 GHz bolometers<br />
File:group14_545_1.jpg | 545 GHz bolometers<br />
</gallery><br />
[[Media:PercentUnvalid2.pdf|The complete set of plots is here]]<br />
<br />
=== Thermal template for decorrelation ===<br />
[[File:T90.jpg|400px|Thermal template used for the decorrelation from bolometer plate temperature fluctuations]]<br />
<br />
A simple linear decorrelation is performed using the 2 dark bolometers as a proxy of the bolometer plate temperature. Coupling coefficients were measured during the CPV phase.<br />
<br />
=== 4K cooler lines variability ===<br />
The amplitude of the nine 4K cooler lines in aW at 10, 20, 30, 40, 50, 60, 70, 80 and 17 Hz is shown for 2 bolometers in the following figures. The trend is smoothed over 31 ring values after having discarded measurements done at a ring which is discarded for all bolometers.<br />
<gallery widths=450px perrow=2 caption="Amplitude of the nine 4K cooler lines"><br />
File:10_143_5_4Klines.jpg | 143_5 bolometer<br />
File:14_545_1_4Klines.jpg | 545_1 bolometer<br />
</gallery><br />
[[Media:Lines4K.pdf|The 4K cooler line coefficients of all bolometers are shown in this file]]<br />
<br />
=== jump correction ===<br />
A piecewise constant value is removed to the TOI if a jump is detected. See a jump example in this figure:<br />
<br />
[[File:jump_exe.png|200px|An example of a jump seen on the rmsigTOI]]<br />
<br />
The number of jumps per day (all bolometers included) is shown in this figure:<br />
<br />
[[File:jumps_per_day.png|200px|Evolution of jump number during the mission]]<br />
<br />
The jumps are uncorrelated from bolometer to bolometer. The total number of jumps detected in the nominal and full mission is shown here:<br />
<br />
[[File:jumps_per_bolometer.png|200px|Number of jumps per bolometer]]<br />
<br />
<br />
=== Trends in noise and signal ===<br />
<gallery widths=450px perrow=2 caption="Signal (top) and noise (bottom) smoothed at 1 minute. All values falling in a discarded ring are not plotted."><br />
File:10_143_5_smooth_Watt.jpg | 143_5 bolometer<br />
File:14_545_1_smooth_Watt.jpg | 545_1 bolometer<br />
</gallery><br />
<br />
[[Media:check_smooth.pdf|The smooth TOIs of all detectors are shown in this file]]<br />
<br />
=== Noise stationarity ===<br />
This is not the final version but gives a good idea of power spectra at the detector level of rmsigTOIs.<br />
[[Media:v53_meanSpectra_bySurvey.pdf|All PSDs can be seen in this file]]<br />
<br />
The standard deviation per ring corrected by ring duration bias is given here, one per bolometer using only valid rings. No smoothing is applied (except a 31-point smoothing for the 545<br />
and 857 GHz channels) but values for rings discarded for all bolometers (see below) are not used. The standard deviation is computed on samples valid for map-making which are also not affected by the Galaxy or the point-sources using the usual flags. Two examples are given here.<br />
<gallery widths=450px perrow=2 caption="Standard deviation per ring"><br />
File:stddev_group10_143_5.jpg | 143_5 bolometer<br />
File:stddev_group_14_545_1.jpg | 545_1 bolometer<br />
</gallery><br />
<br />
<br />
The full series of plots is here:<br />
[[Media:StationarityInROIoutput2.pdf|Standard deviation of rmsig TOIs at the ring level]]<br />
<br />
Note the presence for 3 bolometers of a two-level noise system. No correction can be done for that effect. See one example here:<br />
<br />
[[File:23_353_TwoLevel.jpg|300px|An example of two-level noise system is seen in bolometer 23_353_3a]]<br />
<br />
<br />
An example of the higher order statistics which are used to unveil rings affected by RTS problems.<br />
<br />
[[File:HFI_4_4_2_RTSexample13.jpg|400px|Example of RTS detection]]<br />
<br />
=== Discarded rings ===<br />
<br />
Some rings are discarded (flagged) from further use (beam making, map making) by using ring statistics (see above and the HFI data processing paper REF). For each statistic, we compare each ring value to the ring values averaged (RVA) over a large selection of rings (between 3000 and 21700). We also define the modified standard deviation (MSD) of a ring quantity as the standard deviation of that quantity over the rings that deviate by less than five nominal standard deviations. This truncation is necessary to be robust against extreme deviant rings.<br />
<br />
More specifically, a ring is discarded if it matches one of the following criteria:<br />
<br />
* the | mean-median | deviates from the RVA by more than fifteen times the MSD.<br />
* the standard deviation deviates from the RVA by more than -5 times the MSD (all cases corresponding to almost empty rings) or +15 times the MSD.<br />
* the Kolmogorov-Smirnov test deviates from the RVA by more than 15 times the MSD.<br />
* the ring duration is more than 90 min.<br />
* the ring is contaminated by RTS with an amplitude of more than one standard deviation of the noise. It concerns a few hundreds of rings for 3 bolometers (44_217_6b, 71_217_8a, 74_857_4). Notice that two bolometers are completely discarded for maps: 55_545_3 and 70_143_8, which present RTS at all time.<br />
<br />
For the three first criteria, a visual inspection of the rmsigTOI at each of the incriminated rings has shown that all these anomalous rings are due to either a drift, a small jump in the TOI trend or a sudden change of noise level, the origin of which is unknown at present.<br />
<br />
Once the list of discarded rings per bolometer is produced, a common list of discarded rings can be extracted for all bolometers (by using discarded rings for at least half the bolometers). Such rings correspond to identified phenomena, as can be seen on the following table.<br />
<br />
Furthermore, an isolated valid ring stuck between two common discarded rings becomes discarded as well.<br />
<br />
Table of '''common discarded rings''' of the nominal mission (rings 240-14723).<br />
<br />
{|table border=1 align=center cellpadding="3" cellspacing="0" style=text-align:center<br />
! '''Cause''' !! '''ring number'''<br />
|-<br />
!align="left"| manoeuvre<br />
|align="center"|304 1312 3590 3611 3642 3922 4949 6379 8456 11328<br />
|-<br />
! align="left"| Sorption Cooler Switchover <br />
|align="center"| 11149 11150 11151 11152 <br />
|-<br />
! align="left"| Rings too long<br />
| align="center"| 440 474 509 544 897 898 3589 13333 14627 14653 <br />
|-<br />
! align="left"| Star tracker switchover <br />
| align="center"| 14628 14654 <br />
|-<br />
! align="left"| Massive Glitch Event<br />
| align="center"| 7665 <br />
|-<br />
! align="left"| Solar Flare<br />
| align="center"| 11235 <br />
|}<br />
<br />
<br />
<br />
The following figure is a summary of the impact of the discarding process '''for each bolometer''' (the solid black line is the common discarded ring percentage). The outlier bolometers have some RTS problems as mentionned above.<br />
<br />
[[File:Fraction_discarded_rings_v53.png|300px|Fraction of discarded rings]]<br />
[[File:Fraction_discarded_time_v53.png|300px|Fraction of time due to discarded rings]]<br />
<br />
== Flag description ==<br />
<br />
==== Input flags ====<br />
<br />
These flags are used as inputs to the TOI processing<br />
<br />
* The point-source flag (PSflag):<br />
<br />
An earlier version of HFI point-source catalog is read back into a flag TOIs, at a given frequency. In practice, 5 sigma sources are masked within a radius of 1.3 FWHM (9, 7, 5, 5, 5, 5 arcmin at 100,143,217,353,545,857 GHz) TBC.<br />
<br />
* the galactic flag (Galflag):<br />
<br />
An earlier version of HFI maps is thresholded and apodized. The produced masks are read into flag TOIs. The retained threshold corresponds to a sky coverage of respectively 70, 70, 80, 90, 90, 90% at 100,143,217,353,545,857 GHz.<br />
<br />
* Solar System Object flag <br />
<br />
For the TOI flag, Mars, Jupiter, Saturn are flagged up to a radius of NbBeam= 2,3,3,4,4,4 times the fiducial SSO_FWHM with SSO_FWHM= 9, 7, 5, 5, 5, 5 arcmin at 100,143,217,353,545,857 GHz.<br />
<br />
As an input to planet mask for maps, Mars, Jupiter, Saturn are flagged with a radius computed as a coefficient depending on the planet (Factor_per_source) times NbBeam times SSO_FWHM, with Factor_per_source = 1.1, 2.25, 1.25 for Mars, Jupiter, Saturn respectively and NbBeam = 2.25, 4.25, 4.0, 5.0,.6.0, 8.0 at 100, 143, 217, 353, 545, 857 GHz. This flag is called SSOflag4map.<br />
<br />
A small trailing tail is added to the mask to take into account the non-deconvolution of the planet signal which has been replaced by background values. The width of that tail is 10 % of the main flag diameter. The number of samples which are additionnaly flagged are the Factor_per_Source times AddSNafter with 10, 30, 20, 20, 30, 40 samples at 100, 143, 217, 353, 545, 857 GHz.<br />
<br />
Uranus and Neptune together with detected asteroids are masked by HFI. They are masked at the TOI level using an exclusion radius of 1.5 SSO_FWHM. At 857 GHz, 24 asteroids have been detected with HFI : 1Ceres, 2Pallas, 3Juno, 4Vesta, 7Iris, 8Flora, 9Metis, 10Hygiea, 11Parthenope, 12Victoria, 13Egeria, 14Irene, 15Eunomia, 16Psyche, 18Melpomene, 19Fortuna, 20Massalia, 29Amphitrite, 41Daphne, 45Eugenia, 52Europa, 88Thisbe, 704Interamnia, 324Bamberga.<br />
<br />
<br />
=== Output flags ===<br />
<br />
A FlagTOIproc is produced by the TOIprocessing. It marks measurements which are not reliable for ny of the following reasons :<br />
* gap (no valid input data), enlarged by one sample on each side. It flags less than 0.00044% (resp. 0.00062%) of the nominal (resp. complete) mission. It is equivallen to less thena 3 (resp. 8) minutes of data.<br />
* glitch : samples where the signal from a cosmic ray hit dominates the sky signal at more than 3 $\sigma$ are discarded.<br />
* jump as 100 samples are flagged around the computed position of the jump to take into account the error on this position.<br />
<br />
With and without point-sources, identical to a and b PSBs.<br />
<br />
<br />
Flags produced for the map making :<br />
* UnstablePointing Flag OR FlagTOIproc OR SSOflag4map OR SSOflag seen<br />
* FlagTOIproc = glitch OR jump OR flag thermal template <br />
* PSB flag glitch = flag glitch A OR flag glitch B<br />
<br />
Flag used in the destriping: same + point-source flag<br />
<br />
<br />
<br />
<br />
[[Category:Data processing]]</div>
Fdesert
https://wiki.cosmos.esa.int/planckpla2015/index.php?title=TOI_processing&diff=3531
TOI processing
2013-01-25T09:36:56Z
<p>Fdesert: </p>
<hr />
<div>This Section is kept short as this Planck release does not contain TOIs. The main information in the HFI processing paper is not duplicated here.<br />
<br />
==Overview==<br />
<br />
We describe here the how the TOIs are processed in order to be used for map production. We do not repeat the general features of the pipeline which are given in the HFI Data Processing article (REF). Here we give complementary explanations on some details. The TOI of each bolometer is processed independently of the other bolometers, so as to keep the noise properties as uncorrelated as possible. The processing involves modifying the TOI itself for what concerns the conversion to absorbed power and the correction of glitch tails. It also adds a flag TOI that masks the TOI samples that are not to be projected on maps for various reasons.<br />
<br />
<br />
=== Acronyms and definitions ===<br />
<br />
4K lines: EMI/EMC influence of the 4K cooler mechanical motion on the readout electronics.<br />
ADC : analog to digital converter<br />
IMO : instrument model<br />
Jump : sudden change of the baseline level inside a ring<br />
LFER : low frequency excess response<br />
PBR : phase binned ring<br />
RIMO : reduced IMO<br />
Ring : pointing period<br />
ROI : ring ordered information<br />
RTS : randomn telegraphic signal<br />
SSO : solar system object<br />
TOI : time ordered information<br />
<br />
== Input TOI ==<br />
<br />
The input TOI consists in the AC modulated voltage output of the readout of each bolometer. The input has previously been decompressed, and converted from internal digital units to voltage via a constant factor. The TOI has a regular sampling at the acquisition frequency of facq=180.373700+-0.000050 Hz. There are almost no missing data in the TOIs, except for few hundred samples of 545 and 857GHz TOIs which are lost in the on-board compression due to saturation on the Galactic Center crossings. <br />
<br />
== General Pipeline Structure ==<br />
<br />
The figure on the right shows how the initial ccTOI is transformed and how flags are produced:<br />
<br />
[[File:HFI_4_4_2_ToiProc_ExplS.png|thumb|600px|A schematic of the TOI processing pipeline]]<br />
<br />
<br />
== Output TOIs and products ==<br />
<br />
A TOI of clean calibrated samples (ccTOI) and a combined flag TOI (fTOI) are the outputs of the processing. The ccTOI is calibrated so as to represent the instantaneous power absorbed by the detector up to a constant (which will be determined by the map-making destriper). It is worth mentioning how the ccTOI is changed with respect to the input TOI, beyond the harmless constant conversion factor from voltage to absorbed power. The demodulation stage allows to get the demodulated bolometer voltage. The non-linearity correction is a second-order polynomial correction based on the physical but static bolometer model. In order to avoid too much masking after glitches, a glitch tail is subtracted after an occurrence of a glitch in the TOI. The 4K cooler lines noise is substituted at a series of 9 single temporal frequencies. Finally, the temporal response of the bolometer is deconvolved. This affects mostly the high-temporal frequency part of the TOI, although a small but significant low frequency (the long time response) tail is corrected too. Although flagged samples are not projected, their value influences the valid samples somehow. Hence interpolation procedures introduce some indirect modifications of the TOI.<br />
The flag TOI is a combination a dozen flags with an OR logic. Only unflagged data are projected. The exhaustive list of flags is given here: CompressionError, NoData, SSO, UnstablePointing, Glitch, BoloPlateFluctuation, RTS, Jump, PSBab. A complete qualification of the data is obtained at the ring level. If the TOI shows an anomalous behaviour on a given ring, this ring is discarded from projection. A special production of TOIs is also made as an input to the beam analysis with Mars, Jupiter and Saturn.<br />
<br />
<br />
== Examples of clean TOIs ==<br />
<gallery widths=300px perrow=3><br />
File:10_143_5_PBR_10800RING_LFER4_JC.jpg|Phase-binned signal (PBR) for 6 consecutive rings from 4000 to 4005<br />
File:10_143_5_rcircles_LFER4_JC.jpg| Valid signal (up) and noise (bottom) for 2 consecutive circles of the ring 4000<br />
File:10_143_5_fft_LFER4_JC.jpg| Fourier power spectral density of the first 30 minutes of the ring 4005 with signal before (pink) and after (black) deconvolution (top) and noise (bottom)<br />
File:14_545_1_PBR_10800RING_LFER4_JC.jpg|Phase-binned signal (PBR) for 6 consecutive rings from 4000 to 4005<br />
File:14_545_1_rcircles_LFER4_JC.jpg| Valid signal (up) and noise (bottom) for 2 consecutive circles of the ring 4000<br />
File:14_545_1_fft_LFER4_JC.jpg| Fourier power spectral density of the first 30 minutes of the ring 4005 with signal before (pink) and after (black) deconvolution (top) and noise (bottom)<br />
</gallery><br />
<br />
[[Media:check_ring.pdf|Samples of PBR, TOIs, and PSDs of all detectors are shown in this file]]<br />
<br />
== Trends in the output processing variables ==<br />
<br />
Here we intend to show the trend of the systematic effects that are dealt with in the TOI processing. The full impact of each of them is analyzed in [[HFI-Validation]].<br />
<br />
=== ADC baseline ===<br />
The following figure shows the ADC baseline which is used prior to demodulation (a constant offset is removed for clarity). This baseline is obtained by smoothing on an hour block average the undemodulated TOI on unflagged samples.<br />
<br />
[[File:baseline.jpg|400px|ADC baseline for all bolometers]]<br />
<br />
=== Glitch statistics ===<br />
The glitch rate per channel is shown in this figure. For details, see copap. <br />
<br />
[[File:figIntermPaperGR.jpg|400px|Glitch rate evolution]]<br />
<br />
The percentage of flagged data (mostly due to Cosmic Rays) at the ring level is shown in these examples. No smoothing was applied. Only valid rings are shown.<br />
<gallery widths=450px perrow=2 caption="Percentage of flagged data"><br />
File:group10_143_5.jpg | 143 GHz bolometers<br />
File:group14_545_1.jpg | 545 GHz bolometers<br />
</gallery><br />
[[Media:PercentUnvalid2.pdf|The complete set of plots is here]]<br />
<br />
=== Thermal template for decorrelation ===<br />
[[File:T90.jpg|400px|Thermal template used for the decorrelation from bolometer plate temperature fluctuations]]<br />
<br />
A simple linear decorrelation is performed using the 2 dark bolometers as a proxy of the bolometer plate temperature. Coupling coefficients were measured during the CPV phase.<br />
<br />
=== 4K cooler lines variability ===<br />
The amplitude of the nine 4K cooler lines in aW at 10, 20, 30, 40, 50, 60, 70, 80 and 17 Hz is shown for 2 bolometers in the following figures. The trend is smoothed over 31 ring values after having discarded measurements done at a ring which is discarded for all bolometers.<br />
<gallery widths=450px perrow=2 caption="Amplitude of the nine 4K cooler lines"><br />
File:10_143_5_4Klines.jpg | 143_5 bolometer<br />
File:14_545_1_4Klines.jpg | 545_1 bolometer<br />
</gallery><br />
[[Media:Lines4K.pdf|The 4K cooler line coefficients of all bolometers are shown in this file]]<br />
<br />
=== jump correction ===<br />
A piecewise constant value is removed to the TOI if a jump is detected. See a jump example in this figure:<br />
<br />
[[File:jump_exe.png|200px|An example of a jump seen on the rmsigTOI]]<br />
<br />
The number of jumps per day (all bolometers included) is shown in this figure:<br />
<br />
[[File:jumps_per_day.png|200px|Evolution of jump number during the mission]]<br />
<br />
The jumps are uncorrelated from bolometer to bolometer. The total number of jumps detected in the nominal and full mission is shown here:<br />
<br />
[[File:jumps_per_bolometer.png|200px|Number of jumps per bolometer]]<br />
<br />
<br />
=== Trends in noise and signal ===<br />
<gallery widths=450px perrow=2 caption="Signal (top) and noise (bottom) smoothed at 1 minute. All values falling in a discarded ring are not plotted."><br />
File:10_143_5_smooth_Watt.jpg | 143_5 bolometer<br />
File:14_545_1_smooth_Watt.jpg | 545_1 bolometer<br />
</gallery><br />
<br />
[[Media:check_smooth.pdf|The smooth TOIs of all detectors are shown in this file]]<br />
<br />
=== Noise stationarity ===<br />
This is not the final version but gives a good idea of power spectra at the detector level of rmsigTOIs.<br />
[[Media:v53_meanSpectra_bySurvey.pdf|All PSDs can be seen in this file]]<br />
<br />
The standard deviation per ring corrected by ring duration bias is given here, one per bolometer using only valid rings. No smoothing is applied (except a 31-point smoothing for the 545<br />
and 857 GHz channels) but values for rings discarded for all bolometers (see below) are not used. The standard deviation is computed on samples valid for map-making which are also not affected by the Galaxy or the point-sources using the usual flags. Two examples are given here.<br />
<gallery widths=450px perrow=2 caption="Standard deviation per ring"><br />
File:stddev_group10_143_5.jpg | 143_5 bolometer<br />
File:stddev_group_14_545_1.jpg | 545_1 bolometer<br />
</gallery><br />
<br />
<br />
The full series of plots is here:<br />
[[Media:StationarityInROIoutput2.pdf|Standard deviation of rmsig TOIs at the ring level]]<br />
<br />
Note the presence for 3 bolometers of a two-level noise system. No correction can be done for that effect. See one example here:<br />
<br />
[[File:23_353_TwoLevel.jpg|300px|An example of two-level noise system is seen in bolometer 23_353_3a]]<br />
<br />
<br />
An example of the higher order statistics which are used to unveil rings affected by RTS problems.<br />
<br />
[[File:HFI_4_4_2_RTSexample13.jpg|400px|Example of RTS detection]]<br />
<br />
=== Discarded rings ===<br />
<br />
Some rings are discarded (flagged) from further use (beam making, map making) by using ring statistics (see above and the HFI data processing paper REF). For each statistic, we compare each ring value to the ring values averaged (RVA) over a large selection of rings (between 3000 and 21700). We also define the modified standard deviation (MSD) of a ring quantity as the standard deviation of that quantity over the rings that deviate by less than five nominal standard deviations. This truncation is necessary to be robust against extreme deviant rings.<br />
<br />
More specifically, a ring is discarded if it matches one of the following criteria:<br />
<br />
* the | mean-median | deviates from the RVA by more than fifteen times the MSD.<br />
* the standard deviation deviates from the RVA by more than -5 times the MSD (all cases corresponding to almost empty rings) or +15 times the MSD.<br />
* the Kolmogorov-Smirnov test deviates from the RVA by more than 15 times the MSD.<br />
* the ring duration is more than 90 min.<br />
* the ring is contaminated by RTS with an amplitude of more than one standard deviation of the noise. It concerns a few hundreds of rings for 3 bolometers (44_217_6b, 71_217_8a, 74_857_4). Notice that two bolometers are completely discarded for maps: 55_545_3 and 70_143_8, which present RTS at all time.<br />
<br />
For the three first criteria, a visual inspection of the rmsigTOI at each of the incriminated rings has shown that all these anomalous rings are due to either a drift, a small jump in the TOI trend or a sudden change of noise level, the origin of which is unknown at present.<br />
<br />
Once the list of discarded rings per bolometer is produced, a common list of discarded rings can be extracted for all bolometers (by using discarded rings for at least half the bolometers). Such rings correspond to identified phenomena, as can be seen on the following table.<br />
<br />
Furthermore, an isolated valid ring stuck between two common discarded rings becomes discarded as well.<br />
<br />
Table of '''common discarded rings''' of the nominal mission (rings 240-14723).<br />
<br />
{|table border=1 align=center cellpadding="3" cellspacing="0" style=text-align:center<br />
! '''Cause''' !! '''ring number'''<br />
|-<br />
!align="left"| manoeuvre<br />
|align="center"|304 1312 3590 3611 3642 3922 4949 6379 8456 11328<br />
|-<br />
! align="left"| Sorption Cooler Switchover <br />
|align="center"| 11149 11150 11151 11152 <br />
|-<br />
! align="left"| Rings too long<br />
| align="center"| 440 474 509 544 897 898 3589 13333 14627 14653 <br />
|-<br />
! align="left"| Star tracker switchover <br />
| align="center"| 14628 14654 <br />
|-<br />
! align="left"| Massive Glitch Event<br />
| align="center"| 7665 <br />
|-<br />
! align="left"| Solar Flare<br />
| align="center"| 11235 <br />
|}<br />
<br />
<br />
<br />
The following figure is a summary of the impact of the discarding process '''for each bolometer''' (the solid black line is the common discarded ring percentage). The outlier bolometers have some RTS problems as mentionned above.<br />
<br />
[[File:Fraction_discarded_rings_v53.png|300px|Fraction of discarded rings]]<br />
[[File:Fraction_discarded_time_v53.png|300px|Fraction of time due to discarded rings]]<br />
<br />
== Flag description ==<br />
<br />
==== Input flags ====<br />
<br />
These flags are used as inputs to the TOI processing<br />
<br />
* The point-source flag (PSflag):<br />
<br />
An earlier version of HFI point-source catalog is read back into a flag TOIs, at a given frequency. In practice, 5 sigma sources are masked within a radius of 1.3 FWHM (9, 7, 5, 5, 5, 5 arcmin at 100,143,217,353,545,857 GHz) TBC.<br />
<br />
* the galactic flag (Galflag):<br />
<br />
An earlier version of HFI maps is thresholded and apodized. The produced masks are read into flag TOIs. The retained threshold corresponds to a sky coverage of respectively 70, 70, 80, 90, 90, 90% at 100,143,217,353,545,857 GHz.<br />
<br />
* Solar System Object flag <br />
<br />
For the TOI flag, Mars, Jupiter, Saturn are flagged up to a radius of NbBeam= 2,3,3,4,4,4 times the fiducial SSO_FWHM with SSO_FWHM= 9, 7, 5, 5, 5, 5 arcmin at 100,143,217,353,545,857 GHz.<br />
<br />
As an input to planet mask for maps, Mars, Jupiter, Saturn are flagged with a radius computed as a coefficient depending on the planet (Factor_per_source) times NbBeam times SSO_FWHM, with Factor_per_source = 1.1, 2.25, 1.25 for Mars, Jupiter, Saturn respectively and NbBeam = 2.25, 4.25, 4.0, 5.0,.6.0, 8.0 at 100, 143, 217, 353, 545, 857 GHz. This flag is called SSOflag4map.<br />
<br />
A small trailing tail is added to the mask to take into account the non-deconvolution of the planet signal which has been replaced by background values. The width of that tail is 10 % of the main flag diameter. The number of samples which are additionnaly flagged are the Factor_per_Source times AddSNafter with 10, 30, 20, 20, 30, 40 samples at 100, 143, 217, 353, 545, 857 GHz.<br />
<br />
Uranus and Neptune together with detected asteroids are masked by HFI. They are masked at the TOI level using an exclusion radius of 1.5 SSO_FWHM. At 857 GHz, 24 asteroids have been detected with HFI : 1Ceres, 2Pallas, 3Juno, 4Vesta, 7Iris, 8Flora, 9Metis, 10Hygiea, 11Parthenope, 12Victoria, 13Egeria, 14Irene, 15Eunomia, 16Psyche, 18Melpomene, 19Fortuna, 20Massalia, 29Amphitrite, 41Daphne, 45Eugenia, 52Europa, 88Thisbe, 704Interamnia, 324Bamberga.<br />
<br />
<br />
=== Output flags ===<br />
<br />
A FlagTOIproc is produced by the TOIprocessing. It marks measurements which are not reliable for ny of the following reasons :<br />
* gap (no valid input data), enlarged by one sample on each side. It flags less than 0.00044% (resp. 0.00062%) of the nominal (resp. complete) mission. It is equivallen to less thena 3 (resp. 8) minutes of data.<br />
* glitch : samples where the signal from a cosmic ray hit dominates the sky signal at more than 3 $\sigma$ are discarded.<br />
* jump as 100 samples are flagged around the computed position of the jump to take into account the error on this position.<br />
<br />
With and without point-sources, identical to a and b PSBs.<br />
<br />
<br />
Flags produced for the map making :<br />
* UnstablePointing Flag OR FlagTOIproc OR SSOflag4map OR SSOflag seen<br />
* FlagTOIproc = glitch OR jump OR flag thermal template <br />
* PSB flag glitch = flag glitch A OR flag glitch B<br />
<br />
Flag used in the destriping: same + point-source flag<br />
<br />
<br />
<br />
<br />
[[Category:Data processing]]</div>
Fdesert
https://wiki.cosmos.esa.int/planckpla2015/index.php?title=TOI_processing&diff=3284
TOI processing
2013-01-14T09:07:50Z
<p>Fdesert: </p>
<hr />
<div>This Section is kept short as this Planck release does not contain TOIs. The main information in the HFI processing paper is not duplicated here.<br />
<br />
==Overview==<br />
<br />
We describe here the how the TOIs are processed in order to be used for map production. We do not repeat the general features of the pipeline which are given in the HFI Data Processing article (REF). Here we give complementary explanations on some details. The TOI of each bolometer is processed independently of the other bolometers, so as to keep the noise properties as uncorrelated as possible. The processing involves modifying the TOI itself for what concerns the conversion to absorbed power and the correction of glitch tails. It also adds a flag TOI that masks the TOI samples that are not to be projected on maps for various reasons.<br />
<br />
<br />
== Input TOI ==<br />
<br />
The input TOI consists in the AC modulated voltage output of the readout of each bolometer. The input has previously been decompressed, and converted from internal digital units to voltage via a constant factor. The TOI has a regular sampling at the acquisition frequency of facq=180.373700+-0.000050 Hz. There are almost no missing data in the TOIs, except for few hundred samples of 545 and 857GHz TOIs which are lost in the on-board compression due to saturation on the Galactic Center crossings. <br />
<br />
== General Pipeline Structure ==<br />
<br />
The figure on the right shows how the initial ccTOI is transformed and how flags are produced:<br />
<br />
[[File:HFI_4_4_2_ToiProc_ExplS.png|thumb|600px|A schematic of the TOI processing pipeline]]<br />
<br />
<br />
== Output TOIs and products ==<br />
<br />
A TOI of clean calibrated samples (ccTOI) and a combined flag TOI (fTOI) are the outputs of the processing. The ccTOI is calibrated so as to represent the instantaneous power absorbed by the detector up to a constant (which will be determined by the map-making destriper). It is worth mentioning how the ccTOI is changed with respect to the input TOI, beyond the harmless constant conversion factor from voltage to absorbed power. The demodulation stage allows to get the demodulated bolometer voltage. The non-linearity correction is a second-order polynomial correction based on the physical but static bolometer model. In order to avoid too much masking after glitches, a glitch tail is subtracted after an occurrence of a glitch in the TOI. The 4K cooler lines noise is substituted at a series of 9 single temporal frequencies. Finally, the temporal response of the bolometer is deconvolved. This affects mostly the high-temporal frequency part of the TOI, although a small but significant low frequency (the long time response) tail is corrected too. Although flagged samples are not projected, their value influences the valid samples somehow. Hence interpolation procedures introduce some indirect modifications of the TOI.<br />
The flag TOI is a combination a dozen flags with an OR logic. Only unflagged data are projected. The exhaustive list of flags is given here: CompressionError, NoData, SSO, UnstablePointing, Glitch, BoloPlateFluctuation, RTS, Jump, PSBab. A complete qualification of the data is obtained at the ring level. If the TOI shows an anomalous behaviour on a given ring, this ring is discarded from projection. A special production of TOIs is also made as an input to the beam analysis with Mars, Jupiter and Saturn.<br />
<br />
<br />
== Examples of clean TOIs ==<br />
<gallery widths=300px perrow=3><br />
File:10_143_5_PBR_10800RING_LFER4_JC.jpg|Phase-binned signal (PBR) for 6 consecutive rings from 4000 to 4005<br />
File:10_143_5_rcircles_LFER4_JC.jpg| Valid signal (up) and noise (bottom) for 2 consecutive circles of the ring 4000<br />
File:10_143_5_fft_LFER4_JC.jpg| Fourier power spectral density of the first 30 minutes of the ring 4005 with signal before (pink) and after (black) deconvolution (top) and noise (bottom)<br />
File:14_545_1_PBR_10800RING_LFER4_JC.jpg|Phase-binned signal (PBR) for 6 consecutive rings from 4000 to 4005<br />
File:14_545_1_rcircles_LFER4_JC.jpg| Valid signal (up) and noise (bottom) for 2 consecutive circles of the ring 4000<br />
File:14_545_1_fft_LFER4_JC.jpg| Fourier power spectral density of the first 30 minutes of the ring 4005 with signal before (pink) and after (black) deconvolution (top) and noise (bottom)<br />
</gallery><br />
<br />
[[Media:check_ring.pdf|Samples of PBR, TOIs, and PSDs of all detectors are shown in this file]]<br />
<br />
== Trends in the output processing variables ==<br />
<br />
Here we intend to show the trend of the systematic effects that are dealt with in the TOI processing. The full impact of each of them is analyzed in [[HFI-Validation]].<br />
<br />
=== ADC baseline ===<br />
The following figure shows the ADC baseline which is used prior to demodulation (a constant offset is removed for clarity). This baseline is obtained by smoothing on an hour block average the undemodulated TOI on unflagged samples.<br />
[[File:baseline.jpg|400px|ADC baseline for all bolometers]]<br />
<br />
=== Glitch statistics ===<br />
The glitch rate per channel is shown in this figure. For details, see copap. <br />
[[File:figIntermPaperGR.jpg|400px|Glitch rate evolution]]<br />
<br />
The percentage of flagged data (mostly due to Cosmic Rays) at the ring level is shown in these examples. No smoothing was applied. Only valid rings are shown.<br />
<gallery widths=450px perrow=2 caption="Percentage of flagged data"><br />
File:group10_143_5.jpg | 143 GHz bolometers<br />
File:group14_545_1.jpg | 545 GHz bolometers<br />
</gallery><br />
[[Media:PercentUnvalid2.pdf|The complete set of plots is here]]<br />
<br />
=== Thermal template for decorrelation ===<br />
[[File:T90.jpg|400px|Thermal template used for the decorrelation from bolometer plate temperature fluctuations]]<br />
<br />
A simple linear decorrelation is performed using the 2 dark bolometers as a proxy of the bolometer plate temperature. Coupling coefficients were measured during the CPV phase.<br />
<br />
=== 4K cooler lines variability ===<br />
The amplitude of the nine 4K cooler lines in aW at 10, 20, 30, 40, 50, 60, 70, 80 and 17 Hz is shown for 2 bolometers in the following figures. The trend is smoothed over 31 ring values after having discarded measurements done at a ring which is discarded for all bolometers.<br />
<gallery widths=450px perrow=2 caption="Amplitude of the nine 4K cooler lines"><br />
File:10_143_5_4Klines.jpg | 143_5 bolometer<br />
File:14_545_1_4Klines.jpg | 545_1 bolometer<br />
</gallery><br />
[[Media:Lines4K.pdf|The 4K cooler line coefficients of all bolometers are shown in this file]]<br />
<br />
=== jump correction ===<br />
A piecewise constant value is removed to the TOI if a jump is detected. See a jump example in this figure:<br />
[[File:jump_exe.png|200px|An example of a jump seen on the rmsigTOI]]<br />
<br />
The number of jumps per day (all bolometers included) is shown in this figure:<br />
[[File:jumps_per_day.png|200px|Evolution of jump number during the mission]]<br />
<br />
The jumps are uncorrelated from bolometer to bolometer. The total number of jumps detected in the nominal and full mission is shown here:<br />
[[File:jumps_per_bolometer.png|200px|Number of jumps per bolometer]]<br />
<br />
<br />
=== Trends in noise and signal ===<br />
<gallery widths=450px perrow=2 caption="Signal (top) and noise (bottom) smoothed at 1 minute. All values falling in a discarded ring are not plotted."><br />
File:10_143_5_smooth_Watt.jpg | 143_5 bolometer<br />
File:14_545_1_smooth_Watt.jpg | 545_1 bolometer<br />
</gallery><br />
<br />
[[Media:check_smooth.pdf|The smooth TOIs of all detectors are shown in this file]]<br />
<br />
=== Noise stationarity ===<br />
This is not the final version but gives a good idea of power spectra at the detector level of rmsigTOIs.<br />
[[Media:v53_meanSpectra_bySurvey.pdf|All PSDs can be seen in this file]]<br />
<br />
The standard deviation per ring corrected by ring duration bias is given here, one per bolometer using only valid rings. No smoothing is applied (except a 31-point smoothing for the 545<br />
and 857 GHz channels) but values for rings discarded for all bolometers (see below) are not used. The standard deviation is computed on samples valid for map-making which are also not affected by the Galaxy or the point-sources using the usual flags. Two examples are given here.<br />
<gallery widths=450px perrow=2 caption="Standard deviation per ring"><br />
File:stddev_group10_143_5.jpg | 143_5 bolometer<br />
File:stddev_group_14_545_1.jpg | 545_1 bolometer<br />
</gallery><br />
<br />
<br />
The full series of plots is here:<br />
[[Media:StationarityInROIoutput2.pdf|Standard deviation of rmsig TOIs at the ring level]]<br />
<br />
Note the presence for 3 bolometers of a two-level noise system. No correction can be done for that effect. See one example here:<br />
[[File:23_353_TwoLevel.jpg|300px|An example of two-level noise system is seen in bolometer 23_353_3a]]<br />
<br />
<br />
An example of the higher order statistics which are used to unveil rings affected by RTS problems.<br />
[[File:HFI_4_4_2_RTSexample13.jpg|400px|Example of RTS detection]]<br />
<br />
=== Discarded rings ===<br />
<br />
Some rings are discarded (flagged) from further use (beam making, map making) by using ring statistics (see above and the HFI data processing paper REF). For each statistic, we compare each ring value to the ring values averaged (RVA) over a large selection of rings (between 3000 and 21700). We also define the modified standard deviation (MSD) of a ring quantity as the standard deviation of that quantity over the rings that deviate by less than five nominal standard deviations. This truncation is necessary to be robust against extreme deviant rings.<br />
<br />
More specifically, a ring is discarded if it matches one of the following criteria:<br />
<br />
* the | mean-median | deviates from the RVA by more than fifteen times the MSD.<br />
* the standard deviation deviates from the RVA by more than -5 times the MSD (all cases corresponding to almost empty rings) or +15 times the MSD.<br />
* the Kolmogorov-Smirnov test deviates from the RVA by more than 15 times the MSD.<br />
* the ring duration is more than 90 min.<br />
* the ring is contaminated by popcorn noise with an amplitude of more than one standard deviation of the noise. It concerns a few hundreds of rings for 5 bolometers.<br />
<br />
For the three first criteria, a visual inspection of the rmsigTOI at each of the incriminated rings has shown that all these anomalous rings are due to either a drift, a small jump in the TOI trend or a sudden change of noise level, the origin of which is unknown at present.<br />
<br />
Once the list of discarded rings per bolometer is produced, a common list of discarded rings can be produced for all bolometers (by using discarded rings for at least half the bolometers). Such rings correspond to identified phenomena, as can be seen on the following table.<br />
<br />
Furthermore, an isolated valid ring stuck between two common discarded rings becomes discarded as well.<br />
<br />
Table of common discarded rings (v47, 240-21890), truncated to 240-14723 to have nominal mission, to be updated to v53, nominal mission.<br />
<br />
{|table border=1 align=center cellpadding="3" cellspacing="0" style=text-align:center<br />
! '''Cause''' !! '''ring number'''<br />
|-<br />
!align="left"| manoeuvre<br />
|align="center"|304 1312 3590 3611 3642 3922 4949 6379 8456 11328<br />
|-<br />
! align="left"| Sorption Cooler Switchover <br />
|align="center"| 11149 11150 11151 11152 <br />
|-<br />
! align="left"| Rings too long<br />
| align="center"| 440 474 509 544 897 898 3589 13333 14627 14653 <br />
|-<br />
! align="left"| Star tracker switchover <br />
| align="center"| 14628 14654 <br />
|-<br />
! align="left"| Glitch seen by all of the bolometers, but the darks<br />
| align="center"| 7665 <br />
|-<br />
! align="left"| Ring stuck between two common discarded rings<br />
| align="center"| 11150 <br />
|}<br />
<br />
The following figure is a summary of the impact of the discarding process (the solid black line is the common discarded ring percentage).<br />
<br />
<gallery widths=450px perrow=2 caption="Percentage of flagged data due to ring discarding "><br />
File:Fraction_discarded_rings_v53.png | Fraction of discarded rings<br />
File:Fraction_discarded_time_v53.png | Fraction of time due to discarded rings<br />
</gallery><br />
<br />
== Flag description ==<br />
<br />
==== Input flags ====<br />
<br />
These flags are used as inputs to the TOI processing<br />
<br />
* The point-source flag (PSflag):<br />
<br />
An earlier version of HFI point-source catalog is read back into a flag TOIs, at a given frequency. In practice, 5 sigma sources are masked within a radius of 1.3 FWHM (9, 7, 5, 5, 5, 5 arcmin at 100,143,217,353,545,857 GHz) TBC.<br />
<br />
* the galactic flag (Galflag):<br />
<br />
An earlier version of HFI maps is thresholded and apodized. The produced masks are read into flag TOIs. The retained threshold corresponds to a sky coverage of respectively 70, 70, 80, 90, 90, 90% at 100,143,217,353,545,857 GHz.<br />
<br />
* Solar System Object flag <br />
<br />
For the TOI flag, Mars, Jupiter, Saturn are flagged up to a radius of NbBeam= 2,3,3,4,4,4 times the fiducial SSO_FWHM with SSO_FWHM= 9, 7, 5, 5, 5, 5 arcmin at 100,143,217,353,545,857 GHz.<br />
<br />
As an input to planet mask for maps, Mars, Jupiter, Saturn are flagged with a radius computed as a coefficient depending on the planet (Factor_per_source) times NbBeam times SSO_FWHM, with Factor_per_source = 1.1, 2.25, 1.25 for Mars, Jupiter, Saturn respectively and NbBeam = 2.25, 4.25, 4.0, 5.0,.6.0, 8.0 at 100, 143, 217, 353, 545, 857 GHz. This flag is called SSOflag4map.<br />
<br />
A small trailing tail is added to the mask to take into account the non-deconvolution of the planet signal which has been replaced by background values. The width of that tail is 10 % of the main flag diameter. The number of samples which are additionnaly flagged are the Factor_per_Source times AddSNafter with 10, 30, 20, 20, 30, 40 samples at 100, 143, 217, 353, 545, 857 GHz.<br />
<br />
Uranus and Neptune together with detected asteroids are masked by HFI. They are masked at the TOI level using an exclusion radius of 1.5 SSO_FWHM. At 857 GHz, 24 asteroids have been detected with HFI : 1Ceres, 2Pallas, 3Juno, 4Vesta, 7Iris, 8Flora, 9Metis, 10Hygiea, 11Parthenope, 12Victoria, 13Egeria, 14Irene, 15Eunomia, 16Psyche, 18Melpomene, 19Fortuna, 20Massalia, 29Amphitrite, 41Daphne, 45Eugenia, 52Europa, 88Thisbe, 704Interamnia, 324Bamberga.<br />
<br />
<br />
=== Output flags ===<br />
<br />
A FlagTOIproc is produced by the TOIprocessing. It marks measurements which are not reliable for ny of the following reasons :<br />
* gap (no valid input data), enlarged by one sample on each side. It flags less than 0.00044% (resp. 0.00062%) of the nominal (resp. complete) mission. It is equivallen to less thena 3 (resp. 8) minutes of data.<br />
* glitch : samples where the signal from a cosmic ray hit dominates the sky signal at more than 3 $\sigma$ are discarded.<br />
* jump as 100 samples are flagged around the computed position of the jump to take into account the error on this position.<br />
<br />
With and without point-sources, identical to a and b PSBs.<br />
<br />
<br />
Flags produced for the map making :<br />
* UnstablePointing Flag OR FlagTOIproc OR SSOflag4map OR SSOflag seen<br />
* FlagTOIproc = glitch OR jump OR flag thermal template <br />
* PSB flag glitch = flag glitch A OR flag glitch B<br />
<br />
Flag used in the destriping: same + point-source flag<br />
<br />
<br />
<br />
<br />
[[Category:Data processing]]</div>
Fdesert
https://wiki.cosmos.esa.int/planckpla2015/index.php?title=TOI_processing&diff=3283
TOI processing
2013-01-14T09:02:43Z
<p>Fdesert: </p>
<hr />
<div>This Section is kept short as this Planck release does not contain TOIs. The main information in the HFI processing paper is not duplicated here.<br />
<br />
==Overview==<br />
<br />
We describe here the how the TOIs are processed in order to be used for map production. We do not repeat the general features of the pipeline which are given in the HFI Data Processing article (REF). Here we give complementary explanations on some details. The TOI of each bolometer is processed independently of the other bolometers, so as to keep the noise properties as uncorrelated as possible. The processing involves modifying the TOI itself for what concerns the conversion to absorbed power and the correction of glitch tails. It also adds a flag TOI that masks the TOI samples that are not to be projected on maps for various reasons.<br />
<br />
<br />
== Input TOI ==<br />
<br />
The input TOI consists in the AC modulated voltage output of the readout of each bolometer. The input has previously been decompressed, and converted from internal digital units to voltage via a constant factor. The TOI has a regular sampling at the acquisition frequency of facq=180.373700+-0.000050 Hz. There are almost no missing data in the TOIs, except for few hundred samples of 545 and 857GHz TOIs which are lost in the on-board compression due to saturation on the Galactic Center crossings. <br />
<br />
== General Pipeline Structure ==<br />
<br />
The figure on the right shows how the initial ccTOI is transformed and how flags are produced:<br />
<br />
[[File:HFI_4_4_2_ToiProc_ExplS.png|thumb|600px|A schematic of the TOI processing pipeline]]<br />
<br />
<br />
== Output TOIs and products ==<br />
<br />
A TOI of clean calibrated samples (ccTOI) and a combined flag TOI (fTOI) are the outputs of the processing. The ccTOI is calibrated so as to represent the instantaneous power absorbed by the detector up to a constant (which will be determined by the map-making destriper). It is worth mentioning how the ccTOI is changed with respect to the input TOI, beyond the harmless constant conversion factor from voltage to absorbed power. The demodulation stage allows to get the demodulated bolometer voltage. The non-linearity correction is a second-order polynomial correction based on the physical but static bolometer model. In order to avoid too much masking after glitches, a glitch tail is subtracted after an occurrence of a glitch in the TOI. The 4K cooler lines noise is substituted at a series of 9 single temporal frequencies. Finally, the temporal response of the bolometer is deconvolved. This affects mostly the high-temporal frequency part of the TOI, although a small but significant low frequency (the long time response) tail is corrected too. Although flagged samples are not projected, their value influences the valid samples somehow. Hence interpolation procedures introduce some indirect modifications of the TOI.<br />
The flag TOI is a combination a dozen flags with an OR logic. Only unflagged data are projected. The exhaustive list of flags is given here: CompressionError, NoData, SSO, UnstablePointing, Glitch, BoloPlateFluctuation, RTS, Jump, PSBab. A complete qualification of the data is obtained at the ring level. If the TOI shows an anomalous behaviour on a given ring, this ring is discarded from projection. A special production of TOIs is also made as an input to the beam analysis with Mars, Jupiter and Saturn.<br />
<br />
<br />
== Examples of clean TOIs ==<br />
<gallery widths=300px perrow=3><br />
File:10_143_5_PBR_10800RING_LFER4_JC.jpg|Phase-binned signal (PBR) for 6 consecutive rings from 4000 to 4005<br />
File:10_143_5_rcircles_LFER4_JC.jpg| Valid signal (up) and noise (bottom) for 2 consecutive circles of the ring 4000<br />
File:10_143_5_fft_LFER4_JC.jpg| Fourier power spectral density of the first 30 minutes of the ring 4005 with signal before (pink) and after (black) deconvolution (top) and noise (bottom)<br />
File:14_545_1_PBR_10800RING_LFER4_JC.jpg|Phase-binned signal (PBR) for 6 consecutive rings from 4000 to 4005<br />
File:14_545_1_rcircles_LFER4_JC.jpg| Valid signal (up) and noise (bottom) for 2 consecutive circles of the ring 4000<br />
File:14_545_1_fft_LFER4_JC.jpg| Fourier power spectral density of the first 30 minutes of the ring 4005 with signal before (pink) and after (black) deconvolution (top) and noise (bottom)<br />
</gallery><br />
<br />
[[Media:check_ring.pdf|Samples of PBR, TOIs, and PSDs of all detectors are shown in this file]]<br />
<br />
== Trends in the output processing variables ==<br />
<br />
Here we intend to show the trend of the systematic effects that are dealt with in the TOI processing. The full impact of each of them is analyzed in [[HFI-Validation]].<br />
<br />
=== ADC baseline ===<br />
The following figure shows the ADC baseline which is used prior to demodulation (a constant offset is removed for clarity). This baseline is obtained by smoothing on an hour block average the undemodulated TOI on unflagged samples.<br />
[[File:baseline.jpg|400px|ADC baseline for all bolometers]]<br />
<br />
=== Glitch statistics ===<br />
The glitch rate per channel is shown in this figure. For details, see copap. <br />
[[File:figIntermPaperGR.jpg|400px|Glitch rate evolution]]<br />
<br />
The percentage of flagged data (mostly due to Cosmic Rays) at the ring level is shown in these examples. No smoothing was applied. Only valid rings are shown.<br />
<gallery widths=450px perrow=2 caption="Percentage of flagged data"><br />
File:group10_143_5.jpg | 143 GHz bolometers<br />
File:group14_545_1.jpg | 545 GHz bolometers<br />
</gallery><br />
[[Media:PercentUnvalid2.pdf|The complete set of plots is here]]<br />
<br />
=== Thermal template for decorrelation ===<br />
[[File:T90.jpg|400px|Thermal template used for the decorrelation from bolometer plate temperature fluctuations]]<br />
<br />
A simple linear decorrelation is performed using the 2 dark bolometers as a proxy of the bolometer plate temperature. Coupling coefficients were measured during the CPV phase.<br />
<br />
=== 4K cooler lines variability ===<br />
The amplitude of the nine 4K cooler lines in aW at 10, 20, 30, 40, 50, 60, 70, 80 and 17 Hz is shown for 2 bolometers in the following figures. The trend is smoothed over 31 ring values after having discarded measurements done at a ring which is discarded for all bolometers.<br />
<gallery widths=450px perrow=2 caption="Amplitude of the nine 4K cooler lines"><br />
File:10_143_5_4Klines.jpg | 143_5 bolometer<br />
File:14_545_1_4Klines.jpg | 545_1 bolometer<br />
</gallery><br />
[[Media:Lines4K.pdf|The 4K cooler line coefficients of all bolometers are shown in this file]]<br />
<br />
=== jump correction ===<br />
A piecewise constant value is removed to the TOI if a jump is detected. See a jump example in this figure:<br />
[[File:jump_exe.png|200px|An example of a jump seen on the rmsigTOI]]<br />
<br />
The number of jumps per day (all bolometers included) is shown in this figure:<br />
[[File:jumps_per_day.png|200px|Evolution of jump number during the mission]]<br />
<br />
The jumps are uncorrelated from bolometer to bolometer. The total number of jumps detected in the nominal and full mission is shown here:<br />
[[File:jumps_per_bolometer.png|200px|Number of jumps per bolometer]]<br />
<br />
<br />
=== Trends in noise and signal ===<br />
<gallery widths=450px perrow=2 caption="Signal (top) and noise (bottom) smoothed at 1 minute. All values falling in a discarded ring are not plotted."><br />
File:10_143_5_smooth_Watt.jpg | 143_5 bolometer<br />
File:14_545_1_smooth_Watt.jpg | 545_1 bolometer<br />
</gallery><br />
<br />
[[Media:check_smooth.pdf|The smooth TOIs of all detectors are shown in this file]]<br />
<br />
=== Noise stationarity ===<br />
This is not the final version but gives a good idea of power spectra at the detector level of rmsigTOIs.<br />
[[Media:v53_meanSpectra_bySurvey.pdf|All PSDs can be seen in this file]]<br />
<br />
The standard deviation per ring corrected by ring duration bias is given here, one per bolometer using only valid rings. No smoothing is applied (except a 31-point smoothing for the 545<br />
and 857 GHz channels) but values for rings discarded for all bolometers (see below) are not used. The standard deviation is computed on samples valid for map-making which are also not affected by the Galaxy or the point-sources using the usual flags. Two examples are given here.<br />
<gallery widths=450px perrow=2 caption="Standard deviation per ring"><br />
File:stddev_group10_143_5.jpg | 143_5 bolometer<br />
File:stddev_group_14_545_1.jpg | 545_1 bolometer<br />
</gallery><br />
<br />
<br />
The full series of plots is here:<br />
[[Media:StationarityInROIoutput2.pdf|Standard deviation of rmsig TOIs at the ring level]]<br />
<br />
Note the presence for 3 bolometers of a two-level noise system. No correction can be done for that effect. See one example here:<br />
[[File:23_353_TwoLevel.jpg|300px|An example of two-level noise system is seen in bolometer 23_353_3a]]<br />
<br />
<br />
An example of the higher order statistics which are used to unveil rings affected by RTS problems.<br />
[[File:HFI_4_4_2_RTSexample13.jpg|400px|Example of RTS detection]]<br />
<br />
=== Discarded rings ===<br />
<br />
Some rings are discarded (flagged) from further use (beam making, map making) by using ring statistics (see above and the HFI data processing paper REF). For each statistic, we compare each ring value to the ring values averaged (RVA) over a large selection of rings (between 3000 and 21700). We also define the modified standard deviation (MSD) of a ring quantity as the standard deviation of that quantity over the rings that deviate by less than five nominal standard deviations. This truncation is necessary to be robust against extreme deviant rings.<br />
<br />
More specifically, a ring is discarded if it matches one of the following criteria:<br />
<br />
* the | mean-median | deviates from the RVA by more than fifteen times the MSD.<br />
* the standard deviation deviates from the RVA by more than -5 times the MSD (all cases corresponding to almost empty rings) or +15 times the MSD.<br />
* the Kolmogorov-Smirnov test deviates from the RVA by more than 15 times the MSD.<br />
* the ring duration is more than 90 min.<br />
* the ring is contaminated by popcorn noise with an amplitude of more than one standard deviation of the noise. It concerns a few hundreds of rings for 5 bolometers.<br />
<br />
For the three first criteria, a visual inspection of the rmsigTOI at each of the incriminated rings has shown that all these anomalous rings are due to either a drift, a small jump in the TOI trend or a sudden change of noise level, the origin of which is unknown at present.<br />
<br />
Once the list of discarded rings per bolometer is produced, a common list of discarded rings can be produced for all bolometers (by using discarded rings for at least half the bolometers). Such rings correspond to identified phenomena, as can be seen on the following table.<br />
<br />
Furthermore, an isolated valid ring stuck between two common discarded rings becomes discarded as well.<br />
<br />
Table of common discarded rings (v47, 240-21890), truncated to 240-14723 to have nominal mission, to be updated to v53, nominal mission.<br />
<br />
{|table border=1 align=center cellpadding="3" cellspacing="0" style=text-align:center<br />
! '''Cause''' !! '''ring number'''<br />
|-<br />
!align="left"| manoeuvre<br />
|align="center"|304 1312 3590 3611 3642 3922 4949 6379 8456 11328<br />
|-<br />
! align="left"| Sorption Cooler Switchover <br />
|align="center"| 11149 11150 11151 11152 <br />
|-<br />
! align="left"| Rings too long<br />
| align="center"| 440 474 509 544 897 898 3589 13333 14627 14653 <br />
|-<br />
! align="left"| Star tracker switchover <br />
| align="center"| 14628 14654 <br />
|-<br />
! align="left"| Glitch seen by all of the bolometers, but the darks<br />
| align="center"| 7665 <br />
|-<br />
! align="left"| Ring stuck between two common discarded rings<br />
| align="center"| 11150 <br />
|}<br />
<br />
The following figure is a summary of the impact of the discarding process (the solid black line is the common discarded ring percentage).<br />
<br />
<gallery widths=450px perrow=2 caption="Percentage of flagged data due to ring discarding "><br />
File:Fraction_discarded_rings_v53.png | Fraction of discarded rings<br />
File:Fraction_discarded_time_v53.png | Fraction of time due to discarded rings<br />
</gallery><br />
<br />
== Flag description ==<br />
<br />
==== Input flags ====<br />
<br />
These flags are used as inputs to the TOI processing<br />
<br />
- The point-source flag (PSflag):<br />
<br />
An earlier version of HFI point-source catalog is read back into a flag TOIs, at a given frequency. In practice, 5 sigma sources are masked within a radius of 1.3 FWHM (9, 7, 5, 5, 5, 5 arcmin at 100,143,217,353,545,857 GHz) TBC.<br />
<br />
- the galactic flag (Galflag):<br />
<br />
An earlier version of HFI maps is thresholded and apodized. The produced masks are read into flag TOIs. The retained threshold corresponds to a sky coverage of respectively 70, 70, 80, 90, 90, 90% at 100,143,217,353,545,857 GHz.<br />
<br />
- Solar System Object flag <br />
<br />
For the TOI flag, Mars, Jupiter, Saturn are flagged up to a radius of NbBeam= 2,3,3,4,4,4 times the fiducial SSO_FWHM with SSO_FWHM= 9, 7, 5, 5, 5, 5 arcmin at 100,143,217,353,545,857 GHz.<br />
<br />
As an input to planet mask for maps, Mars, Jupiter, Saturn are flagged with a radius computed as a coefficient depending on the planet (Factor_per_source) times NbBeam times SSO_FWHM, with Factor_per_source = 1.1, 2.25, 1.25 for Mars, Jupiter, Saturn respectively and NbBeam = 2.25, 4.25, 4.0, 5.0,.6.0, 8.0 at 100, 143, 217, 353, 545, 857 GHz. This flag is called SSOflag4map.<br />
<br />
A small trailing tail is added to the mask to take into account the non-deconvolution of the planet signal which has been replaced by background values. The width of that tail is 10 % of the main flag diameter. The number of samples which are additionnaly flagged are the Factor_per_Source times AddSNafter with 10, 30, 20, 20, 30, 40 samples at 100, 143, 217, 353, 545, 857 GHz.<br />
<br />
Uranus and Neptune together with detected asteroids are masked by HFI. They are masked at the TOI level using an exclusion radius of 1.5 SSO_FWHM. At 857 GHz, 24 asteroids have been detected with HFI : 1Ceres, 2Pallas, 3Juno, 4Vesta, 7Iris, 8Flora, 9Metis, 10Hygiea, 11Parthenope, 12Victoria, 13Egeria, 14Irene, 15Eunomia, 16Psyche, 18Melpomene, 19Fortuna, 20Massalia, 29Amphitrite, 41Daphne, 45Eugenia, 52Europa, 88Thisbe, 704Interamnia, 324Bamberga.<br />
<br />
<br />
=== Output flags ===<br />
<br />
A FlagTOIproc is produced by the TOIprocessing. It marks measurements which are not reliable for ny of the following reasons :<br />
* gap (no valid input data), enlarged by one sample on each side. It flags less than 0.00044% (resp. 0.00062%) of the nominal (resp. complete) mission. It is equivallen to less thena 3 (resp. 8) minutes of data.<br />
* glitch : samples where the signal from a cosmic ray hit dominates the sky signal at more than 3 $\sigma$ are discarded.<br />
* jump as 100 samples are flagged around the computed position of the jump to take into account the error on this position.<br />
<br />
With and without point-sources, identical to a and b PSBs.<br />
<br />
<br />
Flags produced for the map making :<br />
* UnstablePointing Flag OR FlagTOIproc OR SSOflag4map OR SSOflag seen<br />
* FlagTOIproc = glitch OR jump OR flag thermal template <br />
* PSB flag glitch = flag glitch A OR flag glitch B<br />
<br />
Flag used in the destriping: same + point-source flag<br />
<br />
<br />
<br />
<br />
[[Category:Data processing]]</div>
Fdesert
https://wiki.cosmos.esa.int/planckpla2015/index.php?title=TOI_processing&diff=3282
TOI processing
2013-01-14T08:57:25Z
<p>Fdesert: </p>
<hr />
<div>This Section is kept short as this Planck release does not contain TOIs. The main information in the HFI processing paper is not duplicated here.<br />
<br />
==Overview==<br />
<br />
We describe here the how the TOIs are processed in order to be used for map production. We do not repeat the general features of the pipeline which are given in the HFI Data Processing article (REF). Here we give complementary explanations on some details. The TOI of each bolometer is processed independently of the other bolometers, so as to keep the noise properties as uncorrelated as possible. The processing involves modifying the TOI itself for what concerns the conversion to absorbed power and the correction of glitch tails. It also adds a flag TOI that masks the TOI samples that are not to be projected on maps for various reasons.<br />
<br />
<br />
== Input TOI ==<br />
<br />
The input TOI consists in the AC modulated voltage output of the readout of each bolometer. The input has previously been decompressed, and converted from internal digital units to voltage via a constant factor. The TOI has a regular sampling at the acquisition frequency of facq=180.373700+-0.000050 Hz. There are almost no missing data in the TOIs, except for few hundred samples of 545 and 857GHz TOIs which are lost in the on-board compression due to saturation on the Galactic Center crossings. <br />
<br />
== General Pipeline Structure ==<br />
<br />
The figure on the right shows how the initial ccTOI is transformed and how flags are produced:<br />
<br />
[[File:HFI_4_4_2_ToiProc_ExplS.png|thumb|600px|A schematic of the TOI processing pipeline]]<br />
<br />
<br />
== Output TOIs and products ==<br />
<br />
A TOI of clean calibrated samples (ccTOI) and a combined flag TOI (fTOI) are the outputs of the processing. The ccTOI is calibrated so as to represent the instantaneous power absorbed by the detector up to a constant (which will be determined by the map-making destriper). It is worth mentioning how the ccTOI is changed with respect to the input TOI, beyond the harmless constant conversion factor from voltage to absorbed power. The demodulation stage allows to get the demodulated bolometer voltage. The non-linearity correction is a second-order polynomial correction based on the physical but static bolometer model. In order to avoid too much masking after glitches, a glitch tail is subtracted after an occurrence of a glitch in the TOI. The 4K cooler lines noise is substituted at a series of 9 single temporal frequencies. Finally, the temporal response of the bolometer is deconvolved. This affects mostly the high-temporal frequency part of the TOI, although a small but significant low frequency (the long time response) tail is corrected too. Although flagged samples are not projected, their value influences the valid samples somehow. Hence interpolation procedures introduce some indirect modifications of the TOI.<br />
The flag TOI is a combination a dozen flags with an OR logic. Only unflagged data are projected. The exhaustive list of flags is given here: CompressionError, NoData, SSO, UnstablePointing, Glitch, BoloPlateFluctuation, RTS, Jump, PSBab. A complete qualification of the data is obtained at the ring level. If the TOI shows an anomalous behaviour on a given ring, this ring is discarded from projection. A special production of TOIs is also made as an input to the beam analysis with Mars, Jupiter and Saturn.<br />
<br />
<br />
== Examples of clean TOIs ==<br />
<gallery widths=300px perrow=3><br />
File:10_143_5_PBR_10800RING_LFER4_JC.jpg|Phase-binned signal (PBR) for 6 consecutive rings from 4000 to 4005<br />
File:10_143_5_rcircles_LFER4_JC.jpg| Valid signal (up) and noise (bottom) for 2 consecutive circles of the ring 4000<br />
File:10_143_5_fft_LFER4_JC.jpg| Fourier power spectral density of the first 30 minutes of the ring 4005 with signal before (pink) and after (black) deconvolution (top) and noise (bottom)<br />
File:14_545_1_PBR_10800RING_LFER4_JC.jpg|Phase-binned signal (PBR) for 6 consecutive rings from 4000 to 4005<br />
File:14_545_1_rcircles_LFER4_JC.jpg| Valid signal (up) and noise (bottom) for 2 consecutive circles of the ring 4000<br />
File:14_545_1_fft_LFER4_JC.jpg| Fourier power spectral density of the first 30 minutes of the ring 4005 with signal before (pink) and after (black) deconvolution (top) and noise (bottom)<br />
</gallery><br />
<br />
[[Media:check_ring.pdf|Samples of PBR, TOIs, and PSDs of all detectors are shown in this file]]<br />
<br />
== Trends in the output processing variables ==<br />
<br />
Here we intend to show the trend of the systematic effects that are dealt with in the TOI processing. The full impact of each of them is analyzed in [[HFI-Validation]].<br />
<br />
=== ADC baseline ===<br />
The following figure shows the ADC baseline which is used prior to demodulation (a constant offset is removed for clarity). This baseline is obtained by smoothing on an hour block average the undemodulated TOI on unflagged samples.<br />
[[File:baseline.jpg|400px|ADC baseline for all bolometers]]<br />
<br />
=== Glitch statistics ===<br />
The glitch rate per channel is shown in this figure. For details, see copap. <br />
[[File:figIntermPaperGR.jpg|400px|Glitch rate evolution]]<br />
<br />
The percentage of flagged data (mostly due to Cosmic Rays) at the ring level is shown in these examples. No smoothing was applied. Only valid rings are shown.<br />
<gallery widths=450px perrow=2 caption="Percentage of flagged data"><br />
File:group10_143_5.jpg | 143 GHz bolometers<br />
File:group14_545_1.jpg | 545 GHz bolometers<br />
</gallery><br />
[[Media:PercentUnvalid2.pdf|The complete set of plots is here]]<br />
<br />
=== Thermal template for decorrelation ===<br />
[[File:T90.jpg|400px|Thermal template used for the decorrelation from bolometer plate temperature fluctuations]]<br />
<br />
A simple linear decorrelation is performed using the 2 dark bolometers as a proxy of the bolometer plate temperature. Coupling coefficients were measured during the CPV phase.<br />
<br />
=== 4K cooler lines variability ===<br />
The amplitude of the nine 4K cooler lines in aW at 10, 20, 30, 40, 50, 60, 70, 80 and 17 Hz is shown for 2 bolometers in the following figures. The trend is smoothed over 31 ring values after having discarded measurements done at a ring which is discarded for all bolometers.<br />
<gallery widths=450px perrow=2 caption="Amplitude of the nine 4K cooler lines"><br />
File:10_143_5_4Klines.jpg | 143_5 bolometer<br />
File:14_545_1_4Klines.jpg | 545_1 bolometer<br />
</gallery><br />
[[Media:Lines4K.pdf|The 4K cooler line coefficients of all bolometers are shown in this file]]<br />
<br />
=== jump correction ===<br />
A piecewise constant value is removed to the TOI if a jump is detected. See a jump example in this figure:<br />
[[File:jump_exe.png|200px|An example of a jump seen on the rmsigTOI]]<br />
<br />
The number of jumps per day (all bolometers included) is shown in this figure:<br />
[[File:jumps_per_day.png|200px|Evolution of jump number during the mission]]<br />
<br />
The jumps are uncorrelated from bolometer to bolometer. The total number of jumps detected in the nominal and full mission is shown here:<br />
[[File:jumps_per_bolometer.png|200px|Number of jumps per bolometer]]<br />
<br />
<br />
=== Trends in noise and signal ===<br />
<gallery widths=450px perrow=2 caption="Signal (top) and noise (bottom) smoothed at 1 minute. All values falling in a discarded ring are not plotted."><br />
File:10_143_5_smooth_Watt.jpg | 143_5 bolometer<br />
File:14_545_1_smooth_Watt.jpg | 545_1 bolometer<br />
</gallery><br />
<br />
[[Media:check_smooth.pdf|The smooth TOIs of all detectors are shown in this file]]<br />
<br />
=== Noise stationarity ===<br />
This is not the final version but gives a good idea of power spectra at the detector level of rmsigTOIs.<br />
[[Media:v53_meanSpectra_bySurvey.pdf|All PSDs can be seen in this file]]<br />
<br />
The standard deviation per ring corrected by ring duration bias is given here, one per bolometer using only valid rings. No smoothing is applied (except a 31-point smoothing for the 545<br />
and 857 GHz channels) but values for rings discarded for all bolometers (see below) are not used. The standard deviation is computed on samples valid for map-making which are also not affected by the Galaxy or the point-sources using the usual flags. Two examples are given here.<br />
<gallery widths=450px perrow=2 caption="Standard deviation per ring"><br />
File:stddev_group10_143_5.jpg | 143_5 bolometer<br />
File:stddev_group_14_545_1.jpg | 545_1 bolometer<br />
</gallery><br />
<br />
<br />
The full series of plots is here:<br />
[[Media:StationarityInROIoutput2.pdf|Standard deviation of rmsig TOIs at the ring level]]<br />
<br />
Note the presence for 3 bolometers of a two-level noise system. No correction can be done for that effect. See one example here:<br />
[[File:23_353_TwoLevel.jpg|300px|An example of two-level noise system is seen in bolometer 23_353_3a]]<br />
<br />
<br />
An example of the higher order statistics which are used to unveil rings affected by RTS problems.<br />
[[File:HFI_4_4_2_RTSexample13.jpg|400px|Example of RTS detection]]<br />
<br />
=== Discarded rings ===<br />
<br />
Some rings are discarded (flagged) from further use (beam making, map making) by using ring statistics (see above and the HFI data processing paper REF). For each statistic, we compare each ring value to the ring values averaged (RVA) over a large selection of rings (between 3000 and 21700). We also define the modified standard deviation (MSD) of a ring quantity as the standard deviation of that quantity over the rings that deviate by less than five nominal standard deviations. This truncation is necessary to be robust against extreme deviant rings.<br />
<br />
More specifically, a ring is discarded if it matches one of the following criteria:<br />
<br />
* the | mean-median | deviates from the RVA by more than fifteen times the MSD.<br />
* the standard deviation deviates from the RVA by more than -5 times the MSD (all cases corresponding to almost empty rings) or +15 times the MSD.<br />
* the Kolmogorov-Smirnov test deviates from the RVA by more than 15 times the MSD.<br />
* the ring duration is more than 90 min.<br />
* the ring is contaminated by popcorn noise with an amplitude of more than one standard deviation of the noise. It concerns a few hundreds of rings for 5 bolometers.<br />
<br />
For the three first criteria, a visual inspection of the rmsigTOI has shown that all these anomalous rings are due to either a drift or a jump in the TOI trend.<br />
<br />
Once the list of discarded rings per bolometer is produced, a common list of discarded rings can be produced for all bolometers (by using discarded rings for at least half the bolometers). Such rings correspond to identified phenomena, as can be seen on the following table.<br />
<br />
Furthermore, an isolated valid ring stuck between two common discarded rings becomes discarded as well.<br />
<br />
Table of common discarded rings (v47, 240-21890), truncated to 240-14723 to have nominal mission, to be updated to v53, nominal mission.<br />
<br />
{|table border=1 align=center cellpadding="3" cellspacing="0" style=text-align:center<br />
! '''Cause''' !! '''ring number'''<br />
|-<br />
!align="left"| manoeuvre<br />
|align="center"|304 1312 3590 3611 3642 3922 4949 6379 8456 11328<br />
|-<br />
! align="left"| Sorption Cooler Switchover <br />
|align="center"| 11149 11150 11151 11152 <br />
|-<br />
! align="left"| Rings too long<br />
| align="center"| 440 474 509 544 897 898 3589 13333 14627 14653 <br />
|-<br />
! align="left"| Star tracker switchover <br />
| align="center"| 14628 14654 <br />
|-<br />
! align="left"| Glitch seen by all of the bolometers, but the darks<br />
| align="center"| 7665 <br />
|-<br />
! align="left"| Ring stuck between two common discarded rings<br />
| align="center"| 11150 <br />
|}<br />
<br />
The following figure is a summary of the impact of the discarding process (the solid black line is the common discarded ring percentage).<br />
<br />
<gallery widths=450px perrow=2 caption="Percentage of flagged data due to ring discarding "><br />
File:Fraction_discarded_rings_v53.png | Fraction of discarded rings<br />
File:Fraction_discarded_time_v53.png | Fraction of time due to discarded rings<br />
</gallery><br />
<br />
== Flag description ==<br />
<br />
==== Input flags ====<br />
<br />
These flags are used as inputs to the TOI processing<br />
<br />
- The point-source flag (PSflag):<br />
<br />
An earlier version of HFI point-source catalog is read back into a flag TOIs, at a given frequency. In practice, 5 sigma sources are masked within a radius of 1.3 FWHM (9, 7, 5, 5, 5, 5 arcmin at 100,143,217,353,545,857 GHz) TBC.<br />
<br />
- the galactic flag (Galflag):<br />
<br />
An earlier version of HFI maps is thresholded and apodized. The produced masks are read into flag TOIs. The retained threshold corresponds to a sky coverage of respectively 70, 70, 80, 90, 90, 90% at 100,143,217,353,545,857 GHz.<br />
<br />
- Solar System Object flag <br />
<br />
For the TOI flag, Mars, Jupiter, Saturn are flagged up to a radius of NbBeam= 2,3,3,4,4,4 times the fiducial SSO_FWHM with SSO_FWHM= 9, 7, 5, 5, 5, 5 arcmin at 100,143,217,353,545,857 GHz.<br />
<br />
As an input to planet mask for maps, Mars, Jupiter, Saturn are flagged with a radius computed as a coefficient depending on the planet (Factor_per_source) times NbBeam times SSO_FWHM, with Factor_per_source = 1.1, 2.25, 1.25 for Mars, Jupiter, Saturn respectively and NbBeam = 2.25, 4.25, 4.0, 5.0,.6.0, 8.0 at 100, 143, 217, 353, 545, 857 GHz. This flag is called SSOflag4map.<br />
<br />
A small trailing tail is added to the mask to take into account the non-deconvolution of the planet signal which has been replaced by background values. The width of that tail is 10 % of the main flag diameter. The number of samples which are additionnaly flagged are the Factor_per_Source times AddSNafter with 10, 30, 20, 20, 30, 40 samples at 100, 143, 217, 353, 545, 857 GHz.<br />
<br />
Uranus and Neptune together with detected asteroids are masked by HFI. They are masked at the TOI level using an exclusion radius of 1.5 SSO_FWHM. At 857 GHz, 24 asteroids have been detected with HFI : 1Ceres, 2Pallas, 3Juno, 4Vesta, 7Iris, 8Flora, 9Metis, 10Hygiea, 11Parthenope, 12Victoria, 13Egeria, 14Irene, 15Eunomia, 16Psyche, 18Melpomene, 19Fortuna, 20Massalia, 29Amphitrite, 41Daphne, 45Eugenia, 52Europa, 88Thisbe, 704Interamnia, 324Bamberga.<br />
<br />
<br />
=== Output flags ===<br />
<br />
A FlagTOIproc is produced by the TOIprocessing. It marks measurements which are not reliable for ny of the following reasons :<br />
* gap (no valid input data), enlarged by one sample on each side. It flags less than 0.00044% (resp. 0.00062%) of the nominal (resp. complete) mission. It is equivallen to less thena 3 (resp. 8) minutes of data.<br />
* glitch : samples where the signal from a cosmic ray hit dominates the sky signal at more than 3 $\sigma$ are discarded.<br />
* jump as 100 samples are flagged around the computed position of the jump to take into account the error on this position.<br />
<br />
With and without point-sources, identical to a and b PSBs.<br />
<br />
<br />
Flags produced for the map making :<br />
* UnstablePointing Flag OR FlagTOIproc OR SSOflag4map OR SSOflag seen<br />
* FlagTOIproc = glitch OR jump OR flag thermal template <br />
* PSB flag glitch = flag glitch A OR flag glitch B<br />
<br />
Flag used in the destriping: same + point-source flag<br />
<br />
<br />
<br />
<br />
[[Category:Data processing]]</div>
Fdesert
https://wiki.cosmos.esa.int/planckpla2015/index.php?title=TOI_processing&diff=3263
TOI processing
2013-01-10T13:53:54Z
<p>Fdesert: </p>
<hr />
<div>This Section is kept short as this Planck release does not contain TOIs. The main information in the HFI processing paper is not duplicated here.<br />
<br />
==Overview==<br />
<br />
We describe here the how the TOIs are processed in order to be used for map production. We do not repeat the general features of the pipeline which are given in the HFI Data Processing article (REF). Here we give complementary explanations on some details. The TOI of each bolometer is processed independently of the other bolometers, so as to keep the noise properties as uncorrelated as possible. The processing involves modifying the TOI itself for what concerns the conversion to absorbed power and the correction of glitch tails. It also adds a flag TOI that masks the TOI samples that are not to be projected on maps for various reasons.<br />
<br />
<br />
== Input TOI ==<br />
<br />
The input TOI consists in the AC modulated voltage output of the readout of each bolometer. The input has previously been decompressed, and converted from internal digital units to voltage via a constant factor. The TOI has a regular sampling at the acquisition frequency of facq=180.373700+-0.000050 Hz. There are almost no missing data in the TOIs, except for few hundred samples of 545 and 857GHz TOIs which are lost in the on-board compression due to saturation on the Galactic Center crossings. <br />
<br />
== General Pipeline Structure ==<br />
<br />
The figure on the right shows how the initial ccTOI is transformed and how flags are produced:<br />
<br />
[[File:HFI_4_4_2_ToiProc_ExplS.png|thumb|600px|A schematic of the TOI processing pipeline]]<br />
<br />
<br />
== Output TOIs and products ==<br />
<br />
A TOI of clean calibrated samples (ccTOI) and a combined flag TOI (fTOI) are the outputs of the processing. The ccTOI is calibrated so as to represent the instantaneous power absorbed by the detector up to a constant (which will be determined by the map-making destriper). It is worth mentioning how the ccTOI is changed with respect to the input TOI, beyond the harmless constant conversion factor from voltage to absorbed power. The demodulation stage allows to get the demodulated bolometer voltage. The non-linearity correction is a second-order polynomial correction based on the physical but static bolometer model. In order to avoid too much masking after glitches, a glitch tail is subtracted after an occurrence of a glitch in the TOI. The 4K cooler lines noise is substituted at a series of 9 single temporal frequencies. Finally, the temporal response of the bolometer is deconvolved. This affects mostly the high-temporal frequency part of the TOI, although a small but significant low frequency (the long time response) tail is corrected too. Although flagged samples are not projected, their value influences the valid samples somehow. Hence interpolation procedures introduce some indirect modifications of the TOI.<br />
The flag TOI is a combination a dozen flags with an OR logic. Only unflagged data are projected. The exhaustive list of flags is given here: CompressionError, NoData, SSO, UnstablePointing, Glitch, BoloPlateFluctuation, RTS, Jump, PSBab. A complete qualification of the data is obtained at the ring level. If the TOI shows an anomalous behaviour on a given ring, this ring is discarded from projection. A special production of TOIs is also made as an input to the beam analysis with Mars, Jupiter and Saturn.<br />
<br />
<br />
== Examples of clean TOIs ==<br />
<gallery widths=300px perrow=3><br />
File:10_143_5_PBR_10800RING_LFER4_JC.jpg|Phase-binned signal (PBR) for 6 consecutive rings from 4000 to 4005<br />
File:10_143_5_rcircles_LFER4_JC.jpg| Valid signal (up) and noise (bottom) for 2 consecutive circles of the ring 4000<br />
File:10_143_5_fft_LFER4_JC.jpg| Fourier power spectral density of the first 30 minutes of the ring 4005 with signal before (pink) and after (black) deconvolution (top) and noise (bottom)<br />
File:14_545_1_PBR_10800RING_LFER4_JC.jpg|Phase-binned signal (PBR) for 6 consecutive rings from 4000 to 4005<br />
File:14_545_1_rcircles_LFER4_JC.jpg| Valid signal (up) and noise (bottom) for 2 consecutive circles of the ring 4000<br />
File:14_545_1_fft_LFER4_JC.jpg| Fourier power spectral density of the first 30 minutes of the ring 4005 with signal before (pink) and after (black) deconvolution (top) and noise (bottom)<br />
</gallery><br />
<br />
[[Media:check_ring.pdf|Samples of PBR, TOIs, and PSDs of all detectors are shown in this file]]<br />
<br />
== Trends in the output processing variables ==<br />
<br />
Here we intend to show the trend of the systematic effects that are dealt with in the TOI processing. The full impact of each of them is analyzed in [[HFI-Validation]].<br />
<br />
=== ADC baseline ===<br />
The following figure shows the ADC baseline which is used prior to demodulation (a constant offset is removed for clarity). This baseline is obtained by smoothing on an hour block average the undemodulated TOI on unflagged samples.<br />
[[File:baseline.jpg|400px|ADC baseline for all bolometers]]<br />
<br />
=== Glitch statistics ===<br />
The glitch rate per channel is shown in this figure. For details, see copap. <br />
[[File:figIntermPaperGR.jpg|400px|Glitch rate evolution]]<br />
<br />
The percentage of flagged data (mostly due to Cosmic Rays) at the ring level is shown in these examples. No smoothing was applied. Only valid rings are shown.<br />
<gallery widths=450px perrow=2 caption="Percentage of flagged data"><br />
File:group10_143_5.jpg | 143 GHz bolometers<br />
File:group14_545_1.jpg | 545 GHz bolometers<br />
</gallery><br />
[[Media:PercentUnvalid2.pdf|The complete set of plots is here]]<br />
<br />
=== Thermal template for decorrelation ===<br />
[[File:T90.jpg|400px|Thermal template used for the decorrelation from bolometer plate temperature fluctuations]]<br />
<br />
A simple linear decorrelation is performed using the 2 dark bolometers as a proxy of the bolometer plate temperature. Coupling coefficients were measured during the CPV phase.<br />
<br />
=== 4K cooler lines variability ===<br />
The amplitude of the nine 4K cooler lines in aW at 10, 20, 30, 40, 50, 60, 70, 80 and 17 Hz is shown for 2 bolometers in the following figures. The trend is smoothed over 31 ring values after having discarded measurements done at a ring which is discarded for all bolometers.<br />
<gallery widths=450px perrow=2 caption="Amplitude of the nine 4K cooler lines"><br />
File:10_143_5_4Klines.jpg | 143_5 bolometer<br />
File:14_545_1_4Klines.jpg | 545_1 bolometer<br />
</gallery><br />
[[Media:Lines4K.pdf|The 4K cooler line coefficients of all bolometers are shown in this file]]<br />
<br />
=== jump correction ===<br />
A piecewise constant value is removed to the TOI if a jump is detected. See a jump example in this figure:<br />
[[File:jump_exe.png|200px|An example of a jump seen on the rmsigTOI]]<br />
<br />
The number of jumps per day (all bolometers included) is shown in this figure:<br />
[[File:jumps_per_day.png|200px|Evolution of jump number during the mission]]<br />
<br />
The jumps are uncorrelated from bolometer to bolometer. The total number of jumps detected in the nominal and full mission is shown here:<br />
[[File:jumps_per_bolometer.png|200px|Number of jumps per bolometer]]<br />
<br />
<br />
=== Trends in noise and signal ===<br />
<gallery widths=450px perrow=2 caption="Signal (top) and noise (bottom) smoothed at 1 minute. All values falling in a discarded ring are not plotted."><br />
File:10_143_5_smooth_Watt.jpg | 143_5 bolometer<br />
File:14_545_1_smooth_Watt.jpg | 545_1 bolometer<br />
</gallery><br />
<br />
[[Media:check_smooth.pdf|The smooth TOIs of all detectors are shown in this file]]<br />
<br />
=== Noise stationarity ===<br />
This is not the final version but gives a good idea of power spectra at the detector level of rmsigTOIs.<br />
[[Media:v53_meanSpectra_bySurvey.pdf|All PSDs can be seen in this file]]<br />
<br />
The standard deviation per ring corrected by ring duration bias is given here, one per bolometer using only valid rings. No smoothing is applied (except a 31-point smoothing for the 545<br />
and 857 GHz channels) but values for rings discarded for all bolometers are not used. The standard deviation is computed on samples valid for map-making which are also not affected by the Galaxy or the point-sources using the usual flags. Two examples are given here.<br />
<gallery widths=450px perrow=2 caption="Standard deviation per ring"><br />
File:stddev_group10_143_5.jpg | 143_5 bolometer<br />
File:stddev_group_14_545_1.jpg | 545_1 bolometer<br />
</gallery><br />
<br />
<br />
The full series of plots is here:<br />
[[Media:StationarityInROIoutput2.pdf|Standard deviation of rmsig TOIs at the ring level]]<br />
<br />
Note the presence for 3 bolometers of a two-level noise system. No correction can be done for that effect. See one example here:<br />
[[File:23_353_TwoLevel.jpg|300px|An example of two-level noise system is seen in bolometer 23_353_3a]]<br />
<br />
<br />
An example of the higher order statistics which are used to unveil rings affected by RTS problems.<br />
[[File:HFI_4_4_2_RTSexample13.jpg|400px|Example of RTS detection]]<br />
<br />
=== Anomalous rings ===<br />
<br />
Some rings are thus discarded (flagged) from further use (beam making, map making) by using these ring statistics (see HFI data processing paper REF).<br />
For each statistics, we compare each ring value to the ring values averaged (RVA) over all rings. We also define the modified standard deviation (MSD) of a ring quantity as the standard deviation of that quantity over the rings that deviate by less than five nominal standard deviations. This truncation is necessary to be robust against extreme deviant rings.<br />
<br />
More specifically, these are the full criteria (with the OR logic) to decide whether a ring is discarded:<br />
<br />
- if |mean-median| deviates from the RVA by more than fifteen times the MSD.<br />
<br />
- if the standard deviation deviates from the RVA by more than -5 times the MSD (all cases corresponding to almost empty rings) and +15 times the MSD.<br />
<br />
- if the Kolmogorov-Smirnov test deviates from the RVA by more than 15 times the MSD.<br />
<br />
- if the ring duration is outside some definite bounds. Bounds are 25 min and 90 min.<br />
<br />
<br />
Once the list of discarded rings is done per bolometer, a common list of discarded rings can be produced for all bolometers (by using discarded rings for at least half the bolometers). See the following table.<br />
<br />
Furthermore, an isolated valid ring stuck between two discarded rings becomes discarded as well.<br />
<br />
Table of common discarded rings (v47, 240-21890), truncated to 240-14723 to have nominal mission, to be updated to v53, nominal mission.<br />
<br />
{|table border=1 align=center cellpadding="3" cellspacing="0" style=text-align:center<br />
! '''Cause''' !! '''ring number'''<br />
|-<br />
!align="left"| manoeuvre<br />
|align="center"|304 1312 3590 3611 3642 3922 4949 6379 8456 11328<br />
|-<br />
! align="left"| Sorption Cooler Switchover <br />
|align="center"| 11149 11150 11151 11152 <br />
|-<br />
! align="left"| Rings too long<br />
| align="center"| 440 474 509 544 897 898 3589 13333 14627 14653 <br />
|-<br />
! align="left"| fully flagged as unstable <br />
| align="center"| 14628 14654 <br />
|-<br />
! align="left"| common jump <br />
| align="center"| 7665 <br />
|}<br />
<br />
== Flag description ==<br />
<br />
==== Input flags ====<br />
<br />
These flags are used as inputs to the TOI processing<br />
<br />
- The point-source flag (PSflag):<br />
<br />
An earlier version of HFI point-source catalog is read back into a flag TOIs, at a given frequency. In practice, 5 sigma sources are masked within a radius of 1.3 FWHM (9, 7, 5, 5, 5, 5 arcmin at 100,143,217,353,545,857 GHz) TBC.<br />
<br />
- the galactic flag (Galflag):<br />
<br />
An earlier version of HFI maps is thresholded and apodized. The produced masks are read into flag TOIs. The retained threshold corresponds to a sky coverage of respectively 70, 70, 80, 90, 90, 90% at 100,143,217,353,545,857 GHz.<br />
<br />
- Solar System Object flag <br />
<br />
For the TOI flag, Mars, Jupiter, Saturn are flagged up to a radius of NbBeam= 2,3,3,4,4,4 times the fiducial SSO_FWHM with SSO_FWHM= 9, 7, 5, 5, 5, 5 arcmin at 100,143,217,353,545,857 GHz.<br />
<br />
As an input to planet mask for maps, Mars, Jupiter, Saturn are flagged with a radius computed as a coefficient depending on the planet (Factor_per_source) times NbBeam times SSO_FWHM, with Factor_per_source = 1.1, 2.25, 1.25 for Mars, Jupiter, Saturn respectively and NbBeam = 2.25, 4.25, 4.0, 5.0,.6.0, 8.0 at 100, 143, 217, 353, 545, 857 GHz. This flag is called SSOflag4map.<br />
<br />
A small trailing tail is added to the mask to take into account the non-deconvolution of the planet signal which has been replaced by background values. The width of that tail is 10 % of the main flag diameter. The number of samples which are additionnaly flagged are the Factor_per_Source times AddSNafter with 10, 30, 20, 20, 30, 40 samples at 100, 143, 217, 353, 545, 857 GHz.<br />
<br />
Uranus and Neptune together with detected asteroids are masked by HFI. They are masked at the TOI level using an exclusion radius of 1.5 SSO_FWHM. At 857 GHz, 24 asteroids have been detected with HFI : 1Ceres, 2Pallas, 3Juno, 4Vesta, 7Iris, 8Flora, 9Metis, 10Hygiea, 11Parthenope, 12Victoria, 13Egeria, 14Irene, 15Eunomia, 16Psyche, 18Melpomene, 19Fortuna, 20Massalia, 29Amphitrite, 41Daphne, 45Eugenia, 52Europa, 88Thisbe, 704Interamnia, 324Bamberga.<br />
<br />
<br />
=== Output flags ===<br />
<br />
A FlagTOIproc is produced by the TOIprocessing. It marks measurements which are not reliable for ny of the following reasons :<br />
* gap (no valid input data), enlarged by one sample on each side. It flags less than 0.00044% (resp. 0.00062%) of the nominal (resp. complete) mission. It is equivallen to less thena 3 (resp. 8) minutes of data.<br />
* glitch : samples where the signal from a cosmic ray hit dominates the sky signal at more than 3 $\sigma$ are discarded.<br />
* jump as 100 samples are flagged around the computed position of the jump to take into account the error on this position.<br />
<br />
With and without point-sources, identical to a and b PSBs.<br />
<br />
<br />
Flags produced for the map making :<br />
* UnstablePointing Flag OR FlagTOIproc OR SSOflag4map OR SSOflag seen<br />
* FlagTOIproc = glitch OR jump OR flag thermal template <br />
* PSB flag glitch = flag glitch A OR flag glitch B<br />
<br />
Flag used in the destriping: same + point-source flag<br />
<br />
<br />
<br />
<br />
[[Category:Data processing]]</div>
Fdesert
https://wiki.cosmos.esa.int/planckpla2015/index.php?title=TOI_processing&diff=3262
TOI processing
2013-01-10T13:53:04Z
<p>Fdesert: </p>
<hr />
<div>This Section is kept short as this Planck release does not contain TOIs. The main information in the HFI processing paper is not duplicated here.<br />
<br />
==Overview==<br />
<br />
We describe here the how the TOIs are processed in order to be used for map production. We do not repeat the general features of the pipeline which are given in the HFI Data Processing article (REF). Here we give complementary explanations on some details. The TOI of each bolometer is processed independently of the other bolometers, so as to keep the noise properties as uncorrelated as possible. The processing involves modifying the TOI itself for what concerns the conversion to absorbed power and the correction of glitch tails. It also adds a flag TOI that masks the TOI samples that are not to be projected on maps for various reasons.<br />
<br />
<br />
== Input TOI ==<br />
<br />
The input TOI consists in the AC modulated voltage output of the readout of each bolometer. The input has previously been decompressed, and converted from internal digital units to voltage via a constant factor. The TOI has a regular sampling at the acquisition frequency of facq=180.373700+-0.000050 Hz. There are almost no missing data in the TOIs, except for few hundred samples of 545 and 857GHz TOIs which are lost in the on-board compression due to saturation on the Galactic Center crossings. <br />
<br />
== General Pipeline Structure ==<br />
<br />
The figure on the right shows how the initial ccTOI is transformed and how flags are produced:<br />
<br />
[[File:HFI_4_4_2_ToiProc_ExplS.png|thumb|600px|A schematic of the TOI processing pipeline]]<br />
<br />
<br />
== Output TOIs and products ==<br />
<br />
A TOI of clean calibrated samples (ccTOI) and a combined flag TOI (fTOI) are the outputs of the processing. The ccTOI is calibrated so as to represent the instantaneous power absorbed by the detector up to a constant (which will be determined by the map-making destriper). It is worth mentioning how the ccTOI is changed with respect to the input TOI, beyond the harmless constant conversion factor from voltage to absorbed power. The demodulation stage allows to get the demodulated bolometer voltage. The non-linearity correction is a second-order polynomial correction based on the physical but static bolometer model. In order to avoid too much masking after glitches, a glitch tail is subtracted after an occurrence of a glitch in the TOI. The 4K cooler lines noise is substituted at a series of 9 single temporal frequencies. Finally, the temporal response of the bolometer is deconvolved. This affects mostly the high-temporal frequency part of the TOI, although a small but significant low frequency (the long time response) tail is corrected too. Although flagged samples are not projected, their value influences the valid samples somehow. Hence interpolation procedures introduce some indirect modifications of the TOI.<br />
The flag TOI is a combination a dozen flags with an OR logic. Only unflagged data are projected. The exhaustive list of flags is given here: CompressionError, NoData, SSO, UnstablePointing, Glitch, BoloPlateFluctuation, RTS, Jump, PSBab. A complete qualification of the data is obtained at the ring level. If the TOI shows an anomalous behaviour on a given ring, this ring is discarded from projection. A special production of TOIs is also made as an input to the beam analysis with Mars, Jupiter and Saturn.<br />
<br />
<br />
== Examples of clean TOIs ==<br />
<gallery widths=300px perrow=3><br />
File:10_143_5_PBR_10800RING_LFER4_JC.jpg|Phase-binned signal (PBR) for 6 consecutive rings from 4000 to 4005<br />
File:10_143_5_rcircles_LFER4_JC.jpg| Valid signal (up) and noise (bottom) for 2 consecutive circles of the ring 4000<br />
File:10_143_5_fft_LFER4_JC.jpg| Fourier power spectral density of the first 30 minutes of the ring 4005 with signal before (pink) and after (black) deconvolution (top) and noise (bottom)<br />
File:14_545_1_PBR_10800RING_LFER4_JC.jpg|Phase-binned signal (PBR) for 6 consecutive rings from 4000 to 4005<br />
File:14_545_1_rcircles_LFER4_JC.jpg| Valid signal (up) and noise (bottom) for 2 consecutive circles of the ring 4000<br />
File:14_545_1_fft_LFER4_JC.jpg| Fourier power spectral density of the first 30 minutes of the ring 4005 with signal before (pink) and after (black) deconvolution (top) and noise (bottom)<br />
</gallery><br />
<br />
[[Media:check_ring.pdf|Samples of PBR, TOIs, and PSDs of all detectors are shown in this file]]<br />
<br />
== Trends in the output processing variables ==<br />
<br />
Here we intend to show the trend of the systematic effects that are dealt with in the TOI processing. The full impact of each of them is analyzed in [[HFI-Validation]].<br />
<br />
=== ADC baseline ===<br />
The following figure shows the ADC baseline which is used prior to demodulation (a constant offset is removed for clarity). This baseline is obtained by smoothing on an hour block average the undemodulated TOI on unflagged samples.<br />
[[File:baseline.jpg|400px|ADC baseline for all bolometers]]<br />
<br />
=== Glitch statistics ===<br />
The glitch rate per channel is shown in this figure. For details, see copap. <br />
[[File:figIntermPaperGR.jpg|400px|Glitch rate evolution]]<br />
<br />
The percentage of flagged data (mostly due to Cosmic Rays) at the ring level is shown in these examples. No smoothing was applied. Only valid rings are shown.<br />
<gallery widths=450px perrow=2 caption="Percentage of flagged data"><br />
File:group10_143_5.jpg | 143 GHz bolometers<br />
File:group14_545_1.jpg | 545 GHz bolometers<br />
</gallery><br />
[[Media:PercentUnvalid2.pdf|The complete set of plots is here]]<br />
<br />
=== Thermal template for decorrelation ===<br />
[[File:T90.jpg|400px|Thermal template used for the decorrelation from bolometer plate temperature fluctuations]]<br />
<br />
A simple linear decorrelation is performed using the 2 dark bolometers as a proxy of the bolometer plate temperature. Coupling coefficients were measured during the CPV phase.<br />
<br />
=== 4K cooler lines variability ===<br />
The amplitude of the nine 4K cooler lines in aW at 10, 20, 30, 40, 50, 60, 70, 80 and 17 Hz is shown for 2 bolometers in the following figures. The trend is smoothed over 31 ring values after having discarded measurements done at a ring which is discarded for all bolometers.<br />
<gallery widths=450px perrow=2 caption="Amplitude of the nine 4K cooler lines"><br />
File:10_143_5_4Klines.jpg | 143_5 bolometer<br />
File:14_545_1_4Klines.jpg | 545_1 bolometer<br />
</gallery><br />
[[Media:Lines4K.pdf|The 4K cooler line coefficients of all bolometers are shown in this file]]<br />
<br />
=== jump correction ===<br />
A piecewise constant value is removed to the TOI if a jump is detected. See a jump example in this figure:<br />
[[File:jump_exe.png|200px|An example of a jump seen on the rmsigTOI]]<br />
<br />
The number of jumps per day (all bolometers included) is shown in this figure:<br />
[[File:jumps_per_day.png|200px|Evolution of jump number during the mission]]<br />
<br />
The jumps are uncorrelated from bolometer to bolometer. The total number of jumps detected in the nominal and full mission is shown here:<br />
[[File:jumps_per_bolometer.png|200px|Number of jumps per bolometer]]<br />
<br />
<br />
=== Trends in noise and signal ===<br />
<gallery widths=450px perrow=2 caption="Signal (top) and noise (bottom) smoothed at 1 minute. All values falling in a discarded ring are not plotted."><br />
File:10_143_5_smooth_Watt.jpg | 143_5 bolometer<br />
File:14_545_1_smooth_Watt.jpg | 545_1 bolometer<br />
</gallery><br />
<br />
[[Media:check_smooth.pdf|The smooth TOIs of all detectors are shown in this file]]<br />
<br />
=== Noise stationarity ===<br />
This is not the final version but gives a good idea of power spectra at the detector level of rmsigTOIs.<br />
[[Media:v53_meanSpectra_bySurvey.pdf|All PSDs can be seen in this file]]<br />
<br />
The standard deviation per ring corrected by ring duration bias is given here, one per bolometer using only valid rings. No smoothing is applied (except a 31-point smoothing for the 545<br />
and 857 GHz channels) but values for rings discarded for all bolometers are not used. The standard deviation is computed on samples valid for map-making which are also not affected by the Galaxy or the point-sources using the usual flags. Two examples are given here.<br />
<gallery widths=450px perrow=2 caption="Standard deviation per ring"><br />
File:stddev_group10_143_5.jpg | 143_5 bolometer<br />
File:stddev_group_14_545_1.jpg | 545_1 bolometer<br />
</gallery><br />
<br />
<br />
The full series of plots is here:<br />
[[Media:StationarityInROIoutput2.pdf|Standard deviation of rmsig TOIs at the ring level]]<br />
<br />
Note the presence for 3 bolometers of a two-level noise system. No correction can be done for that effect. See one example here:<br />
[[File:23_353_TwoLevel.jpg|300px|An example of two-level noise system is seen in bolometer 23_353_3a]]<br />
<br />
<br />
An example of the higher order statistics which are used to unveil rings affected by RTS problems.<br />
[[File:HFI_4_4_2_RTSexample13.jpg|400px|Example of RTS detection]]<br />
<br />
== Anomalous rings ==<br />
<br />
Some rings are thus discarded (flagged) from further use (beam making, map making) by using these ring statistics (see HFI data processing paper REF).<br />
For each statistics, we compare each ring value to the ring values averaged (RVA) over all rings. We also define the modified standard deviation (MSD) of a ring quantity as the standard deviation of that quantity over the rings that deviate by less than five nominal standard deviations. This truncation is necessary to be robust against extreme deviant rings.<br />
<br />
More specifically, these are the full criteria (with the OR logic) to decide whether a ring is discarded:<br />
<br />
- if |mean-median| deviates from the RVA by more than fifteen times the MSD.<br />
<br />
- if the standard deviation deviates from the RVA by more than -5 times the MSD (all cases corresponding to almost empty rings) and +15 times the MSD.<br />
<br />
- if the Kolmogorov-Smirnov test deviates from the RVA by more than 15 times the MSD.<br />
<br />
- if the ring duration is outside some definite bounds. Bounds are 25 min and 90 min.<br />
<br />
<br />
Once the list of discarded rings is done per bolometer, a common list of discarded rings can be produced for all bolometers (by using discarded rings for at least half the bolometers). See the following table.<br />
<br />
Furthermore, an isolated valid ring stuck between two discarded rings becomes discarded as well.<br />
<br />
Table of common discarded rings (v47, 240-21890), truncated to 240-14723 to have nominal mission, to be updated to v53, nominal mission.<br />
<br />
{|table border=1 align=center cellpadding="3" cellspacing="0" style=text-align:center<br />
! '''Cause''' !! '''ring number'''<br />
|-<br />
!align="left"| manoeuvre<br />
|align="center"|304 1312 3590 3611 3642 3922 4949 6379 8456 11328<br />
|-<br />
! align="left"| Sorption Cooler Switchover <br />
|align="center"| 11149 11150 11151 11152 <br />
|-<br />
! align="left"| Rings too long<br />
| align="center"| 440 474 509 544 897 898 3589 13333 14627 14653 <br />
|-<br />
! align="left"| fully flagged as unstable <br />
| align="center"| 14628 14654 <br />
|-<br />
! align="left"| common jump <br />
| align="center"| 7665 <br />
|}<br />
<br />
== Flag description ==<br />
<br />
==== Input flags ====<br />
<br />
These flags are used as inputs to the TOI processing<br />
<br />
- The point-source flag (PSflag):<br />
<br />
An earlier version of HFI point-source catalog is read back into a flag TOIs, at a given frequency. In practice, 5 sigma sources are masked within a radius of 1.3 FWHM (9, 7, 5, 5, 5, 5 arcmin at 100,143,217,353,545,857 GHz) TBC.<br />
<br />
- the galactic flag (Galflag):<br />
<br />
An earlier version of HFI maps is thresholded and apodized. The produced masks are read into flag TOIs. The retained threshold corresponds to a sky coverage of respectively 70, 70, 80, 90, 90, 90% at 100,143,217,353,545,857 GHz.<br />
<br />
- Solar System Object flag <br />
<br />
For the TOI flag, Mars, Jupiter, Saturn are flagged up to a radius of NbBeam= 2,3,3,4,4,4 times the fiducial SSO_FWHM with SSO_FWHM= 9, 7, 5, 5, 5, 5 arcmin at 100,143,217,353,545,857 GHz.<br />
<br />
As an input to planet mask for maps, Mars, Jupiter, Saturn are flagged with a radius computed as a coefficient depending on the planet (Factor_per_source) times NbBeam times SSO_FWHM, with Factor_per_source = 1.1, 2.25, 1.25 for Mars, Jupiter, Saturn respectively and NbBeam = 2.25, 4.25, 4.0, 5.0,.6.0, 8.0 at 100, 143, 217, 353, 545, 857 GHz. This flag is called SSOflag4map.<br />
<br />
A small trailing tail is added to the mask to take into account the non-deconvolution of the planet signal which has been replaced by background values. The width of that tail is 10 % of the main flag diameter. The number of samples which are additionnaly flagged are the Factor_per_Source times AddSNafter with 10, 30, 20, 20, 30, 40 samples at 100, 143, 217, 353, 545, 857 GHz.<br />
<br />
Uranus and Neptune together with detected asteroids are masked by HFI. They are masked at the TOI level using an exclusion radius of 1.5 SSO_FWHM. At 857 GHz, 24 asteroids have been detected with HFI : 1Ceres, 2Pallas, 3Juno, 4Vesta, 7Iris, 8Flora, 9Metis, 10Hygiea, 11Parthenope, 12Victoria, 13Egeria, 14Irene, 15Eunomia, 16Psyche, 18Melpomene, 19Fortuna, 20Massalia, 29Amphitrite, 41Daphne, 45Eugenia, 52Europa, 88Thisbe, 704Interamnia, 324Bamberga.<br />
<br />
<br />
=== Output flags ===<br />
<br />
A FlagTOIproc is produced by the TOIprocessing. It marks measurements which are not reliable for ny of the following reasons :<br />
* gap (no valid input data), enlarged by one sample on each side. It flags less than 0.00044% (resp. 0.00062%) of the nominal (resp. complete) mission. It is equivallen to less thena 3 (resp. 8) minutes of data.<br />
* glitch : samples where the signal from a cosmic ray hit dominates the sky signal at more than 3 $\sigma$ are discarded.<br />
* jump as 100 samples are flagged around the computed position of the jump to take into account the error on this position.<br />
<br />
With and without point-sources, identical to a and b PSBs.<br />
<br />
<br />
Flags produced for the map making :<br />
* UnstablePointing Flag OR FlagTOIproc OR SSOflag4map OR SSOflag seen<br />
* FlagTOIproc = glitch OR jump OR flag thermal template <br />
* PSB flag glitch = flag glitch A OR flag glitch B<br />
<br />
Flag used in the destriping: same + point-source flag<br />
<br />
<br />
<br />
<br />
[[Category:Data processing]]</div>
Fdesert
https://wiki.cosmos.esa.int/planckpla2015/index.php?title=TOI_processing&diff=3261
TOI processing
2013-01-10T13:51:02Z
<p>Fdesert: </p>
<hr />
<div>This Section is kept short as this Planck release does not contain TOIs. The main information in the HFI processing paper is not duplicated here.<br />
<br />
==Overview==<br />
<br />
We describe here the how the TOIs are processed in order to be used for map production. We do not repeat the general features of the pipeline which are given in the HFI Data Processing article (REF). Here we give complementary explanations on some details. The TOI of each bolometer is processed independently of the other bolometers, so as to keep the noise properties as uncorrelated as possible. The processing involves modifying the TOI itself for what concerns the conversion to absorbed power and the correction of glitch tails. It also adds a flag TOI that masks the TOI samples that are not to be projected on maps for various reasons.<br />
<br />
<br />
== Input TOI ==<br />
<br />
The input TOI consists in the AC modulated voltage output of the readout of each bolometer. The input has previously been decompressed, and converted from internal digital units to voltage via a constant factor. The TOI has a regular sampling at the acquisition frequency of facq=180.373700+-0.000050 Hz. There are almost no missing data in the TOIs, except for few hundred samples of 545 and 857GHz TOIs which are lost in the on-board compression due to saturation on the Galactic Center crossings. <br />
<br />
== General Pipeline Structure ==<br />
<br />
The figure on the right shows how the initial ccTOI is transformed and how flags are produced:<br />
<br />
[[File:HFI_4_4_2_ToiProc_ExplS.png|thumb|600px|A schematic of the TOI processing pipeline]]<br />
<br />
<br />
== Output TOIs and products ==<br />
<br />
A TOI of clean calibrated samples (ccTOI) and a combined flag TOI (fTOI) are the outputs of the processing. The ccTOI is calibrated so as to represent the instantaneous power absorbed by the detector up to a constant (which will be determined by the map-making destriper). It is worth mentioning how the ccTOI is changed with respect to the input TOI, beyond the harmless constant conversion factor from voltage to absorbed power. The demodulation stage allows to get the demodulated bolometer voltage. The non-linearity correction is a second-order polynomial correction based on the physical but static bolometer model. In order to avoid too much masking after glitches, a glitch tail is subtracted after an occurrence of a glitch in the TOI. The 4K cooler lines noise is substituted at a series of 9 single temporal frequencies. Finally, the temporal response of the bolometer is deconvolved. This affects mostly the high-temporal frequency part of the TOI, although a small but significant low frequency (the long time response) tail is corrected too. Although flagged samples are not projected, their value influences the valid samples somehow. Hence interpolation procedures introduce some indirect modifications of the TOI.<br />
The flag TOI is a combination a dozen flags with an OR logic. Only unflagged data are projected. The exhaustive list of flags is given here: CompressionError, NoData, SSO, UnstablePointing, Glitch, BoloPlateFluctuation, RTS, Jump, PSBab. A complete qualification of the data is obtained at the ring level. If the TOI shows an anomalous behaviour on a given ring, this ring is discarded from projection. A special production of TOIs is also made as an input to the beam analysis with Mars, Jupiter and Saturn.<br />
<br />
<br />
== Examples of clean TOIs ==<br />
<gallery widths=300px perrow=3><br />
File:10_143_5_PBR_10800RING_LFER4_JC.jpg|Phase-binned signal (PBR) for 6 consecutive rings from 4000 to 4005<br />
File:10_143_5_rcircles_LFER4_JC.jpg| Valid signal (up) and noise (bottom) for 2 consecutive circles of the ring 4000<br />
File:10_143_5_fft_LFER4_JC.jpg| Fourier power spectral density of the first 30 minutes of the ring 4005 with signal before (pink) and after (black) deconvolution (top) and noise (bottom)<br />
File:14_545_1_PBR_10800RING_LFER4_JC.jpg|Phase-binned signal (PBR) for 6 consecutive rings from 4000 to 4005<br />
File:14_545_1_rcircles_LFER4_JC.jpg| Valid signal (up) and noise (bottom) for 2 consecutive circles of the ring 4000<br />
File:14_545_1_fft_LFER4_JC.jpg| Fourier power spectral density of the first 30 minutes of the ring 4005 with signal before (pink) and after (black) deconvolution (top) and noise (bottom)<br />
</gallery><br />
<br />
[[Media:check_ring.pdf|Samples of PBR, TOIs, and PSDs of all detectors are shown in this file]]<br />
<br />
== Trends in the output processing variables ==<br />
<br />
Here we intend to show the trend of the systematic effects that are dealt with in the TOI processing. The full impact of each of them is analyzed in [[HFI-Validation]].<br />
<br />
=== ADC baseline ===<br />
The following figure shows the ADC baseline which is used prior to demodulation (a constant offset is removed for clarity). This baseline is obtained by smoothing on an hour block average the undemodulated TOI on unflagged samples.<br />
[[File:baseline.jpg|400px|ADC baseline for all bolometers]]<br />
<br />
=== Glitch statistics ===<br />
The glitch rate per channel is shown in this figure. For details, see copap. <br />
[[File:figIntermPaperGR.jpg|400px|Glitch rate evolution]]<br />
<br />
The percentage of flagged data (mostly due to Cosmic Rays) at the ring level is shown in these examples. No smoothing was applied. Only valid rings are shown.<br />
<gallery widths=450px perrow=2 caption="Percentage of flagged data"><br />
File:group10_143_5.jpg | 143 GHz bolometers<br />
File:group14_545_1.jpg | 545 GHz bolometers<br />
</gallery><br />
[[Media:PercentUnvalid2.pdf|The complete set of plots is here]]<br />
<br />
=== Thermal template for decorrelation ===<br />
[[File:T90.jpg|400px|Thermal template used for the decorrelation from bolometer plate temperature fluctuations]]<br />
<br />
A simple linear decorrelation is performed using the 2 dark bolometers as a proxy of the bolometer plate temperature. Coupling coefficients were measured during the CPV phase.<br />
<br />
=== 4K cooler lines variability ===<br />
The amplitude of the nine 4K cooler lines in aW at 10, 20, 30, 40, 50, 60, 70, 80 and 17 Hz is shown for 2 bolometers in the following figures. The trend is smoothed over 31 ring values after having discarded measurements done at a ring which is discarded for all bolometers.<br />
<gallery widths=450px perrow=2 caption="Amplitude of the nine 4K cooler lines"><br />
File:10_143_5_4Klines.jpg | 143_5 bolometer<br />
File:14_545_1_4Klines.jpg | 545_1 bolometer<br />
</gallery><br />
[[Media:Lines4K.pdf|The 4K cooler line coefficients of all bolometers are shown in this file]]<br />
<br />
=== jump correction ===<br />
A piecewise constant value is removed to the TOI if a jump is detected. See a jump example in this figure:<br />
[[File:jump_exe.png|200px|An example of a jump seen on the rmsigTOI]]<br />
<br />
The number of jumps per day (all bolometers included) is shown in this figure:<br />
[[File:jumps_per_day.png|200px|Evolution of jump number during the mission]]<br />
<br />
The jumps are uncorrelated from bolometer to bolometer. The total number of jumps detected in the nominal and full mission is shown here:<br />
[[File:jumps_per_bolometer.png|200px|Number of jumps per bolometer]]<br />
<br />
<br />
=== Trends in noise and signal ===<br />
<gallery widths=450px perrow=2 caption="Signal (top) and noise (bottom) smoothed at 1 minute. All values falling in a discarded ring are not plotted."><br />
File:10_143_5_smooth_Watt.jpg | 143_5 bolometer<br />
File:14_545_1_smooth_Watt.jpg | 545_1 bolometer<br />
</gallery><br />
<br />
[[Media:check_smooth.pdf|The smooth TOIs of all detectors are shown in this file]]<br />
<br />
=== Noise stationarity ===<br />
This is not the final version but gives a good idea of power spectra at the detector level of rmsigTOIs.<br />
[[Media:v53_meanSpectra_bySurvey.pdf|All PSDs can be seen in this file]]<br />
<br />
The standard deviation per ring corrected by ring duration bias is given here, one per bolometer using only valid rings. No smoothing is applied (except a 31-point smoothing for the 545<br />
and 857 GHz channels) but values for rings discarded for all bolometers are not used. The standard deviation is computed on samples valid for map-making which are also not affected by the Galaxy or the point-sources using the usual flags. Two examples are given here.<br />
<gallery widths=450px perrow=2 caption="Standard deviation per ring"><br />
File:stddev_group10_143_5.jpg | 143_5 bolometer<br />
File:stddev_group_14_545_1.jpg | 545_1 bolometer<br />
</gallery><br />
<br />
<br />
The full series of plots is here:<br />
[[Media:StationarityInROIoutput2.pdf|Standard deviation of rmsig TOIs at the ring level]]<br />
<br />
Note the presence for 3 bolometers of a two-level noise system. No correction can be done for that effect. See one example here:<br />
[[File:23_353_TwoLevel.jpg|300px|An example of two-level noise system is seen in bolometer 23_353_3a]]<br />
<br />
<br />
An example of the higher order statistics which are used to unveil rings affected by RTS problems.<br />
[[File:HFI_4_4_2_RTSexample13.jpg|400px|Example of RTS detection]]<br />
<br />
== Anomalous rings ==<br />
<br />
Some rings are thus discarded (flagged) from further use (beam making, map making) by using these ring statistics (see HFI data processing paper REF).<br />
For each statistics, we compare each ring value to the ring values averaged (RVA) over all rings. We also define the modified standard deviation (MSD) of a ring quantity as the standard deviation of that quantity over the rings that deviate by less than five nominal standard deviations. This truncation is necessary to be robust against extreme deviant rings.<br />
<br />
More specifically, these are the full criteria (with the OR logic) to decide whether a ring is discarded:<br />
<br />
- if |mean-median| deviates from the RVA by more than fifteen times the MSD.<br />
<br />
- if the standard deviation deviates from the RVA by more than -5 times the MSD (all cases corresponding to almost empty rings) and +15 times the MSD.<br />
<br />
- if the Kolmogorov-Smirnov test deviates from the RVA by more than 15 times the MSD.<br />
<br />
- if the ring duration is outside some definite bounds. Bounds are 25 min and 90 min.<br />
<br />
<br />
Once the list of discarded rings is done per bolometer, a common list of discarded rings can be produced for all bolometers (by using discarded rings for at least half the bolometers). See the following table.<br />
<br />
Furthermore, an isolated valid ring stuck between two discarded rings becomes discarded as well.<br />
<br />
Table of common discarded rings (v47, 240-21890) to be updated to v53, nominal mission.<br />
<br />
{|table border=1 align=center cellpadding="3" cellspacing="0" style=text-align:center<br />
! '''Cause''' !! '''ring number'''<br />
|-<br />
!align="left"| manoeuvre<br />
|align="center"|304 1312 3590 3611 3642 3922 4949 6379 8456 11328<br />
|-<br />
! align="left"| Sorption Cooler Switchover <br />
|align="center"| 11149 11150 11151 11152 <br />
|-<br />
! align="left"| Rings too long<br />
| align="center"| 440 474 509 544 897 898 3589 13333 14627 14653 <br />
|-<br />
! align="left"| fully flagged as unstable <br />
| align="center"| 14628 14654 14842 16407 18210 <br />
|-<br />
! align="left"| common jump <br />
| align="center"| 7665 <br />
|-<br />
! align="left"| solar flare <br />
| align="center"| 20451 20452 20453 20454 20455 20456 <br />
|}<br />
<br />
<br />
<br />
<br />
{|table border=1 align=center cellpadding="3" cellspacing="0" style=text-align:center<br />
!definition!!Operational day #!!Date!!HFI ring #!!pointing-IDs<br />
|-<br />
!align="left"| Launch / first HFI data<br />
|align="center"|-||align="center"|14/05/2009 13:12:00 / 14/05/2009 14:20:18||align="center"|-||align="center"|-<br />
|-<br />
!align="left"| CPV phase<br />
|align="center"|1 - 90||align="center"|14/05/2009 14:20:18 - 12/08/2009 14:13:44||align="center"|0 - 239||align="center"|not significant<br />
|}<br />
<br />
<br />
== Flag description ==<br />
<br />
==== Input flags ====<br />
<br />
These flags are used as inputs to the TOI processing<br />
<br />
- The point-source flag (PSflag):<br />
<br />
An earlier version of HFI point-source catalog is read back into a flag TOIs, at a given frequency. In practice, 5 sigma sources are masked within a radius of 1.3 FWHM (9, 7, 5, 5, 5, 5 arcmin at 100,143,217,353,545,857 GHz) TBC.<br />
<br />
- the galactic flag (Galflag):<br />
<br />
An earlier version of HFI maps is thresholded and apodized. The produced masks are read into flag TOIs. The retained threshold corresponds to a sky coverage of respectively 70, 70, 80, 90, 90, 90% at 100,143,217,353,545,857 GHz.<br />
<br />
- Solar System Object flag <br />
<br />
For the TOI flag, Mars, Jupiter, Saturn are flagged up to a radius of NbBeam= 2,3,3,4,4,4 times the fiducial SSO_FWHM with SSO_FWHM= 9, 7, 5, 5, 5, 5 arcmin at 100,143,217,353,545,857 GHz.<br />
<br />
As an input to planet mask for maps, Mars, Jupiter, Saturn are flagged with a radius computed as a coefficient depending on the planet (Factor_per_source) times NbBeam times SSO_FWHM, with Factor_per_source = 1.1, 2.25, 1.25 for Mars, Jupiter, Saturn respectively and NbBeam = 2.25, 4.25, 4.0, 5.0,.6.0, 8.0 at 100, 143, 217, 353, 545, 857 GHz. This flag is called SSOflag4map.<br />
<br />
A small trailing tail is added to the mask to take into account the non-deconvolution of the planet signal which has been replaced by background values. The width of that tail is 10 % of the main flag diameter. The number of samples which are additionnaly flagged are the Factor_per_Source times AddSNafter with 10, 30, 20, 20, 30, 40 samples at 100, 143, 217, 353, 545, 857 GHz.<br />
<br />
Uranus and Neptune together with detected asteroids are masked by HFI. They are masked at the TOI level using an exclusion radius of 1.5 SSO_FWHM. At 857 GHz, 24 asteroids have been detected with HFI : 1Ceres, 2Pallas, 3Juno, 4Vesta, 7Iris, 8Flora, 9Metis, 10Hygiea, 11Parthenope, 12Victoria, 13Egeria, 14Irene, 15Eunomia, 16Psyche, 18Melpomene, 19Fortuna, 20Massalia, 29Amphitrite, 41Daphne, 45Eugenia, 52Europa, 88Thisbe, 704Interamnia, 324Bamberga.<br />
<br />
<br />
=== Output flags ===<br />
<br />
A FlagTOIproc is produced by the TOIprocessing. It marks measurements which are not reliable for ny of the following reasons :<br />
* gap (no valid input data), enlarged by one sample on each side. It flags less than 0.00044% (resp. 0.00062%) of the nominal (resp. complete) mission. It is equivallen to less thena 3 (resp. 8) minutes of data.<br />
* glitch : samples where the signal from a cosmic ray hit dominates the sky signal at more than 3 $\sigma$ are discarded.<br />
* jump as 100 samples are flagged around the computed position of the jump to take into account the error on this position.<br />
<br />
With and without point-sources, identical to a and b PSBs.<br />
<br />
<br />
Flags produced for the map making :<br />
* UnstablePointing Flag OR FlagTOIproc OR SSOflag4map OR SSOflag seen<br />
* FlagTOIproc = glitch OR jump OR flag thermal template <br />
* PSB flag glitch = flag glitch A OR flag glitch B<br />
<br />
Flag used in the destriping: same + point-source flag<br />
<br />
<br />
<br />
<br />
[[Category:Data processing]]</div>
Fdesert
https://wiki.cosmos.esa.int/planckpla2015/index.php?title=TOI_processing&diff=3260
TOI processing
2013-01-10T13:49:29Z
<p>Fdesert: </p>
<hr />
<div>This Section is kept short as this Planck release does not contain TOIs. The main information in the HFI processing paper is not duplicated here.<br />
<br />
==Overview==<br />
<br />
We describe here the how the TOIs are processed in order to be used for map production. We do not repeat the general features of the pipeline which are given in the HFI Data Processing article (REF). Here we give complementary explanations on some details. The TOI of each bolometer is processed independently of the other bolometers, so as to keep the noise properties as uncorrelated as possible. The processing involves modifying the TOI itself for what concerns the conversion to absorbed power and the correction of glitch tails. It also adds a flag TOI that masks the TOI samples that are not to be projected on maps for various reasons.<br />
<br />
<br />
== Input TOI ==<br />
<br />
The input TOI consists in the AC modulated voltage output of the readout of each bolometer. The input has previously been decompressed, and converted from internal digital units to voltage via a constant factor. The TOI has a regular sampling at the acquisition frequency of facq=180.373700+-0.000050 Hz. There are almost no missing data in the TOIs, except for few hundred samples of 545 and 857GHz TOIs which are lost in the on-board compression due to saturation on the Galactic Center crossings. <br />
<br />
== General Pipeline Structure ==<br />
<br />
The figure on the right shows how the initial ccTOI is transformed and how flags are produced:<br />
<br />
[[File:HFI_4_4_2_ToiProc_ExplS.png|thumb|600px|A schematic of the TOI processing pipeline]]<br />
<br />
<br />
== Output TOIs and products ==<br />
<br />
A TOI of clean calibrated samples (ccTOI) and a combined flag TOI (fTOI) are the outputs of the processing. The ccTOI is calibrated so as to represent the instantaneous power absorbed by the detector up to a constant (which will be determined by the map-making destriper). It is worth mentioning how the ccTOI is changed with respect to the input TOI, beyond the harmless constant conversion factor from voltage to absorbed power. The demodulation stage allows to get the demodulated bolometer voltage. The non-linearity correction is a second-order polynomial correction based on the physical but static bolometer model. In order to avoid too much masking after glitches, a glitch tail is subtracted after an occurrence of a glitch in the TOI. The 4K cooler lines noise is substituted at a series of 9 single temporal frequencies. Finally, the temporal response of the bolometer is deconvolved. This affects mostly the high-temporal frequency part of the TOI, although a small but significant low frequency (the long time response) tail is corrected too. Although flagged samples are not projected, their value influences the valid samples somehow. Hence interpolation procedures introduce some indirect modifications of the TOI.<br />
The flag TOI is a combination a dozen flags with an OR logic. Only unflagged data are projected. The exhaustive list of flags is given here: CompressionError, NoData, SSO, UnstablePointing, Glitch, BoloPlateFluctuation, RTS, Jump, PSBab. A complete qualification of the data is obtained at the ring level. If the TOI shows an anomalous behaviour on a given ring, this ring is discarded from projection. A special production of TOIs is also made as an input to the beam analysis with Mars, Jupiter and Saturn.<br />
<br />
<br />
== Examples of clean TOIs ==<br />
<gallery widths=300px perrow=3><br />
File:10_143_5_PBR_10800RING_LFER4_JC.jpg|Phase-binned signal (PBR) for 6 consecutive rings from 4000 to 4005<br />
File:10_143_5_rcircles_LFER4_JC.jpg| Valid signal (up) and noise (bottom) for 2 consecutive circles of the ring 4000<br />
File:10_143_5_fft_LFER4_JC.jpg| Fourier power spectral density of the first 30 minutes of the ring 4005 with signal before (pink) and after (black) deconvolution (top) and noise (bottom)<br />
File:14_545_1_PBR_10800RING_LFER4_JC.jpg|Phase-binned signal (PBR) for 6 consecutive rings from 4000 to 4005<br />
File:14_545_1_rcircles_LFER4_JC.jpg| Valid signal (up) and noise (bottom) for 2 consecutive circles of the ring 4000<br />
File:14_545_1_fft_LFER4_JC.jpg| Fourier power spectral density of the first 30 minutes of the ring 4005 with signal before (pink) and after (black) deconvolution (top) and noise (bottom)<br />
</gallery><br />
<br />
[[Media:check_ring.pdf|Samples of PBR, TOIs, and PSDs of all detectors are shown in this file]]<br />
<br />
== Trends in the output processing variables ==<br />
<br />
Here we intend to show the trend of the systematic effects that are dealt with in the TOI processing. The full impact of each of them is analyzed in [[HFI-Validation]].<br />
<br />
=== ADC baseline ===<br />
The following figure shows the ADC baseline which is used prior to demodulation (a constant offset is removed for clarity). This baseline is obtained by smoothing on an hour block average the undemodulated TOI on unflagged samples.<br />
[[File:baseline.jpg|400px|ADC baseline for all bolometers]]<br />
<br />
=== Glitch statistics ===<br />
The glitch rate per channel is shown in this figure. For details, see copap. <br />
[[File:figIntermPaperGR.jpg|400px|Glitch rate evolution]]<br />
<br />
The percentage of flagged data (mostly due to Cosmic Rays) at the ring level is shown in these examples. No smoothing was applied. Only valid rings are shown.<br />
<gallery widths=450px perrow=2 caption="Percentage of flagged data"><br />
File:group10_143_5.jpg | 143 GHz bolometers<br />
File:group14_545_1.jpg | 545 GHz bolometers<br />
</gallery><br />
[[Media:PercentUnvalid2.pdf|The complete set of plots is here]]<br />
<br />
=== Thermal template for decorrelation ===<br />
[[File:T90.jpg|400px|Thermal template used for the decorrelation from bolometer plate temperature fluctuations]]<br />
<br />
A simple linear decorrelation is performed using the 2 dark bolometers as a proxy of the bolometer plate temperature. Coupling coefficients were measured during the CPV phase.<br />
<br />
=== 4K cooler lines variability ===<br />
The amplitude of the nine 4K cooler lines in aW at 10, 20, 30, 40, 50, 60, 70, 80 and 17 Hz is shown for 2 bolometers in the following figures. The trend is smoothed over 31 ring values after having discarded measurements done at a ring which is discarded for all bolometers.<br />
<gallery widths=450px perrow=2 caption="Amplitude of the nine 4K cooler lines"><br />
File:10_143_5_4Klines.jpg | 143_5 bolometer<br />
File:14_545_1_4Klines.jpg | 545_1 bolometer<br />
</gallery><br />
[[Media:Lines4K.pdf|The 4K cooler line coefficients of all bolometers are shown in this file]]<br />
<br />
=== jump correction ===<br />
A piecewise constant value is removed to the TOI if a jump is detected. See a jump example in this figure:<br />
[[File:jump_exe.png|200px|An example of a jump seen on the rmsigTOI]]<br />
<br />
The number of jumps per day (all bolometers included) is shown in this figure:<br />
[[File:jumps_per_day.png|200px|Evolution of jump number during the mission]]<br />
<br />
The jumps are uncorrelated from bolometer to bolometer. The total number of jumps detected in the nominal and full mission is shown here:<br />
[[File:jumps_per_bolometer.png|200px|Number of jumps per bolometer]]<br />
<br />
<br />
=== Trends in noise and signal ===<br />
<gallery widths=450px perrow=2 caption="Signal (top) and noise (bottom) smoothed at 1 minute. All values falling in a discarded ring are not plotted."><br />
File:10_143_5_smooth_Watt.jpg | 143_5 bolometer<br />
File:14_545_1_smooth_Watt.jpg | 545_1 bolometer<br />
</gallery><br />
<br />
[[Media:check_smooth.pdf|The smooth TOIs of all detectors are shown in this file]]<br />
<br />
=== Noise stationarity ===<br />
This is not the final version but gives a good idea of power spectra at the detector level of rmsigTOIs.<br />
[[Media:v53_meanSpectra_bySurvey.pdf|All PSDs can be seen in this file]]<br />
<br />
The standard deviation per ring corrected by ring duration bias is given here, one per bolometer using only valid rings. No smoothing is applied (except a 31-point smoothing for the 545<br />
and 857 GHz channels) but values for rings discarded for all bolometers are not used. The standard deviation is computed on samples valid for map-making which are also not affected by the Galaxy or the point-sources using the usual flags. Two examples are given here.<br />
<gallery widths=450px perrow=2 caption="Standard deviation per ring"><br />
File:stddev_group10_143_5.jpg | 143_5 bolometer<br />
File:stddev_group_14_545_1.jpg | 545_1 bolometer<br />
</gallery><br />
<br />
<br />
The full series of plots is here:<br />
[[Media:StationarityInROIoutput2.pdf|Standard deviation of rmsig TOIs at the ring level]]<br />
<br />
Note the presence for 3 bolometers of a two-level noise system. No correction can be done for that effect. See one example here:<br />
[[File:23_353_TwoLevel.jpg|300px|An example of two-level noise system is seen in bolometer 23_353_3a]]<br />
<br />
<br />
An example of the higher order statistics which are used to unveil rings affected by RTS problems.<br />
[[File:HFI_4_4_2_RTSexample13.jpg|400px|Example of RTS detection]]<br />
<br />
== Anomalous rings ==<br />
<br />
Some rings are thus discarded (flagged) from further use (beam making, map making) by using these ring statistics (see HFI data processing paper REF).<br />
For each statistics, we compare each ring value to the ring values averaged (RVA) over all rings. We also define the modified standard deviation (MSD) of a ring quantity as the standard deviation of that quantity over the rings that deviate by less than five nominal standard deviations. This truncation is necessary to be robust against extreme deviant rings.<br />
<br />
More specifically, these are the full criteria (with the OR logic) to decide whether a ring is discarded:<br />
<br />
- if |mean-median| deviates from the RVA by more than fifteen times the MSD.<br />
<br />
- if the standard deviation deviates from the RVA by more than -5 times the MSD (all cases corresponding to almost empty rings) and +15 times the MSD.<br />
<br />
- if the Kolmogorov-Smirnov test deviates from the RVA by more than 15 times the MSD.<br />
<br />
- if the ring duration is outside some definite bounds. Bounds are 25 min and 90 min.<br />
<br />
<br />
Once the list of discarded rings is done per bolometer, a common list of discarded rings can be produced for all bolometers (by using discarded rings for at least half the bolometers). See the following table.<br />
<br />
Furthermore, an isolated valid ring stuck between two discarded rings becomes discarded as well.<br />
<br />
Table of common discarded rings (v47, 240-21890) to be updated to v53, nominal mission.<br />
<br />
{|table border=1 align=center cellpadding="3" cellspacing="0" style=text-align:center<br />
! '''Cause''' !! '''ring number'''<br />
|-<br />
!align="left"| manoeuvre<br />
|align="center"|304 1312 3590 3611 3642 3922 4949 6379 8456 11328<br />
|-<br />
! align="left"| Sorption Cooler Switchover <br />
|align="center"| 11149 11150 11151 11152 <br />
|-<br />
! Rings too long| 440 474 509 544 897 898 3589 13333 14627 14653 <br />
|-<br />
! fully flagged as unstable | 14628 14654 14842 16407 18210 <br />
|-<br />
! common jump | 7665 <br />
|-<br />
! solar flare | 20451 20452 20453 20454 20455 20456 <br />
|}<br />
<br />
<br />
<br />
<br />
{|table border=1 align=center cellpadding="3" cellspacing="0" style=text-align:center<br />
!definition!!Operational day #!!Date!!HFI ring #!!pointing-IDs<br />
|-<br />
!align="left"| Launch / first HFI data<br />
|align="center"|-||align="center"|14/05/2009 13:12:00 / 14/05/2009 14:20:18||align="center"|-||align="center"|-<br />
|-<br />
!align="left"| CPV phase<br />
|align="center"|1 - 90||align="center"|14/05/2009 14:20:18 - 12/08/2009 14:13:44||align="center"|0 - 239||align="center"|not significant<br />
|}<br />
<br />
<br />
== Flag description ==<br />
<br />
==== Input flags ====<br />
<br />
These flags are used as inputs to the TOI processing<br />
<br />
- The point-source flag (PSflag):<br />
<br />
An earlier version of HFI point-source catalog is read back into a flag TOIs, at a given frequency. In practice, 5 sigma sources are masked within a radius of 1.3 FWHM (9, 7, 5, 5, 5, 5 arcmin at 100,143,217,353,545,857 GHz) TBC.<br />
<br />
- the galactic flag (Galflag):<br />
<br />
An earlier version of HFI maps is thresholded and apodized. The produced masks are read into flag TOIs. The retained threshold corresponds to a sky coverage of respectively 70, 70, 80, 90, 90, 90% at 100,143,217,353,545,857 GHz.<br />
<br />
- Solar System Object flag <br />
<br />
For the TOI flag, Mars, Jupiter, Saturn are flagged up to a radius of NbBeam= 2,3,3,4,4,4 times the fiducial SSO_FWHM with SSO_FWHM= 9, 7, 5, 5, 5, 5 arcmin at 100,143,217,353,545,857 GHz.<br />
<br />
As an input to planet mask for maps, Mars, Jupiter, Saturn are flagged with a radius computed as a coefficient depending on the planet (Factor_per_source) times NbBeam times SSO_FWHM, with Factor_per_source = 1.1, 2.25, 1.25 for Mars, Jupiter, Saturn respectively and NbBeam = 2.25, 4.25, 4.0, 5.0,.6.0, 8.0 at 100, 143, 217, 353, 545, 857 GHz. This flag is called SSOflag4map.<br />
<br />
A small trailing tail is added to the mask to take into account the non-deconvolution of the planet signal which has been replaced by background values. The width of that tail is 10 % of the main flag diameter. The number of samples which are additionnaly flagged are the Factor_per_Source times AddSNafter with 10, 30, 20, 20, 30, 40 samples at 100, 143, 217, 353, 545, 857 GHz.<br />
<br />
Uranus and Neptune together with detected asteroids are masked by HFI. They are masked at the TOI level using an exclusion radius of 1.5 SSO_FWHM. At 857 GHz, 24 asteroids have been detected with HFI : 1Ceres, 2Pallas, 3Juno, 4Vesta, 7Iris, 8Flora, 9Metis, 10Hygiea, 11Parthenope, 12Victoria, 13Egeria, 14Irene, 15Eunomia, 16Psyche, 18Melpomene, 19Fortuna, 20Massalia, 29Amphitrite, 41Daphne, 45Eugenia, 52Europa, 88Thisbe, 704Interamnia, 324Bamberga.<br />
<br />
<br />
=== Output flags ===<br />
<br />
A FlagTOIproc is produced by the TOIprocessing. It marks measurements which are not reliable for ny of the following reasons :<br />
* gap (no valid input data), enlarged by one sample on each side. It flags less than 0.00044% (resp. 0.00062%) of the nominal (resp. complete) mission. It is equivallen to less thena 3 (resp. 8) minutes of data.<br />
* glitch : samples where the signal from a cosmic ray hit dominates the sky signal at more than 3 $\sigma$ are discarded.<br />
* jump as 100 samples are flagged around the computed position of the jump to take into account the error on this position.<br />
<br />
With and without point-sources, identical to a and b PSBs.<br />
<br />
<br />
Flags produced for the map making :<br />
* UnstablePointing Flag OR FlagTOIproc OR SSOflag4map OR SSOflag seen<br />
* FlagTOIproc = glitch OR jump OR flag thermal template <br />
* PSB flag glitch = flag glitch A OR flag glitch B<br />
<br />
Flag used in the destriping: same + point-source flag<br />
<br />
<br />
<br />
<br />
[[Category:Data processing]]</div>
Fdesert
https://wiki.cosmos.esa.int/planckpla2015/index.php?title=TOI_processing&diff=3259
TOI processing
2013-01-10T13:44:41Z
<p>Fdesert: </p>
<hr />
<div>This Section is kept short as this Planck release does not contain TOIs. The main information in the HFI processing paper is not duplicated here.<br />
<br />
==Overview==<br />
<br />
We describe here the how the TOIs are processed in order to be used for map production. We do not repeat the general features of the pipeline which are given in the HFI Data Processing article (REF). Here we give complementary explanations on some details. The TOI of each bolometer is processed independently of the other bolometers, so as to keep the noise properties as uncorrelated as possible. The processing involves modifying the TOI itself for what concerns the conversion to absorbed power and the correction of glitch tails. It also adds a flag TOI that masks the TOI samples that are not to be projected on maps for various reasons.<br />
<br />
<br />
== Input TOI ==<br />
<br />
The input TOI consists in the AC modulated voltage output of the readout of each bolometer. The input has previously been decompressed, and converted from internal digital units to voltage via a constant factor. The TOI has a regular sampling at the acquisition frequency of facq=180.373700+-0.000050 Hz. There are almost no missing data in the TOIs, except for few hundred samples of 545 and 857GHz TOIs which are lost in the on-board compression due to saturation on the Galactic Center crossings. <br />
<br />
== General Pipeline Structure ==<br />
<br />
The figure on the right shows how the initial ccTOI is transformed and how flags are produced:<br />
<br />
[[File:HFI_4_4_2_ToiProc_ExplS.png|thumb|600px|A schematic of the TOI processing pipeline]]<br />
<br />
<br />
== Output TOIs and products ==<br />
<br />
A TOI of clean calibrated samples (ccTOI) and a combined flag TOI (fTOI) are the outputs of the processing. The ccTOI is calibrated so as to represent the instantaneous power absorbed by the detector up to a constant (which will be determined by the map-making destriper). It is worth mentioning how the ccTOI is changed with respect to the input TOI, beyond the harmless constant conversion factor from voltage to absorbed power. The demodulation stage allows to get the demodulated bolometer voltage. The non-linearity correction is a second-order polynomial correction based on the physical but static bolometer model. In order to avoid too much masking after glitches, a glitch tail is subtracted after an occurrence of a glitch in the TOI. The 4K cooler lines noise is substituted at a series of 9 single temporal frequencies. Finally, the temporal response of the bolometer is deconvolved. This affects mostly the high-temporal frequency part of the TOI, although a small but significant low frequency (the long time response) tail is corrected too. Although flagged samples are not projected, their value influences the valid samples somehow. Hence interpolation procedures introduce some indirect modifications of the TOI.<br />
The flag TOI is a combination a dozen flags with an OR logic. Only unflagged data are projected. The exhaustive list of flags is given here: CompressionError, NoData, SSO, UnstablePointing, Glitch, BoloPlateFluctuation, RTS, Jump, PSBab. A complete qualification of the data is obtained at the ring level. If the TOI shows an anomalous behaviour on a given ring, this ring is discarded from projection. A special production of TOIs is also made as an input to the beam analysis with Mars, Jupiter and Saturn.<br />
<br />
<br />
== Examples of clean TOIs ==<br />
<gallery widths=300px perrow=3><br />
File:10_143_5_PBR_10800RING_LFER4_JC.jpg|Phase-binned signal (PBR) for 6 consecutive rings from 4000 to 4005<br />
File:10_143_5_rcircles_LFER4_JC.jpg| Valid signal (up) and noise (bottom) for 2 consecutive circles of the ring 4000<br />
File:10_143_5_fft_LFER4_JC.jpg| Fourier power spectral density of the first 30 minutes of the ring 4005 with signal before (pink) and after (black) deconvolution (top) and noise (bottom)<br />
File:14_545_1_PBR_10800RING_LFER4_JC.jpg|Phase-binned signal (PBR) for 6 consecutive rings from 4000 to 4005<br />
File:14_545_1_rcircles_LFER4_JC.jpg| Valid signal (up) and noise (bottom) for 2 consecutive circles of the ring 4000<br />
File:14_545_1_fft_LFER4_JC.jpg| Fourier power spectral density of the first 30 minutes of the ring 4005 with signal before (pink) and after (black) deconvolution (top) and noise (bottom)<br />
</gallery><br />
<br />
[[Media:check_ring.pdf|Samples of PBR, TOIs, and PSDs of all detectors are shown in this file]]<br />
<br />
== Trends in the output processing variables ==<br />
<br />
Here we intend to show the trend of the systematic effects that are dealt with in the TOI processing. The full impact of each of them is analyzed in [[HFI-Validation]].<br />
<br />
=== ADC baseline ===<br />
The following figure shows the ADC baseline which is used prior to demodulation (a constant offset is removed for clarity). This baseline is obtained by smoothing on an hour block average the undemodulated TOI on unflagged samples.<br />
[[File:baseline.jpg|400px|ADC baseline for all bolometers]]<br />
<br />
=== Glitch statistics ===<br />
The glitch rate per channel is shown in this figure. For details, see copap. <br />
[[File:figIntermPaperGR.jpg|400px|Glitch rate evolution]]<br />
<br />
The percentage of flagged data (mostly due to Cosmic Rays) at the ring level is shown in these examples. No smoothing was applied. Only valid rings are shown.<br />
<gallery widths=450px perrow=2 caption="Percentage of flagged data"><br />
File:group10_143_5.jpg | 143 GHz bolometers<br />
File:group14_545_1.jpg | 545 GHz bolometers<br />
</gallery><br />
[[Media:PercentUnvalid2.pdf|The complete set of plots is here]]<br />
<br />
=== Thermal template for decorrelation ===<br />
[[File:T90.jpg|400px|Thermal template used for the decorrelation from bolometer plate temperature fluctuations]]<br />
<br />
A simple linear decorrelation is performed using the 2 dark bolometers as a proxy of the bolometer plate temperature. Coupling coefficients were measured during the CPV phase.<br />
<br />
=== 4K cooler lines variability ===<br />
The amplitude of the nine 4K cooler lines in aW at 10, 20, 30, 40, 50, 60, 70, 80 and 17 Hz is shown for 2 bolometers in the following figures. The trend is smoothed over 31 ring values after having discarded measurements done at a ring which is discarded for all bolometers.<br />
<gallery widths=450px perrow=2 caption="Amplitude of the nine 4K cooler lines"><br />
File:10_143_5_4Klines.jpg | 143_5 bolometer<br />
File:14_545_1_4Klines.jpg | 545_1 bolometer<br />
</gallery><br />
[[Media:Lines4K.pdf|The 4K cooler line coefficients of all bolometers are shown in this file]]<br />
<br />
=== jump correction ===<br />
A piecewise constant value is removed to the TOI if a jump is detected. See a jump example in this figure:<br />
[[File:jump_exe.png|200px|An example of a jump seen on the rmsigTOI]]<br />
<br />
The number of jumps per day (all bolometers included) is shown in this figure:<br />
[[File:jumps_per_day.png|200px|Evolution of jump number during the mission]]<br />
<br />
The jumps are uncorrelated from bolometer to bolometer. The total number of jumps detected in the nominal and full mission is shown here:<br />
[[File:jumps_per_bolometer.png|200px|Number of jumps per bolometer]]<br />
<br />
<br />
=== Trends in noise and signal ===<br />
<gallery widths=450px perrow=2 caption="Signal (top) and noise (bottom) smoothed at 1 minute. All values falling in a discarded ring are not plotted."><br />
File:10_143_5_smooth_Watt.jpg | 143_5 bolometer<br />
File:14_545_1_smooth_Watt.jpg | 545_1 bolometer<br />
</gallery><br />
<br />
[[Media:check_smooth.pdf|The smooth TOIs of all detectors are shown in this file]]<br />
<br />
=== Noise stationarity ===<br />
This is not the final version but gives a good idea of power spectra at the detector level of rmsigTOIs.<br />
[[Media:v53_meanSpectra_bySurvey.pdf|All PSDs can be seen in this file]]<br />
<br />
The standard deviation per ring corrected by ring duration bias is given here, one per bolometer using only valid rings. No smoothing is applied (except a 31-point smoothing for the 545<br />
and 857 GHz channels) but values for rings discarded for all bolometers are not used. The standard deviation is computed on samples valid for map-making which are also not affected by the Galaxy or the point-sources using the usual flags. Two examples are given here.<br />
<gallery widths=450px perrow=2 caption="Standard deviation per ring"><br />
File:stddev_group10_143_5.jpg | 143_5 bolometer<br />
File:stddev_group_14_545_1.jpg | 545_1 bolometer<br />
</gallery><br />
<br />
<br />
The full series of plots is here:<br />
[[Media:StationarityInROIoutput2.pdf|Standard deviation of rmsig TOIs at the ring level]]<br />
<br />
Note the presence for 3 bolometers of a two-level noise system. No correction can be done for that effect. See one example here:<br />
[[File:23_353_TwoLevel.jpg|300px|An example of two-level noise system is seen in bolometer 23_353_3a]]<br />
<br />
<br />
An example of the higher order statistics which are used to unveil rings affected by RTS problems.<br />
[[File:HFI_4_4_2_RTSexample13.jpg|400px|Example of RTS detection]]<br />
<br />
== Anomalous rings ==<br />
<br />
Some rings are thus discarded (flagged) from further use (beam making, map making) by using these ring statistics (see HFI data processing paper REF).<br />
For each statistics, we compare each ring value to the ring values averaged (RVA) over all rings. We also define the modified standard deviation (MSD) of a ring quantity as the standard deviation of that quantity over the rings that deviate by less than five nominal standard deviations. This truncation is necessary to be robust against extreme deviant rings.<br />
<br />
More specifically, these are the full criteria (with the OR logic) to decide whether a ring is discarded:<br />
<br />
- if |mean-median| deviates from the RVA by more than fifteen times the MSD.<br />
<br />
- if the standard deviation deviates from the RVA by more than -5 times the MSD (all cases corresponding to almost empty rings) and +15 times the MSD.<br />
<br />
- if the Kolmogorov-Smirnov test deviates from the RVA by more than 15 times the MSD.<br />
<br />
- if the ring duration is outside some definite bounds. Bounds are 25 min and 90 min.<br />
<br />
<br />
Once the list of discarded rings is done per bolometer, a common list of discarded rings can be produced for all bolometers (by using discarded rings for at least half the bolometers). See the following table.<br />
<br />
Furthermore, an isolated valid ring stuck between two discarded rings becomes discarded as well.<br />
<br />
Table of common discarded rings (v47, 240-21890) to be updated to v53, nominal mission.<br />
<br />
{|table border=1 align=center cellpadding="3" cellspacing="0" style=text-align:center<br />
! '''Cause''' !! '''ring number'''<br />
|-<br />
!align="left"| manoeuvre<br />
|align="center"|304 1312 3590 3611 3642 3922 4949 6379 8456 11328<br />
|-<br />
| manoeuvre | 304 1312 3590 3611 3642 3922 4949 6379 8456 11328 |<br />
| Sorption Cooler Switchover | 11149 11150 11151 11152 |<br />
| Rings too long| 440 474 509 544 897 898 3589 13333 14627 14653 |<br />
| fully flagged as unstable | 14628 14654 14842 16407 18210 |<br />
| common jump | 7665 |<br />
| solar flare | 20451 20452 20453 20454 20455 20456 |<br />
|}<br />
<br />
<br />
{|table border=1 align=center cellpadding="3" cellspacing="0" style=text-align:center<br />
!definition!!Operational day #!!Date!!HFI ring #!!pointing-IDs<br />
|-<br />
!align="left"| Launch / first HFI data<br />
|align="center"|-||align="center"|14/05/2009 13:12:00 / 14/05/2009 14:20:18||align="center"|-||align="center"|-<br />
|-<br />
!align="left"| CPV phase<br />
|align="center"|1 - 90||align="center"|14/05/2009 14:20:18 - 12/08/2009 14:13:44||align="center"|0 - 239||align="center"|not significant<br />
|}<br />
<br />
<br />
== Flag description ==<br />
<br />
==== Input flags ====<br />
<br />
These flags are used as inputs to the TOI processing<br />
<br />
- The point-source flag (PSflag):<br />
<br />
An earlier version of HFI point-source catalog is read back into a flag TOIs, at a given frequency. In practice, 5 sigma sources are masked within a radius of 1.3 FWHM (9, 7, 5, 5, 5, 5 arcmin at 100,143,217,353,545,857 GHz) TBC.<br />
<br />
- the galactic flag (Galflag):<br />
<br />
An earlier version of HFI maps is thresholded and apodized. The produced masks are read into flag TOIs. The retained threshold corresponds to a sky coverage of respectively 70, 70, 80, 90, 90, 90% at 100,143,217,353,545,857 GHz.<br />
<br />
- Solar System Object flag <br />
<br />
For the TOI flag, Mars, Jupiter, Saturn are flagged up to a radius of NbBeam= 2,3,3,4,4,4 times the fiducial SSO_FWHM with SSO_FWHM= 9, 7, 5, 5, 5, 5 arcmin at 100,143,217,353,545,857 GHz.<br />
<br />
As an input to planet mask for maps, Mars, Jupiter, Saturn are flagged with a radius computed as a coefficient depending on the planet (Factor_per_source) times NbBeam times SSO_FWHM, with Factor_per_source = 1.1, 2.25, 1.25 for Mars, Jupiter, Saturn respectively and NbBeam = 2.25, 4.25, 4.0, 5.0,.6.0, 8.0 at 100, 143, 217, 353, 545, 857 GHz. This flag is called SSOflag4map.<br />
<br />
A small trailing tail is added to the mask to take into account the non-deconvolution of the planet signal which has been replaced by background values. The width of that tail is 10 % of the main flag diameter. The number of samples which are additionnaly flagged are the Factor_per_Source times AddSNafter with 10, 30, 20, 20, 30, 40 samples at 100, 143, 217, 353, 545, 857 GHz.<br />
<br />
Uranus and Neptune together with detected asteroids are masked by HFI. They are masked at the TOI level using an exclusion radius of 1.5 SSO_FWHM. At 857 GHz, 24 asteroids have been detected with HFI : 1Ceres, 2Pallas, 3Juno, 4Vesta, 7Iris, 8Flora, 9Metis, 10Hygiea, 11Parthenope, 12Victoria, 13Egeria, 14Irene, 15Eunomia, 16Psyche, 18Melpomene, 19Fortuna, 20Massalia, 29Amphitrite, 41Daphne, 45Eugenia, 52Europa, 88Thisbe, 704Interamnia, 324Bamberga.<br />
<br />
<br />
=== Output flags ===<br />
<br />
A FlagTOIproc is produced by the TOIprocessing. It marks measurements which are not reliable for ny of the following reasons :<br />
* gap (no valid input data), enlarged by one sample on each side. It flags less than 0.00044% (resp. 0.00062%) of the nominal (resp. complete) mission. It is equivallen to less thena 3 (resp. 8) minutes of data.<br />
* glitch : samples where the signal from a cosmic ray hit dominates the sky signal at more than 3 $\sigma$ are discarded.<br />
* jump as 100 samples are flagged around the computed position of the jump to take into account the error on this position.<br />
<br />
With and without point-sources, identical to a and b PSBs.<br />
<br />
<br />
Flags produced for the map making :<br />
* UnstablePointing Flag OR FlagTOIproc OR SSOflag4map OR SSOflag seen<br />
* FlagTOIproc = glitch OR jump OR flag thermal template <br />
* PSB flag glitch = flag glitch A OR flag glitch B<br />
<br />
Flag used in the destriping: same + point-source flag<br />
<br />
<br />
<br />
<br />
[[Category:Data processing]]</div>
Fdesert
https://wiki.cosmos.esa.int/planckpla2015/index.php?title=TOI_processing&diff=3258
TOI processing
2013-01-10T13:30:20Z
<p>Fdesert: </p>
<hr />
<div>This Section is kept short as this Planck release does not contain TOIs. The main information in the HFI processing paper is not duplicated here.<br />
<br />
==Overview==<br />
<br />
We describe here the how the TOIs are processed in order to be used for map production. We do not repeat the general features of the pipeline which are given in the HFI Data Processing article (REF). Here we give complementary explanations on some details. The TOI of each bolometer is processed independently of the other bolometers, so as to keep the noise properties as uncorrelated as possible. The processing involves modifying the TOI itself for what concerns the conversion to absorbed power and the correction of glitch tails. It also adds a flag TOI that masks the TOI samples that are not to be projected on maps for various reasons.<br />
<br />
<br />
== Input TOI ==<br />
<br />
The input TOI consists in the AC modulated voltage output of the readout of each bolometer. The input has previously been decompressed, and converted from internal digital units to voltage via a constant factor. The TOI has a regular sampling at the acquisition frequency of facq=180.373700+-0.000050 Hz. There are almost no missing data in the TOIs, except for few hundred samples of 545 and 857GHz TOIs which are lost in the on-board compression due to saturation on the Galactic Center crossings. <br />
<br />
== General Pipeline Structure ==<br />
<br />
The figure on the right shows how the initial ccTOI is transformed and how flags are produced:<br />
<br />
[[File:HFI_4_4_2_ToiProc_ExplS.png|thumb|600px|A schematic of the TOI processing pipeline]]<br />
<br />
<br />
== Output TOIs and products ==<br />
<br />
A TOI of clean calibrated samples (ccTOI) and a combined flag TOI (fTOI) are the outputs of the processing. The ccTOI is calibrated so as to represent the instantaneous power absorbed by the detector up to a constant (which will be determined by the map-making destriper). It is worth mentioning how the ccTOI is changed with respect to the input TOI, beyond the harmless constant conversion factor from voltage to absorbed power. The demodulation stage allows to get the demodulated bolometer voltage. The non-linearity correction is a second-order polynomial correction based on the physical but static bolometer model. In order to avoid too much masking after glitches, a glitch tail is subtracted after an occurrence of a glitch in the TOI. The 4K cooler lines noise is substituted at a series of 9 single temporal frequencies. Finally, the temporal response of the bolometer is deconvolved. This affects mostly the high-temporal frequency part of the TOI, although a small but significant low frequency (the long time response) tail is corrected too. Although flagged samples are not projected, their value influences the valid samples somehow. Hence interpolation procedures introduce some indirect modifications of the TOI.<br />
The flag TOI is a combination a dozen flags with an OR logic. Only unflagged data are projected. The exhaustive list of flags is given here: CompressionError, NoData, SSO, UnstablePointing, Glitch, BoloPlateFluctuation, RTS, Jump, PSBab. A complete qualification of the data is obtained at the ring level. If the TOI shows an anomalous behaviour on a given ring, this ring is discarded from projection. A special production of TOIs is also made as an input to the beam analysis with Mars, Jupiter and Saturn.<br />
<br />
<br />
== Examples of clean TOIs ==<br />
<gallery widths=300px perrow=3><br />
File:10_143_5_PBR_10800RING_LFER4_JC.jpg|Phase-binned signal (PBR) for 6 consecutive rings from 4000 to 4005<br />
File:10_143_5_rcircles_LFER4_JC.jpg| Valid signal (up) and noise (bottom) for 2 consecutive circles of the ring 4000<br />
File:10_143_5_fft_LFER4_JC.jpg| Fourier power spectral density of the first 30 minutes of the ring 4005 with signal before (pink) and after (black) deconvolution (top) and noise (bottom)<br />
File:14_545_1_PBR_10800RING_LFER4_JC.jpg|Phase-binned signal (PBR) for 6 consecutive rings from 4000 to 4005<br />
File:14_545_1_rcircles_LFER4_JC.jpg| Valid signal (up) and noise (bottom) for 2 consecutive circles of the ring 4000<br />
File:14_545_1_fft_LFER4_JC.jpg| Fourier power spectral density of the first 30 minutes of the ring 4005 with signal before (pink) and after (black) deconvolution (top) and noise (bottom)<br />
</gallery><br />
<br />
[[Media:check_ring.pdf|Samples of PBR, TOIs, and PSDs of all detectors are shown in this file]]<br />
<br />
== Trends in the output processing variables ==<br />
<br />
Here we intend to show the trend of the systematic effects that are dealt with in the TOI processing. The full impact of each of them is analyzed in [[HFI-Validation]].<br />
<br />
=== ADC baseline ===<br />
The following figure shows the ADC baseline which is used prior to demodulation (a constant offset is removed for clarity). This baseline is obtained by smoothing on an hour block average the undemodulated TOI on unflagged samples.<br />
[[File:baseline.jpg|400px|ADC baseline for all bolometers]]<br />
<br />
=== Glitch statistics ===<br />
The glitch rate per channel is shown in this figure. For details, see copap. <br />
[[File:figIntermPaperGR.jpg|400px|Glitch rate evolution]]<br />
<br />
The percentage of flagged data (mostly due to Cosmic Rays) at the ring level is shown in these examples. No smoothing was applied. Only valid rings are shown.<br />
<gallery widths=450px perrow=2 caption="Percentage of flagged data"><br />
File:group10_143_5.jpg | 143 GHz bolometers<br />
File:group14_545_1.jpg | 545 GHz bolometers<br />
</gallery><br />
[[Media:PercentUnvalid2.pdf|The complete set of plots is here]]<br />
<br />
=== Thermal template for decorrelation ===<br />
[[File:T90.jpg|400px|Thermal template used for the decorrelation from bolometer plate temperature fluctuations]]<br />
<br />
A simple linear decorrelation is performed using the 2 dark bolometers as a proxy of the bolometer plate temperature. Coupling coefficients were measured during the CPV phase.<br />
<br />
=== 4K cooler lines variability ===<br />
The amplitude of the nine 4K cooler lines in aW at 10, 20, 30, 40, 50, 60, 70, 80 and 17 Hz is shown for 2 bolometers in the following figures. The trend is smoothed over 31 ring values after having discarded measurements done at a ring which is discarded for all bolometers.<br />
<gallery widths=450px perrow=2 caption="Amplitude of the nine 4K cooler lines"><br />
File:10_143_5_4Klines.jpg | 143_5 bolometer<br />
File:14_545_1_4Klines.jpg | 545_1 bolometer<br />
</gallery><br />
[[Media:Lines4K.pdf|The 4K cooler line coefficients of all bolometers are shown in this file]]<br />
<br />
=== jump correction ===<br />
A piecewise constant value is removed to the TOI if a jump is detected. See a jump example in this figure:<br />
[[File:jump_exe.png|200px|An example of a jump seen on the rmsigTOI]]<br />
<br />
The number of jumps per day (all bolometers included) is shown in this figure:<br />
[[File:jumps_per_day.png|200px|Evolution of jump number during the mission]]<br />
<br />
The jumps are uncorrelated from bolometer to bolometer. The total number of jumps detected in the nominal and full mission is shown here:<br />
[[File:jumps_per_bolometer.png|200px|Number of jumps per bolometer]]<br />
<br />
<br />
=== Trends in noise and signal ===<br />
<gallery widths=450px perrow=2 caption="Signal (top) and noise (bottom) smoothed at 1 minute. All values falling in a discarded ring are not plotted."><br />
File:10_143_5_smooth_Watt.jpg | 143_5 bolometer<br />
File:14_545_1_smooth_Watt.jpg | 545_1 bolometer<br />
</gallery><br />
<br />
[[Media:check_smooth.pdf|The smooth TOIs of all detectors are shown in this file]]<br />
<br />
=== Noise stationarity ===<br />
This is not the final version but gives a good idea of power spectra at the detector level of rmsigTOIs.<br />
[[Media:v53_meanSpectra_bySurvey.pdf|All PSDs can be seen in this file]]<br />
<br />
The standard deviation per ring corrected by ring duration bias is given here, one per bolometer using only valid rings. No smoothing is applied (except a 31-point smoothing for the 545<br />
and 857 GHz channels) but values for rings discarded for all bolometers are not used. The standard deviation is computed on samples valid for map-making which are also not affected by the Galaxy or the point-sources using the usual flags. Two examples are given here.<br />
<gallery widths=450px perrow=2 caption="Standard deviation per ring"><br />
File:stddev_group10_143_5.jpg | 143_5 bolometer<br />
File:stddev_group_14_545_1.jpg | 545_1 bolometer<br />
</gallery><br />
<br />
<br />
The full series of plots is here:<br />
[[Media:StationarityInROIoutput2.pdf|Standard deviation of rmsig TOIs at the ring level]]<br />
<br />
Note the presence for 3 bolometers of a two-level noise system. No correction can be done for that effect. See one example here:<br />
[[File:23_353_TwoLevel.jpg|300px|An example of two-level noise system is seen in bolometer 23_353_3a]]<br />
<br />
<br />
An example of the higher order statistics which are used to unveil rings affected by RTS problems.<br />
[[File:HFI_4_4_2_RTSexample13.jpg|400px|Example of RTS detection]]<br />
<br />
== Anomalous rings ==<br />
<br />
Some rings are thus discarded (flagged) from further use (beam making, map making) by using these ring statistics (see HFI data processing paper REF).<br />
For each statistics, we compare each ring value to the ring values averaged (RVA) over all rings. We also define the modified standard deviation (MSD) of a ring quantity as the standard deviation of that quantity over the rings that deviate by less than five nominal standard deviations. This truncation is necessary to be robust against extreme deviant rings.<br />
<br />
More specifically, these are the full criteria (with the OR logic) to decide whether a ring is discarded:<br />
<br />
- if |mean-median| deviates from the RVA by more than fifteen times the MSD.<br />
<br />
- if the standard deviation deviates from the RVA by more than -5 times the MSD (all cases corresponding to almost empty rings) and +15 times the MSD.<br />
<br />
- if the Kolmogorov-Smirnov test deviates from the RVA by more than 15 times the MSD.<br />
<br />
- if the ring duration is outside some definite bounds. Bounds are 25 min and 90 min.<br />
<br />
<br />
Once the list of discarded rings is done per bolometer, a common list of discarded rings can be produced for all bolometers (by using discarded rings for at least half the bolometers). See the following table.<br />
<br />
Furthermore, an isolated valid ring stuck between two discarded rings becomes discarded as well.<br />
<br />
Table of common discarded rings (v47, 240-21890) to be updated to v53, nominal mission.<br />
| '''Cause''' | '''ring number''' |<br />
| manoeuvre | 304 1312 3590 3611 3642 3922 4949 6379 8456 11328 |<br />
| Sorption Cooler Switchover | 11149 11150 11151 11152 |<br />
| Rings too long| 440 474 509 544 897 898 3589 13333 14627 14653 |<br />
| fully flagged as unstable | 14628 14654 14842 16407 18210 |<br />
| common jump | 7665 |<br />
| solar flare | 20451 20452 20453 20454 20455 20456 |<br />
<br />
<br />
<br />
== Flag description ==<br />
<br />
==== Input flags ====<br />
<br />
These flags are used as inputs to the TOI processing<br />
<br />
- The point-source flag (PSflag):<br />
<br />
An earlier version of HFI point-source catalog is read back into a flag TOIs, at a given frequency. In practice, 5 sigma sources are masked within a radius of 1.3 FWHM (9, 7, 5, 5, 5, 5 arcmin at 100,143,217,353,545,857 GHz) TBC.<br />
<br />
- the galactic flag (Galflag):<br />
<br />
An earlier version of HFI maps is thresholded and apodized. The produced masks are read into flag TOIs. The retained threshold corresponds to a sky coverage of respectively 70, 70, 80, 90, 90, 90% at 100,143,217,353,545,857 GHz.<br />
<br />
- Solar System Object flag <br />
<br />
For the TOI flag, Mars, Jupiter, Saturn are flagged up to a radius of NbBeam= 2,3,3,4,4,4 times the fiducial SSO_FWHM with SSO_FWHM= 9, 7, 5, 5, 5, 5 arcmin at 100,143,217,353,545,857 GHz.<br />
<br />
As an input to planet mask for maps, Mars, Jupiter, Saturn are flagged with a radius computed as a coefficient depending on the planet (Factor_per_source) times NbBeam times SSO_FWHM, with Factor_per_source = 1.1, 2.25, 1.25 for Mars, Jupiter, Saturn respectively and NbBeam = 2.25, 4.25, 4.0, 5.0,.6.0, 8.0 at 100, 143, 217, 353, 545, 857 GHz. This flag is called SSOflag4map.<br />
<br />
A small trailing tail is added to the mask to take into account the non-deconvolution of the planet signal which has been replaced by background values. The width of that tail is 10 % of the main flag diameter. The number of samples which are additionnaly flagged are the Factor_per_Source times AddSNafter with 10, 30, 20, 20, 30, 40 samples at 100, 143, 217, 353, 545, 857 GHz.<br />
<br />
Uranus and Neptune together with detected asteroids are masked by HFI. They are masked at the TOI level using an exclusion radius of 1.5 SSO_FWHM. At 857 GHz, 24 asteroids have been detected with HFI : 1Ceres, 2Pallas, 3Juno, 4Vesta, 7Iris, 8Flora, 9Metis, 10Hygiea, 11Parthenope, 12Victoria, 13Egeria, 14Irene, 15Eunomia, 16Psyche, 18Melpomene, 19Fortuna, 20Massalia, 29Amphitrite, 41Daphne, 45Eugenia, 52Europa, 88Thisbe, 704Interamnia, 324Bamberga.<br />
<br />
<br />
=== Output flags ===<br />
<br />
A FlagTOIproc is produced by the TOIprocessing. It marks measurements which are not reliable for ny of the following reasons :<br />
* gap (no valid input data), enlarged by one sample on each side. It flags less than 0.00044% (resp. 0.00062%) of the nominal (resp. complete) mission. It is equivallen to less thena 3 (resp. 8) minutes of data.<br />
* glitch : samples where the signal from a cosmic ray hit dominates the sky signal at more than 3 $\sigma$ are discarded.<br />
* jump as 100 samples are flagged around the computed position of the jump to take into account the error on this position.<br />
<br />
With and without point-sources, identical to a and b PSBs.<br />
<br />
<br />
Flags produced for the map making :<br />
* UnstablePointing Flag OR FlagTOIproc OR SSOflag4map OR SSOflag seen<br />
* FlagTOIproc = glitch OR jump OR flag thermal template <br />
* PSB flag glitch = flag glitch A OR flag glitch B<br />
<br />
Flag used in the destriping: same + point-source flag<br />
<br />
<br />
<br />
<br />
[[Category:Data processing]]</div>
Fdesert
https://wiki.cosmos.esa.int/planckpla2015/index.php?title=TOI_processing&diff=3257
TOI processing
2013-01-10T13:25:52Z
<p>Fdesert: </p>
<hr />
<div>This Section is kept short as this Planck release does not contain TOIs. The main information in the HFI processing paper is not duplicated here.<br />
<br />
==Overview==<br />
<br />
We describe here the how the TOIs are processed in order to be used for map production. We do not repeat the general features of the pipeline which are given in the HFI Data Processing article (REF). Here we give complementary explanations on some details. The TOI of each bolometer is processed independently of the other bolometers, so as to keep the noise properties as uncorrelated as possible. The processing involves modifying the TOI itself for what concerns the conversion to absorbed power and the correction of glitch tails. It also adds a flag TOI that masks the TOI samples that are not to be projected on maps for various reasons.<br />
<br />
<br />
== Input TOI ==<br />
<br />
The input TOI consists in the AC modulated voltage output of the readout of each bolometer. The input has previously been decompressed, and converted from internal digital units to voltage via a constant factor. The TOI has a regular sampling at the acquisition frequency of facq=180.373700+-0.000050 Hz. There are almost no missing data in the TOIs, except for few hundred samples of 545 and 857GHz TOIs which are lost in the on-board compression due to saturation on the Galactic Center crossings. <br />
<br />
== General Pipeline Structure ==<br />
<br />
The figure on the right shows how the initial ccTOI is transformed and how flags are produced:<br />
<br />
[[File:HFI_4_4_2_ToiProc_ExplS.png|thumb|600px|A schematic of the TOI processing pipeline]]<br />
<br />
<br />
== Output TOIs and products ==<br />
<br />
A TOI of clean calibrated samples (ccTOI) and a combined flag TOI (fTOI) are the outputs of the processing. The ccTOI is calibrated so as to represent the instantaneous power absorbed by the detector up to a constant (which will be determined by the map-making destriper). It is worth mentioning how the ccTOI is changed with respect to the input TOI, beyond the harmless constant conversion factor from voltage to absorbed power. The demodulation stage allows to get the demodulated bolometer voltage. The non-linearity correction is a second-order polynomial correction based on the physical but static bolometer model. In order to avoid too much masking after glitches, a glitch tail is subtracted after an occurrence of a glitch in the TOI. The 4K cooler lines noise is substituted at a series of 9 single temporal frequencies. Finally, the temporal response of the bolometer is deconvolved. This affects mostly the high-temporal frequency part of the TOI, although a small but significant low frequency (the long time response) tail is corrected too. Although flagged samples are not projected, their value influences the valid samples somehow. Hence interpolation procedures introduce some indirect modifications of the TOI.<br />
The flag TOI is a combination a dozen flags with an OR logic. Only unflagged data are projected. The exhaustive list of flags is given here: CompressionError, NoData, SSO, UnstablePointing, Glitch, BoloPlateFluctuation, RTS, Jump, PSBab. A complete qualification of the data is obtained at the ring level. If the TOI shows an anomalous behaviour on a given ring, this ring is discarded from projection. A special production of TOIs is also made as an input to the beam analysis with Mars, Jupiter and Saturn.<br />
<br />
<br />
== Examples of clean TOIs ==<br />
<gallery widths=300px perrow=3><br />
File:10_143_5_PBR_10800RING_LFER4_JC.jpg|Phase-binned signal (PBR) for 6 consecutive rings from 4000 to 4005<br />
File:10_143_5_rcircles_LFER4_JC.jpg| Valid signal (up) and noise (bottom) for 2 consecutive circles of the ring 4000<br />
File:10_143_5_fft_LFER4_JC.jpg| Fourier power spectral density of the first 30 minutes of the ring 4005 with signal before (pink) and after (black) deconvolution (top) and noise (bottom)<br />
File:14_545_1_PBR_10800RING_LFER4_JC.jpg|Phase-binned signal (PBR) for 6 consecutive rings from 4000 to 4005<br />
File:14_545_1_rcircles_LFER4_JC.jpg| Valid signal (up) and noise (bottom) for 2 consecutive circles of the ring 4000<br />
File:14_545_1_fft_LFER4_JC.jpg| Fourier power spectral density of the first 30 minutes of the ring 4005 with signal before (pink) and after (black) deconvolution (top) and noise (bottom)<br />
</gallery><br />
<br />
[[Media:check_ring.pdf|Samples of PBR, TOIs, and PSDs of all detectors are shown in this file]]<br />
<br />
== Trends in the output processing variables ==<br />
<br />
Here we intend to show the trend of the systematic effects that are dealt with in the TOI processing. The full impact of each of them is analyzed in [[HFI-Validation]].<br />
<br />
=== ADC baseline ===<br />
The following figure shows the ADC baseline which is used prior to demodulation (a constant offset is removed for clarity). This baseline is obtained by smoothing on an hour block average the undemodulated TOI on unflagged samples.<br />
[[File:baseline.jpg|400px|ADC baseline for all bolometers]]<br />
<br />
=== Glitch statistics ===<br />
The glitch rate per channel is shown in this figure. For details, see copap. <br />
[[File:figIntermPaperGR.jpg|400px|Glitch rate evolution]]<br />
<br />
The percentage of flagged data (mostly due to Cosmic Rays) at the ring level is shown in these examples. No smoothing was applied. Only valid rings are shown.<br />
<gallery widths=450px perrow=2 caption="Percentage of flagged data"><br />
File:group10_143_5.jpg | 143 GHz bolometers<br />
File:group14_545_1.jpg | 545 GHz bolometers<br />
</gallery><br />
[[Media:PercentUnvalid2.pdf|The complete set of plots is here]]<br />
<br />
=== Thermal template for decorrelation ===<br />
[[File:T90.jpg|400px|Thermal template used for the decorrelation from bolometer plate temperature fluctuations]]<br />
<br />
A simple linear decorrelation is performed using the 2 dark bolometers as a proxy of the bolometer plate temperature. Coupling coefficients were measured during the CPV phase.<br />
<br />
=== 4K cooler lines variability ===<br />
The amplitude of the nine 4K cooler lines in aW at 10, 20, 30, 40, 50, 60, 70, 80 and 17 Hz is shown for 2 bolometers in the following figures. The trend is smoothed over 31 ring values after having discarded measurements done at a ring which is discarded for all bolometers.<br />
<gallery widths=450px perrow=2 caption="Amplitude of the nine 4K cooler lines"><br />
File:10_143_5_4Klines.jpg | 143_5 bolometer<br />
File:14_545_1_4Klines.jpg | 545_1 bolometer<br />
</gallery><br />
[[Media:Lines4K.pdf|The 4K cooler line coefficients of all bolometers are shown in this file]]<br />
<br />
=== jump correction ===<br />
A piecewise constant value is removed to the TOI if a jump is detected. See a jump example in this figure:<br />
[[File:jump_exe.png|200px|An example of a jump seen on the rmsigTOI]]<br />
<br />
The number of jumps per day (all bolometers included) is shown in this figure:<br />
[[File:jumps_per_day.png|200px|Evolution of jump number during the mission]]<br />
<br />
The jumps are uncorrelated from bolometer to bolometer. The total number of jumps detected in the nominal and full mission is shown here:<br />
[[File:jumps_per_bolometer.png|200px|Number of jumps per bolometer]]<br />
<br />
<br />
=== Trends in noise and signal ===<br />
<gallery widths=450px perrow=2 caption="Signal (top) and noise (bottom) smoothed at 1 minute. All values falling in a discarded ring are not plotted."><br />
File:10_143_5_smooth_Watt.jpg | 143_5 bolometer<br />
File:14_545_1_smooth_Watt.jpg | 545_1 bolometer<br />
</gallery><br />
<br />
[[Media:check_smooth.pdf|The smooth TOIs of all detectors are shown in this file]]<br />
<br />
=== Noise stationarity ===<br />
This is not the final version but gives a good idea of power spectra at the detector level of rmsigTOIs.<br />
[[Media:v53_meanSpectra_bySurvey.pdf|All PSDs can be seen in this file]]<br />
<br />
The standard deviation per ring corrected by ring duration bias is given here, one per bolometer using only valid rings. No smoothing is applied (except a 31-point smoothing for the 545<br />
and 857 GHz channels) but values for rings discarded for all bolometers are not used. The standard deviation is computed on samples valid for map-making which are also not affected by the Galaxy or the point-sources using the usual flags. Two examples are given here.<br />
<gallery widths=450px perrow=2 caption="Standard deviation per ring"><br />
File:stddev_group10_143_5.jpg | 143_5 bolometer<br />
File:stddev_group_14_545_1.jpg | 545_1 bolometer<br />
</gallery><br />
<br />
<br />
The full series of plots is here:<br />
[[Media:StationarityInROIoutput2.pdf|Standard deviation of rmsig TOIs at the ring level]]<br />
<br />
Note the presence for 3 bolometers of a two-level noise system. No correction can be done for that effect. See one example here:<br />
[[File:23_353_TwoLevel.jpg|300px|An example of two-level noise system is seen in bolometer 23_353_3a]]<br />
<br />
<br />
An example of the higher order statistics which are used to unveil rings affected by RTS problems.<br />
[[File:HFI_4_4_2_RTSexample13.jpg|400px|Example of RTS detection]]<br />
<br />
== Anomalous rings ==<br />
<br />
Some rings are thus discarded (flagged) from further use (beam making, map making) by using these ring statistics (see HFI data processing paper REF).<br />
For each statistics, we compare each ring value to the ring values averaged (RVA) over all rings. We also define the modified standard deviation (MSD) of a ring quantity as the standard deviation of that quantity over the rings that deviate by less than five nominal standard deviations. This truncation is necessary to be robust against extreme deviant rings.<br />
<br />
More specifically, these are the full criteria (with the OR logic) to decide whether a ring is discarded:<br />
<br />
- if |mean-median| deviates from the RVA by more than fifteen times the MSD.<br />
<br />
- if the standard deviation deviates from the RVA by more than -5 times the MSD (all cases corresponding to almost empty rings) and +15 times the MSD.<br />
<br />
- if the Kolmogorov-Smirnov test deviates from the RVA by more than 15 times the MSD.<br />
<br />
- if the ring duration is outside some definite bounds. Bounds are 25 min and 90 min.<br />
<br />
<br />
Once the list of discarded rings is done per bolometer, a common list of discarded rings can be produced for all bolometers (by using discarded rings for at least half the bolometers). See the following table.<br />
<br />
Furthermore, an isolated valid ring stuck between two discarded rings becomes discarded as well.<br />
<br />
Table of common discarded rings (v47, 240-21890) to be updated to v53, nominal mission.<br />
|| border="1"<br />
|| '''Cause''' || '''ring number''' ||<br />
|| manoeuvre || 304 1312 3590 3611 3642 3922 4949 6379 8456 11328 ||<br />
|| Sorption Cooler Switchover || 11149 11150 11151 11152 ||<br />
|| Rings too long|| 440 474 509 544 897 898 3589 13333 14627 14653 ||<br />
|| fully flagged as unstable || 14628 14654 14842 16407 18210 ||<br />
|| common jump || 7665 ||<br />
|| solar flare || 20451 20452 20453 20454 20455 20456 ||<br />
<br />
<br />
<br />
== Flag description ==<br />
<br />
==== Input flags ====<br />
<br />
These flags are used as inputs to the TOI processing<br />
<br />
- The point-source flag (PSflag):<br />
<br />
An earlier version of HFI point-source catalog is read back into a flag TOIs, at a given frequency. In practice, 5 sigma sources are masked within a radius of 1.3 FWHM (9, 7, 5, 5, 5, 5 arcmin at 100,143,217,353,545,857 GHz) TBC.<br />
<br />
- the galactic flag (Galflag):<br />
<br />
An earlier version of HFI maps is thresholded and apodized. The produced masks are read into flag TOIs. The retained threshold corresponds to a sky coverage of respectively 70, 70, 80, 90, 90, 90% at 100,143,217,353,545,857 GHz.<br />
<br />
- Solar System Object flag <br />
<br />
For the TOI flag, Mars, Jupiter, Saturn are flagged up to a radius of NbBeam= 2,3,3,4,4,4 times the fiducial SSO_FWHM with SSO_FWHM= 9, 7, 5, 5, 5, 5 arcmin at 100,143,217,353,545,857 GHz.<br />
<br />
As an input to planet mask for maps, Mars, Jupiter, Saturn are flagged with a radius computed as a coefficient depending on the planet (Factor_per_source) times NbBeam times SSO_FWHM, with Factor_per_source = 1.1, 2.25, 1.25 for Mars, Jupiter, Saturn respectively and NbBeam = 2.25, 4.25, 4.0, 5.0,.6.0, 8.0 at 100, 143, 217, 353, 545, 857 GHz. This flag is called SSOflag4map.<br />
<br />
A small trailing tail is added to the mask to take into account the non-deconvolution of the planet signal which has been replaced by background values. The width of that tail is 10 % of the main flag diameter. The number of samples which are additionnaly flagged are the Factor_per_Source times AddSNafter with 10, 30, 20, 20, 30, 40 samples at 100, 143, 217, 353, 545, 857 GHz.<br />
<br />
Uranus and Neptune together with detected asteroids are masked by HFI. They are masked at the TOI level using an exclusion radius of 1.5 SSO_FWHM. At 857 GHz, 24 asteroids have been detected with HFI : 1Ceres, 2Pallas, 3Juno, 4Vesta, 7Iris, 8Flora, 9Metis, 10Hygiea, 11Parthenope, 12Victoria, 13Egeria, 14Irene, 15Eunomia, 16Psyche, 18Melpomene, 19Fortuna, 20Massalia, 29Amphitrite, 41Daphne, 45Eugenia, 52Europa, 88Thisbe, 704Interamnia, 324Bamberga.<br />
<br />
<br />
=== Output flags ===<br />
<br />
A FlagTOIproc is produced by the TOIprocessing. It marks measurements which are not reliable for ny of the following reasons :<br />
* gap (no valid input data), enlarged by one sample on each side. It flags less than 0.00044% (resp. 0.00062%) of the nominal (resp. complete) mission. It is equivallen to less thena 3 (resp. 8) minutes of data.<br />
* glitch : samples where the signal from a cosmic ray hit dominates the sky signal at more than 3 $\sigma$ are discarded.<br />
* jump as 100 samples are flagged around the computed position of the jump to take into account the error on this position.<br />
<br />
With and without point-sources, identical to a and b PSBs.<br />
<br />
<br />
Flags produced for the map making :<br />
* UnstablePointing Flag OR FlagTOIproc OR SSOflag4map OR SSOflag seen<br />
* FlagTOIproc = glitch OR jump OR flag thermal template <br />
* PSB flag glitch = flag glitch A OR flag glitch B<br />
<br />
Flag used in the destriping: same + point-source flag<br />
<br />
<br />
<br />
<br />
[[Category:Data processing]]</div>
Fdesert
https://wiki.cosmos.esa.int/planckpla2015/index.php?title=TOI_processing&diff=3256
TOI processing
2013-01-10T13:18:30Z
<p>Fdesert: </p>
<hr />
<div>This Section is kept short as this Planck release does not contain TOIs. The main information in the HFI processing paper is not duplicated here.<br />
<br />
==Overview==<br />
<br />
We describe here the how the TOIs are processed in order to be used for map production. We do not repeat the general features of the pipeline which are given in the HFI Data Processing article (REF). Here we give complementary explanations on some details. The TOI of each bolometer is processed independently of the other bolometers, so as to keep the noise properties as uncorrelated as possible. The processing involves modifying the TOI itself for what concerns the conversion to absorbed power and the correction of glitch tails. It also adds a flag TOI that masks the TOI samples that are not to be projected on maps for various reasons.<br />
<br />
<br />
== Input TOI ==<br />
<br />
The input TOI consists in the AC modulated voltage output of the readout of each bolometer. The input has previously been decompressed, and converted from internal digital units to voltage via a constant factor. The TOI has a regular sampling at the acquisition frequency of facq=180.373700+-0.000050 Hz. There are almost no missing data in the TOIs, except for few hundred samples of 545 and 857GHz TOIs which are lost in the on-board compression due to saturation on the Galactic Center crossings. <br />
<br />
== General Pipeline Structure ==<br />
<br />
The figure on the right shows how the initial ccTOI is transformed and how flags are produced:<br />
<br />
[[File:HFI_4_4_2_ToiProc_ExplS.png|thumb|600px|A schematic of the TOI processing pipeline]]<br />
<br />
<br />
== Output TOIs and products ==<br />
<br />
A TOI of clean calibrated samples (ccTOI) and a combined flag TOI (fTOI) are the outputs of the processing. The ccTOI is calibrated so as to represent the instantaneous power absorbed by the detector up to a constant (which will be determined by the map-making destriper). It is worth mentioning how the ccTOI is changed with respect to the input TOI, beyond the harmless constant conversion factor from voltage to absorbed power. The demodulation stage allows to get the demodulated bolometer voltage. The non-linearity correction is a second-order polynomial correction based on the physical but static bolometer model. In order to avoid too much masking after glitches, a glitch tail is subtracted after an occurrence of a glitch in the TOI. The 4K cooler lines noise is substituted at a series of 9 single temporal frequencies. Finally, the temporal response of the bolometer is deconvolved. This affects mostly the high-temporal frequency part of the TOI, although a small but significant low frequency (the long time response) tail is corrected too. Although flagged samples are not projected, their value influences the valid samples somehow. Hence interpolation procedures introduce some indirect modifications of the TOI.<br />
The flag TOI is a combination a dozen flags with an OR logic. Only unflagged data are projected. The exhaustive list of flags is given here: CompressionError, NoData, SSO, UnstablePointing, Glitch, BoloPlateFluctuation, RTS, Jump, PSBab. A complete qualification of the data is obtained at the ring level. If the TOI shows an anomalous behaviour on a given ring, this ring is discarded from projection. A special production of TOIs is also made as an input to the beam analysis with Mars, Jupiter and Saturn.<br />
<br />
<br />
== Examples of clean TOIs ==<br />
<gallery widths=300px perrow=3><br />
File:10_143_5_PBR_10800RING_LFER4_JC.jpg|Phase-binned signal (PBR) for 6 consecutive rings from 4000 to 4005<br />
File:10_143_5_rcircles_LFER4_JC.jpg| Valid signal (up) and noise (bottom) for 2 consecutive circles of the ring 4000<br />
File:10_143_5_fft_LFER4_JC.jpg| Fourier power spectral density of the first 30 minutes of the ring 4005 with signal before (pink) and after (black) deconvolution (top) and noise (bottom)<br />
File:14_545_1_PBR_10800RING_LFER4_JC.jpg|Phase-binned signal (PBR) for 6 consecutive rings from 4000 to 4005<br />
File:14_545_1_rcircles_LFER4_JC.jpg| Valid signal (up) and noise (bottom) for 2 consecutive circles of the ring 4000<br />
File:14_545_1_fft_LFER4_JC.jpg| Fourier power spectral density of the first 30 minutes of the ring 4005 with signal before (pink) and after (black) deconvolution (top) and noise (bottom)<br />
</gallery><br />
<br />
[[Media:check_ring.pdf|Samples of PBR, TOIs, and PSDs of all detectors are shown in this file]]<br />
<br />
== Trends in the output processing variables ==<br />
<br />
Here we intend to show the trend of the systematic effects that are dealt with in the TOI processing. The full impact of each of them is analyzed in [[HFI-Validation]].<br />
<br />
=== ADC baseline ===<br />
The following figure shows the ADC baseline which is used prior to demodulation (a constant offset is removed for clarity). This baseline is obtained by smoothing on an hour block average the undemodulated TOI on unflagged samples.<br />
[[File:baseline.jpg|400px|ADC baseline for all bolometers]]<br />
<br />
=== Glitch statistics ===<br />
The glitch rate per channel is shown in this figure. For details, see copap. <br />
[[File:figIntermPaperGR.jpg|400px|Glitch rate evolution]]<br />
<br />
The percentage of flagged data (mostly due to Cosmic Rays) at the ring level is shown in these examples. No smoothing was applied. Only valid rings are shown.<br />
<gallery widths=450px perrow=2 caption="Percentage of flagged data"><br />
File:group10_143_5.jpg | 143 GHz bolometers<br />
File:group14_545_1.jpg | 545 GHz bolometers<br />
</gallery><br />
[[Media:PercentUnvalid2.pdf|The complete set of plots is here]]<br />
<br />
=== Thermal template for decorrelation ===<br />
[[File:T90.jpg|400px|Thermal template used for the decorrelation from bolometer plate temperature fluctuations]]<br />
<br />
A simple linear decorrelation is performed using the 2 dark bolometers as a proxy of the bolometer plate temperature. Coupling coefficients were measured during the CPV phase.<br />
<br />
=== 4K cooler lines variability ===<br />
The amplitude of the nine 4K cooler lines in aW at 10, 20, 30, 40, 50, 60, 70, 80 and 17 Hz is shown for 2 bolometers in the following figures. The trend is smoothed over 31 ring values after having discarded measurements done at a ring which is discarded for all bolometers.<br />
<gallery widths=450px perrow=2 caption="Amplitude of the nine 4K cooler lines"><br />
File:10_143_5_4Klines.jpg | 143_5 bolometer<br />
File:14_545_1_4Klines.jpg | 545_1 bolometer<br />
</gallery><br />
[[Media:Lines4K.pdf|The 4K cooler line coefficients of all bolometers are shown in this file]]<br />
<br />
=== jump correction ===<br />
A piecewise constant value is removed to the TOI if a jump is detected. See a jump example in this figure:<br />
[[File:jump_exe.png|200px|An example of a jump seen on the rmsigTOI]]<br />
<br />
The number of jumps per day (all bolometers included) is shown in this figure:<br />
[[File:jumps_per_day.png|200px|Evolution of jump number during the mission]]<br />
<br />
The jumps are uncorrelated from bolometer to bolometer. The total number of jumps detected in the nominal and full mission is shown here:<br />
[[File:jumps_per_bolometer.png|200px|Number of jumps per bolometer]]<br />
<br />
<br />
=== Trends in noise and signal ===<br />
<gallery widths=450px perrow=2 caption="Signal (top) and noise (bottom) smoothed at 1 minute. All values falling in a discarded ring are not plotted."><br />
File:10_143_5_smooth_Watt.jpg | 143_5 bolometer<br />
File:14_545_1_smooth_Watt.jpg | 545_1 bolometer<br />
</gallery><br />
<br />
[[Media:check_smooth.pdf|The smooth TOIs of all detectors are shown in this file]]<br />
<br />
=== Noise stationarity ===<br />
This is not the final version but gives a good idea of power spectra at the detector level of rmsigTOIs.<br />
[[Media:v53_meanSpectra_bySurvey.pdf|All PSDs can be seen in this file]]<br />
<br />
The standard deviation per ring corrected by ring duration bias is given here, one per bolometer using only valid rings. No smoothing is applied (except a 31-point smoothing for the 545<br />
and 857 GHz channels) but values for rings discarded for all bolometers are not used. The standard deviation is computed on samples valid for map-making which are also not affected by the Galaxy or the point-sources using the usual flags. Two examples are given here.<br />
<gallery widths=450px perrow=2 caption="Standard deviation per ring"><br />
File:stddev_group10_143_5.jpg | 143_5 bolometer<br />
File:stddev_group_14_545_1.jpg | 545_1 bolometer<br />
</gallery><br />
<br />
<br />
The full series of plots is here:<br />
[[Media:StationarityInROIoutput2.pdf|Standard deviation of rmsig TOIs at the ring level]]<br />
<br />
Note the presence for 3 bolometers of a two-level noise system. No correction can be done for that effect. See one example here:<br />
[[File:23_353_TwoLevel.jpg|300px|An example of two-level noise system is seen in bolometer 23_353_3a]]<br />
<br />
<br />
An example of the higher order statistics which are used to unveil rings affected by RTS problems.<br />
[[File:HFI_4_4_2_RTSexample13.jpg|400px|Example of RTS detection]]<br />
<br />
== Anomalous rings ==<br />
<br />
Some rings are thus discarded (flagged) from further use (beam making, map making) by using these ring statistics (see HFI data processing paper REF).<br />
For each statistics, we compare each ring value to the ring values averaged (RVA) over all rings. We also define the modified standard deviation (MSD) of a ring quantity as the standard deviation of that quantity over the rings that deviate by less than five nominal standard deviations. This truncation is necessary to be robust against extreme deviant rings.<br />
<br />
More specifically, these are the full criteria (with the OR logic) to decide whether a ring is discarded:<br />
<br />
- if |mean-median| deviates from the RVA by more than fifteen times the MSD.<br />
<br />
- if the standard deviation deviates from the RVA by more than -5 times the MSD (all cases corresponding to almost empty rings) and +15 times the MSD.<br />
<br />
- if the Kolmogorov-Smirnov test deviates from the RVA by more than 15 times the MSD.<br />
<br />
- if the ring duration is outside some definite bounds. Bounds are 15 min and 70 min.<br />
<br />
<br />
<br />
Once the list of discarded rings is done per bolometer, a common list of discarded rings can be produced for all bolometers (discarded rings for at least half the bolometers). Furthermore, an isolated valid ring stuck between two discarded rings becomes discarded as well.<br />
<br />
<br />
<br />
<br />
== Flag description ==<br />
<br />
==== Input flags ====<br />
<br />
These flags are used as inputs to the TOI processing<br />
<br />
- The point-source flag (PSflag):<br />
<br />
An earlier version of HFI point-source catalog is read back into a flag TOIs, at a given frequency. In practice, 5 sigma sources are masked within a radius of 1.3 FWHM (9, 7, 5, 5, 5, 5 arcmin at 100,143,217,353,545,857 GHz) TBC.<br />
<br />
- the galactic flag (Galflag):<br />
<br />
An earlier version of HFI maps is thresholded and apodized. The produced masks are read into flag TOIs. The retained threshold corresponds to a sky coverage of respectively 70, 70, 80, 90, 90, 90% at 100,143,217,353,545,857 GHz.<br />
<br />
- Solar System Object flag <br />
<br />
For the TOI flag, Mars, Jupiter, Saturn are flagged up to a radius of NbBeam= 2,3,3,4,4,4 times the fiducial SSO_FWHM with SSO_FWHM= 9, 7, 5, 5, 5, 5 arcmin at 100,143,217,353,545,857 GHz.<br />
<br />
As an input to planet mask for maps, Mars, Jupiter, Saturn are flagged with a radius computed as a coefficient depending on the planet (Factor_per_source) times NbBeam times SSO_FWHM, with Factor_per_source = 1.1, 2.25, 1.25 for Mars, Jupiter, Saturn respectively and NbBeam = 2.25, 4.25, 4.0, 5.0,.6.0, 8.0 at 100, 143, 217, 353, 545, 857 GHz. This flag is called SSOflag4map.<br />
<br />
A small trailing tail is added to the mask to take into account the non-deconvolution of the planet signal which has been replaced by background values. The width of that tail is 10 % of the main flag diameter. The number of samples which are additionnaly flagged are the Factor_per_Source times AddSNafter with 10, 30, 20, 20, 30, 40 samples at 100, 143, 217, 353, 545, 857 GHz.<br />
<br />
Uranus and Neptune together with detected asteroids are masked by HFI. They are masked at the TOI level using an exclusion radius of 1.5 SSO_FWHM. At 857 GHz, 24 asteroids have been detected with HFI : 1Ceres, 2Pallas, 3Juno, 4Vesta, 7Iris, 8Flora, 9Metis, 10Hygiea, 11Parthenope, 12Victoria, 13Egeria, 14Irene, 15Eunomia, 16Psyche, 18Melpomene, 19Fortuna, 20Massalia, 29Amphitrite, 41Daphne, 45Eugenia, 52Europa, 88Thisbe, 704Interamnia, 324Bamberga.<br />
<br />
<br />
=== Output flags ===<br />
<br />
A FlagTOIproc is produced by the TOIprocessing. It marks measurements which are not reliable for ny of the following reasons :<br />
* gap (no valid input data), enlarged by one sample on each side. It flags less than 0.00044% (resp. 0.00062%) of the nominal (resp. complete) mission. It is equivallen to less thena 3 (resp. 8) minutes of data.<br />
* glitch : samples where the signal from a cosmic ray hit dominates the sky signal at more than 3 $\sigma$ are discarded.<br />
* jump as 100 samples are flagged around the computed position of the jump to take into account the error on this position.<br />
<br />
With and without point-sources, identical to a and b PSBs.<br />
<br />
<br />
Flags produced for the map making :<br />
* UnstablePointing Flag OR FlagTOIproc OR SSOflag4map OR SSOflag seen<br />
* FlagTOIproc = glitch OR jump OR flag thermal template <br />
* PSB flag glitch = flag glitch A OR flag glitch B<br />
<br />
Flag used in the destriping: same + point-source flag<br />
<br />
<br />
<br />
<br />
[[Category:Data processing]]</div>
Fdesert