Difference between revisions of "Timelines"

From Planck Legacy Archive Wiki
Jump to: navigation, search
m (General description)
(HFI processing)
Line 45: Line 45:
 
Throughout this processing, some planet signals are masked in the timeline in order to avoid ringing effects in the processing. It concerns only Jupiter at 100 and 143 GHZ, Jupiter and Saturn at 217 GHz, Jupiter, Saturn and Mars at 353, 545 and 857 GHz (thanks to various improvements in the despiking and in the transfer functions it is no longer necessary to flag these three planets in the CMB channels as was done in Release 1).  Since they move on the sky, the portion of the sky masked during one survey is observed during one, and no hole is left in the final map.  In parallel, the planet data are processed in a similar way, but with different parameters for the despiking step. These results are processed separately to determine the beam shapes and the focal plane geometry.
 
Throughout this processing, some planet signals are masked in the timeline in order to avoid ringing effects in the processing. It concerns only Jupiter at 100 and 143 GHZ, Jupiter and Saturn at 217 GHz, Jupiter, Saturn and Mars at 353, 545 and 857 GHz (thanks to various improvements in the despiking and in the transfer functions it is no longer necessary to flag these three planets in the CMB channels as was done in Release 1).  Since they move on the sky, the portion of the sky masked during one survey is observed during one, and no hole is left in the final map.  In parallel, the planet data are processed in a similar way, but with different parameters for the despiking step. These results are processed separately to determine the beam shapes and the focal plane geometry.
  
Finally, the TOIs are calibrated in astrophysical units using the results of the calibration the calibration pipeline, described in the [[Map-making | Map-making and calibration]] section, which are stored in the IMO, and the best estimate of the zero-point offsets (a constant level for each bolometer) and of the far-side lobes (another signal TOI determined from the input TOI) are removed. These TOIs are ready for projection onto a map.  That projection normally requires that the low-frequency noise be removed via a "destriping" (sometimes called "baseline removal") step.  The HFI-DPC does its destriping at ring-level; should the user want to use the DPC's offsets, they are provided separately ([##REF##]). Due to a technical reason, for a ring with a duration larger than 72 minutes, only the first 72 minutes of the ring are used to make the HFI maps which are officially delivered (the loss of data only amounts to a total of 1/2 hour for the 29 months of HFI data).
+
Next, the TOIs are calibrated in astrophysical units using the results of the calibration the calibration pipeline, described in the [[Map-making | Map-making and calibration]] section, which are stored in the IMO, and the best estimate of the zero-point offsets (a constant level for each bolometer) and of the far-side lobes (another signal TOI determined from the input TOI) are removed. Then the TOIs are multiplied by factors of 1.00087, 1.00046, and 1.00043,  at 100-217 GHz respectively, to compensate for the non-removal of the far-side lobes. 
 +
 
 +
Finally the low-ell modes are filtered out the PBS TOIs  [##REF to reason for this##]. This is done by low-pass filtering a CMB map (produced by the component separation effort).  This map is then ''scanned'' to produce TOIs of the low-ell modes, and these TOIs are subtracted from the TOIs produced above. 
 +
 
 +
These TOIs are ready for projection onto a map.  That projection normally requires that the low-frequency noise be removed via a "destriping" (sometimes called "baseline removal") step.  The HFI-DPC does its destriping at ring-level; should the user want to use the DPC's offsets, they are provided separately ([##REF##]). Due to a technical reason, for a ring with a duration larger than 72 minutes, only the first 72 minutes of the ring are used to make the HFI maps which are officially delivered (the loss of data only amounts to a total of 1/2 hour for the 29 months of HFI data).
 +
 
  
 
The pointing (see common [[Detector_pointing | Detector Pointing]] section for details) is determined from the AHF produced by MOC, which gives the direction and orientation of the LOS of a fiducial position in the focal plane at frequencies of 8Hz during stable pointing and 4 Hz during maneuvers.  This is interpolated to the timestamps of data samples, corrected for the wobble and other time-dependent offsets determined from the observed positions of a large number of sources around the sky, and finally converted to the LOS of each detector using the quaternions in the IMO.
 
The pointing (see common [[Detector_pointing | Detector Pointing]] section for details) is determined from the AHF produced by MOC, which gives the direction and orientation of the LOS of a fiducial position in the focal plane at frequencies of 8Hz during stable pointing and 4 Hz during maneuvers.  This is interpolated to the timestamps of data samples, corrected for the wobble and other time-dependent offsets determined from the observed positions of a large number of sources around the sky, and finally converted to the LOS of each detector using the quaternions in the IMO.

Revision as of 16:06, 30 October 2014

General description[edit]

The timelines, or TOIs for Time-Ordered Information, are vectors of signal or of pointing or of some other quantity giving the value of that quantity as a function of time during the mission. The TOIs described here are sampled regularly at the (instrument dependent) detector sampling frequency and span the full science mission. They thus consist of a large amount of data. This section is concerned with the signal TOIs and their associated pointing TOIs. For each bolometer the DPCs provide a single timeline of calibrated signal and 3 timelines of coordinates, corresponding to the two angular coordinates and one position angle.

Obviously these vectors are very long (~1.38E10 samples for HFI, nnnn for LFI) and thus need to be split into multiple files for export. Here the data are split by Operational Day (OD) as follows:

  • HFI: 12 files per OD, one for pointing and one for signal for each channel, covering ODs 91 -- 974, all sampled at Fsamp = 180.3737 Hz.
  • LFI: 6 files per OD, 3 each for signal and pointing, since the sampling frequency is channel dependent, and covering ODs 91--NNN.

The signal timelines are encoded as Real*4, but the pointing vectors had to be encoded as Real*8 to maintain the required accuracy. The result is that the total volume of the full dataset is ~30 TB. All files also contain the OBT.

The signal timelines have been cleaned of all known instrumental systematic effects, they have been calibrated in flux, and the solar and earth motion dipole signals have been removed. But they have not been cleaned of the low-frequency, 1/f noise that needs to be removed via a destriping tool. The methods consists of removing offsets or baselines determined by minimizing the differences in the signals at points on the sky where they intersect. For HFI, an offset per ring is determined; For LFI ..... . These offsets are determined using the full mission and all the valid detectors per channel, and they are then used for all the maps produced, i.e., for those maps using any fraction of the mission (year, survey) or any subset of the detectors (single detector, detector set), and it was shown that using offsets determined from a limited part of the mission and/or a subset of the detectors yields maps that are less consistent than otherwise. The offsets are delivered separately, as described below.

Maybe insert a figure for example?

Furthermore, for HFI, there are three sets of offsets produced: the primary set using the full rings, and the secondary ones using the first and second half rings, only. The difference between the primary and the secondary sets are fairly minor, but they are necessary to rebuild the maps as they were built by the HFI-DPC. Overall the offsets are useful for users wishing to build a map of a small area of the sky.

TBW: baselines for LFI

The HFI delivers its offsets in a ROI, or "Ring-Ordered Information" file. That file contains a table of Nrings = 26766 rows by Nbolometers = 50 columns in which each cell contains a 3-elements vector containing the full-ring, first half-ring, and second half-ring offsets. The indices of the signal TOI delimiting the rings are given in a second ROI file with the same Nrings rows containing global parameters.

Indexing[edit]

Each FITS file contains keywords giving the first and last index of the section of the full TOI contained in the file. These indices are those of the internal DPC representation; they begin at 0 corresponding to the switch-on of the instrument, and run to some very large number (~25 billion for HFI) representing the full mission. Only the science part of the mission is exported, which is about 60% of the total for HFI. Are left out the early mission phase (cool-down and validation phases) and the warm phase of HFI, during which LFI continued to collect data. The HFI keywords also indicated which rings are included in each OD, but be aware that normally an OD does not begin at the same time (with the same index) as a ring. The HFI offsets are determined on a per-ring basis, and must be applied ring-by-ring, and for this purpose the begin and end ring indices are given in


TBW: HFI: explain TOI indices and (BEG/ENDINDEX kwds) and same for ROI

Production process[edit]

HFI processing[edit]

The input TOIs are in ADUs representing the voltage signal at the end of the electronics (see Detection chain for details). The processing applied to remove instrumental effects and to calibrate them is described in detail in the HFI TOI processing section, and we give a very brief summary here for convenience. That pipeline performs the following operations:

ADC correction
corrects for the uneven size of the ADC bins.
demodulation
this is performed around a variable level which is determined from the valid input data (using a validity flag from a previous version of the processing), and the data are converted to engineering units (V) using known conversion coefficients.
despiking
the redundancy within a ring is used to determine where glitches occur. Once identified, the glitches are fitted with templates. A glitch flag is produced that identifies the strongest part of the glitches, and a timeline of glitch tails. produced from the template fits, is subtracted from the demodulated timeline. Finally, the flagged regions are replaced with data from an average over the pointing period in order to not leave holes in the timeline that would perturb the usage of Fourier methods on them
dark template removal
the two dark bolometers are demodulated and despiked as above; the resulting timelines are then smoothed and used as an indicator of the overall temperature variations of the bolometer plate. Where the variations are consistent with each other, they are combined and removed from the bolometer signal timelines using appropriate coupling coefficients. The few percent of the data where they are not consistent are flagged as not valid on the timelines.
conversion to absorbed power
the timeline is converted to watts of absorbed power using the bolometer function. This includes a non-linearity correction; removal of the 4K cooler lines: the electromagnetic interference of the 4K cooler with the bolometer readout wires induces some sharp lines in the signal power spectra at frequencies of the 4K cooler's fundamental and its multiples, folded by the signal modulations. Fourier coefficients of the relevant lines are determined on a per-ring basis, and then removed from the data. The quality of the removal depends on the bolometer.
deconvolution by the time transfer function
this is done to correct for the non-instantaneous time response of the bolometers. The function itself is modeled using up to 8 time constants which are adjusted primarily on the planet data and also from comparisons of the northward and southward scans of the Galactic Plane. It is then removed using Fourier techniques, which has the side-effect of increasing the noise at high frequencies.
jump correction
removes some (relatively rare: 0.3 jumps per bolometer per pointing period, on average) jumps in the signal baseline. The jumps are detected characterized on smoothed TOIs, and corrected by adding a constant to part of the signal timeline. The origin of the jumps is not known.

The results of this processing are a timeline of signal (in absorbed watts) and a valid data flag timeline for each of the 50 valid bolometers which are processed, out of the 52 HFI bolometers; these timelines contain the full sky signal, i.e., including the solar and orbital dipoles, the Zodiacal light, and contributions from the Far-Side lobes. The dipoles are necessary for the flux calibration and are removed at the mapmaking stage. The remaining two bolometers (143-8 and 545-3) show semi-random jumps in the signal level, typically jumping over 2-5 different pseudo-baseline levels, a behavior known as Random Telegraphic Signal, so that these are commonly called the RTS bolometers. Finally, ring-level statistics of different types (mean, median, rms, kurtosis, etc.) are determined on a per-ring basis for all timelines, and a selection based on these statistics is used to discard anomalous rings, which are recorded in a ring-level flag for each valid bolometer timeline (see the Discarded rings section).

Throughout this processing, some planet signals are masked in the timeline in order to avoid ringing effects in the processing. It concerns only Jupiter at 100 and 143 GHZ, Jupiter and Saturn at 217 GHz, Jupiter, Saturn and Mars at 353, 545 and 857 GHz (thanks to various improvements in the despiking and in the transfer functions it is no longer necessary to flag these three planets in the CMB channels as was done in Release 1). Since they move on the sky, the portion of the sky masked during one survey is observed during one, and no hole is left in the final map. In parallel, the planet data are processed in a similar way, but with different parameters for the despiking step. These results are processed separately to determine the beam shapes and the focal plane geometry.

Next, the TOIs are calibrated in astrophysical units using the results of the calibration the calibration pipeline, described in the Map-making and calibration section, which are stored in the IMO, and the best estimate of the zero-point offsets (a constant level for each bolometer) and of the far-side lobes (another signal TOI determined from the input TOI) are removed. Then the TOIs are multiplied by factors of 1.00087, 1.00046, and 1.00043, at 100-217 GHz respectively, to compensate for the non-removal of the far-side lobes.

Finally the low-ell modes are filtered out the PBS TOIs [##REF to reason for this##]. This is done by low-pass filtering a CMB map (produced by the component separation effort). This map is then scanned to produce TOIs of the low-ell modes, and these TOIs are subtracted from the TOIs produced above.

These TOIs are ready for projection onto a map. That projection normally requires that the low-frequency noise be removed via a "destriping" (sometimes called "baseline removal") step. The HFI-DPC does its destriping at ring-level; should the user want to use the DPC's offsets, they are provided separately ([##REF##]). Due to a technical reason, for a ring with a duration larger than 72 minutes, only the first 72 minutes of the ring are used to make the HFI maps which are officially delivered (the loss of data only amounts to a total of 1/2 hour for the 29 months of HFI data).


The pointing (see common Detector Pointing section for details) is determined from the AHF produced by MOC, which gives the direction and orientation of the LOS of a fiducial position in the focal plane at frequencies of 8Hz during stable pointing and 4 Hz during maneuvers. This is interpolated to the timestamps of data samples, corrected for the wobble and other time-dependent offsets determined from the observed positions of a large number of sources around the sky, and finally converted to the LOS of each detector using the quaternions in the IMO.

File Names[edit]

The file names are of the form:

{H,L}FI_TOI_{fff}-{SCI,PTG}_R2.nn_ODxxxx.fits

where

  • fff denotes the frequency
  • SCI or PTG denote signal or pointing TOIs
  • R2.nn is the version, and
  • ODxxxx indicates the OD.

FITS file structure[edit]

All FITS files begin with a primary extension containing no data and a minimal header, which is followed by one or more BINTABLE extensions with data, and with a description of the data in the header keywords. In what follows will usually ignore the primary extension, and count only the extensions containing data.

TOI files[edit]

The signal FITS files contain N ,'BINTABLE', data extensions, where N is stye number of detectors in that frequency channel. The first extension contains the OBT and its flags, called the global flags since they apply to all detectors, which is followed by one extension for each detector, containing the signal and its local flag. The flags columns are written as byte in which each of the 8 bits (max) encodes one flag timeline. The meaning of each bit is given in the header comments and is subject to change. For the global flag they include:

  • Unstable pointing: 1= pointing is not stable (e.g., during repointing maneuvers)
  • Dark correlation: 1 = darks are uncorrelated and data are flagged
  • First/Second half ring: which samples are in which half (only covers the stable pointing part of the ring)
  • HCM: in HCM mode (unstable pointing)
  • and more

And for the local flag they include

  • Total Flag: a combination of the various flags that is the one finally used in the map-making (all samples with Total Flag different from zero should not be used)
  • Data not valid: glitched samples
  • Despike Common: (for PSBs only) glitch on current or other of PSB pair
  • StrongSignal: on Galactic Plane
  • Strong Source: on point source
  • other


TOI file data structure
1. EXTNAME = 'OBT' : Data columns
Column Name Data Type Units Description
OBT Int*8 2-16 sec On-board time
FLAG Byte none the various bit-level flags
Keyword Data Type Value Description
OD Int OD covered (as in filename)
BEGIDX Int first sample index of given OD
ENDIDX Int last sample index of given OD
BEGRING Int first ring in given OD
ENDRING Int last ring in given OD
TIMEZERO String 1958-01-01z00:00 Origin of OBT
n. EXTNAME = DETNAME : Data columns
Column Name Data Type Units Description
SIGNAL Real*4 Kcmb or MJy/sr Value of signal
FLAG Byte none the various bit-level flags
Keyword Data Type Value Description
UNIT String Units of signal
DESTRIPE 1/0 whether timeline is destriped
OD Int OD covered (as in filename)
BEGIDX Int first sample index of given OD
ENDIDX Int last sample index of given OD
BEGRING Int first ring in given OD
ENDRING Int last ring in given OD


The pointing files have a similar structure, except that DETNAME extensions contain 3 columns of Real*8 with the phi, theta, and psi Galactic spherical coordinates of each sample in radians. There is no local flag for the coordinates.

ROI files[edit]

The files provided by HFI are

  • HFI_ROI_GlobalParams_RelNum_full.fits
  • HFI_ROI_DestripeOffsets_RelNum_full.fits

which are described below.

Global parameters
this ROI file contains a single BINTABLE extension with 3 columns that give, for each ring, the index number of the first sample, the ESA pointing ID of the ring, and the ring start time. All three are encoded as Int*8, and the time is given in units of 2-16 sec from TIMEZERO, 1958-01-01z00:00, as in the TOI files. Rings are numbered from 240 to 27005, as given in the BEGRING and ENDRING keywords
Destriping offsets
this ROI file contains a single BINTABLE extension with 3 columns that give, for each ring, the offsets to subtract from each bolometer signal timeline or vector. There are three offsets available for each bolometer: full-ring, first half-ring, and second half-ring, for building the full-ring or the two half-ring maps, respectively. The offsets must be subtracted from Index(N) to Index(N+1)-1 of the corresponding signal timeline, where N is stye ring number, and the indices are given in the Global parameters file. The offsets are given in the same units as the signal vectors, that is Kcmv for the 100-353 GHz channels, and MJy/sr for the 545 and 857 GHz channels. All values are encoded as Real*4. The rings for which the offsets are set to 0.000 are badrings which are rejected in the mapmaking process.

TBW: Note on half-ring offsets, which differ by 1-2% from full ring offsets

(Planck) High Frequency Instrument

(Planck) Low Frequency Instrument

Operation Day definition is geometric visibility driven as it runs from the start of a DTCP (satellite Acquisition Of Signal) to the start of the next DTCP. Given the different ground stations and spacecraft will takes which station for how long, the OD duration varies but it is basically once a day.

On-Board Time

Data Processing Center

Ring-Ordered Information (DMC group/object)

Flexible Image Transfer Specification

analog to digital converter

sudden change of the baseline level inside a ring

random telegraphic signal

Cosmic Microwave background

Attitude History File

[ESA's] Mission Operation Center [Darmstadt, Germany]

Line Of Sight

Angular momentum Control Mode

European Space Agency