About the EDP2 HATS Catalogs

About the EDP2 HATS Catalogs#

This page describes the HATS distribution of Rubin’s Early Data Preview 2 (EDP2) — the early release of the Rubin Data Preview 2 (DP2) data products, repackaged into the HATS format for use with LSDB. In addition to mirroring the Rubin Data Release catalogs, the HATS distribution adds a number of value-added columns intended to make common science workflow easier.

More information is available from the Observatory at dp2.lsst.io.

What’s in the release#

The EDP2 HATS distribution provides three catalogs, all of which can be opened with LSDB:

  • object_collection — the DRP Object table with its nested objectForcedSource light curves.

  • dia_object_collection — the DRP DiaObject table with its nested diaSource and diaObjectForcedSource light curves.

  • object_photoz — photometric-redshift estimates for EDP2 objects (see object_photoz below).

The build also publishes an auxiliary visit–detector metadata table under public-files/ (see Visit–detector metadata table below).

These catalogs are available to Rubin data-rights holders. For step-by-step instructions on accesing them see the Accessing Rubin Data Preview 2 tutorial.

Light curves are nested within their objects#

Each catalog is a single table with one row per object, and every object carries its own light curve as a nested column — the per-epoch measurements are stored inline with the parent object rather than in a separate source table that you would have to join yourself. LSDB loads these as a NestedFrame (backed by nested-pandas).

  • object_collection — each Object row (indexed by objectId) has a nested objectForcedSource column holding its forced-photometry light curve.

  • dia_object_collection — each DiaObject row (indexed by diaObjectId) has nested diaSource and diaObjectForcedSource columns.

Under the hood a nested column is a struct of lists: each sub-column (for example psfFlux, psfMag, band, midpointMjdTai) is a list with one entry per epoch, all aligned to the same parent row. You can keep the light curves nested and operate on them in place, select an individual object’s light curve, or flatten a nested column back into a flat per-epoch source table — see Exploding Lightcurves to flat Source Tables.

How the HATS catalogs differ from the standard catalogs#

Relative to the standard Rubin data products, the HATS object_collection and dia_object_collection catalogs add derived columns and downcast a subset of columns. The changes are summarized below.

Warning

Do not use the downcast position columns where full precision matters. This applies to all the per-band and per-model RA/Dec columns: {band}_ra / {band}_dec for the ugrizy bands, sersic_ra / sersic_dec, and exponential_ra / exponential_dec are all stored as float32. The float32 rounding error can be larger than the corresponding *_raErr / *_decErr values, so these columns are unsuitable for precision astrometry or tight positional cross-matching. Use the full-precision object position (coord_ra / coord_dec in object_collection, ra / dec in dia_object_collection, both kept as float64) for those tasks.

Warning

The per-band epoch columns are downsampled in time. Storing an MJD as float32 puts it on a grid roughly 5 minutes wide, so a {band}_epoch value can differ from the true observation time. The same applies to diaSource.timeProcessedMjdTai. These columns are not suitable for timing work. Use midpointMjdTai on the nested light curves (objectForcedSource, diaObjectForcedSource, and diaSource), which is kept as float64.

Changes common to both collections#

Magnitude and magnitude-error columns. Magnitude and magnitude-error columns converted from the flux columns are added. The magnitude error is defined as half the difference between the magnitudes corresponding to Flux + FluxErr and Flux - FluxErr. The full list of magnitude columns added for each collection is given in the per-collection sections below.

Corrected flux-error subcolumns. Four subcolumns are added to the nested forced-source columns (objectForcedSource and diaObjectForcedSource):

  • psfFluxErr_corrected

  • psfDiffFluxErr_corrected

  • psfFluxErr_corrected_flag

  • psfDiffFluxErr_corrected_flag

These are produced by a model trained to correct the flux errors of non-variable objects so that their reduced χ² is close to unity. The corrected values may be useful for light-curve fitting. The *_flag columns are True when the ratio between the original and corrected error falls outside the [0.1, 50] interval. The method is described in Malanchev et al. (in prep); the source code is available in the uncle-val repository.

Corrected magnitude error. A psfMagErr_corrected subcolumn is added to the nested forced-source columns, derived from psfFluxErr_corrected in the same way the magnitude errors are derived from the original flux-error columns.

Visit timestamp. A midpointMjdTai column is added to both objectForcedSource and diaObjectForcedSource, joined from the visit table for the corresponding visit in the original catalog. That visit table is itself published with the release; see the visit–detector table below.

Downcasting. Some float64 columns are downcast to float32 (see the warning above and the per-collection lists below).

object_collection#

Magnitude columns added:

  • Object (base level): {band}_psfMag, {band}_kronMag, and {band}_cModelMag for the ugrizy bands, plus the corresponding error columns.

  • objectForcedSource: psfMag plus the corresponding error column.

Downcast columns (float64 → float32):

  • Base level (22 columns):

    exponential_dec  exponential_ra
g_dec  g_epoch  g_ra
i_dec  i_epoch  i_ra
r_dec  r_epoch  r_ra
sersic_dec  sersic_ra
u_dec  u_epoch  u_ra
y_dec  y_epoch  y_ra
z_dec  z_epoch  z_ra

  • objectForcedSource: none.

All of the position columns listed above ({band}_ra / {band}_dec, sersic_ra / sersic_dec, exponential_ra / exponential_dec) and the {band}_epoch columns are downcast from float64 to float32; see the warnings above regarding the resulting precision loss. coord_ra / coord_dec are not downcast.

dia_object_collection#

Magnitude columns added:

  • diaSource: psfMag and scienceMag, plus the corresponding error columns.

  • diaObjectForcedSource: psfMag plus the corresponding error column.

diaSource rows whose diaObjectId does not appear in the DiaObject table are not included in the nested light curves: 8,136,150 of 1,000,825,975 rows (0.81%).

DiaSources that were assigned to Solar System objects are among those not carried over — they are associated with an SSObject rather than a DiaObject, so they will not be found anywhere in dia_object_collection. Do not treat this collection as a complete inventory of DiaSources.

Downcast columns (float64 → float32):

  • Base level (48 columns):

    g_psfFluxMax  g_psfFluxMaxSlope  g_psfFluxMean  g_psfFluxMeanErr  g_psfFluxMin  g_psfFluxSigma  g_scienceFluxMean  g_scienceFluxMeanErr
i_psfFluxMax  i_psfFluxMaxSlope  i_psfFluxMean  i_psfFluxMeanErr  i_psfFluxMin  i_psfFluxSigma  i_scienceFluxMean  i_scienceFluxMeanErr
r_psfFluxMax  r_psfFluxMaxSlope  r_psfFluxMean  r_psfFluxMeanErr  r_psfFluxMin  r_psfFluxSigma  r_scienceFluxMean  r_scienceFluxMeanErr
u_psfFluxMax  u_psfFluxMaxSlope  u_psfFluxMean  u_psfFluxMeanErr  u_psfFluxMin  u_psfFluxSigma  u_scienceFluxMean  u_scienceFluxMeanErr
y_psfFluxMax  y_psfFluxMaxSlope  y_psfFluxMean  y_psfFluxMeanErr  y_psfFluxMin  y_psfFluxSigma  y_scienceFluxMean  y_scienceFluxMeanErr
z_psfFluxMax  z_psfFluxMaxSlope  z_psfFluxMean  z_psfFluxMeanErr  z_psfFluxMin  z_psfFluxSigma  z_scienceFluxMean  z_scienceFluxMeanErr

  • diaSource (20 columns):

    dipoleAngle  dipoleChi2  dipoleFluxDiff  dipoleFluxDiffErr  dipoleLength
dipoleMeanFlux  dipoleMeanFluxErr  extendedness
ixx  ixxPSF  ixy  ixyPSF  iyy  iyyPSF
snr  timeProcessedMjdTai  trailAngle  trailDec  trailLength  trailRa

  • diaObjectForcedSource: none.

object_photoz#

The object_photoz catalog provides photometric-redshift estimates for EDP2 objects. A full description of the photo-z estimator(s), output columns, and caveats will be provided in the RTN-124 technical note.

Visit–detector metadata table#

Alongside the three catalogs, the EDP2 build publishes an auxiliary public-files/visit_detector.parquet table — one row per (visit, detector), 5,150,198 rows and 59 columns. This is the visit-level metadata table from which the midpointMjdTai column on the forced-source light curves is joined. It is a plain Parquet file (not a HATS catalog) and can be loaded directly with, for example, pandas or pyarrow for additional per-visit and per-detector information: observation-time MJDs, PSF and image-quality metrics, photometric zero points, sky background and noise, effective-time metrics, and WCS residuals.

It is keyed by visitId and detectorId (also available as ccdVisitId).

How to cite#

If you use LSDB or these catalogs in published research, please follow the citation instructions on the citation page. If you use the corrected flux-error columns, please also cite Malanchev et al. (in prep).

Caveats#

  • Position precision. Every per-band and per-model RA/Dec column ({band}_ra / {band}_dec, sersic_ra / sersic_dec, exponential_ra / exponential_dec) is downcast to float32 and can carry a rounding error larger than its reported error (see the warning above). coord_ra / coord_dec retain full float64 precision.

  • Time precision. {band}_epoch and diaSource.timeProcessedMjdTai are downcast to float32, which downsamples them onto a grid roughly 5 minutes wide. Use the float64 midpointMjdTai column on the nested light curves for any time-domain work.

  • Unassociated DIA sources. 0.81% of diaSource rows (8,136,150 of 1,000,825,975) are dropped from the nested light curves because their diaObjectId has no match in the DiaObject table.

  • Solar System objects. DiaSources assigned to Solar System objects are associated with an SSObject rather than a DiaObject, so they are not present in dia_object_collection.

How to ask for help#

There are several places to get help, depending on the nature of your question. For the full list of channels, events, and office hours, see the Contact Us / Getting Help page.

Questions about LSDB / HATS tooling (loading the catalogs, cross-matching, performance):

  • Open an issue on the LSDB GitHub repository.

  • Ask in the LSST Discovery Alliance Slack, channel #lincc-frameworks-lsdb.

  • Drop in to the weekly “HATS/LSDB Drop-in” co-working / office hours (Fridays, 10:00–11:00 AM US Pacific Time).

Questions about the Rubin data / science (the underlying DP2 measurements, data rights, pipelines):

Future plans#

  • Additional catalogs. We plan to add HATS distributions of the ShearObject and IsolatedStarStellarMotions tables.

  • Source table. The Source table has been imported to HATS, but there are currently no immediate plans to serve it due to its size.

  • Storage migration. The data currently hosted on the US Rubin Science Platform (RSP) will be moved to a Google Cloud Storage bucket. This will change the access path (the catalog URLs), but should not impact read performance.