Using Index Tables

HATS catalogs are partitioned spatially, typically on right ascension and declination. This makes finding objects in a particular area of the sky very straightforward. However, you may occassionally only know the object by the survey-assigned identifier.

HATS supports creating additional secondary index tables, stored separately from the HATS catalog. You can find more information about creating these index tables in the hats-import documentation, or request that your archive provider create an index table for your use.

In this notebook, we explore using index tables to find GAIA objects, if you know their designation.

First, let’s create some list of designations that we’re interested in. The ones here have no special signficance.

ids = [
    6350084614282952448,
    4684296460655809408,
    4684296460655681280,
    6350084408124522368,
    6379030430412267264,
    6397962092201358080,
    999999988604363776,
]

import hats

catalog_index = hats.read_hats("/data3/epyc/data3/hats/catalogs/gaia_dr3/gaia_source_id_index/")
catalog_index.schema

Norder: uint8
Dir: uint64
Npix: uint64
source_id: int64
-- schema metadata --
pandas: '{"index_columns": ["source_id"], "column_indexes": [{"name": nul' + 641

This small set of GAIA designations are in just a handful of partitions. Let’s check the index catalog, and see if we can reduce this list of IDs into just those pixels of interest.

We’re still only using the HATS api, so the reading is single-threaded.

%%time
partitions = catalog_index.loc_partitions(ids)
partitions

CPU times: user 2min 51s, sys: 1min 41s, total: 4min 32s
Wall time: 33.2 s

array([Order: 3, Pixel: 111, Order: 5, Pixel: 8320, Order: 3, Pixel: 705,
       Order: 3, Pixel: 708], dtype=object)

Faster selection with LSDB

Now let’s try the search and selection with LSDB. This will use dask to fetch the full rows of these objects.

%%time
import lsdb

gaia = lsdb.read_hats("/data3/epyc/data3/hats/catalogs/gaia_dr3/gaia")

CPU times: user 769 ms, sys: 54.8 ms, total: 824 ms
Wall time: 1 s

%%time
gaia.index_search(ids, catalog_index).compute()

CPU times: user 2min 53s, sys: 1min 58s, total: 4min 51s
Wall time: 24.5 s

	solution_id	designation	source_id	random_index	ref_epoch	ra	ra_error	dec	dec_error	parallax	...	ag_gspphot	ag_gspphot_lower	ag_gspphot_upper	ebpminrp_gspphot	ebpminrp_gspphot_lower	ebpminrp_gspphot_upper	libname_gspphot	Norder	Dir	Npix
_healpix_29
500000002326108747	1636148068921376768	Gaia DR3 999999988604363776	999999988604363776	1146519553	2016.0	104.910499	0.416448	55.969686	0.392039	1.501113	...	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	3	0	111
2342148232994970763	1636148068921376768	Gaia DR3 4684296460655681280	4684296460655681280	935088391	2016.0	7.756788	0.075319	-75.621828	0.069098	0.249487	...	0.005	0.001	0.013	0.0027	0.0005	0.0071	MARCS	5	0	8320
2342148241066740563	1636148068921376768	Gaia DR3 4684296460655809408	4684296460655809408	930596526	2016.0	7.74122	3.736654	-75.619753	3.646013	<NA>	...	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	5	0	8320
3175042221187998424	1636148068921376768	Gaia DR3 6350084408124522368	6350084408124522368	64296681	2016.0	359.838284	0.249825	-83.92197	0.239047	0.656007	...	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	3	0	705
3175042313799520523	1636148068921376768	Gaia DR3 6350084614282952448	6350084614282952448	1750910513	2016.0	359.435728	0.510138	-83.921882	0.492162	-0.272154	...	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	3	0	705
3189515220382416186	1636148068921376768	Gaia DR3 6379030430412267264	6379030430412267264	1257926240	2016.0	348.261377	0.101749	-73.622147	0.093109	0.199036	...	0.005	0.0012	0.0129	0.0027	0.0006	0.007	MARCS	3	0	708
3192976252233877374	1636148068921376768	Gaia DR3 6397962092201358080	6397962092201358080	150735067	2016.0	336.476393	0.079348	-67.482415	0.09224	0.239489	...	0.0021	0.0004	0.0073	0.0011	0.0002	0.0039	MARCS	3	0	708

7 rows × 155 columns

_ = gaia.index_search(ids, catalog_index).plot_pixels(plot_title="Pixels containing objects of interest")