Lazy Operations in LSDB

Lazy Operations in LSDB#

In this tutorial, we will:

  • explore what lazy operations are and how LSDB uses them to run pipelines at scale

  • preview a small part of the data

Introduction - What are Lazy Operations?#

In the previous tutorial we looked at loading a catalog and inspecting it’s metadata. When we call open_catalog(), only the catalog’s metadata is loaded, not any of the data in the rows of the catalog. This is because operations in LSDB are lazy: when you call the operation it isn’t actually executed immediately.

As explained in the video above, instead of executing the operation when you call it, the operation is just planned. The catalog object keeps track of the pipeline of operations by building a task graph - an object that keeps track of the pipeline of operations you want to perform on the catalog. This way, you can write the code for the pipeline locally, and the task graph will be sent to the workers to execute the pipeline in parallel. We can also perform optimizations to the task graph to make sure the workflow is as efficient as possible. This is how LSDB can scale from working on your local machine, to running pipelines on clusters or in the cloud without having to make any code changes.

To actually execute the operations, you call the catalog.compute() method, which will execute the pipeline and return the resulting data as a pandas DataFrame.

You will find that most use cases start with LAZY loading and planning operations, followed by more expensive COMPUTE operations. The data is only loaded into memory when we trigger the workflow computations, usually with a compute call.

Lazy workflow diagram

[1]:

import lsdb

[2]:

gaia = lsdb.open_catalog("s3://stpubdata/gaia/gaia_dr3/public/hats")
gaia

[2]:
lsdb Catalog gaia:
solution_id designation source_id random_index ref_epoch ra ra_error dec dec_error parallax parallax_error parallax_over_error pm pmra pmra_error pmdec pmdec_error ra_dec_corr ra_parallax_corr ra_pmra_corr ra_pmdec_corr dec_parallax_corr dec_pmra_corr dec_pmdec_corr parallax_pmra_corr parallax_pmdec_corr pmra_pmdec_corr astrometric_n_obs_al astrometric_n_obs_ac astrometric_n_good_obs_al astrometric_n_bad_obs_al astrometric_gof_al astrometric_chi2_al astrometric_excess_noise astrometric_excess_noise_sig astrometric_params_solved astrometric_primary_flag nu_eff_used_in_astrometry pseudocolour pseudocolour_error ra_pseudocolour_corr dec_pseudocolour_corr parallax_pseudocolour_corr pmra_pseudocolour_corr pmdec_pseudocolour_corr astrometric_matched_transits visibility_periods_used astrometric_sigma5d_max matched_transits new_matched_transits matched_transits_removed ipd_gof_harmonic_amplitude ipd_gof_harmonic_phase ipd_frac_multi_peak ipd_frac_odd_win ruwe scan_direction_strength_k1 scan_direction_strength_k2 scan_direction_strength_k3 scan_direction_strength_k4 scan_direction_mean_k1 scan_direction_mean_k2 scan_direction_mean_k3 scan_direction_mean_k4 duplicated_source phot_g_n_obs phot_g_mean_flux phot_g_mean_flux_error phot_g_mean_flux_over_error phot_g_mean_mag phot_bp_n_obs phot_bp_mean_flux phot_bp_mean_flux_error phot_bp_mean_flux_over_error phot_bp_mean_mag phot_rp_n_obs phot_rp_mean_flux phot_rp_mean_flux_error phot_rp_mean_flux_over_error phot_rp_mean_mag phot_bp_rp_excess_factor phot_bp_n_contaminated_transits phot_bp_n_blended_transits phot_rp_n_contaminated_transits phot_rp_n_blended_transits phot_proc_mode bp_rp bp_g g_rp radial_velocity radial_velocity_error rv_method_used rv_nb_transits rv_nb_deblended_transits rv_visibility_periods_used rv_expected_sig_to_noise rv_renormalised_gof rv_chisq_pvalue rv_time_duration rv_amplitude_robust rv_template_teff rv_template_logg rv_template_fe_h rv_atm_param_origin vbroad vbroad_error vbroad_nb_transits grvs_mag grvs_mag_error grvs_mag_nb_transits rvs_spec_sig_to_noise phot_variable_flag l b ecl_lon ecl_lat in_qso_candidates in_galaxy_candidates non_single_star has_xp_continuous has_xp_sampled has_rvs has_epoch_photometry has_epoch_rv has_mcmc_gspphot has_mcmc_msc in_andromeda_survey classprob_dsc_combmod_quasar classprob_dsc_combmod_galaxy classprob_dsc_combmod_star teff_gspphot teff_gspphot_lower teff_gspphot_upper logg_gspphot logg_gspphot_lower logg_gspphot_upper mh_gspphot mh_gspphot_lower mh_gspphot_upper distance_gspphot distance_gspphot_lower distance_gspphot_upper azero_gspphot azero_gspphot_lower azero_gspphot_upper ag_gspphot ag_gspphot_lower ag_gspphot_upper ebpminrp_gspphot ebpminrp_gspphot_lower ebpminrp_gspphot_upper libname_gspphot
npartitions=2016
Order: 2, Pixel: 0 int64[pyarrow] string[pyarrow] int64[pyarrow] int64[pyarrow] double[pyarrow] double[pyarrow] float[pyarrow] double[pyarrow] float[pyarrow] double[pyarrow] float[pyarrow] float[pyarrow] float[pyarrow] double[pyarrow] float[pyarrow] double[pyarrow] float[pyarrow] float[pyarrow] float[pyarrow] float[pyarrow] float[pyarrow] float[pyarrow] float[pyarrow] float[pyarrow] float[pyarrow] float[pyarrow] float[pyarrow] int16[pyarrow] int16[pyarrow] int16[pyarrow] int16[pyarrow] float[pyarrow] float[pyarrow] float[pyarrow] float[pyarrow] int8[pyarrow] bool[pyarrow] float[pyarrow] float[pyarrow] float[pyarrow] float[pyarrow] float[pyarrow] float[pyarrow] float[pyarrow] float[pyarrow] int16[pyarrow] int16[pyarrow] float[pyarrow] int16[pyarrow] int16[pyarrow] int16[pyarrow] float[pyarrow] float[pyarrow] int8[pyarrow] int8[pyarrow] float[pyarrow] float[pyarrow] float[pyarrow] float[pyarrow] float[pyarrow] float[pyarrow] float[pyarrow] float[pyarrow] float[pyarrow] bool[pyarrow] int16[pyarrow] double[pyarrow] float[pyarrow] float[pyarrow] float[pyarrow] int16[pyarrow] double[pyarrow] float[pyarrow] float[pyarrow] float[pyarrow] int16[pyarrow] double[pyarrow] float[pyarrow] float[pyarrow] float[pyarrow] float[pyarrow] int16[pyarrow] int16[pyarrow] int16[pyarrow] int16[pyarrow] int8[pyarrow] float[pyarrow] float[pyarrow] float[pyarrow] float[pyarrow] float[pyarrow] int8[pyarrow] int16[pyarrow] int16[pyarrow] int16[pyarrow] float[pyarrow] float[pyarrow] float[pyarrow] float[pyarrow] float[pyarrow] float[pyarrow] float[pyarrow] float[pyarrow] int16[pyarrow] float[pyarrow] float[pyarrow] int16[pyarrow] float[pyarrow] float[pyarrow] int16[pyarrow] float[pyarrow] string[pyarrow] double[pyarrow] double[pyarrow] double[pyarrow] double[pyarrow] bool[pyarrow] bool[pyarrow] int16[pyarrow] bool[pyarrow] bool[pyarrow] bool[pyarrow] bool[pyarrow] bool[pyarrow] bool[pyarrow] bool[pyarrow] bool[pyarrow] float[pyarrow] float[pyarrow] float[pyarrow] float[pyarrow] float[pyarrow] float[pyarrow] float[pyarrow] float[pyarrow] float[pyarrow] float[pyarrow] float[pyarrow] float[pyarrow] float[pyarrow] float[pyarrow] float[pyarrow] float[pyarrow] float[pyarrow] float[pyarrow] float[pyarrow] float[pyarrow] float[pyarrow] float[pyarrow] float[pyarrow] float[pyarrow] string[pyarrow]
Order: 2, Pixel: 1 ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
Order: 3, Pixel: 766 ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
Order: 3, Pixel: 767 ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
152 out of 152 available columns in the catalog have been loaded lazily, meaning no data has been read, only the catalog schema
This catalog has an estimated size of 765.3 GB

We can see above from the ... as placeholders for the data, and the warning at the bottom that this catalog has been loaded lazily. Now that we have the object we’re ready to start performing operations!

Operating on the Catalog#

Once we have a catalog object, we can start planning operations on it. In the rest of the tutorials, we’ll look deeper into exactly what kind of operations you can do with a catalog. The catalog is based on pandas DataFrames so you’ll see some functions that work the same as in pandas, such as columns, dtypes, query, and selecting columns or filtering with [].

There are also a lot of astronomy specific functions, such as Spatial filters like cone search or box search, and Crossmatching.

After you’ve performed your operations, you can call catalog.compute() to perform the pipeline, but this will run on the entire catalog!

Previewing part of the data#

Computing an entire catalog will result in loading all of its data into memory on your local machine after the workers have computed it, which is expensive and may lead to out-of-memory issues.

Often, our goal is to have a peek at a slice of data to make sure the workflow output is reasonable (e.g., to assess if some new created columns are present and their values have been properly processed). head() is a pandas-like method which allows us to preview part of the data for this purpose. It runs the pipeline on the catalog partitions one by one, and finds the first few rows of the results.

Making a Dask client#

LSDB is built on top of the Dask framework, which allows the pipelines to be executed on distributed workers. Before we do anything that executes the pipeline such as head() or compute(), we recommend making a dask client.

For additional information on dask client creation, please read our tutorial on Setting up a Dask Client.

For now, we’ll make a simple Client that uses 4 workers on our local machine.

[3]:

from dask.distributed import Client

client = Client(n_workers=4, memory_limit="auto")
client

[3]:

Client

Client-ec2bf232-7be3-11f1-88a8-aa5817dd4ea4

Connection method: Cluster object Cluster type: distributed.LocalCluster
Dashboard: http://127.0.0.1:8787/status

Cluster Info

 
[4]:

gaia.head()

2026-07-09 22:17:26,991 - distributed.worker.memory - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker-memory.html#memory-not-released-back-to-the-os for more information. -- Unmanaged memory: 2.00 GiB -- Worker memory limit: 3.27 GiB

[4]:
solution_id designation source_id random_index ref_epoch ra ra_error dec dec_error parallax parallax_error parallax_over_error pm pmra pmra_error pmdec pmdec_error ra_dec_corr ra_parallax_corr ra_pmra_corr ra_pmdec_corr dec_parallax_corr dec_pmra_corr dec_pmdec_corr parallax_pmra_corr parallax_pmdec_corr pmra_pmdec_corr astrometric_n_obs_al astrometric_n_obs_ac astrometric_n_good_obs_al astrometric_n_bad_obs_al astrometric_gof_al astrometric_chi2_al astrometric_excess_noise astrometric_excess_noise_sig astrometric_params_solved astrometric_primary_flag nu_eff_used_in_astrometry pseudocolour pseudocolour_error ra_pseudocolour_corr dec_pseudocolour_corr parallax_pseudocolour_corr pmra_pseudocolour_corr pmdec_pseudocolour_corr astrometric_matched_transits visibility_periods_used astrometric_sigma5d_max matched_transits new_matched_transits matched_transits_removed ipd_gof_harmonic_amplitude ipd_gof_harmonic_phase ipd_frac_multi_peak ipd_frac_odd_win ruwe scan_direction_strength_k1 scan_direction_strength_k2 scan_direction_strength_k3 scan_direction_strength_k4 scan_direction_mean_k1 scan_direction_mean_k2 scan_direction_mean_k3 scan_direction_mean_k4 duplicated_source phot_g_n_obs phot_g_mean_flux phot_g_mean_flux_error phot_g_mean_flux_over_error phot_g_mean_mag phot_bp_n_obs phot_bp_mean_flux phot_bp_mean_flux_error phot_bp_mean_flux_over_error phot_bp_mean_mag phot_rp_n_obs phot_rp_mean_flux phot_rp_mean_flux_error phot_rp_mean_flux_over_error phot_rp_mean_mag phot_bp_rp_excess_factor phot_bp_n_contaminated_transits phot_bp_n_blended_transits phot_rp_n_contaminated_transits phot_rp_n_blended_transits phot_proc_mode bp_rp bp_g g_rp radial_velocity radial_velocity_error rv_method_used rv_nb_transits rv_nb_deblended_transits rv_visibility_periods_used rv_expected_sig_to_noise rv_renormalised_gof rv_chisq_pvalue rv_time_duration rv_amplitude_robust rv_template_teff rv_template_logg rv_template_fe_h rv_atm_param_origin vbroad vbroad_error vbroad_nb_transits grvs_mag grvs_mag_error grvs_mag_nb_transits rvs_spec_sig_to_noise phot_variable_flag l b ecl_lon ecl_lat in_qso_candidates in_galaxy_candidates non_single_star has_xp_continuous has_xp_sampled has_rvs has_epoch_photometry has_epoch_rv has_mcmc_gspphot has_mcmc_msc in_andromeda_survey classprob_dsc_combmod_quasar classprob_dsc_combmod_galaxy classprob_dsc_combmod_star teff_gspphot teff_gspphot_lower teff_gspphot_upper logg_gspphot logg_gspphot_lower logg_gspphot_upper mh_gspphot mh_gspphot_lower mh_gspphot_upper distance_gspphot distance_gspphot_lower distance_gspphot_upper azero_gspphot azero_gspphot_lower azero_gspphot_upper ag_gspphot ag_gspphot_lower ag_gspphot_upper ebpminrp_gspphot ebpminrp_gspphot_lower ebpminrp_gspphot_upper libname_gspphot
_healpix_29
3121269471 1636148068921376768 Gaia DR3 4295806720 4295806720 545300884 2016.0 44.996155 0.101618 0.005615 0.101334 0.354331 0.122664 2.888632 12.616485 11.938352 0.137942 -4.080619 0.13317 0.122935 0.132028 -0.08891 0.022551 -0.365342 -0.036904 -0.244838 0.063012 0.135709 0.334337 184 0 183 1 2.672049 242.20697 0.380619 2.076508 31 False 1.508939 <NA> <NA> <NA> <NA> <NA> <NA> <NA> 22 16 0.217802 22 9 0 0.017597 90.239342 0 0 1.142952 0.307951 0.197659 0.430101 0.842078 -87.754784 -30.69455 -46.201908 30.174356 False 182 1653.394716 2.075764 796.523376 17.641426 18 800.429546 12.601409 63.519051 18.080235 20 1187.588004 15.823832 75.050598 17.061232 1.202385 0 0 0 2 0 1.019003 0.438808 0.580194 <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> NOT_AVAILABLE 176.951076 -48.901521 42.533726 -16.329574 False False 0 True False False False False True True False 0.0 0.0 0.999988 5052.976074 5040.658203 5066.628418 4.7793 4.7613 4.7924 -1.4592 -1.5732 -1.3439 1497.054688 1466.734619 1536.13208 0.0064 0.0016 0.0176 0.0052 0.0013 0.0143 0.0028 0.0007 0.0078 MARCS
29153808338 1636148068921376768 Gaia DR3 38655544960 38655544960 1757259052 2016.0 45.004978 0.017885 0.01988 0.018772 3.13917 0.022347 140.471313 35.308208 29.686339 0.023772 19.1152 0.02383 0.115263 0.073231 -0.106919 -0.030214 -0.448866 -0.155514 -0.379279 0.181846 0.26367 0.355281 183 0 182 1 0.264343 181.438461 0.0 0.0 31 False 1.455016 <NA> <NA> <NA> <NA> <NA> <NA> <NA> 21 15 0.039295 21 9 0 0.024302 98.629005 0 0 1.012191 0.306561 0.205788 0.452994 0.846566 -96.318893 -34.497215 -44.825779 30.34742 False 180 42030.600439 11.392837 3689.212891 14.128453 20 17955.479377 26.03932 689.552551 14.70305 19 34263.48754 36.75135 932.305542 13.410816 1.242404 0 3 0 2 0 1.292233 0.574596 0.717637 41.187176 3.113034 2 10 1 8 7.034563 <NA> <NA> 749.919922 <NA> 4500.0 3.0 -0.25 111 <NA> <NA> <NA> 13.068616 0.049816 10 <NA> NOT_AVAILABLE 176.944762 -48.88527 42.546872 -16.318522 False False 0 True True False False False True True False 0.0 0.0 0.999806 4708.794434 4659.062012 4723.277344 4.5588 4.5261 4.5654 -0.087 -0.1218 -0.0681 332.832214 330.470886 347.172913 0.2345 0.184 0.2516 0.182 0.1425 0.1955 0.0961 0.0752 0.1032 MARCS
29640498453 1636148068921376768 Gaia DR3 34361129088 34361129088 894504938 2016.0 45.00432 0.09732 0.021048 0.101753 3.235017 0.12045 26.857704 35.230515 29.518344 0.133693 19.231655 0.133922 0.163253 0.000643 -0.073663 -0.012017 -0.403893 -0.101522 -0.315934 0.14065 0.231426 0.381754 172 0 171 1 1.081467 194.599335 0.267413 1.032802 31 False 1.285487 <NA> <NA> <NA> <NA> <NA> <NA> <NA> 20 15 0.220539 20 9 0 0.057373 84.542816 0 0 1.05789 0.284316 0.182422 0.42349 0.848356 -101.856606 -31.586445 -44.381237 29.909302 False 170 1763.191387 2.121236 831.209595 17.571619 18 389.997136 9.491409 41.089485 18.860889 19 2178.214858 15.074686 144.494873 16.402643 1.45657 0 2 0 1 0 2.458246 1.28927 1.168976 <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> NOT_AVAILABLE 176.942789 -48.884934 42.546573 -16.317212 False False 0 False False False False False True True False 0.0 0.0 0.999389 3478.540771 3461.147461 3497.578369 4.7 4.6405 4.7734 -0.6143 -0.7064 -0.4964 302.234711 292.532501 312.637299 0.7643 0.7292 0.7975 0.505 0.4815 0.5273 0.3096 0.2956 0.3228 MARCS
162211426005 1636148068921376768 Gaia DR3 309238066432 309238066432 327609758 2016.0 44.995037 0.322039 0.038152 0.283505 1.38315 0.367907 3.759512 1.472751 0.710339 0.427646 -1.290122 0.364729 0.031063 0.200474 0.053647 0.059748 -0.22102 -0.016839 -0.13663 0.179964 0.052951 0.211478 191 0 191 0 -0.241182 192.304047 0.0 0.0 95 False <NA> 1.286628 0.090775 -0.157192 0.010362 -0.082226 -0.106753 -0.088081 22 16 0.619914 22 9 0 0.035551 87.96405 0 0 0.985694 0.269229 0.189581 0.319531 0.846794 -67.001999 -0.060676 -50.141193 28.862438 False 191 272.914296 1.123318 242.953751 19.597301 15 70.174394 8.844292 7.934428 20.723095 15 353.699104 9.905063 35.70892 18.37631 1.553138 0 0 0 1 0 2.346785 1.125793 1.220991 <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> NOT_AVAILABLE 176.914265 -48.879747 42.542548 -16.298139 False False 0 False False False False False False False False 0.0 0.0 0.99998 <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA>
187874219371 1636148068921376768 Gaia DR3 343597448960 343597448960 1791600903 2016.0 44.963896 0.117176 0.043595 0.109004 0.196148 0.133464 1.469662 6.845276 6.567299 0.15458 -1.930905 0.136051 0.090632 0.164648 -0.104231 -0.006402 -0.171425 -0.023352 -0.235325 0.071847 0.014762 0.158318 200 0 198 2 -0.380676 193.063904 0.0 0.0 31 False 1.459356 <NA> <NA> <NA> <NA> <NA> <NA> <NA> 23 15 0.221527 23 10 0 0.032448 87.084869 0 0 0.978971 0.26817 0.140934 0.26649 0.876896 -59.657413 -17.643497 -47.965027 30.37381 False 190 1049.382036 1.745514 601.187927 18.135033 19 442.641353 11.305161 39.153919 18.723412 18 827.857708 14.239632 58.137577 17.453007 1.210712 0 0 0 0 0 1.270405 0.588379 0.682026 <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> NOT_AVAILABLE 176.875424 -48.898379 42.513192 -16.283806 False False 0 False False False False False True True False 0.0 0.0 0.999991 4536.664062 4516.359375 4578.298828 4.6276 4.5718 4.6635 -0.5605 -0.737 -0.3866 1741.91272 1622.824707 1946.818726 0.0085 0.0024 0.0204 0.0065 0.0018 0.0158 0.0035 0.0009 0.0084 PHOENIX

5 rows × 152 columns

By default, the first 5 rows of data will be shown, but we can specify a higher number if we need.

[5]:

gaia.head(n=10)

2026-07-09 22:17:41,518 - distributed.worker.memory - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker-memory.html#memory-not-released-back-to-the-os for more information. -- Unmanaged memory: 1.97 GiB -- Worker memory limit: 3.27 GiB

[5]:
solution_id designation source_id random_index ref_epoch ra ra_error dec dec_error parallax parallax_error parallax_over_error pm pmra pmra_error pmdec pmdec_error ra_dec_corr ra_parallax_corr ra_pmra_corr ra_pmdec_corr dec_parallax_corr dec_pmra_corr dec_pmdec_corr parallax_pmra_corr parallax_pmdec_corr pmra_pmdec_corr astrometric_n_obs_al astrometric_n_obs_ac astrometric_n_good_obs_al astrometric_n_bad_obs_al astrometric_gof_al astrometric_chi2_al astrometric_excess_noise astrometric_excess_noise_sig astrometric_params_solved astrometric_primary_flag nu_eff_used_in_astrometry pseudocolour pseudocolour_error ra_pseudocolour_corr dec_pseudocolour_corr parallax_pseudocolour_corr pmra_pseudocolour_corr pmdec_pseudocolour_corr astrometric_matched_transits visibility_periods_used astrometric_sigma5d_max matched_transits new_matched_transits matched_transits_removed ipd_gof_harmonic_amplitude ipd_gof_harmonic_phase ipd_frac_multi_peak ipd_frac_odd_win ruwe scan_direction_strength_k1 scan_direction_strength_k2 scan_direction_strength_k3 scan_direction_strength_k4 scan_direction_mean_k1 scan_direction_mean_k2 scan_direction_mean_k3 scan_direction_mean_k4 duplicated_source phot_g_n_obs phot_g_mean_flux phot_g_mean_flux_error phot_g_mean_flux_over_error phot_g_mean_mag phot_bp_n_obs phot_bp_mean_flux phot_bp_mean_flux_error phot_bp_mean_flux_over_error phot_bp_mean_mag phot_rp_n_obs phot_rp_mean_flux phot_rp_mean_flux_error phot_rp_mean_flux_over_error phot_rp_mean_mag phot_bp_rp_excess_factor phot_bp_n_contaminated_transits phot_bp_n_blended_transits phot_rp_n_contaminated_transits phot_rp_n_blended_transits phot_proc_mode bp_rp bp_g g_rp radial_velocity radial_velocity_error rv_method_used rv_nb_transits rv_nb_deblended_transits rv_visibility_periods_used rv_expected_sig_to_noise rv_renormalised_gof rv_chisq_pvalue rv_time_duration rv_amplitude_robust rv_template_teff rv_template_logg rv_template_fe_h rv_atm_param_origin vbroad vbroad_error vbroad_nb_transits grvs_mag grvs_mag_error grvs_mag_nb_transits rvs_spec_sig_to_noise phot_variable_flag l b ecl_lon ecl_lat in_qso_candidates in_galaxy_candidates non_single_star has_xp_continuous has_xp_sampled has_rvs has_epoch_photometry has_epoch_rv has_mcmc_gspphot has_mcmc_msc in_andromeda_survey classprob_dsc_combmod_quasar classprob_dsc_combmod_galaxy classprob_dsc_combmod_star teff_gspphot teff_gspphot_lower teff_gspphot_upper logg_gspphot logg_gspphot_lower logg_gspphot_upper mh_gspphot mh_gspphot_lower mh_gspphot_upper distance_gspphot distance_gspphot_lower distance_gspphot_upper azero_gspphot azero_gspphot_lower azero_gspphot_upper ag_gspphot ag_gspphot_lower ag_gspphot_upper ebpminrp_gspphot ebpminrp_gspphot_lower ebpminrp_gspphot_upper libname_gspphot
_healpix_29
3121269471 1636148068921376768 Gaia DR3 4295806720 4295806720 545300884 2016.0 44.996155 0.101618 0.005615 0.101334 0.354331 0.122664 2.888632 12.616485 11.938352 0.137942 -4.080619 0.13317 0.122935 0.132028 -0.08891 0.022551 -0.365342 -0.036904 -0.244838 0.063012 0.135709 0.334337 184 0 183 1 2.672049 242.20697 0.380619 2.076508 31 False 1.508939 <NA> <NA> <NA> <NA> <NA> <NA> <NA> 22 16 0.217802 22 9 0 0.017597 90.239342 0 0 1.142952 0.307951 0.197659 0.430101 0.842078 -87.754784 -30.69455 -46.201908 30.174356 False 182 1653.394716 2.075764 796.523376 17.641426 18 800.429546 12.601409 63.519051 18.080235 20 1187.588004 15.823832 75.050598 17.061232 1.202385 0 0 0 2 0 1.019003 0.438808 0.580194 <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> NOT_AVAILABLE 176.951076 -48.901521 42.533726 -16.329574 False False 0 True False False False False True True False 0.0 0.0 0.999988 5052.976074 5040.658203 5066.628418 4.7793 4.7613 4.7924 -1.4592 -1.5732 -1.3439 1497.054688 1466.734619 1536.13208 0.0064 0.0016 0.0176 0.0052 0.0013 0.0143 0.0028 0.0007 0.0078 MARCS
29153808338 1636148068921376768 Gaia DR3 38655544960 38655544960 1757259052 2016.0 45.004978 0.017885 0.01988 0.018772 3.13917 0.022347 140.471313 35.308208 29.686339 0.023772 19.1152 0.02383 0.115263 0.073231 -0.106919 -0.030214 -0.448866 -0.155514 -0.379279 0.181846 0.26367 0.355281 183 0 182 1 0.264343 181.438461 0.0 0.0 31 False 1.455016 <NA> <NA> <NA> <NA> <NA> <NA> <NA> 21 15 0.039295 21 9 0 0.024302 98.629005 0 0 1.012191 0.306561 0.205788 0.452994 0.846566 -96.318893 -34.497215 -44.825779 30.34742 False 180 42030.600439 11.392837 3689.212891 14.128453 20 17955.479377 26.03932 689.552551 14.70305 19 34263.48754 36.75135 932.305542 13.410816 1.242404 0 3 0 2 0 1.292233 0.574596 0.717637 41.187176 3.113034 2 10 1 8 7.034563 <NA> <NA> 749.919922 <NA> 4500.0 3.0 -0.25 111 <NA> <NA> <NA> 13.068616 0.049816 10 <NA> NOT_AVAILABLE 176.944762 -48.88527 42.546872 -16.318522 False False 0 True True False False False True True False 0.0 0.0 0.999806 4708.794434 4659.062012 4723.277344 4.5588 4.5261 4.5654 -0.087 -0.1218 -0.0681 332.832214 330.470886 347.172913 0.2345 0.184 0.2516 0.182 0.1425 0.1955 0.0961 0.0752 0.1032 MARCS
29640498453 1636148068921376768 Gaia DR3 34361129088 34361129088 894504938 2016.0 45.00432 0.09732 0.021048 0.101753 3.235017 0.12045 26.857704 35.230515 29.518344 0.133693 19.231655 0.133922 0.163253 0.000643 -0.073663 -0.012017 -0.403893 -0.101522 -0.315934 0.14065 0.231426 0.381754 172 0 171 1 1.081467 194.599335 0.267413 1.032802 31 False 1.285487 <NA> <NA> <NA> <NA> <NA> <NA> <NA> 20 15 0.220539 20 9 0 0.057373 84.542816 0 0 1.05789 0.284316 0.182422 0.42349 0.848356 -101.856606 -31.586445 -44.381237 29.909302 False 170 1763.191387 2.121236 831.209595 17.571619 18 389.997136 9.491409 41.089485 18.860889 19 2178.214858 15.074686 144.494873 16.402643 1.45657 0 2 0 1 0 2.458246 1.28927 1.168976 <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> NOT_AVAILABLE 176.942789 -48.884934 42.546573 -16.317212 False False 0 False False False False False True True False 0.0 0.0 0.999389 3478.540771 3461.147461 3497.578369 4.7 4.6405 4.7734 -0.6143 -0.7064 -0.4964 302.234711 292.532501 312.637299 0.7643 0.7292 0.7975 0.505 0.4815 0.5273 0.3096 0.2956 0.3228 MARCS
162211426005 1636148068921376768 Gaia DR3 309238066432 309238066432 327609758 2016.0 44.995037 0.322039 0.038152 0.283505 1.38315 0.367907 3.759512 1.472751 0.710339 0.427646 -1.290122 0.364729 0.031063 0.200474 0.053647 0.059748 -0.22102 -0.016839 -0.13663 0.179964 0.052951 0.211478 191 0 191 0 -0.241182 192.304047 0.0 0.0 95 False <NA> 1.286628 0.090775 -0.157192 0.010362 -0.082226 -0.106753 -0.088081 22 16 0.619914 22 9 0 0.035551 87.96405 0 0 0.985694 0.269229 0.189581 0.319531 0.846794 -67.001999 -0.060676 -50.141193 28.862438 False 191 272.914296 1.123318 242.953751 19.597301 15 70.174394 8.844292 7.934428 20.723095 15 353.699104 9.905063 35.70892 18.37631 1.553138 0 0 0 1 0 2.346785 1.125793 1.220991 <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> NOT_AVAILABLE 176.914265 -48.879747 42.542548 -16.298139 False False 0 False False False False False False False False 0.0 0.0 0.99998 <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA>
187874219371 1636148068921376768 Gaia DR3 343597448960 343597448960 1791600903 2016.0 44.963896 0.117176 0.043595 0.109004 0.196148 0.133464 1.469662 6.845276 6.567299 0.15458 -1.930905 0.136051 0.090632 0.164648 -0.104231 -0.006402 -0.171425 -0.023352 -0.235325 0.071847 0.014762 0.158318 200 0 198 2 -0.380676 193.063904 0.0 0.0 31 False 1.459356 <NA> <NA> <NA> <NA> <NA> <NA> <NA> 23 15 0.221527 23 10 0 0.032448 87.084869 0 0 0.978971 0.26817 0.140934 0.26649 0.876896 -59.657413 -17.643497 -47.965027 30.37381 False 190 1049.382036 1.745514 601.187927 18.135033 19 442.641353 11.305161 39.153919 18.723412 18 827.857708 14.239632 58.137577 17.453007 1.210712 0 0 0 0 0 1.270405 0.588379 0.682026 <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> NOT_AVAILABLE 176.875424 -48.898379 42.513192 -16.283806 False False 0 False False False False False True True False 0.0 0.0 0.999991 4536.664062 4516.359375 4578.298828 4.6276 4.5718 4.6635 -0.5605 -0.737 -0.3866 1741.91272 1622.824707 1946.818726 0.0085 0.0024 0.0204 0.0065 0.0018 0.0158 0.0035 0.0009 0.0084 PHOENIX
268252741041 1636148068921376768 Gaia DR3 515396233856 515396233856 791118203 2016.0 44.998327 0.325203 0.066333 0.325257 0.242394 0.374324 0.647551 9.078264 4.473011 0.413431 -7.899814 0.364344 0.258 0.080822 -0.119447 0.001248 -0.226218 0.002904 -0.205003 0.054882 0.028036 0.206339 194 0 193 1 3.412331 273.874664 1.333511 2.700941 31 False 1.480195 <NA> <NA> <NA> <NA> <NA> <NA> <NA> 22 15 0.602228 22 10 0 0.022699 135.080292 0 0 1.179168 0.258251 0.198696 0.324514 0.836216 -67.002907 -28.493275 -48.898422 29.621029 False 189 317.343123 1.20997 262.273529 19.433544 19 145.8467 9.856594 14.796865 19.928801 17 216.133697 9.56562 22.594843 18.911089 1.140659 0 0 0 0 0 1.017712 0.495256 0.522455 <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> NOT_AVAILABLE 176.886903 -48.857815 42.554429 -16.272159 False False 0 False False False False False False False False 0.0 0.000001 0.999992 <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA>
282950616891 1636148068921376768 Gaia DR3 549755818112 549755818112 1176142027 2016.0 45.048282 0.027804 0.048254 0.0265 1.583477 0.034425 45.99728 16.465364 0.843128 0.038817 -16.443764 0.03292 0.150414 -0.141034 0.058549 0.17611 -0.4741 0.199068 0.040113 -0.144958 -0.137338 0.258817 186 0 185 1 1.73613 220.668442 0.072145 0.96563 31 True 1.464321 <NA> <NA> <NA> <NA> <NA> <NA> <NA> 21 13 0.057412 21 8 0 0.0284 136.842896 0 0 1.090977 0.331172 0.18609 0.49345 0.842158 -121.244049 -9.218482 -38.814762 29.860806 False 182 19047.581229 6.548358 2908.756592 14.987767 18 8336.447382 16.801348 496.177307 15.53609 17 15362.299345 18.731024 820.152649 14.281754 1.244187 0 0 0 0 0 1.254335 0.548323 0.706013 <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> NOT_AVAILABLE 176.959366 -48.834398 42.598673 -16.304077 False False 0 False False False False False True True False 0.0 0.0 0.999712 4837.496094 4819.394043 4859.164551 4.4109 4.3971 4.426 -0.6022 -0.6293 -0.5715 621.201111 607.664307 634.602417 0.3668 0.3498 0.3862 0.2886 0.275 0.3042 0.155 0.1476 0.1633 MARCS
425704743710 1636148068921376768 Gaia DR3 828929527040 828929527040 602360354 2016.0 45.02362 0.054348 0.068419 0.057793 1.203095 0.066816 18.006063 17.646046 13.952005 0.078203 -10.803909 0.07721 0.151767 0.035847 -0.174849 -0.019222 -0.43819 -0.130195 -0.382967 0.187084 0.243698 0.374865 180 0 174 6 0.674236 189.075348 0.0 0.0 31 False 1.42808 <NA> <NA> <NA> <NA> <NA> <NA> <NA> 21 15 0.128491 21 9 0 0.021228 100.855522 0 0 1.034901 0.315109 0.211116 0.46352 0.841352 -94.506332 -34.891853 -45.023621 30.44096 False 175 4394.201552 3.34694 1312.901245 16.580168 15 1758.203878 16.53475 106.333862 17.225868 17 3789.788093 18.455858 205.343369 15.801358 1.262571 0 2 0 0 0 1.42451 0.6457 0.77881 <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> NOT_AVAILABLE 176.911306 -48.838163 42.580258 -16.277574 False False 0 False False False False False True True False 0.0 0.0 0.99985 4333.865234 4303.369141 4382.055176 4.6641 4.6466 4.6782 -0.3251 -0.3924 -0.2468 690.560425 669.14801 719.154175 0.0405 0.0079 0.0907 0.0304 0.0059 0.0684 0.0162 0.0031 0.0363 PHOENIX
468807926017 1636148068921376768 Gaia DR3 927713095040 927713095040 1643621005 2016.0 45.026727 0.303744 0.081699 0.322892 -0.122728 0.368763 -0.332811 3.980225 3.762188 0.472112 -1.299282 0.482444 0.190339 0.121188 -0.113716 -0.09805 -0.390787 -0.244701 -0.448713 0.224337 0.185155 0.506009 169 0 169 0 1.71375 203.084595 0.826392 1.198147 95 False <NA> 1.454281 0.094649 -0.253413 -0.054037 -0.163671 -0.088179 0.047459 19 14 0.825556 19 8 0 0.015569 139.410172 0 0 1.094294 0.363257 0.229992 0.505837 0.834506 -83.60305 -36.492233 -47.43565 30.496729 False 168 336.699427 1.377869 244.362473 19.369261 18 100.588275 9.140166 11.005081 20.332174 19 340.960457 7.565854 45.065693 18.416136 1.311403 0 3 0 1 0 1.916039 0.962914 0.953125 <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> NOT_AVAILABLE 176.900061 -48.826707 42.587405 -16.265786 False False 0 False False False False False False False False 0.0 0.0 0.999996 <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA>
493675273613 1636148068921376768 Gaia DR3 966367933184 966367933184 1173244990 2016.0 45.03908 0.401012 0.086855 0.397626 1.467697 0.459791 3.192093 4.099111 2.191341 0.609582 3.464207 0.577576 0.186754 0.009441 -0.065326 0.097503 -0.269008 0.049176 -0.167165 0.102291 0.082944 0.398819 185 8 182 3 -0.883311 178.540543 0.0 0.0 95 False <NA> 1.163643 0.124589 -0.102592 -0.001083 -0.065867 -0.09567 -0.047096 22 16 0.972654 22 9 0 0.034764 81.918793 0 0 0.951427 <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> False 180 196.075981 1.043136 187.967804 19.956306 13 41.570137 8.182405 5.08043 21.291588 18 239.9443 14.923929 16.077824 18.797619 1.435742 0 1 0 1 0 2.493969 1.335281 1.158688 <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> NOT_AVAILABLE 176.907451 -48.814236 42.601282 -16.264474 False False 0 False False False False False False False False 0.0 0.0 0.999987 <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA>

10 rows × 152 columns

Closing the Dask client#

[6]:

client.close()

About#

Authors: Sean McGuire

Last updated on: June 27, 2025

If you use lsdb for published research, please cite following instructions.