Why Lazy Evaluation?

You might wonder why LSDB loads catalogs lazily. Why not just download the data? Let’s explore this with a metaphor.

A thoughtful gift

Imagine it’s your birthday. Your friend shows up to your party in a moving truck. After greeting you, she says, “Happy birthday! My gift to you is a houseplant, but I wasn’t sure which one you would like.”

She continues, “Inside this truck is every plant they had at the store. All you have to do is help me move all these plants inside your house so you can choose the ones you want. Then whatever you don’t want, you can throw away! Also, I’m renting this truck by the hour, so can you stop what you’re doing and help me bring these all in?”

A forest of stars

The plants in this story represent the data available through Rubin. Over a decade of the survey, dozens or hundreds of petabytes of data will be generated—a vast forest of information compared to your friend’s moving truck full of plants. Lazy evaluation is a practical way to handle this volume of data.

The catalog

Imagine that your friend arrives instead carrying a small booklet, World of Plants on its glossy cover. A catalog.

As you open it, she says, “I want to get you a plant. Take a look through this catalog. You can see each plant’s dimensions, weight, color, and care instructions, and decide which ones you want for your home. Once you pick out which plants you want, I’ll schedule a time to help you move them in.”

You and your friend enjoy the rest of the party. A few days later, she helps you move a gorgeous Bird of Paradise plant into your living room.

Lazy evaluation

This is the principle of lazy evaluation. When a catalog is loaded lazily, no data has been loaded, only the catalog schema. You can refine your search, ensuring that your query will select the data you want, all before you ever download it. Finally, you can compute the catalog to get the objects that match your query.

LSDB example

Let’s use lazy evaluation in LSDB.

# Setup

import lsdb

import astropy.units as u
from astropy.coordinates import SkyCoord
from upath import UPath
import matplotlib.pyplot as plt
import warnings

warnings.filterwarnings("ignore")

import logging

logging.getLogger().setLevel(logging.WARNING)
logging.getLogger("distributed").setLevel(logging.WARNING)

from dask.distributed import Client

client = Client(n_workers=4, memory_limit="4GiB", threads_per_worker=1)
print(f"Dask dashboard: {client.dashboard_link}")

# Path on RSP
base_path = UPath("/rubin/lsdb_data")

Dask dashboard: https://ncaplar.nb.data-int.lsst.cloud/nb/user/ncaplar/proxy/8787/status

# Load Rubin catalog
cat = lsdb.open_catalog(base_path / "object_collection")

We have loaded the Rubin catalog. This loading is lazy, so it takes only a few seconds.

Now we see all the columns available, the number of partitions, and an estimated total size of the data.

We can also see our catalog’s coverage in the sky.

cat

lsdb Catalog object_lc:

	coord_dec	coord_decErr	coord_ra	coord_raErr	g_psfFlux	g_psfFluxErr	g_psfMag	g_psfMagErr	i_psfFlux	i_psfFluxErr	i_psfMag	i_psfMagErr	objectId	patch	r_psfFlux	r_psfFluxErr	r_psfMag	r_psfMagErr	refBand	shape_flag	shape_xx	shape_xy	shape_yy	tract	u_psfFlux	u_psfFluxErr	u_psfMag	u_psfMagErr	y_psfFlux	y_psfFluxErr	y_psfMag	y_psfMagErr	z_psfFlux	z_psfFluxErr	z_psfMag	z_psfMagErr	objectForcedSource
npartitions=8840
Order: 8, Pixel: 10240	double[pyarrow]	float[pyarrow]	double[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	int64[pyarrow]	int64[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	string[pyarrow]	bool[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	int64[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	nested<band: [string], coord_dec: [double], co...
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
Order: 5, Pixel: 12283	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
Order: 5, Pixel: 12284	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...

37 out of 1261 available columns in the catalog have been loaded lazily, meaning no data has been read, only the catalog schema

This catalog has an estimated size of 1.5 TB

cat.plot_pixels();

At this point, we may realize that we only need a subset of this data:

• A certain set of columns

• A certain region in the sky

• A certain filter based on column values

Because of lazy evaluation, we were able to see that we had too much data, and can filter it appropriately.

cat = (
    lsdb.open_catalog(
        base_path / "object_collection",
        # Select columns
        columns=["g_psfMag", "r_psfMag"],
        # Select a sky region
    )
    .cone_search(
        ra=270.0,
        dec=-30.0,
        radius_arcsec=2 * 3600,
        # Query on column values
    )
    .query("g_psfMag < 28.0 and r_psfMag < 28.0")
)

cat

lsdb Catalog object_lc:

	g_psfMag	r_psfMag	coord_ra	coord_dec
npartitions=10
Order: 3, Pixel: 448	float[pyarrow]	float[pyarrow]	double[pyarrow]	double[pyarrow]
...	...	...	...	...
Order: 5, Pixel: 7217	...	...	...	...
Order: 6, Pixel: 28872	...	...	...	...

4 out of 1261 available columns in the catalog have been loaded lazily, meaning no data has been read, only the catalog schema

This catalog has an estimated size of 28.5 MB

Our lazy catalog took only a few seconds to filter. Now we know that we will get only the data we need.

Finally, we can compute the result.

df = cat.compute()

df

	g_psfMag	r_psfMag	coord_ra	coord_dec
_healpix_29
2021535611241012678	20.514009	19.290812	271.034835	-31.772878
2021535673342214295	20.53277	19.314535	271.043658	-31.750023
...	...	...	...	...
2031705148308382094	21.398901	19.407282	269.063239	-28.175253
2031705159509858786	21.041615	18.884104	269.066058	-28.17461

22086 rows × 4 columns

# Clean up
client.close()

About

Authors: Heather Sestili

Last run: July 27, 2026

If you use lsdb for published research, please cite following instructions.