Column filtering (e.g. columns=)#
In this tutorial, we will:
load a HATS catalog with the default set of columns
load a HATS catalog with all available columns
load a HATS catalog with a specified subset of columns
Introduction#
To improve performance and readability when working with large catalogs, users can specify the column names they want to load instead of loading the entire dataset. This approach reduces memory usage and speeds up data processing by avoiding unnecessary data retrieval.
[1]:
# Import LSDB.
import lsdb
[2]:
# Start a Dask client.
from dask.distributed import Client
client = Client(n_workers=4, memory_limit="auto")
[3]:
# Specify the path to the LSDB catalog you want to use.
surveys_path = "https://data.lsdb.io/hats/"
ztf_object_path = f"{surveys_path}/ztf_dr14/ztf_object"
1. Load the catalog with default columns#
[4]:
ztf_object = lsdb.open_catalog(ztf_object_path)
ztf_object
[4]:
lsdb Catalog ztf_dr14:
| ps1_objid | ra | dec | ps1_gMeanPSFMag | ps1_rMeanPSFMag | ps1_iMeanPSFMag | nobs_g | nobs_r | nobs_i | mean_mag_g | mean_mag_r | mean_mag_i | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| npartitions=2352 | ||||||||||||
| Order: 3, Pixel: 0 | int64[pyarrow] | double[pyarrow] | double[pyarrow] | double[pyarrow] | double[pyarrow] | double[pyarrow] | int32[pyarrow] | int32[pyarrow] | int32[pyarrow] | double[pyarrow] | double[pyarrow] | double[pyarrow] |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| Order: 4, Pixel: 3070 | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| Order: 4, Pixel: 3071 | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
12 out of 15 available columns in the catalog have been loaded lazily, meaning no data has been read, only the catalog schema
This catalog has an estimated size of 56.0 GB
2. Load the catalog with all columns#
[5]:
ztf_object = lsdb.open_catalog(ztf_object_path, columns="all")
ztf_object
[5]:
lsdb Catalog ztf_dr14:
| ps1_objid | ra | dec | ps1_gMeanPSFMag | ps1_rMeanPSFMag | ps1_iMeanPSFMag | nobs_g | nobs_r | nobs_i | mean_mag_g | mean_mag_r | mean_mag_i | Norder | Dir | Npix | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| npartitions=2352 | |||||||||||||||
| Order: 3, Pixel: 0 | int64[pyarrow] | double[pyarrow] | double[pyarrow] | double[pyarrow] | double[pyarrow] | double[pyarrow] | int32[pyarrow] | int32[pyarrow] | int32[pyarrow] | double[pyarrow] | double[pyarrow] | double[pyarrow] | int8[pyarrow] | int64[pyarrow] | int64[pyarrow] |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| Order: 4, Pixel: 3070 | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| Order: 4, Pixel: 3071 | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
15 out of 15 available columns in the catalog have been loaded lazily, meaning no data has been read, only the catalog schema
This catalog has an estimated size of 66.3 GB
3. Load the catalog with a specific subset of columns#
[6]:
ztf_object = lsdb.open_catalog(ztf_object_path, columns=["ps1_objid", "ra", "dec", "mean_mag_r"])
ztf_object
[6]:
lsdb Catalog ztf_dr14:
| ps1_objid | ra | dec | mean_mag_r | |
|---|---|---|---|---|
| npartitions=2352 | ||||
| Order: 3, Pixel: 0 | int64[pyarrow] | double[pyarrow] | double[pyarrow] | double[pyarrow] |
| ... | ... | ... | ... | ... |
| Order: 4, Pixel: 3070 | ... | ... | ... | ... |
| Order: 4, Pixel: 3071 | ... | ... | ... | ... |
4 out of 15 available columns in the catalog have been loaded lazily, meaning no data has been read, only the catalog schema
This catalog has an estimated size of 24.3 GB
Closing the Dask client#
[7]:
client.close()
2026-09-21 20:59:54,773 - distributed.worker - ERROR - Failed to communicate with scheduler during heartbeat.
Traceback (most recent call last):
File "/home/docs/checkouts/readthedocs.org/user_builds/lsdb/envs/stable/lib/python3.12/site-packages/distributed/comm/tcp.py", line 226, in read
frames_nosplit_nbytes_bin = await stream.read_bytes(fmt_size)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
tornado.iostream.StreamClosedError: Stream is closed
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/docs/checkouts/readthedocs.org/user_builds/lsdb/envs/stable/lib/python3.12/site-packages/distributed/worker.py", line 1165, in heartbeat
response = await retry_operation(
^^^^^^^^^^^^^^^^^^^^^^
File "/home/docs/checkouts/readthedocs.org/user_builds/lsdb/envs/stable/lib/python3.12/site-packages/distributed/utils_comm.py", line 420, in retry_operation
return await retry(
^^^^^^^^^^^^
File "/home/docs/checkouts/readthedocs.org/user_builds/lsdb/envs/stable/lib/python3.12/site-packages/distributed/utils_comm.py", line 399, in retry
return await coro()
^^^^^^^^^^^^
File "/home/docs/checkouts/readthedocs.org/user_builds/lsdb/envs/stable/lib/python3.12/site-packages/distributed/core.py", line 1212, in send_recv_from_rpc
return await send_recv(comm=comm, op=key, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/docs/checkouts/readthedocs.org/user_builds/lsdb/envs/stable/lib/python3.12/site-packages/distributed/core.py", line 982, in send_recv
response = await comm.read(deserializers=deserializers)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/docs/checkouts/readthedocs.org/user_builds/lsdb/envs/stable/lib/python3.12/site-packages/distributed/comm/tcp.py", line 237, in read
convert_stream_closed_error(self, e)
File "/home/docs/checkouts/readthedocs.org/user_builds/lsdb/envs/stable/lib/python3.12/site-packages/distributed/comm/tcp.py", line 137, in convert_stream_closed_error
raise CommClosedError(f"in {obj}: {exc}") from exc
distributed.comm.core.CommClosedError: in <TCP (closed) ConnectionPool.heartbeat_worker local=tcp://127.0.0.1:40782 remote=tcp://127.0.0.1:44543>: Stream is closed
About#
Authors: Olivia Lynn
Last updated on: May 20, 2025
If you use lsdb for published research, please cite following instructions.