LSDB Guide

LSDB Guide#

Last updated: 2026-05-28 | LSDB version: v0.9.0

Canonical reference for AI coding assistants working on LSDB. Tool-specific files (CLAUDE.md, .github/copilot-instructions.md) contain only tool-specific overrides and reference this file for shared guidance. Edit this file for changes that should apply to all AI assistants; edit tool-specific files only for tool-specific behavior.

Version notice for AI assistants: If anything in this guide conflicts with what you observe in the actual code (missing methods, different signatures, changed behaviour), the guide may be outdated. Check the version above against the installed package version (python -c "import lsdb; print(lsdb.__version__)") and alert the user about the changes that LSDB_GUIDE.md might need. Do not silently assume the guide is correct.

What is LSDB#

LSDB is a Python library for scalable spatial analysis of large astronomical catalogs, designed to handle order of billion sources from upcoming surveys like LSST, Euclid, and Roman. It is built on Dask for distributed parallelization and uses the HATS (Hierarchical Adaptive Tiling Scheme) partitioning format for efficient spatial operations. LSDB handles the boilerplate around loading, querying, cross-matching, and transforming sky catalogs so astronomers can focus on science.

For a deep dive into the underlying file format and HATS library classes, see HATS_GUIDE.md.

Design goals and north stars#

CRITICAL: Always keep these design principles in mind when making changes to LSDB.

Lazy evaluation by default. All operations on a Catalog build a Dask task graph and return immediately - nothing is computed until .compute() is called or a result is written to disk. Never eagerly compute results inside library methods; preserve laziness so users can chain operations before triggering execution.

HATS partitioning is the backbone. Every catalog is partitioned by HEALPix pixel and the underlying pixel data is stored on disk in parquet. Cross-partition operations (crossmatching, joins) require margin catalogs to avoid missing objects near partition edges.

Catalog collections: Catalogs can be stand-alone or part of a collection. A collection groups related catalogs together (main, margin, and index catalogs). When loading a collection, the default margin is loaded automatically and attached to the main catalog (catalog.margin).

Margins matter for correctness. Any spatial operation that can involve objects within some radius of a partition boundary (crossmatch, cone search near edges) must use a MarginCatalog. Do not silently drop margin handling - incorrect results at partition edges are worse than a clear error.

Slim API surface. Do not add new public API methods unless asked. Prefer composing existing primitives. If you think a new method is needed, propose it first and get an agreement on the design before implementing. When adding a new method, ensure it is added to the docs API reference and has a complete docstring with examples.

Scale from laptop to cluster. LSDB code must work on a single-machine local Dask scheduler and on a distributed dask.distributed cluster without code changes. Scalability is critical. Avoid patterns that only work on small datasets or require loading everything into memory. Use Dask best practices to ensure good performance at scale.

Backwards compatibility. Maintain backward compatibility where possible! If breaking changes are necessary, be loud about it.

Document current behavior. When migrating away from old patterns, use @deprecated with a helpful message rather than silently removing behavior. Error messages should point users to the documentation or the correct alternative.

Docstrings and type safety. All public methods must have complete NumPy-style docstrings and accurate type annotations.

Coding advice#

Do not push or open PRs unless explicitly asked.
When changing code, ensure that the current assumptions of the change appear to have always been true.
Leave code better than you find it over keeping old assumptions around.

Development setup#

Python ≥ 3.11 (see pyproject.toml requires-python)
Create a conda env: conda create -n lsdb python=3.11 && conda activate lsdb
Clone and install: git clone https://github.com/astronomy-commons/lsdb.git && cd lsdb
Run the setup script: echo 'y' | bash .setup_dev.sh
- Installs the package in editable mode with dev extras
- Installs pre-commit hooks
Alternative manual install: pip install -e .'[dev]' && pre-commit install
For full optional features (e.g. plotting, polygon search, Lance): pip install -e '.[full]'
For S3 / object-store support: pip install -e '.[s3fs]'
For bleeding-edge dependency versions (hats + nested-pandas from main): pip install -r requirements.txt
For documentation dependencies: pip install -r docs/requirements.txt

Common commands#

# Run the full test suite (includes doctests in src/ and docs/)
python -m pytest

# Run only unit tests (skip doctest collection from docs/)
python -m pytest tests/

# Run with coverage reporting
python -m pytest --cov=lsdb --cov-report=xml

# Lint
pylint src/ --rcfile=./src/.pylintrc
pylint tests/ --rcfile=./tests/.pylintrc

# Format
black src/ tests/ && isort src/ tests/

# Type check
mypy src/ tests/ --ignore-missing-imports

# Pre-commit (runs black, isort, pylint, mypy ...)
pre-commit run --all-files

# Build docs. Requires `docs/requirements.txt` dependencies installed.
cd docs && make html

# Run ASV benchmarks
cd benchmarks && asv run --quick

Repository structure#

src/lsdb/               Main package
src/lsdb/catalog/       Catalog class hierarchy (Catalog, MarginCatalog, MapCatalog, AssociationCatalog)
src/lsdb/core/          Core algorithms (crossmatch, spatial search, plotting)
src/lsdb/dask/          Dask-level operations and task graph construction
src/lsdb/loaders/       Catalog loading (from HATS, pandas DataFrame, astropy Table)
src/lsdb/io/            Write paths (to_hats, to_lance, to_association)
src/lsdb/nested/        NestedFrame: Dask DataFrame extension for nested data
src/lsdb/streams/       CatalogStream for iterating over partitions in chunks
tests/lsdb/             Test suite (mirrors src/ layout)
tests/data/             Small HATS-formatted test catalogs and other validation files
benchmarks/             ASV performance benchmarks
docs/                   Sphinx documentation sources
docs/reference/         API reference (auto-generated from docstrings)
docs/tutorials/         Jupyter notebook tutorials

Key files:

File	Purpose
`pyproject.toml`	Project metadata, dependencies, pytest/black/mypy config
`src/lsdb/__init__.py`	Public API - everything exported here is stable public surface
`src/lsdb/catalog/catalog.py`	Main `Catalog` class
`src/lsdb/catalog/dataset/healpix_dataset.py`	Base `HealpixDataset` class, shared by all catalog types
`src/lsdb/loaders/hats/read_hats.py`	`open_catalog` and `read_hats` entry points
`src/lsdb/loaders/dataframe/from_dataframe.py`	`from_dataframe` and `from_astropy` entry points
`src/lsdb/core/crossmatch/crossmatch.py`	Top-level `crossmatch()` function
`src/lsdb/core/crossmatch/abstract_crossmatch_algorithm.py`	Abstract class to implement custom crossmatch algorithms
`src/lsdb/core/search/abstract_search.py`	Abstract class to implement custom spatial searches
`src/lsdb/dask/`	Internal Dask graph construction - not public API

Architecture: Catalog class hierarchy#

All catalog types inherit from HealpixDataset and carry three core attributes:

ddf - a NestedFrame (Dask DataFrame with nested-pandas extension)
ddf_pixel_map - Dict[HealpixPixel, int] mapping HEALPix pixels to Dask partition indices
hc_structure - the corresponding hats metadata object (no actual data)

Catalog#

Key file: src/lsdb/catalog/catalog.py

The primary user-facing class. Wraps a HATS-partitioned sky catalog.

Key method groups:

Introspection: est_size, get_healpix_pixels, plot_coverage, plot_pixels, plot_points
Query and spatial search: cone_search, box_search, id_search, moc_search, pixel_search, polygon_search, query, search (generic)
Cross-catalog operations: concat, crossmatch, crossmatch_nested, join, join_nested, merge, merge_asof, merge_map
Partition and row level transforms: map_partitions, map_rows
Column and pixel statistics: aggregate_column_statistics, per_partition_statistics
Sampling (eager execution): head, tail, sample, random_sample
Computing (eager execution): compute, write_catalog

MarginCatalog#

Key file: src/lsdb/catalog/margin_catalog.py

Holds objects from neighboring partitions within a specified angular radius of each partition boundary. A Catalog has often a corresponding margin which can be access via catalog.margin.

MapCatalog#

Key file: src/lsdb/catalog/map_catalog.py

Represent non-point-source data (e.g. dust extinction, survey depth values) in a continuous map.

AssociationCatalog#

Key file: src/lsdb/catalog/association_catalog.py

Represents the pre-computed result of a crossmatch or join operation. Carries the pixels to join to recreate the full crossmatch/join result and any extra columns (e.g. object distance in arcseconds). Each association catalog is defined by a max_separation (arcseconds) for which the association catalog is valid. When joining catalogs through an association, the right catalog must have a margin threshold at least as high as the max_separation to ensure all matches are captured.

Architecture: search and crossmatch#

AbstractSearch#

Key file: src/lsdb/core/search/abstract_search.py

Subclass to implement a custom spatial filter. Must implement:

filter_hc_catalog(hc_structure) - return the subset of HEALPix pixels that could contain matching objects.
search_dataframe(df, catalog_info) - filter a single-partition NestedFrame.

Pass an instance to catalog.search(my_search).

AbstractCrossmatchAlgorithm#

Key file: src/lsdb/core/crossmatch/abstract_crossmatch_algorithm.py

Subclass to implement a custom crossmatch algorithm and pass an instance to the crossmatch: catalog.crossmatch(other, algorithm=MyCustomAlgorithm()). The default algorithm is KdTreeCrossmatch, an efficient implementation using scipy.spatial.cKDTree for fast nearest neighbor search.

Typical LSDB Workflow#

A typical LSDB workflow involves:

Opening catalogs from an existing HATS catalog, pd.DataFrame or astropy.Table.
Exploring metadata to understand the data:
- Size (length) of the catalog
- Schema (columns available and their types)
- Estimated size of the catalog
- Sky coverage and partitioning structure
- The columns/pixel statistics (e.g. mean, min, max) to understand the data distribution
Performing spatial queries (cone search, box search, polygon search) to subset the catalog to a region of interest.
Crossmatching with another catalog, ensuring margin handling for edge correctness.
Transforming results with map_partitions or map_rows for custom computations.
Writing results to HATS format for downstream use or sharing.

Open catalogs#

LSDB can handle local or remote catalogs with fsspec. All paths are transformed into UPath objects internally.

import lsdb

# Open a HATS catalog / collection from disk or object store.
# When opening a collection, the default margin is loaded and attached to the main catalog (`cat.margin`).
cat = lsdb.open_catalog("/path/to/hats/catalog", columns=["ra", "dec", "flux_g", "flux_r"])

# Open an auxiliary catalog directly from disk or object store.
margin = lsdb.read_hats("/path/to/hats/margin")

# Load a small pandas DataFrame as a catalog (< 1M rows)
cat = lsdb.from_dataframe(df, ra_column="ra", dec_column="dec")

# Load from an astropy Table
cat = lsdb.from_astropy(table, ra_column="ra", dec_column="dec")

# Generate a synthetic test catalog
# 500 rows in the base layer, 10 rows in the nested layer
cat = lsdb.generate_catalog(500, 10, seed=1)

CRITICAL#

Only load the columns you need when running your analysis to minimize memory usage and speed up operations.
Use the columns argument in open_catalog to specify the desired subset of columns.

Explore metadata#

Explore the distribution of the data before using it and modifying it.

# Number of rows
n = len(cat)

# Column names and dtypes
print(cat.dtypes)

# Sky MOC (multi-order coverage) map
cat.plot_coverage()

# Sky HEALPix map
cat.plot_pixels()

# Sky point density map (using `hats` library)
hats.inspection.plot_density(cat.hc_structure)

# Per-column statistics (mean, min, max, etc.)
column_stats = cat.aggregate_column_statistics()

# Per-pixel statistics (mean, min, max, etc.)
pixel_stats = cat.per_pixel_statistics()

# Estimated size in KB
size_kb = cat.est_size()

Spatial queries#

# Cone search
result = cat.cone_search(ra=180.0, dec=-30.0, radius_arcsec=3600)

# Box search
result = cat.box_search(ra=(170, 190), dec=(-40, -20))

# Polygon search (requires lsst-sphgeom)
result = cat.polygon_search(vertices=[(300, -50), (300, -55), (272, -55), (272, -50)])

Crossmatch#

# Basic crossmatch (inner join by default)
xmatch = cat_a.crossmatch(cat_b, radius_arcsec=1.0)

# Basic crossmatch (left join)
xmatch = cat_a.crossmatch(cat_b, radius_arcsec=1.0, how="left")

# Crossmatch with a custom algorithm
xmatch = cat_a.crossmatch(cat_b, radius_arcsec=1.0, algorithm=MyCustomAlgorithm())

cat_b’s margin_threshold (arcsec) must be greater or equal to radius_arcsec.

Custom partition operations#

def partition_transform(df, *, inc):
    df["new_ra"] = df["ra"] + inc
    return df

result = cat.map_partitions(partition_transform, inc=1)

If partition_transform function does not handle the empty partition case, LSDB might fail to infer the result Dask meta. In that case, manually specify meta: cat.map_partitions(..., meta=).

Custom row operations#

def row_transform(ra, dec, mag):
    return {"ra": ra, "dec": dec, "num_src": len(mag)}

result = cat.map_rows(
    row_transform,
    columns=["ra", "dec", "sources.mag"],
    row_container="args",
    meta={"ra": float, "dec": float, "num_src": int},
    append_columns=False # default
)

Always specify meta. If append_columns=True, meta should only include the new columns returned by row_transform.

Stream large catalogs#

from lsdb.streams import CatalogStream

stream = CatalogStream(cat, partitions_per_chunk=10)
for chunk in stream:
    process(chunk)

Compute and write results#

from dask.distributed import Client

with Client(
    n_workers=4, 
    threads_per_worker=1, 
    memory_limit="4GB", 
    local_directory="/tmp/dask-scratch",
) as client:
  
  # Write to HATS (Parquet-based, re-readable by lsdb.open_catalog)
  cat.write_catalog("/output/path")

  # Write to Lance format
  cat.to_lance("/output/path")
  
  # Compute to memory (use only for small results)
  df = cat.compute()

CRITICAL#

Do not load a full catalog into memory with .compute() unless you are sure it fits in memory!
Plan the computation by chaining lazy operations and only compute/write the results at the very end!
Suggest the creation of a dask.distributed.Client to take advantage of distributed computation.
Make sure the Client specifies appropriate defaults for the n_workers and memory_limit (per worker) according to the available resources. It should also specify a local temporary directory for Dask to use.

Troubleshooting#

The Dask dashboard is the primary tool for diagnosing performance problems. It shows task progress, worker memory, and the task stream in real time.

from dask.distributed import Client
client = Client(n_workers=4, threads_per_worker=1, memory_limit="4GB")
print(client.dashboard_link)  # e.g. http://127.0.0.1:8787/status

The dashboard is available as soon as the Client is created. Open the URL in a browser.
Use a custom port if 8787 is taken: Client(..., dashboard_address=":8790").
On a remote machine or JupyterHub, set up an SSH tunnel: ssh -L 8787:localhost:8787 user@remote-host, then open http://localhost:8787/status.

What to look for#

Progress - shows how many tasks are pending, running, and done per computation. A bar that barely moves means workers are idle or a single task is a bottleneck.
Task Stream - each horizontal bar is a worker; colored blocks are tasks. Gaps between blocks indicate workers waiting on each other or on the scheduler. Aim for dense, uniform coverage across all workers.
Workers - shows CPU utilization and memory per worker. Workers near their memory_limit will start spilling to disk (orange) or get restarted (red). If this happens, try to reduce n_workers and increase memory_limit.
Graph - visualizes the full Dask task graph. The number of tasks tends to scale with the number of partitions.

Common error messages and failures#

For a detailed guide to common Dask log messages, worker OOM scenarios, and memory tuning strategies, see docs/tutorials/dask-messages-guide.rst. It covers:

Worker paused at 80% memory usage
Couldn't gather N keys, rescheduling (fatal silent OOM - restart the kernel)
Unmanaged memory use is high warning
CommClosedError: Stream is closed after a with Client(...) block (cosmetic)
All workers killed at the start of a job (how to get a real Python traceback)
All workers killed in the middle or end of a job (unbalanced task memory)
Dashboard empty for a long time after .compute() (graph construction phase)

Testing Conventions#

File naming: tests/lsdb/test_<name>.py, mirroring the src/lsdb/ layout.
Fixtures: defined in tests/conftest.py. Use existing fixtures, do not duplicate test data. All fixtures are backed by tiny HATS catalogs in tests/data/.
Default test run: python -m pytest
Doctest enforcement: pytest is configured with --doctest-modules and --doctest-glob=*.rst. All public docstring examples must be runnable and correct.
No network in unit tests. Test data lives in tests/data/; do not fetch from the internet in unit tests.

Key Conventions#

Line length: 110 characters (black and isort both enforce this).
Import style: isort with profile = "black". Do not hand-tune import order.
Docstrings: NumPy style. All public functions and methods require a complete docstring including Parameters and Returns. Try to also include an Examples block.
Deprecation: use @deprecated(version="X.Y", reason="...") from the deprecated package. Never silently remove behavior.
_version.py is auto-generated by setuptools_scm from git tags. Never edit it by hand; it is excluded from coverage and linting.
lsst-sphgeom is platform-gated: polygon search requires this package, which is only available on macOS and Linux (sys_platform guard in pyproject.toml). Code using lsst-sphgeom must guard the import and provide a clear error on Windows.
dask/ is internal: functions in src/lsdb/dask/ are implementation details. Do not expose them in __init__.py or reference them in documentation.
Partition index vs. pixel: ddf_pixel_map maps HealpixPixel → int (partition index). Never assume partition indices are contiguous or match pixel order.

CI/CD and GitHub Workflows#

testing-and-coverage.yml - runs on every PR and push to main; matrix over Python 3.11–3.14; uploads coverage to Codecov.
smoke-test.yml - daily at 06:45 UTC; tests both [dev] and [full] extras.
testing-windows.yml - Windows-specific test matrix.
asv-main.yml / asv-pr.yml / asv-nightly.yml - ASV performance benchmarks; PR results are posted back to the PR.
publish-to-pypi.yml - triggered on tagged releases.
pre-commit-ci.yml - automated pre-commit hook checks for format/lint/mypy.

LSDB Guide

Contents

LSDB Guide#

What is LSDB#

Design goals and north stars#

Coding advice#

Development setup#

Common commands#

Repository structure#

Architecture: Catalog class hierarchy#

Catalog#

MarginCatalog#

MapCatalog#

AssociationCatalog#

Architecture: search and crossmatch#

AbstractSearch#

AbstractCrossmatchAlgorithm#

Typical LSDB Workflow#

Open catalogs#

CRITICAL#

Explore metadata#

Spatial queries#

Crossmatch#

Custom partition operations#

Custom row operations#

Stream large catalogs#

Compute and write results#

CRITICAL#

Troubleshooting#

What to look for#

Common error messages and failures#

Testing Conventions#

Key Conventions#

CI/CD and GitHub Workflows#