Small-Scale Analysis

Small-Scale Analysis#

In this tutorial, we will cover strategies for working with a small slice of a large catalog before committing to a full-scale computation:

  • narrow the sky area with a spatial filter (e.g., cone_search)

  • inspect a single partition with .partitions[i]

  • filter rows to a manageable subset (e.g., bright stars)

  • peek at the first few rows of every partition with map_partitions

  • draw a random sample with .random_sample()

Introduction#

Large astronomical catalogs can contain billions of rows spread across thousands of partitions. Running a pipeline on the full dataset is expensive, and it is easy to waste hours on a bug that could have been caught in seconds on a small slice.

A good workflow starts small:

  1. Narrow the sky: work only in the patch of sky you actually care about.

  2. Inspect one partition: confirm the data looks right before processing everything.

  3. Filter aggressively: drop rows you do not need as early as possible.

  4. Peek at multiple partitions: cheaply verify your function behaves correctly across partition boundaries.

  5. Draw a random sample: get a statistically representative preview without a full compute.

Each technique in this tutorial reduces the amount of data you touch, so you can iterate quickly and scale up only once you are confident in the result.

[1]:

import pandas as pd

import lsdb
from dask.distributed import Client

1. Open a catalog#

Additional Help

For additional information on dask client creation, please refer to the official Dask documentation and our Dask cluster configuration page for LSDB-specific tips. Note that dask also provides its own best practices, which may also be useful to consult.

For tips on accessing remote data, see our Accessing remote data guide

[2]:

client = Client(n_workers=4, memory_limit="auto")

We open the Pan-STARRS1 (PS1) object catalog. The catalog is loaded lazily–no row data is read yet.

[3]:

ps1_object = lsdb.open_catalog("s3://stpubdata/panstarrs/ps1/public/hats/otmo")
ps1_object

[3]:
lsdb Catalog otmo:
decMean decMeanErr epochMean gFlags gMeanPSFMag gMeanPSFMagErr iFlags iMeanPSFMag iMeanPSFMagErr nDetections ng ni nr ny nz objID objInfoFlag qualityFlag raMean raMeanErr rFlags rMeanPSFMag rMeanPSFMagErr surveyID yFlags yMeanPSFMag yMeanPSFMagErr zFlags zMeanPSFMag zMeanPSFMagErr
npartitions=9577
Order: 5, Pixel: 0 double[pyarrow] double[pyarrow] double[pyarrow] int32[pyarrow] double[pyarrow] double[pyarrow] int32[pyarrow] double[pyarrow] double[pyarrow] int16[pyarrow] int16[pyarrow] int16[pyarrow] int16[pyarrow] int16[pyarrow] int16[pyarrow] int64[pyarrow] int32[pyarrow] int16[pyarrow] double[pyarrow] double[pyarrow] int32[pyarrow] double[pyarrow] double[pyarrow] int16[pyarrow] int32[pyarrow] double[pyarrow] double[pyarrow] int32[pyarrow] double[pyarrow] double[pyarrow]
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
Order: 5, Pixel: 12286 ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
Order: 5, Pixel: 12287 ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
30 out of 131 available columns in the catalog have been loaded lazily, meaning no data has been read, only the catalog schema
This catalog has an estimated size of 1.9 TB

Note the number of columns shown.

Some catalogs come with a pre-specified set of “default columns” that will be loaded automatically (unless the columns='all' keyword is specified). We can also always manually specify which columns we’d like to load.

Let’s reduce the amount of data we’ll handle as we work with this catalog.

[4]:

ps1_object = lsdb.open_catalog(
    "s3://stpubdata/panstarrs/ps1/public/hats/otmo",
    columns=["objID", "raMean", "decMean", "gMeanPSFMag", "rMeanPSFMag", "iMeanPSFMag", "nDetections"],
)
ps1_object

[4]:
lsdb Catalog otmo:
objID raMean decMean gMeanPSFMag rMeanPSFMag iMeanPSFMag nDetections
npartitions=9577
Order: 5, Pixel: 0 int64[pyarrow] double[pyarrow] double[pyarrow] double[pyarrow] double[pyarrow] double[pyarrow] int16[pyarrow]
... ... ... ... ... ... ... ...
Order: 5, Pixel: 12286 ... ... ... ... ... ... ...
Order: 5, Pixel: 12287 ... ... ... ... ... ... ...
7 out of 131 available columns in the catalog have been loaded lazily, meaning no data has been read, only the catalog schema
This catalog has an estimated size of 612.5 GB

2. Region selection#

The simplest way to reduce the amount of data you work with is to restrict the sky area. A cone_search keeps only the partitions that overlap a circle defined by a center (ra, dec) and radius_arcsec.

Starting with a small cone lets you develop and test your pipeline on a tiny fraction of the catalog. Once the pipeline is correct, you can widen the cone or remove it entirely.

[5]:

ps1_cone = ps1_object.cone_search(ra=200.0, dec=30.0, radius_arcsec=1 * 3600)
ps1_cone

[5]:
lsdb Catalog otmo:
objID raMean decMean gMeanPSFMag rMeanPSFMag iMeanPSFMag nDetections
npartitions=4
Order: 5, Pixel: 2596 int64[pyarrow] double[pyarrow] double[pyarrow] double[pyarrow] double[pyarrow] double[pyarrow] int16[pyarrow]
Order: 5, Pixel: 2597 ... ... ... ... ... ... ...
Order: 5, Pixel: 2598 ... ... ... ... ... ... ...
Order: 5, Pixel: 2599 ... ... ... ... ... ... ...
7 out of 131 available columns in the catalog have been loaded lazily, meaning no data has been read, only the catalog schema
This catalog has an estimated size of 255.8 MB

The npartitions has dropped from thousands to a handful, so every subsequent step is much cheaper.

We can also use a pre-built ConeSearch object, which lets us reuse the same region across multiple catalogs.

[6]:

from lsdb import ConeSearch

cone = ConeSearch(ra=200.0, dec=30.0, radius_arcsec=1 * 3600)
ps1_cone = ps1_object.search(cone)
ps1_cone

[6]:
lsdb Catalog otmo:
objID raMean decMean gMeanPSFMag rMeanPSFMag iMeanPSFMag nDetections
npartitions=4
Order: 5, Pixel: 2596 int64[pyarrow] double[pyarrow] double[pyarrow] double[pyarrow] double[pyarrow] double[pyarrow] int16[pyarrow]
Order: 5, Pixel: 2597 ... ... ... ... ... ... ...
Order: 5, Pixel: 2598 ... ... ... ... ... ... ...
Order: 5, Pixel: 2599 ... ... ... ... ... ... ...
7 out of 131 available columns in the catalog have been loaded lazily, meaning no data has been read, only the catalog schema
This catalog has an estimated size of 255.8 MB

3. Partition selection#

Even within a small region you may want to look at a single partition in isolation. Use .partitions[i] to index into the catalog by partition number. The result is a lazy Dask DataFrame for that one partition.

This is useful when you want to call .compute() on just one chunk to inspect values or test a function without touching the rest of the catalog.

[7]:

# Look at the first partition
first_partition = ps1_cone.partitions[0]
first_partition.compute()

Building task graph...

[7]:
objID raMean decMean gMeanPSFMag rMeanPSFMag iMeanPSFMag nDetections
_healpix_29
730840911935789259 142942005432773466 200.543267 29.119078 -999.0 -999.0 22.1049 1
730840958034592453 142942005448135918 200.544851 29.121114 -999.0 -999.0 -999.0 1
... ... ... ... ... ... ... ...
730990514047383426 143991996861207234 199.686118 29.99723 -999.0 -999.0 -999.0 1
730990514483634856 144001996877830045 199.687771 29.999566 -999.0 -999.0 22.3608 1

181808 rows × 7 columns

You can find the HEALPix pixel that corresponds to a given partition index using get_healpix_pixels().

[8]:

pixels = ps1_cone.get_healpix_pixels()

print(f"All pixels covered: {pixels}")
print(f"Partition 0 covers: {pixels[0]}")

All pixels covered: [Order: 5, Pixel: 2596, Order: 5, Pixel: 2597, Order: 5, Pixel: 2598, Order: 5, Pixel: 2599]
Partition 0 covers: Order: 5, Pixel: 2596

4. Sub-filtering#

Row filters let you trim the data further before any expensive computation. For example, selecting only bright stars reduces the number of rows dramatically and gives you a representative but manageable subset to work with.

[9]:

ps1_cone_and_bright = ps1_cone.query("0 < gMeanPSFMag < 16")
ps1_cone_and_bright

[9]:
lsdb Catalog otmo:
objID raMean decMean gMeanPSFMag rMeanPSFMag iMeanPSFMag nDetections
npartitions=4
Order: 5, Pixel: 2596 int64[pyarrow] double[pyarrow] double[pyarrow] double[pyarrow] double[pyarrow] double[pyarrow] int16[pyarrow]
Order: 5, Pixel: 2597 ... ... ... ... ... ... ...
Order: 5, Pixel: 2598 ... ... ... ... ... ... ...
Order: 5, Pixel: 2599 ... ... ... ... ... ... ...
7 out of 131 available columns in the catalog have been loaded lazily, meaning no data has been read, only the catalog schema
This catalog has an estimated size of 255.8 MB
[10]:

ps1_cone_and_bright.head(5)

Building task graph...

[10]:
objID raMean decMean gMeanPSFMag rMeanPSFMag iMeanPSFMag nDetections
_healpix_29
730844140320140186 142832001473917106 200.147265 29.030416 14.7094 14.0439 13.8106 55
730844900543284476 142842001903509791 200.190449 29.04093 15.1536 14.6864 14.5175 55
730845199124677148 142912002434617897 200.243435 29.097786 15.3492 14.9563 14.8328 69
730845991608593391 142852003372855574 200.337294 29.045833 15.9104 15.4282 15.2398 68
730846292154371814 142912004225284029 200.422788 29.09478 12.0536 11.207 10.9045 49

5 rows × 7 columns

Filters compose: you can chain a spatial filter with a row filter and LSDB will push both into the same pipeline.

5. Peeking at every partition#

map_partitions applies a function to each partition individually. Passing pd.DataFrame.head (or a small wrapper around it) is a cheap way to fetch the first few rows of every partition without loading the full catalog into memory.

This is especially useful for checking that a transformation produces the expected columns and values across all partition boundaries.

[11]:

# Grab the first 3 rows from each partition, then compute
sample_per_partition = ps1_cone.map_partitions(lambda df: df.head(3))
sample_per_partition.compute()

Building task graph...

[11]:
objID raMean decMean gMeanPSFMag rMeanPSFMag iMeanPSFMag nDetections
_healpix_29
730840911935789259 142942005432773466 200.543267 29.119078 -999.0 -999.0 22.1049 1
730840958034592453 142942005448135918 200.544851 29.121114 -999.0 -999.0 -999.0 1
730840958090282340 142942005456926339 200.545683 29.121461 -999.0 -999.0 -999.0 1
731028614286063577 142972005930353041 200.593027 29.143732 -999.0 21.5769 -999.0 1
731028614545892412 142972005940344460 200.594009 29.144912 22.1401 -999.0 -999.0 1
731028614842509175 142972005928714895 200.59283 29.145228 -999.0 21.4786 -999.0 2
731343459159795912 143281990771334860 199.077096 29.40356 -999.0 -999.0 -999.0 1
731343462332759277 143281990795017790 199.079454 29.406004 18.150801 -999.0 -999.0 1
731343462646587519 143281990763998560 199.076392 29.406652 -999.0 21.8297 -999.0 1
731553465279730385 144001996876174113 199.687558 30.002921 22.3853 21.153799 20.791201 50
731553465557381929 144001996902904284 199.690246 30.003063 21.1381 -999.0 -999.0 1
731553465612402991 144001996909224697 199.690881 30.00342 20.8773 -999.0 -999.0 1

12 rows × 7 columns

You can pass pd.DataFrame.head directly as the function, along with n as an extra keyword argument.

[12]:

sample_per_partition = ps1_cone.map_partitions(pd.DataFrame.head, n=3)
sample_per_partition.compute()

Building task graph...

[12]:
objID raMean decMean gMeanPSFMag rMeanPSFMag iMeanPSFMag nDetections
_healpix_29
730840911935789259 142942005432773466 200.543267 29.119078 -999.0 -999.0 22.1049 1
730840958034592453 142942005448135918 200.544851 29.121114 -999.0 -999.0 -999.0 1
730840958090282340 142942005456926339 200.545683 29.121461 -999.0 -999.0 -999.0 1
731028614286063577 142972005930353041 200.593027 29.143732 -999.0 21.5769 -999.0 1
731028614545892412 142972005940344460 200.594009 29.144912 22.1401 -999.0 -999.0 1
731028614842509175 142972005928714895 200.59283 29.145228 -999.0 21.4786 -999.0 2
731343459159795912 143281990771334860 199.077096 29.40356 -999.0 -999.0 -999.0 1
731343462332759277 143281990795017790 199.079454 29.406004 18.150801 -999.0 -999.0 1
731343462646587519 143281990763998560 199.076392 29.406652 -999.0 21.8297 -999.0 1
731553465279730385 144001996876174113 199.687558 30.002921 22.3853 21.153799 20.791201 50
731553465557381929 144001996902904284 199.690246 30.003063 21.1381 -999.0 -999.0 1
731553465612402991 144001996909224697 199.690881 30.00342 20.8773 -999.0 -999.0 1

12 rows × 7 columns

6. Random sample#

.random_sample(n) draws approximately n rows distributed proportionally across all partitions. Unlike .head(), which always returns rows from the first partitions, a random sample is representative of the whole catalog.

Use .random_sample() when you need a statistical cross-section of the data — for example, to estimate a distribution or spot-check the output of a filter.

Pass a seed for reproducible results.

[13]:

sample = ps1_cone.random_sample(n=20, seed=42)
sample

Building task graph...

Building task graph...

[13]:
objID raMean decMean gMeanPSFMag rMeanPSFMag iMeanPSFMag nDetections
_healpix_29
730936837514970211 143031996642025316 199.664071 29.195606 -999.0 -999.0 20.358801 1
730937260098560845 143051996633936840 199.663345 29.213566 -999.0 -999.0 21.446899 1
730981020410294723 143701999367105563 199.936786 29.754175 -999.0 -999.0 17.8957 1
730846636153692221 142912004657185227 200.465743 29.095445 22.0921 -999.0 -999.0 2
730963866856569696 143291995094088739 199.50941 29.41515 -999.0 -999.0 -999.0 0
731187372341540167 144232003704111414 200.370417 30.192346 -999.0 -999.0 -999.0 1
731053897920541906 143582009547747748 200.95475 29.655966 -999.0 -999.0 -999.0 1
731163293637326794 144142008220797529 200.822152 30.12236 -999.0 -999.0 -999.0 1
731238740278871599 144632008234864165 200.823459 30.528015 -999.0 -999.0 21.811399 1
731135719984753069 143412005905491960 200.590549 29.509489 -999.0 -999.0 -999.0 0
731240428581329021 144702007713717233 200.771341 30.588915 -999.0 18.207199 -999.0 1
731182646650581791 144241999015365830 199.90153 30.204389 -999.0 -999.0 20.955601 1
731398542233445251 144251990043859423 199.004367 30.215691 -999.0 -999.0 -999.0 1
731395737863511823 144081989414884602 198.941401 30.070031 -999.0 -999.0 20.711399 2
731400824440579321 144371992230650973 199.223057 30.308635 -999.0 -999.0 -999.0 1
731403442917530770 144501991116254788 199.111624 30.420168 -999.0 -999.0 -999.0 0
731396642545367306 144221991383342669 199.138321 30.185065 -999.0 -999.0 -999.0 1
731608318167675384 145081997520915575 199.752067 30.904151 -999.0 -999.0 21.286699 1
731560930653127513 144441997622930914 199.762249 30.366917 -999.0 -999.0 -999.0 1
731566684980778844 144431996915317320 199.691565 30.363833 21.549801 -999.0 21.535801 2

20 rows × 7 columns

If you only want to sample from a single partition, use .sample(partition_id, n) instead. This avoids touching any other partition.

[14]:

single_partition_sample = ps1_cone.sample(partition_id=0, n=5, seed=42)
single_partition_sample

Building task graph...

[14]:
objID raMean decMean gMeanPSFMag rMeanPSFMag iMeanPSFMag nDetections
_healpix_29
730934257558267668 142921997225540746 199.72242 29.100046 22.0844 -999.0 21.249701 2
730973808190207201 143311996852465172 199.685196 29.428824 -999.0 -999.0 -999.0 1
730985264575674676 143661995028525264 199.502877 29.720583 -999.0 -999.0 -999.0 0
730946255313115900 143322001720581152 200.172059 29.433766 21.757099 20.5362 20.0984 63
730949930420245664 143221998915439684 199.891559 29.357595 -999.0 -999.0 -999.0 1

5 rows × 7 columns

Closing the Dask client#

[15]:

client.close()

About#

Authors: Olivia Lynn

Last updated on: May 18, 2026

If you use lsdb for published research, please cite following instructions.