Accessing EDP2

Accessing EDP2#

In this tutorial, we will:

  • access Rubin’s Data Preview 2 with LSDB

    • at RSP (Rubin Science Platform), based in the USA; and in the UK for the LSST-UK community

    • at NERSC (National Energy Research Scientific Computing Center) for the LSST DESC members

    • at CC-IN2P3 (Centre de Calcul de l’IN2P3), French Data Facility, for the LSST-France community

    • at LIneA (Laboratório Interinstitucional de e-Astronomia), Brazilian Independent Data Access Center

Introduction#

Prerequisites#

In order to access Rubin data, you must be a Rubin data rights holder.

Setup#

Dask emits more advisory logging than we need here. Run the cell below once, before either section.

[ ]:

# Dask puts out more advisory logging than we care for in this tutorial.
# It takes some doing to quiet all of it, but this recipe works.

import dask
import os

dask.config.set({"logging.distributed": "critical"})

import logging

# This also has to be done, for the above to be effective
logger = logging.getLogger("distributed")
logger.setLevel(logging.CRITICAL)

import warnings

# Finally, suppress the specific warning about Dask dashboard port usage
warnings.filterwarnings("ignore", message="Port 8787 is already in use.")

1. Accessing the data on Rubin Science Platform (RSP)#

1.1 Prepare your RSP container#

  • Visit https://data.lsst.cloud, unless you are accessing RSP through the LSST-UK IDAC participation program - in that case, visit https://rsp.lsst.ac.uk.

  • Log in using your identity provider.

  • Once in, you will see Portal, Notebooks, and APIs. Choose Notebooks.

  • When it asks you what container to start, choose one of the latest weekly releases on the left, and “Large” on the right.

    • We always recommend the latest weekly release, so that you get the newest version of LSDB. “Recommended” will give you the version of LSDB frozen at the date of the EDP2 release.

  • Once this has started, create a new notebook.

1.1.1 Ensure your notebook kernel has the right version of lsdb#

Make sure you’ve got at least version 0.10.2 of lsdb. Try the following.

[ ]:

%pip list | grep -E 'lsdb|hats'

[ ]:

# Or for even more detail:

import lsdb

lsdb.show_versions()

Because the recommended kernel version can contain an old version of lsdb, we always suggest using the latest weekly release to get all the features.

1.2 Create a Dask client#

When working on RSP, with a Large container you have access to 32 GB of RAM. To ensure each worker has enough memory to work with the data, we recommend using 4 workers with 1 thread each, and memory limit of “auto” (which will divide the available memory across the workers). Dask also will sometimes spill to disk when it needs to, so we recommend setting the local directory to the /deleted-sundays directory, which is a large scratch space available on RSP that will be cleared every Sunday. You can set up your Dask client with the following code:

[ ]:

from dask.distributed import Client

client = Client(
    n_workers=4,
    threads_per_worker=1,
    memory_limit="auto",
    local_directory=f"/deleted-sundays/{os.environ.get('USER', 'dask_scratch')}",
)
print(f"Dask dashboard: {client.dashboard_link}")
client

Your Dask dashboard will be accessible at https://{username}.nb.data.lsst.cloud/nb/user/{username}/proxy/{port}/status. The dashboard gives a live view of your computation — task progress, worker memory usage, and CPU activity — which is useful for monitoring progress and diagnosing performance issues. See the Dask dashboard documentation for a guide to reading it.

1.3 Open a catalog#

The data is divided into object, dia_object, and object_photoz (photometric redshifts). For a description of what these catalogs contain, see About the EDP2 HATS Catalogs. Let’s open all three catalogs:

[ ]:

from upath import UPath

base_path = UPath("/rubin/lsdb_data")

object_cat = lsdb.open_catalog(base_path / "object_collection")
dia_object_cat = lsdb.open_catalog(base_path / "dia_object_collection")
photoz_cat = lsdb.open_catalog(base_path / "object_photoz")

You’ve accessed the data!

To learn how to use this data, see Using Rubin Data and follow the lsdb quickstart at Getting Started.

If you have questions, the documentation’s How to ask for help section lists the best places to reach us — for LSDB and HATS tooling, and for questions about the Rubin data itself.

Downloading data to your machine

If you want to continue working on your own machine, scp your data out from the terminal. Suppose you have an account myself on a machine big-box.astro.somewhere.edu and the data is in ./some_data:

scp -r ./some_data myself@big-box.astro.somewhere.edu:some_data

2. Accessing the data at NERSC (Perlmutter)#

If you are a part of the LSST DESC collaboration and have a NERSC account, you can access Rubin DP2 via Perlmutter cluster. You can use both batch jobs and jupyter.nersc.gov, bellow we assume that you use NERSC’s Jupyter Hub.

2.1 Launch Jupyter#

Login to NERSC at https://jupyter.nersc.gov. Select “Login Node” for data exploration, configuration and code development. Use “Exclusive CPU Node” for larger tasks, such as full-catalog analysis.

2.2a Start kernel with LSDB#

LSDB is available in the DESC Jupyter kernel desc-td-env.

If you haven’t already set up the DESC Jupyter kernels at NERSC, run the one-time setup step on the Perlmutter command line:

source /global/common/software/lsst/common/miniconda/kernels/setup.sh

The next time you start up jupyter.nersc.gov, you’ll have access to the desc-td-env jupyter kernel. More information can be found here.

2.2b Alternative: Install LSDB#

For conda installation run conda install -c conda-forge lsdb in the terminal. For pip installation run python -m pip install lsdb or the following cell in a Jupyter notebook:

[ ]:

%pip install lsdb

2.2.1 Ensure your notebook kernel has the right version of lsdb#

Restart the kernel, then make sure you’ve got at least version 0.10.2 of lsdb. Try the following.

[ ]:

%pip list | grep -E 'lsdb|hats'

[ ]:

# Or for even more detail:

import lsdb

lsdb.show_versions()

2.3 Create a Dask client#

Create a Dask client sized to the node you launched — a Login Node is shared, so stay modest; an Exclusive CPU Node gives you the whole machine.

Dask’s dashboard has no public URL at NERSC. Use the Dask JupyterLab extension instead: open the Dask tab in the left sidebar, paste the address client prints (for example http://127.0.0.1:46517/status) into the search box, and click the panels you want — Task Stream, Progress, Worker Memory — to open them inside JupyterLab.

[ ]:

from dask.distributed import Client

client = Client(n_workers=4, threads_per_worker=1, memory_limit="auto")
client

2.4 Open a catalog#

The data is divided into object, dia_object, and object_photoz (photometric redshifts). For a description of what these catalogs contain, see About the EDP2 HATS Catalogs. Let’s open all three catalogs:

[ ]:

from upath import UPath

base_path = UPath("/global/cfs/cdirs/lsst/shared/rubin/DP2/HATS/")

object_cat = lsdb.open_catalog(base_path / "object_collection")
dia_object_cat = lsdb.open_catalog(base_path / "dia_object_collection")
photoz_cat = lsdb.open_catalog(base_path / "object_photoz")

You’ve accessed the data!

To learn how to use this data, see Using Rubin Data and follow the lsdb quickstart at Getting Started.

If you have questions, the documentation’s How to ask for help section lists the best places to reach us — for LSDB and HATS tooling, and for questions about the Rubin data itself.

Downloading data to your machine

If you want to continue working on your own machine, scp your data out from the terminal. Suppose you have an account myself on a machine big-box.astro.somewhere.edu and the data is in ./some_data:

scp -r ./some_data myself@big-box.astro.somewhere.edu:some_data

3. Accessing the data at CC-IN2P3 (French Data Facility)#

If you are a member of the LSST-France community and hold Rubin data rights, you can access Rubin DP2 at CC-IN2P3, where the catalogs are available in HATS format under /sps/lsst/datasets/rubin/previews/dp2/catalogs/hats.

Detailed instructions for access via the French Data Facility are available at https://doc.lsst.eu/tutorial/releases.html#data-preview-2-dp2.

4. Accessing the data at LIneA (Laboratório Interinstitucional de e-Astronomia)#

Detailed instructions for access via the Brazilian IDAC will be available at https://data.linea.org.br/en/lsdb/how_to_access_rubin_dp2.html. That page is not published yet.

About#

Authors: Neven Caplar, Sandro Campos, Olivia Lynn, Konstantin Malanchev

Last updated: August 11, 2026

If you use lsdb for published research, please cite following instructions.