Exporting results

Contents

Exporting results#

You can save the results of your workflow to disk, in parquet, using the Catalog.write_catalog call.

You must provide:

  • A base_catalog_path, which is the output path for your catalog directory.

  • The necessary credentials as storage_options, to the UPath construction, when trying to export the catalog to protected remote storage.

You can also (optionally) provide:

  • A name for your catalog, catalog_name. The catalog_name is the HATS catalog’s internal identifier and therefore may differ from the catalog’s base directory name.

  • Which columns should be loaded by default, using default_columns.

  • Whether to resume a previously interrupted write to the same directory. By default, resumption is disabled.

  • Whether to overwrite the contents of the specified directory. By default, overwriting is disabled.

For more available arguments, please read the method’s signature.

Example#

We planned the crossmatching of Gaia with ZTF in the following gaia_x_ztf Catalog:

gaia = lsdb.open_catalog(gaia_path)
ztf = lsdb.open_catalog(ztf_path)
gaia_x_ztf = gaia.crossmatch(ztf)

We can save the results to a local directory ./my_catalogs/gaia_x_ztf, using:

gaia_x_ztf.write_catalog(base_catalog_path="./my_catalogs/gaia_x_ztf", catalog_name="gaia_x_ztf")

If saving to cloud storage, do not forget to specify your credentials:

cloud_path = UPath(f"{bucket_name}/gaia_x_ztf", <your storage_options>)
gaia_x_ztf.write_catalog(base_catalog_path=cloud_path, catalog_name="gaia_x_ztf")

The HATS catalog on disk follows a well-defined directory structure:

gaia_x_ztf/
├── dataset
│   ├── _common_metadata
│   ├── _metadata
│   ├── Norder={}/
│   │   └── Dir={}/
│   │       ├── Npix={}.parquet
|   |       └── ...
│   └── Norder={}/
│       └── Dir={}/
│           ├── Npix={}.parquet
|           └── ...
├── hats.properties
├── partition_info.csv
└── skymap.fits

You can read more about this format at HATS Directory Scheme.

About#

Authors: Neven Caplar, Sandro Campos

Last updated on: April 27, 2025

If you use lsdb for published research, please cite following instructions.

Contents