Gaia Sky data tools

A collection of Python and Shell scripts to automate the packaging, deployment, and metadata management of Gaia Sky datasets.

Included scripts

1. dataset-version.py

The primary deployment tool. It automates the process of turning a local Git repository into a published dataset version on the server. Run it from the root dataset git repository directory:

./dataset-version.py <tag-name>

---

2. update-metadata.py

Synchronizes the state of the remote server with the central gaiasky-data.json registry. To run it, pass it the dataset.json path of the dataset you want to update, and the tag.

./update-metadata.py <tag-name> --dataset-json /path/to/dataset/dataset.json

---

3. gen-repo-index.sh

A utility to generate HTML directory listings for the dataset repository part of the server.

---

Standard workflow

To publish a new version of a dataset (e.g., jwst), follow these steps:

1. Tag the Data: In your dataset repository (e.g., gsdata-spacecraft-jwst), create and push a new Git tag:

   git tag -a v005 -m "Update model"
   git push origin v005
   

2. Upload to Server: Run the versioning script:

   dataset-version.py v005
   

3. Update Registry: Run the metadata script to update the central gaiasky-data.json:

   update-metadata.py v005 --repo ~/Projects/gaiasky-data
   

4. Push Registry: Navigate to your gaiasky-data repository, review the changes, and push:

   cd ~/Projects/gaiasky-data
   git diff gaiasky-data.json
   git commit -am "Update jwst to v005"
   git push
   push-gaiasky-data 017 gaiasky-data.json
   

🛠 Prerequisites

• Python 3.10+
• SSH access: Passwordless SSH key access to the andromeda server.
• Server utilities: The gen-checksums command must be available in the shell path on the remote server.
• Local config: Each dataset repository must contain a .serverpath file indicating its destination on the server.