unfcc_di_data/howto_new_release.md

How to create a new release

1. download and publish new data

Go to https://github.com/mikapfl/unfccc_di_data/actions/workflows/download.yaml and trigger a new run of the "download" workflow. This will download the latest data from the UNFCCC and add it to the data package on gin.hemio.de

The workflow takes about 10 minutes to run.

2. pull changed data, examine changes

In a terminal in your local checkout of the unfccc_di_data package, run:

datalad update -s ginhemio --how merge
datalad update
git pull

to pull all changes.

If you want to compare differences, checkout the base you want to compare against, move the CSVs from it to a temporary folder called old, then checkout the latest state again and copy the CSVs to a temporary folder called new. Then run (in fish):

for i in old/*.csv; echo $i; csvdiff --ignore-columns 0 -p 1,2,3,4,5,7 -o word-diff $i  new/$(basename $i) > diff_$(basename $i); end

you need https://github.com/aswinkarthik/csvdiff for that. Afterwards, you can check the diff_{country}.csv files for changes.

3. release a new version of the data package

If you are happy, start making a new version on zenodo and get a pre-reserved doi. Put the new doi and new citation info into datacite.yml and README.md

Then run:

datalad save
datalad push
datalad push --to origin
datalad push --to ginhemio --data anything
datalad export-archive -t zip "data-$(date --iso).zip"
zip parquet-only.zip data/annexI/*.parquet data/non-annexI/*.parquet

upload the new data (consisting of the data-{date}.zip and parquet-only.zip files) to zenodo.

4. start using the new version

To use the new version of the data package from other places, you should release a new version of the unfccc_di_api package, in particular update the default values for the ZenodoReader class in the file unfccc_di_api.py.