Docs home

Data submission

The measurement sites can submit data files for archiving, processing and publication in the Cloudnet data portal.

We recommend using the cloudnet-submit Python package for the submissions. If the package is missing a feature that you need, please create an issue in GitHub.

If you don't want to use the Python package, you can also use the HTTP API. The documentation for it is below.

API reference

File submission has two stages: metadata and data upload. Metadata of the file must be uploaded before uploading the file itself. You can find sample scripts in examples.

Metadata upload

Metadata is uploaded by sending a POST request to https://cloudnet.fmi.fi/upload/metadata/. The route accepts application/json type data, and requires HTTP Basic authentication. The JSON request should have the following fields:

Example JSON for uploading a file named 201030_020000_P06_ZEN.LV1:

{
  "measurementDate": "2020-10-30",
  "instrument": "rpg-fmcw-94",
  "instrumentPid": "https://hdl.handle.net/21.12132/3.191564170f8a4686",
  "filename": "201030_020000_P06_ZEN.LV1",
  "checksum": "e07910a06a086c83ba41827aa00b26ed",
  "site": "hyytiala"
}

Tags are used to distinguish files with the same filename such as co and cross files from Halo doppler lidar:

{
  "measurementDate": "2022-01-01",
  "instrument": "halo-doppler-lidar",
  "instrumentPid": "https://hdl.handle.net/21.12132/3.421272f219be4f97",
  "filename": "Stare_34_20220101_00.hpl",
  "checksum": "3f19b6bb762af3c61fa0845cfb5fa6d1",
  "site": "hyytiala",
  "tags": ["co"]
}

{
  "measurementDate": "2022-01-01",
  "instrument": "halo-doppler-lidar",
  "instrumentPid": "https://hdl.handle.net/21.12132/3.421272f219be4f97",
  "filename": "Stare_34_20220101_00.hpl",
  "checksum": "de46ec1e32b2965fce285057a8970d16",
  "site": "hyytiala",
  "tags": ["cross"]
}

Example of metadata upload with the curl command:

curl -u USERNAME:PASSWORD \
  -H "Content-Type: application/json" \
  -d '{"measurementDate":"2020-10-30","instrument":"rpg-fmcw-94","instrumentPid": "https://hdl.handle.net/21.12132/3.191564170f8a4686","filename":"201030_020000_P06_ZEN.LV1","checksum":"e07910a06a086c83ba41827aa00b26ed","site":"hyytiala"}' \
  https://cloudnet.fmi.fi/upload/metadata/

Replace USERNAME and PASSWORD with your station’s credentials. You can acquire the credentials by contacting the CLU team at actris-cloudnet@fmi.fi.

Update: It is now possible to acquire single credentials for uploading data from several sites.

Data upload

After uploading the metadata, the file itself is uploaded by sending a PUT request to https://cloudnet.fmi.fi/upload/data/<md5>, where <md5> is replaced by the file’s MD5 checksum. The body of the request should be the file contents. Use the Transfer-Encoding: chunked HTTP header when uploading files.

Example using curl:

curl -u USERNAME:PASSWORD \
  -H "Transfer-Encoding: chunked" \
  --upload-file 201030_020000_P06_ZEN.LV1 \
  https://cloudnet.fmi.fi/upload/data/e07910a06a086c83ba41827aa00b26ed

Examples

Here are some examples for submitting a chmk15k file named file1.nc from the current working directory.

Bash

FILENAME="file1.nc"
USERNAME="example"
PASSWORD="letmein"
HASH=$(md5sum $FILENAME | cut -f 1 -d " ")
JSON=$(cat << EOF
{
 "measurementDate": "2020-10-30",
 "instrument": "chm15k",
 "instrumentPid": "INSTRUMENTS_PID",
 "filename": "$FILENAME",
 "checksum": "$HASH",
 "site": "hyytiala"
}
EOF
)

# Upload metadata
printf "Uploading $FILENAME\t"
MD_RESPONSE=$(curl -s -i -u $USERNAME:$PASSWORD \
   -H "Content-Type: application/json" \
   -d "$JSON" \
   https://cloudnet.fmi.fi/upload/metadata/)

STATUS_CODE=$(echo "$MD_RESPONSE" | head -1 | cut -d' ' -f2)
if test $STATUS_CODE -ne 200 ; then # Handle errors
   RESPONSE_BODY=$(echo "$MD_RESPONSE" | tail -1)
   echo "$RESPONSE_BODY"
else # Upload data
DATA_RESPONSE=$(curl -s -u $USERNAME:$PASSWORD \
   -H "Transfer-Encoding: chunked" \
   --upload-file "$FILENAME" \
   https://cloudnet.fmi.fi/upload/data/$HASH)
echo "$DATA_RESPONSE"
fi

Python

This example uses the Python library requests for submitting requests.

import hashlib
import requests

filename = 'file1.nc'
username = 'example'
password = 'letmein'

# Compute hash
md5_hash = hashlib.md5()
with open(filename, 'rb') as f:
   for byte_block in iter(lambda: f.read(4096), b""):
       md5_hash.update(byte_block)

checksum = md5_hash.hexdigest()

metadata = {
   'filename': filename,
   'checksum': checksum,
   'measurementDate': '2020-10-30',
   'instrument': 'chm15k',
   'instrumentPid': 'INSTRUMENTS_PID',
   'site': 'hyytiala'
}

# Upload metadata
print(f'Uploading {filename}', end='\t')
res = requests.post('https://cloudnet.fmi.fi/upload/metadata/',
   json=metadata,
   auth=(username, password))

if res.status_code != 200: # Handle errors
   print(res.text)
else: # Upload data
   with open(filename, 'rb') as f:
       res = requests.put(f'https://cloudnet.fmi.fi/upload/data/{checksum}',
           data=f,
           auth=(username, password))
   print(res.text)

Responses

The following status codes are used by the server to signal the success/failure of the (meta)data upload. Each response is accompanied by a message elaborating the cause of the status code.

Metadata

Data

This site is open source. Improve this page.