Docs home

Data portal API reference

This is the documentation for the API v2 provided by the Cloudnet data portal.

General

We provide an HTTP REST API for programmatic access to the metadata at the Cloudnet data portal. The API responds with JSON and can be accessed via the base URL https://cloudnet.fmi.fi/api/. Metadata can be queried via the following routes. For examples on how to use the API programmatically, see Examples.

Index

  1. Routes
    1. Sites
    2. Products
    3. Product variables
    4. Instruments
    5. Models
    6. File by UUID
    7. Product files
    8. Model files
    9. Visualization by file UUID
    10. Visualizations
    11. Raw files
    12. Raw model files
  2. Examples
  3. Notes
  4. Errors

Routes

GET /api/sitesSite[]

Fetch information on all the measuring stations in the system. Responds with an array of Site objects, each having the properties:

Example query:

GET https://cloudnet.fmi.fi/api/sites

Response body:

[
  {
    "id": "alomar",
    "humanReadableName": "Alomar",
    "type": [
      "hidden"
    ],
    "latitude": 62.278,
    "longitude": 16.008,
    "altitude": 380,
    "gaw": null,
    "dvasId": null,
    "country": "Norway",
    "countryCode": "NO",
    "countrySubdivisionCode": null
  },
...
]

GET /api/productsProduct[]

Fetch information on the products served in the data portal. Responds with an array of Product objects, each having the properties:

Example query:

GET https://cloudnet.fmi.fi/api/products

Response body:

[
  {
    "id": "categorize",
    "humanReadableName": "Categorize",
    "level": "1c"
  },
...
]

GET /api/products/variablesProductVariables[]

Fetch product variables. Responds with an array of ProductVariable objects, each having the properties:

NOTE: The list of returned variables is not exhaustive, the products may contain more variables than listed. Internally this list is used for plotting.

Example query:

GET https://cloudnet.fmi.fi/api/products/variables

Response body:

[
  {
    "id": "disdrometer",
    "humanReadableName": "Disdrometer",
    "level": "1b",
    "variables": [
      {
        "id": "disdrometer-rainfall_rate",
        "humanReadableName": "Rainfall rate",
        "order": "0"
      },
      {
        "id": "disdrometer-n_particles",
        "humanReadableName": "Number of particles",
        "order": "1"
      }
    ]
  },
...
]

GET /api/instrumentsInstrument[]

Fetch information on the instruments supported by the data portal. Responds with an array of Instrument objects, each having the properties:

Example query:

GET https://cloudnet.fmi.fi/api/instruments

Response body:

[
  {
    "id": "mira",
    "humanReadableName": "METEK MIRA-35 cloud radar",
    "type": "radar",
    "allowedTags": []
  },
...
]

GET /api/modelsModel[]

Fetch information on the different model file types served in the data portal. Responds with an array of Model objects, each having the properties:

Example query:

GET https://cloudnet.fmi.fi/api/models

Response body:

[
  {
    "id": "ecmwf",
    "humanReadableName":"ECMWF IFS forecast",
    "optimumOrder": 0
  },
...
]

GET /api/files/UUIDFile

Fetch metadata for a single data object using its UUID (Universal Unique IDentifier). All the data objects downloaded from the data portal are associated with a UUID, which can be found in the data object’s global attribute file_uuid. To view the global attributes of a NetCDF file, one may use ncdump -h file.nc.

On a successful query the route responds with a File object, which has the following properties:

Example query:

GET https://cloudnet.fmi.fi/api/files/911bd5b1-3104-4732-9bd3-34ed8208adad

Response body:

{
  "uuid": "911bd5b1-3104-4732-9bd3-34ed8208adad",
  "version": "icRZfkXzTHB2-uuRTDaSV-eQu6N5wNm",
  "pid": "https://hdl.handle.net/21.12132/1.911bd5b131044732",
  "volatile": false,
  "legacy": false,
  "measurementDate": "2020-01-05",
  "history": "2020-09-25 02:51:44 - categorize file created\n2020-09-25 02:50:07 - radar file created\n2020-09-25 02:49:27 - ceilometer file created",
  "checksum": "51db399d73e69842988d35e7e14fff815342f6925c70c0bdc9296b2847960738",
  "size": "8836007",
  "format": "HDF5 (NetCDF4)",
  "createdAt": "2020-12-01T09:30:40.016Z",
  "updatedAt": "2020-12-01T09:30:40.016Z",
  "sourceFileIds": [
    "f4cb2b92bd0c49779606b87440afa7b7",
    "6ac91e0483934db2afb3bacc97a7d8c0",
    "3171e11d022549b29e863138c406dce8"
  ],
  "site": {
    "id": "bucharest",
    "humanReadableName": "Bucharest",
    "type": ["cloudnet"],
    "latitude": 44.348,
    "longitude": 26.029,
    "altitude": 93,
    "gaw": "Unknown",
    "dvasId": "INO",
    "country": "Romania"
  },
  "product": {
    "id": "categorize",
    "humanReadableName": "Categorize",
    "level": "1c"
  },
  "software": [
    {
      "id": "cloudnet-processing",
      "version": "2.22.0",
      "title": "Cloudnet processing 2.22.0",
      "url": "https://github.com/actris-cloudnet/cloudnet-processing/tree/v2.22.0"
    },
    {
      "id": "cloudnetpy",
      "version": "1.53.1",
      "title": "CloudnetPy 1.53.1",
      "url": "https://doi.org/10.5281/zenodo.8321630"
    }
  ],
  "downloadUrl": "https://cloudnet.fmi.fi/api/download/product/911bd5b1-3104-4732-9bd3-34ed8208adad/20200105_bucharest_categorize.nc",
  "filename": "20200105_bucharest_categorize.nc",
  "errorLevel": "pass",
  "timeliness": "nrt"
}

GET /api/filesFile[]

Queries the metadata of multiple product files. On a successful query responds with an array of File objects. The results can be filtered with the following parameters:

Note: one or more of the parameters must be issued. A query without any valid parameters will result in a 400 Bad Request error.

Example query for fetching metadata for all classification files from Bucharest starting at 24. April 2020 and ending at the current date:

GET https://cloudnet.fmi.fi/api/files?site=bucharest&dateFrom=2020-04-24&product=classification

Response body:

[
  {
    "uuid": "f63628b4-7e68-4c98-87ea-86a247f7fcfd",
    "version": "",
    "pid": "",
    "volatile": true,
    "legacy": false,
    "measurementDate": "2021-02-21",
    "history": "2021-02-23 08:04:18 - classification file created\n2021-02-23 08:04:01 - categorize file created\n2021-02-23 02:04:13 - radar file created\n2021-02-23 02:04:24 - ceilometer file created",
    "checksum": "9ccd36e4c6ee94c5179e34e08ff7898bf31979277badeb5806b688a30e23feda",
    "size": "82547",
    "format": "HDF5 (NetCDF4)",
    "createdAt": "2021-02-23T02:05:03.970Z",
    "updatedAt": "2021-02-23T08:04:20.186Z",
    "site": {
      "id": "bucharest",
      "humanReadableName": "Bucharest",
      "type": [
        "cloudnet"
      ],
      "latitude": 44.348,
      "longitude": 26.029,
      "altitude": 93,
      "gaw": "Unknown",
      "dvasId": "INO",
      "country": "Romania"
    },
    "product": {
      "id": "classification",
      "humanReadableName": "Classification",
      "level": "2"
    },
    "software": [
      {
        "id":"cloudnet-processing",
        "version":"2.22.0",
        "title":"Cloudnet processing 2.22.0",
        "url":"https://github.com/actris-cloudnet/cloudnet-processing/tree/v2.22.0"
      },
      {
        "id":"cloudnetpy",
        "version":"1.53.1",
        "title":"CloudnetPy 1.53.1",
        "url":"https://doi.org/10.5281/zenodo.8321630"
      }
    ],
    "downloadUrl": "https://cloudnet.fmi.fi/api/download/product/f63628b4-7e68-4c98-87ea-86a247f7fcfd/20210221_bucharest_classification.nc",
    "filename": "20210221_bucharest_classification.nc",
    "quality": "nrt",
    "qualityScore": 0.9
  },
...
]

GET /api/model-filesModelFile[]

Queries the metadata of model files. It offers the following parameters for filtering the results:

The ModelFile response similar to the File response, with an additional model property. The model property containing a Model object (see example response). Furthermore, the ModelFile response omits the fields cloudnetPyVersion and sourceFileIds, as this information is not available for model files.

Example query for fetching metadata for gdas1 model from Lindenberg on 2. March 2021:

GET https://cloudnet.fmi.fi/api/model-files?site=lindenberg&date=2021-02-03&model=gdas1

Response body:

[
  {
    "uuid": "90f95f81-4f45-4efb-a25d-80066652aece",
    "version": "",
    "pid": "",
    "volatile": true,
    "legacy": false,
    "measurementDate": "2021-02-03",
    "history": "2021-02-04 08:10:26 - File content harmonized by the CLU unit.\n03-Feb-2021 18:59:45: Created from GDAS1 profiles produced with the profile binary in the HYSPLIT offline package using convert_gdas12pro.sh.",
    "checksum": "0b8b621ad2d76ca629451f56c94b79b432caba9c2839b3fcc535910544b3b854",
    "size": "194983",
    "format": "HDF5 (NetCDF4)",
    "createdAt": "2021-02-04T08:10:28.001Z",
    "updatedAt": "2021-02-04T08:10:28.001Z",
    "site": {
      "id": "lindenberg",
      "humanReadableName": "Lindenberg",
      "type": ["cloudnet"],
      "latitude": 52.208,
      "longitude": 14.118,
      "altitude": 104,
      "gaw": "LIN",
      "dvasId": "LIN",
      "country": "Germany"
    },
    "product": {
      "id": "model",
      "humanReadableName": "Model",
      "level": "1b"
    },
    "model": {
      "id": "gdas1",
      "optimumOrder": 3
    },
    "software": [
      {
        "id": "cloudnet-processing",
        "version": "2.22.0",
        "title": "Cloudnet processing 2.22.0",
        "url": "https://github.com/actris-cloudnet/cloudnet-processing/tree/v2.22.0"
      }
    ],
    "downloadUrl": "https://cloudnet.fmi.fi/api/download/product/90f95f81-4f45-4efb-a25d-80066652aece/20210203_lindenberg_gdas1.nc",
    "filename": "20210203_lindenberg_gdas1.nc"
  }
]

GET /api/visualizations/UUIDVisualization

Fetch visualization metadata for a single data object using its UUID. On a successful query the route responds with a Visualization object, which has the following properties:

Example query:

GET https://cloudnet.fmi.fi/api/visualizations/cfda5129-0a51-45d4-b674-ccf6c7f1a447

Response body:

{
  "sourceFileId": "cfda5129-0a51-45d4-b674-ccf6c7f1a447",
  "visualizations": [
    {
      "s3key": "20211025_hyytiala_cl61d-cfda5129-beta.png",
      "productVariable": {
        "id": "lidar-beta",
        "humanReadableName": "Attenuated backscatter coefficient",
        "order": "0"
      },
      "dimensions": null
    },
    {
      "s3key": "20211025_hyytiala_cl61d-cfda5129-depolarisation.png",
      "productVariable": {
        "id": "lidar-depolarisation",
        "humanReadableName": "Lidar depolarisation",
        "order": "2"
      },
      "dimensions": null
    }
  ],
  "productHumanReadable": "Lidar",
  "locationHumanReadable": "Hyytiälä",
  "volatile": true,
  "legacy": false
}

GET /api/visualizationsVisualization[]

Queries the metadata of visualizations. It offers the same parameters as the Product files -route for filtering the results, with an additional parameter variable for fetching visualizations for a specific ProductVariable. The route responds with an array of Visualization objects.

Example request:

GET https://cloudnet.fmi.fi/api/visualizations/?date=2021-10-25&site=mindelo

Response body:

[
  {
    "sourceFileId": "36a95fab-f07f-4182-99b8-0082b26b6069",
    "visualizations": [
      {
        "s3key": "20211025_mindelo_hatpro-36a95fab-LWP.png",
        "productVariable": {
          "id": "mwr-LWP",
          "humanReadableName": "Liquid water path",
          "order": "0"
        },
        "dimensions": null
      }
    ],
    "productHumanReadable": "Microwave radiometer",
    "locationHumanReadable": "Mindelo",
    "volatile": true,
    "legacy": false
  },
...
]

GET /api/raw-filesUpload[]

Query the metadata of uploaded instrument files. You can use the following the parameters to filter the search results.

The response is an array of Upload objects. The Upload object has the following properties:

Example query:

GET https://cloudnet.fmi.fi/api/raw-files?site=norunda&updatedAtFrom=2021-09-01

Response body:

[
  {
    "uuid": "890c7130-8cf0-44a2-b5bc-75fcec344d97",
    "checksum": "1a3ab11370e1b3c4833fb812c0a0f82e",
    "filename": "220701_050008_P09_ZEN.LV0",
    "measurementDate": "2022-07-01",
    "size": "975360456",
    "status": "uploaded",
    "createdAt": "2022-07-01T10:24:44.130Z",
    "updatedAt": "2022-07-01T12:25:05.188Z",
    "instrumentPid": "https://hdl.handle.net/21.12132/3.bf6c0c3926c54dbb",
    "tags": [],
    "siteId": "norunda",
    "instrumentId": "rpg-fmcw-94",
    "site": {
      "id": "norunda",
      "humanReadableName": "Norunda",
      "type": [
        "cloudnet"
      ],
      "latitude": 60.086,
      "longitude": 17.479,
      "altitude": 46,
      "gaw": "NOR",
      "dvasId": "NOR",
      "actrisId": null,
      "country": "Sweden",
      "countryCode": "SE",
      "countrySubdivisionCode": null
    },
    "instrument": {
      "id": "rpg-fmcw-94",
      "type": "radar",
      "humanReadableName": "RPG-Radiometer Physics RPG-FMCW-94 cloud radar",
      "allowedTags": []
    },
    "downloadUrl": "https://cloudnet.fmi.fi/api/download/raw/890c7130-8cf0-44a2-b5bc-75fcec344d97/220701_050008_P09_ZEN.LV0"
  }
  ...
]

GET /api/raw-model-filesModelUpload[]

Query the metadata of uploaded model files. Query parameters are as above, with the following exception: Instead of the instrument parameter, it is possible to filter the results with the model parameter, which takes Model id.

Example query:

GET https://cloudnet.fmi.fi/api/raw-model-files?site=norunda&date=2021-09-01

Response body:

[
  {
    "uuid": "375e32c1-d30c-4265-98ff-d07623b0c524",
    "checksum": "f9c59e2db978a2d2b6cb55d3af342b80",
    "filename": "20210904_norunda_ecmwf.nc",
    "measurementDate": "2021-09-04",
    "size": "501500",
    "status": "processed",
    "createdAt": "2021-09-04T18:37:02.852Z",
    "updatedAt": "2021-09-07T18:37:00.945Z",
    "site": {
      "id": "norunda",
      "humanReadableName": "Norunda",
      "type": [
        "cloudnet"
      ],
      "latitude": 60.086,
      "longitude": 17.479,
      "altitude": 46,
      "gaw": "NOR",
      "dvasId": "NOR",
      "country": "Sweden"
    },
    "model": {
      "id": "ecmwf",
      "optimumOrder": 0
    },
    "downloadUrl": "https://cloudnet.fmi.fi/api/download/raw/375e32c1-d30c-4265-98ff-d07623b0c524/20210904_norunda_ecmwf.nc"
  },
  ...
]

Examples

The following examples use the curl and jq applications. They can be installed from the package repositories of most UNIX-based systems.

Multiple sites and products as parameters

Fetch all categorize and classification products from Norunda and Granada:

curl "https://cloudnet.fmi.fi/api/files?site=norunda&site=granada&product=categorize&product=classification"

Fetching all versions

Fetch all versions of the classification product from Granada on 2020-05-20:

curl "https://cloudnet.fmi.fi/api/files?site=granada&product=classification&date=2020-05-20&allVersions"

Model data

Fetch the optimum model file from Bucharest on 2020-12-10:

curl "https://cloudnet.fmi.fi/api/model-files?site=bucharest&date=2020-12-10"

Fetch all available models:

curl "https://cloudnet.fmi.fi/api/model-files?site=bucharest&date=2020-12-10&allModels"

Fetch only the gdas1 model:

curl "https://cloudnet.fmi.fi/api/model-files?site=bucharest&date=2020-12-10&model=gdas1"

Using the API to download all data objects matching a criteria

Download all data measured on 1. October 2020, saving them to the current working directory:

curl "https://cloudnet.fmi.fi/api/files?date=2020-10-01" | jq '.[]["downloadUrl"]' | xargs -n1 curl -O

That is, get the filtered list of file metadata, pick the downloadUrl properties from each of the File objects and pass them on to curl again to download.

Python example

Download one day of classification data from all sites using the requests package:

import requests

url = 'https://cloudnet.fmi.fi/api/files'
payload = {
    'date': '2020-10-01',
    'product': 'classification'
}
metadata = requests.get(url, payload).json()
for row in metadata:
    res = requests.get(row['downloadUrl'])
    with open(row['filename'], 'wb') as f:
        f.write(res.content)

Notes

The API provides gzip compression of responses where applicable. We recommend using the compression for large queries to reduce transmitted response size. The compression can be enabled effortlessly in curl with the --compressed switch.

Errors

The API responds to errors with the appropriate HTTP status code and an Error object, which has the properties:

Example query:

GET https://cloudnet.fmi.fi/api/files?product=sausage

Response body:

{
  "status": 404,
  "errors": ["One or more of the specified products were not found"]
}