The MDEP Data API

The SMDH data API user guide is intended for organisations and individuals who want to upload their batch data into SMDH data lake using an API instead of the UI e.g. for scripting/crond purposes. Allows all CRUD operations on SMDH lake data.

The underlying API is the CKAN API. For more information, refer to the CKAN API documentation. This guide aims to provide a simplified overview of the important data management API endpoints. The Data API Reference

The SMDH data API is an RPC-style API that exposes a set of actions which can be applied to resources.

BASE URL
https://mdep.smdh.uk/api/3/action/

You may also use the staging API for testing purposes.

Staging BASE URL
https://stagingmdep.smdh.uk/api/3/action/

You can use a POST request to invoke an action, the body of the request should be a JSON object containing the parameters of the action. Some of the actions also support calling them through a GET request, in this case the parameters of the action should be sent as URL parameters.

The API response is a JSON object containing the following keys:

"help": the documentation string for the function you called.
"success": true or false. The API will always respond with a 200 OK status code (except in rare cases) even if there was an error, as such you should always check the success key.
"result": the result of calling the action, which depends on the action itself. Only present if the action was successful.
"error": details of the error if there was an error. This is an object containing a "__type" and a "message" properties. The examples below assume a successful response and omit the "help" and "success" properties. The following is an example of an error response:
```
{
    "help": "<https://mdep.smdh.uk/api/3/action/help_show?name=package_create>",
    "success": false,
    "error": {
        "message": "Access denied",
        "__type": "Authorization Error"
    }
}
```

Authentication

Authentication involves sending an API token as a request Authorization header.

Danger

Make sure to safeguard your API tokens as they hold the same privileges as your user account. Avoid exposing your secret API token in places that are accessible to the public, such as GitHub or client-side code.

$ curl -X GET <https://mdep.smdh.uk/api/3/action/dashboard_activity_list> \
    -H "Authorization: <api_token>"
{
  "help": "<https://mdep.smdh.uk/api/3/action/help_show?name=dashboard_activity_list>",
  "success": true,
  "result": ---snip---
}

Replace <api_token> with your API token.

Creating an API Token

To create an API token follow the steps below:

Visit https://mdep.smdh.uk (or https://stagingmdep.smdh.uk for creating an API token that works with the staging API) and go to your profile by clicking on your username in the top bar at the right.

From there, navigate to the API Tokens tab, enter a descriptive name for your API token and click on the “Create API Token” button. We recommend that you name your token according to your use case.

List Datasets

Endpoint

GET/POST /package_list

Parameters

limit (int): if given, the list of datasets will be broken into pages of at most limit datasets per page and only one page will be returned at a time (optional)

offset (int): when limit is given, the offset to start returning packages from

Return Type

list of strings representing package names

Authorization

optional, required for listing private packages

Example

$ curl -X GET <https://mdep.smdh.uk/api/3/action/package_list?limit=3>
{
    "result": [
        "package-1-name",
        "package-2-name"
    ]
}

Get A Dataset

Endpoint

GET/POST /package_show

Parameters

id (string): the dataset id or name

Return Type

package object

Authorization

optional, required for showing a private package

Example

$ curl -X GET <https://mdep.smdh.uk/api/3/action/package_show?id=package-1-name>
{
    "result": {
        "creator_user_id": "12345678-9abc-defg-hijk-lmnopqrstuvw",
        "id": "12345678-9abc-defg-hijk-lmnopqrstuvw",
        "name": "org--package-1-name",
        "organization": { ... },
        "owner_org": "12345678-9abc-defg-hijk-lmnopqrstuvw",
        "resources": [ ... ],
        ...
    }
}

Create A Dataset

Endpoint

POST /package_create

Parameters

name (string): the name of the new dataset, must be between 2 and 100 characters long and contain only lowercase alphanumeric characters, - and_, e.g. 'my_datasets'

owner_org (string): the id/name of the dataset’s owning organisation

title (string): the title of the dataset (optional, default: same as name)

private (bool): If true creates a private dataset

author (string): the name of the dataset’s author (optional)

author_email (string): email address of the author (optional)

maintainer (string): the name of the dataset’s maintainer (optional)

maintainer_email (string): email address of the maintainer (optional)

notes (string): a description of the dataset (optional)

url (string): a URL for the dataset’s source (optional)

version (string): no longer than 100 characters (optional)

resources (list of resource objects): the dataset’s resources, see resource_create for the format of resource objects (optional)

Return Type

newly created package object

Authorization

required

Example

$ curl -X POST <https://mdep.smdh.uk/api/3/action/package_create> \
    -H "Content-Type: application/json" \
    -d '{"name": "my_dataset", "owner_org": "org"}'
{
    "result": {
        "creator_user_id": "12345678-9abc-defg-hijk-lmnopqrstuvw",
        "id": "12345678-9abc-defg-hijk-lmnopqrstuvw",
        "name": "org--my_dataset",
        "organization": { ... },
        "owner_org": "12345678-9abc-defg-hijk-lmnopqrstuvw",
        "resources": [],
        ...
    }
}

Update A Dataset

Warning

package_update methods may delete parameters not explicitly provided in the request body. If you want to edit only a specific attribute use package_patch instead. It's recommended to call package_show and update the package depending on the result of that call if you are going to use the package_update action.

Endpoint

POST /package_update OR POST /package_patch

Parameters

id (string): the name or id of the dataset to update

All the parameters supported by the package_create action

Return Type

The updated package object

Authorization

required

Example

$ curl -X POST <https://mdep.smdh.uk/api/3/action/package_patch> \
    -H "Content-Type: application/json" \
    -d '{"id": "org--my_dataset", "title": "My Dataset"}'
{
    "result": {
        "creator_user_id": "12345678-9abc-defg-hijk-lmnopqrstuvw",
        "id": "12345678-9abc-defg-hijk-lmnopqrstuvw",
        "name": "org--my_dataset",
        "organization": { ... },
        "owner_org": "12345678-9abc-defg-hijk-lmnopqrstuvw",
        "resources": [],
        ...
    }
}

Get A Resource

Endpoint

GET/POST /resource_show

Parameters

id (string): the resource id

Return Type

resource object

Authorization

optional, required for showing a private resource

Example

$ curl -X GET <https://mdep.smdh.uk/api/3/action/resource_show?id=12345678-9abc-defg-hijk-lmnopqrstuvw>
{
    "result": {
        "description": "Resource description",
        "mimetype": "image/png",
        "name": "image.png",
        "package_id": "12345678-9abc-defg-hijk-lmnopqrstuvw",
        "size": 123456,
        "url": "<https://mdep.smdh.uk/dataset/12345678-9abc-defg-hijk-lmnopqrstuvw/resource/12345678-9abc-defg-hijk-lmnopqrstuvw/download/image.png>",
        "url_type": "upload",
        ...
    }
}

Create A Resource

Bug

Because of a known issue with the API upload, creating a resource and uploading a file must be done in two steps. First, create the resource with a placeholder upload file. Then, use the resource_patch endpoint to upload the actual file, providing the resource ID and the upload file as request fields.

A resource could either be a url or a file. For URL resources, include the url parameter, otherwise use the file parameter. The example below shows a file upload request.

Endpoint

POST /resource_create

Parameters

package_id (string): id of package that the resource will be added to.

url (string): URL of resource (required for URL resources)

description (string): (optional)

format (string): (optional)

hash (string): (optional)

name (string): (optional)

mimetype (string): (optional)

upload (FieldStorage): needs multipart/form-data (optional)

Return Type

newly created resource object

Authorization

required

Example

$ curl -X POST <https://mdep.smdh.uk/api/3/action/resource_create> \
    -F "upload=@/path/to/image.png" \
    -F "package_id=12345678-9abc-defg-hijk-lmnopqrstuvw" \
    -F "name=resource.png"

Update A Resource

Warning

resource_update methods may delete parameters not explicitly provided in the request body. If you want to edit only a specific attribute use resource_patch instead.

Endpoint

POST /resource_update OR POST /resource_patch

Parameters

id (string): id of resource to update

All the parameters supported by the resource_create action

Return Type

updated resource object

Authorization

required

Example

$ curl -X POST <https://mdep.smdh.uk/api/3/action/resource_patch> \
    -F "upload=@/path/to/new_image.png" \
    -F "id=12345678-9abc-defg-hijk-lmnopqrstuvw" \

Create A Public Resource Download Link

Create a temporary public resource download link that is valid for a specific period of time.

Warning

resource_update methods may delete parameters not explicitly provided in the request body. If you want to edit only a specific attribute use resource_patch instead.

Endpoint

POST /resource_update OR POST /resource_patch

Parameters

id (string): id of resource to generate a link for

expires_in (int): number of seconds for the link to expire

Return Type

JSON object containing the key "url" which contains the string value of the download link URL

Authorization

required

Example

$ curl -X POST <https://mdep.smdh.uk/api/3/action/resource_create_presigned_url> \
    -H "Content-Type: application/json" \
    -d '{"id": "12345678-9abc-defg-hijk-lmnopqrstuvw", "expires_in": 86400}'
{
    "result": {
        "url": "https://<public download link>"
    }
}