Skip to content

Using the Slurm REST API on NCShare

This guide explains how to talk to the NCShare Slurm scheduler over its REST API (slurmrestd). You will authenticate with a JSON Web Token (JWT), inspect partitions and nodes, submit and cancel jobs, and query accounting data, all with curl and jq instead of sbatch, squeue, and scancel.

The Slurm commands you normally use on NCShare (sinfo, squeue, sbatch, scancel, sacct) are clients over the scheduler. NCShare also exposes the same functionality through slurmrestd, Slurm's REST daemon. This is useful when you want to drive the cluster from something that is not a login shell, for example a web application, a workflow engine, a CI pipeline, a Jupyter notebook running elsewhere, or an AI agent.

The NCShare Slurm REST endpoint is,

http://scheduler-01:6820

The endpoint is plain HTTP on the cluster's private network

scheduler-01 resolves to a private address that is reachable only from inside the cluster, and slurmrestd there speaks HTTP, not HTTPS. Requesting https://scheduler-01:6820 fails with an SSL error rather than connecting. Because there is no TLS, your JWT travels the wire unencrypted, so do not call this endpoint from off-cluster over any path other than an SSH tunnel, which encrypts the whole hop for you.

Treat the token as a password

The REST API acts on your behalf with your Slurm privileges. The JWT you generate below is a bearer credential, meaning anyone who holds it can submit and cancel jobs as you until it expires. Never commit a token, paste it into a shared log or a chat window, or store it in a script.

Where you can call the API from

Because scheduler-01 is on the cluster's internal network, there are two supported ways to reach it,

From How
An NCShare login or compute node Call http://scheduler-01:6820 directly.
Your laptop, or anywhere off-cluster Forward the port through a login node with SSH and call http://localhost:6820. See Connecting through an SSH tunnel.

This differs from clusters that publish slurmrestd on a routable hostname. There is no NCShare address you can point curl at from an arbitrary network.

Prerequisites

You need curl, jq, and ssh, all of which are present on the cluster,

curl --version
jq --version
ssh -V

You also need SSH access to NCShare, because the JWT is minted with scontrol on the cluster.

Set your NCShare UID once so the rest of the commands can reuse it,

export SLURM_USER="your_ncshare_uid"

Optionally, confirm from a login node that JWT authentication is enabled,

scontrol show config | grep -E 'AuthAltTypes|AuthAltParameters'

which should report,

AuthAltTypes            = auth/jwt
AuthAltParameters       = jwt_key=/var/spool/slurm/statesave/jwt_hs256.key

Generate a Slurm JWT

From your local terminal, ask an NCShare login node to mint a token,

ssh "${SLURM_USER}@login.ncshare.org" 'scontrol token lifespan=3600'

This prints a single line,

SLURM_JWT=eyJhbGciOiJIUzI1NiIs...

lifespan is in seconds. Keep it short, request a new token when it expires rather than minting a long-lived one. Export the value in the terminal where you will call the API,

export SLURM_JWT='paste-token-here'

Set connection variables

export SLURM_REST_URL="http://scheduler-01:6820"
export SLURM_REST_VERSION="v0.0.42"

Every authenticated request carries two headers,

X-SLURM-USER-NAME:  your NCShare UID
X-SLURM-USER-TOKEN: your JWT

which in curl looks like,

-H "X-SLURM-USER-NAME: $SLURM_USER" \
-H "X-SLURM-USER-TOKEN: $SLURM_JWT"

NCShare runs Slurm 24.11.5 and serves data parser versions v0.0.40 through v0.0.42. The examples below use v0.0.42, the newest available. Field names and response shapes change between versions, so pin a version explicitly rather than assuming a particular one is present.

Test the connection

curl -sS \
  -H "X-SLURM-USER-NAME: $SLURM_USER" \
  -H "X-SLURM-USER-TOKEN: $SLURM_JWT" \
  "$SLURM_REST_URL/slurm/$SLURM_REST_VERSION/ping/" | jq '.pings'

A working connection returns,

[
  {
    "hostname": "scheduler-01",
    "pinged": "UP",
    "responding": true,
    "latency": 1607,
    "mode": "primary",
    "primary": true
  }
]

Note that there is no -k flag anywhere in this guide. -k disables TLS certificate verification, and this endpoint does not use TLS at all.

Explore the API

The endpoint publishes its own OpenAPI schema, which is the authoritative reference for every field used below.

curl -sS \
  -H "X-SLURM-USER-NAME: $SLURM_USER" \
  -H "X-SLURM-USER-TOKEN: $SLURM_JWT" \
  "$SLURM_REST_URL/openapi/v3" | jq '.info'

{
  "title": "Slurm REST API",
  "description": "API to access and control Slurm",
  "version": "Slurm-24.11.5&openapi/slurmdbd&openapi/slurmctld"
}

List the paths available for your chosen version,

curl -sS \
  -H "X-SLURM-USER-NAME: $SLURM_USER" \
  -H "X-SLURM-USER-TOKEN: $SLURM_JWT" \
  "$SLURM_REST_URL/openapi/v3" \
  | jq -r '.paths | keys[]' \
  | grep "/slurm/$SLURM_REST_VERSION"

Inspect partitions

The equivalent of sinfo,

curl -sS \
  -H "X-SLURM-USER-NAME: $SLURM_USER" \
  -H "X-SLURM-USER-TOKEN: $SLURM_JWT" \
  "$SLURM_REST_URL/slurm/$SLURM_REST_VERSION/partitions/" \
  | jq '.partitions[] | {name, state: .partition.state, nodes: .nodes.total, cpus: .cpus.total}'

{"name": "common",          "state": ["UP"], "nodes": 7, "cpus": 896}
{"name": "interactive",     "state": ["UP"], "nodes": 7, "cpus": 896}
{"name": "gpu",             "state": ["UP"], "nodes": 4, "cpus": 768}
{"name": "interactive-gpu", "state": ["UP"], "nodes": 4, "cpus": 768}
{"name": "gpu-hp",          "state": ["UP"], "nodes": 4, "cpus": 768}

For names only,

curl -sS \
  -H "X-SLURM-USER-NAME: $SLURM_USER" \
  -H "X-SLURM-USER-TOKEN: $SLURM_JWT" \
  "$SLURM_REST_URL/slurm/$SLURM_REST_VERSION/partitions/" \
  | jq -r '.partitions[].name'

See NCShare Cluster Partitions for which of these you can submit to.

Inspect nodes

The equivalent of sinfo -N,

curl -sS \
  -H "X-SLURM-USER-NAME: $SLURM_USER" \
  -H "X-SLURM-USER-TOKEN: $SLURM_JWT" \
  "$SLURM_REST_URL/slurm/$SLURM_REST_VERSION/nodes/" \
  | jq '.nodes[] | {name, partitions, state, cpus, real_memory}'

Filter for GPU nodes by matching on the generic resource string,

curl -sS \
  -H "X-SLURM-USER-NAME: $SLURM_USER" \
  -H "X-SLURM-USER-TOKEN: $SLURM_JWT" \
  "$SLURM_REST_URL/slurm/$SLURM_REST_VERSION/nodes/" \
  | jq '.nodes[] | select((.gres // "") | contains("gpu")) | {name, state, gres}'

{"name": "compute-gpu-01", "state": ["ALLOCATED"],          "gres": "gpu:h200:8(S:0-1)"}
{"name": "compute-gpu-02", "state": ["MIXED", "PLANNED"],   "gres": "gpu:h200:8(S:0-1)"}

Each GPU node carries eight H200s. See the GPU Guide for more.

Look at the queue

The equivalent of squeue,

curl -sS \
  -H "X-SLURM-USER-NAME: $SLURM_USER" \
  -H "X-SLURM-USER-TOKEN: $SLURM_JWT" \
  "$SLURM_REST_URL/slurm/$SLURM_REST_VERSION/jobs/" \
  | jq '.jobs[] | {job_id, name, user_name, partition, job_state, nodes}'

The /jobs/ endpoint has no user filter

Passing jobs/?user=$SLURM_USER returns every job plus the warning Ignoring unknown field "user" of type string. Fetch the full list and filter client-side with jq instead.

curl -sS \
  -H "X-SLURM-USER-NAME: $SLURM_USER" \
  -H "X-SLURM-USER-TOKEN: $SLURM_JWT" \
  "$SLURM_REST_URL/slurm/$SLURM_REST_VERSION/jobs/" \
  | jq --arg user "$SLURM_USER" \
       '.jobs[] | select(.user_name == $user) | {job_id, name, partition, job_state, nodes, reason: .state_reason}'

{"job_id": 717539, "name": "agent-uid", "partition": "interactive", "job_state": ["RUNNING"], "nodes": "compute-08", "reason": "None"}

Returning only jobs updated recently is much cheaper than fetching the whole queue. update_time takes a Unix timestamp,

UPDATE_TIME=$(date -d '1 hour ago' +%s)

UPDATE_TIME=$(date -v-1H +%s)

curl -sS \
  -H "X-SLURM-USER-NAME: $SLURM_USER" \
  -H "X-SLURM-USER-TOKEN: $SLURM_JWT" \
  "$SLURM_REST_URL/slurm/$SLURM_REST_VERSION/jobs/?update_time=$UPDATE_TIME" \
  | jq '.jobs[] | {job_id, name, user_name, job_state}'

Submit a job

Submission is a POST of a JSON body to /slurm/$SLURM_REST_VERSION/job/submit. The body has two parts: script, the batch script as a single string with escaped newlines, and job, the resource request that would otherwise live in #SBATCH directives.

Create a working directory first, since current_working_directory must exist on the cluster before the job starts,

ssh "${SLURM_USER}@login.ncshare.org" 'mkdir -p /work/$USER/slurm_api_test'

Then write the request body,

submit_job.json
{
  "script": "#!/bin/bash\nhostname\ndate\nsleep 30\necho done\n",
  "job": {
    "name": "rest-api-test",
    "partition": "common",
    "tasks": 1,
    "cpus_per_task": 1,
    "memory_per_node": 1024,
    "time_limit": 5,
    "current_working_directory": "/work/your_ncshare_uid/slurm_api_test",
    "standard_output": "/work/your_ncshare_uid/slurm_api_test/rest-api-test-%j.out",
    "standard_error": "/work/your_ncshare_uid/slurm_api_test/rest-api-test-%j.err",
    "environment": [
      "PATH=/usr/local/bin:/usr/bin:/bin"
    ]
  }
}

Note the units, memory_per_node is in MB and time_limit is in minutes. environment is required. A job submitted with an empty environment will fail to launch. There is no account field, because NCShare does not require one; the equivalent sbatch submission takes no -A either.

Submit it and capture the job ID,

JOB_ID=$(curl -sS -X POST \
  -H "Content-Type: application/json" \
  -H "X-SLURM-USER-NAME: $SLURM_USER" \
  -H "X-SLURM-USER-TOKEN: $SLURM_JWT" \
  --data @submit_job.json \
  "$SLURM_REST_URL/slurm/$SLURM_REST_VERSION/job/submit" \
  | jq -r '.job_id')

echo "$JOB_ID"

A successful submission returns the job ID with empty errors and warnings arrays. Always check .errors when a submission does not behave as expected, a rejected job still returns HTTP 200 with the reason in that array.

Requesting GPUs

Generic resources map to the tres_per_node field using Slurm's TRES syntax, so --gres=gpu:h200:1 becomes,

{
  "job": {
    "partition": "gpu",
    "tres_per_node": "gres/gpu:h200:1",
    "cpus_per_task": 1,
    "memory_per_node": 4096,
    "time_limit": 5
  }
}

All NCShare GPU nodes carry H200s, so "gres/gpu:1" works equally well when you do not care to name the model. Access to the gpu, interactive-gpu, and gpu-hp partitions is granted by your institutional representative; see the GPU Guide.

Check a job

The equivalent of scontrol show job <jobid>,

curl -sS \
  -H "X-SLURM-USER-NAME: $SLURM_USER" \
  -H "X-SLURM-USER-TOKEN: $SLURM_JWT" \
  "$SLURM_REST_URL/slurm/$SLURM_REST_VERSION/job/$JOB_ID" \
  | jq '.jobs[0] | {job_id, name, partition, job_state, state_reason, nodes, submit_time, start_time, end_time}'

For polling, the dedicated state endpoint is much lighter,

curl -sS \
  -H "X-SLURM-USER-NAME: $SLURM_USER" \
  -H "X-SLURM-USER-TOKEN: $SLURM_JWT" \
  "$SLURM_REST_URL/slurm/$SLURM_REST_VERSION/jobs/state/?job_id=$JOB_ID" \
  | jq '.jobs'

[
  {
    "job_id": "717553",
    "state": [
      "RUNNING"
    ]
  }
]

state is an array because a job can hold several state flags at once, for example ["CANCELLED", "COMPLETING"].

Cancel a job

The equivalent of scancel <jobid>,

curl -sS -X DELETE \
  -H "X-SLURM-USER-NAME: $SLURM_USER" \
  -H "X-SLURM-USER-TOKEN: $SLURM_JWT" \
  "$SLURM_REST_URL/slurm/$SLURM_REST_VERSION/job/$JOB_ID" \
  | jq '{errors, warnings}'

Then confirm with the state endpoint above.

Query accounting data

Accounting lives under /slurmdb/ rather than /slurm/, and is the equivalent of sacct. Unlike the queue endpoint, it does support server-side filtering.

curl -sS \
  -H "X-SLURM-USER-NAME: $SLURM_USER" \
  -H "X-SLURM-USER-TOKEN: $SLURM_JWT" \
  "$SLURM_REST_URL/slurmdb/$SLURM_REST_VERSION/jobs/?users=$SLURM_USER" \
  | jq '[.jobs[] | {job_id, name, state: .state.current}] | .[0:10]'

[
  {"job_id": 717547, "name": "mpi_ring_test", "state": ["COMPLETED"]},
  {"job_id": 717553, "name": "rest-api-test", "state": ["CANCELLED"]}
]

Details for one finished job,

curl -sS \
  -H "X-SLURM-USER-NAME: $SLURM_USER" \
  -H "X-SLURM-USER-TOKEN: $SLURM_JWT" \
  "$SLURM_REST_URL/slurmdb/$SLURM_REST_VERSION/job/$JOB_ID" | jq

Confirm the accounting database is reachable with,

curl -sS \
  -H "X-SLURM-USER-NAME: $SLURM_USER" \
  -H "X-SLURM-USER-TOKEN: $SLURM_JWT" \
  "$SLURM_REST_URL/slurmdb/$SLURM_REST_VERSION/ping/" | jq '.pings'

A helper function

The header boilerplate gets repetitive. After setting SLURM_USER, SLURM_JWT, and SLURM_REST_URL, define,

slurm_api() {
  local method="$1"
  local path="$2"
  shift 2

  curl -sS -X "$method" \
    -H "X-SLURM-USER-NAME: $SLURM_USER" \
    -H "X-SLURM-USER-TOKEN: $SLURM_JWT" \
    "$@" \
    "$SLURM_REST_URL$path"
}

which reduces the examples above to,

slurm_api GET "/slurm/$SLURM_REST_VERSION/ping/" | jq '.pings'
slurm_api GET "/slurm/$SLURM_REST_VERSION/partitions/" | jq -r '.partitions[].name'
slurm_api GET "/slurm/$SLURM_REST_VERSION/jobs/" | jq '.jobs[] | {job_id, user_name, job_state}'

slurm_api POST "/slurm/$SLURM_REST_VERSION/job/submit" \
  -H "Content-Type: application/json" \
  --data @submit_job.json | jq

slurm_api DELETE "/slurm/$SLURM_REST_VERSION/job/$JOB_ID" | jq

Connecting through an SSH tunnel

scheduler-01 is not routable from outside the cluster, so calling the API from your laptop requires forwarding the port through a login node,

ssh -L 6820:scheduler-01:6820 "${SLURM_USER}@login.ncshare.org"

Leave that session open, and in another local terminal point the API at the forwarded port,

export SLURM_REST_URL="http://localhost:6820"

Everything else, including token generation, is unchanged. The tunnel also solves the encryption problem: although slurmrestd itself speaks plain HTTP, SSH encrypts the entire hop between your laptop and the login node, so your JWT is never exposed on the network in transit.

If port 6820 is already in use locally, forward to a different local port and adjust the URL to match,

ssh -L 16820:scheduler-01:6820 "${SLURM_USER}@login.ncshare.org"
export SLURM_REST_URL="http://localhost:16820"

Troubleshooting

curl: (6) Could not resolve host: scheduler-01

You are not on the cluster. scheduler-01 only resolves on the NCShare internal network. Use an SSH tunnel and point SLURM_REST_URL at http://localhost:6820.

curl: (35) OpenSSL ... wrong version number

You used https://. This endpoint speaks plain HTTP. Use http://scheduler-01:6820.

curl: Failed to connect to login.ncshare.org port 6820

The REST service does not run on the login hosts. It runs on scheduler-01, which the login nodes can reach on your behalf through a tunnel.

401 Authentication failure

The token is missing, expired, malformed, or does not match X-SLURM-USER-NAME. Mint a fresh one with scontrol token. A request with no headers at all also returns 401, which is a quick way to confirm the endpoint is up.

Ignoring unknown field "user"

/slurm/<version>/jobs/ takes no user query parameter. Filter with jq, or use /slurmdb/<version>/jobs/?users= for accounting records.

Submission returns HTTP 200 but no job_id

Read the .errors array. QoS, partition, and memory validation failures all report there. Match the partition, qos, and resource fields to what your normal sbatch submission uses.

Job submits but fails immediately

Check that current_working_directory exists and that environment is populated. Both are common causes of a job that starts and exits at once.

Quick reference

# Connection (on-cluster)
export SLURM_USER="your_ncshare_uid"
export SLURM_REST_URL="http://scheduler-01:6820"
export SLURM_REST_VERSION="v0.0.42"

# Connection (from a laptop, in a second terminal)
# ssh -L 6820:scheduler-01:6820 "${SLURM_USER}@login.ncshare.org"
# export SLURM_REST_URL="http://localhost:6820"

# Token
ssh "${SLURM_USER}@login.ncshare.org" 'scontrol token lifespan=3600'
export SLURM_JWT='paste-token-here'

# Ping
curl -sS -H "X-SLURM-USER-NAME: $SLURM_USER" -H "X-SLURM-USER-TOKEN: $SLURM_JWT" \
  "$SLURM_REST_URL/slurm/$SLURM_REST_VERSION/ping/" | jq '.pings'

# Your jobs in the queue
curl -sS -H "X-SLURM-USER-NAME: $SLURM_USER" -H "X-SLURM-USER-TOKEN: $SLURM_JWT" \
  "$SLURM_REST_URL/slurm/$SLURM_REST_VERSION/jobs/" \
  | jq --arg user "$SLURM_USER" '.jobs[] | select(.user_name == $user) | {job_id, name, job_state}'

# Job state
curl -sS -H "X-SLURM-USER-NAME: $SLURM_USER" -H "X-SLURM-USER-TOKEN: $SLURM_JWT" \
  "$SLURM_REST_URL/slurm/$SLURM_REST_VERSION/jobs/state/?job_id=$JOB_ID" | jq '.jobs'

# Submit
curl -sS -X POST -H "Content-Type: application/json" \
  -H "X-SLURM-USER-NAME: $SLURM_USER" -H "X-SLURM-USER-TOKEN: $SLURM_JWT" \
  --data @submit_job.json \
  "$SLURM_REST_URL/slurm/$SLURM_REST_VERSION/job/submit" | jq

# Cancel
curl -sS -X DELETE -H "X-SLURM-USER-NAME: $SLURM_USER" -H "X-SLURM-USER-TOKEN: $SLURM_JWT" \
  "$SLURM_REST_URL/slurm/$SLURM_REST_VERSION/job/$JOB_ID" | jq

# Accounting
curl -sS -H "X-SLURM-USER-NAME: $SLURM_USER" -H "X-SLURM-USER-TOKEN: $SLURM_JWT" \
  "$SLURM_REST_URL/slurmdb/$SLURM_REST_VERSION/jobs/?users=$SLURM_USER" \
  | jq '[.jobs[] | {job_id, name, state: .state.current}] | .[0:10]'

For the command-line equivalents of everything above, see Cluster Computing.

Comments