Using Duke's AI Gateway on NCShare
This guide demonstrates querying the OpenAI-compatible API on Duke's AI Gateway from NCShare.
The code for this example is available at: https://github.com/NCShare/examples/tree/main/Using-Dukes-AI-Gateway-on-NCShare
Duke University's AI Gateway provides a central endpoint for various AI models through an OpenAI-compatible API. This allows users to leverage their tokens for research workflows. This tutorial is based on Drew Stinnett's guide, Getting Started with Duke's AI Gateway: A Developer's Guide. Users should refer to it for more detailed information.
Unlike the vLLM and Ollama examples, which run a model on NCShare's own H200 GPUs, this workflow sends prompts to models hosted outside the cluster. Your job therefore needs no GPU, only outbound network access, which NCShare compute nodes have.
Duke credentials required
The AI Gateway is a Duke University service, so a Duke NetID is needed to obtain a token. Users from other participating institutions should contact their institutional representative about equivalent access, or use the vLLM and Ollama examples to run models locally on NCShare instead.
Before you start, you need a Duke AI Gateway token (LITELLM_TOKEN). Get it from the AI Dashboard. You should also have a Python environment available (see Conda / Python Install if you do not).
Warning
If you will be performing computationally intensive inference tasks, ensure that you are on a compute node requested through a Slurm interactive session or submit as a Slurm batch job.
Initial setup
Create a working directory in your /work space,
Save your LITELLM_TOKEN in a .env file in this directory and set permissions so that it's only user-accessible,
echo 'LITELLM_TOKEN="your_api_token"' > /work/${USER}/openai-gateway-example/.env
chmod 600 /work/${USER}/openai-gateway-example/.env
Important!
Your token is a credential. Keep it in the .env file, never paste it into a script that you commit to a repository, and do not share the file with others. /work is shared network storage, so the chmod 600 above matters.
Create and activate a Python environment,
Install dependencies,
openai is the API client library and python-dotenv loads secrets from .env.
Listing available models
Use the script, list_models.sh to query the available models,
#!/usr/bin/env bash
#
# A script to query all available models through Duke's AI Gateway.
# Courtesy of Drew Stinnett:https://ai.colab.duke.edu/colab-ai-blog/all-blogs/getting-started-with-dukes-ai-gateway-a-developers-guide
#
# Usage:
# Get your API token from: https://dashboard.ai.duke.edu/api-keys
# Set your API token in a .env file in this directory with the following content:
# LITELLM_TOKEN="your_api_token_here"
# or export the environment variable directly in your shell. Then run,
# ./list_models.sh
set -e
# This is the base URL for all operations
LITELLM_URL="https://litellm.oit.duke.edu/v1"
# Load environment variables from .env file if it exists
if [[ -f ".env" ]]; then
set -a
source .env
set +a
fi
if [[ -z "$LITELLM_TOKEN" ]]; then
echo "Error: LITELLM_TOKEN is not set. Please set it to your LiteLLM API token." 1>&2
exit 1
fi
# Query the API to list all available models
echo "Listing all models available in LiteLLM..."
curl -X GET "${LITELLM_URL}/models" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${LITELLM_TOKEN}" | jq -r .data[].id | sort
Make it executable and run with,
Running AI Prompts
Create openai_example.py with the following content,
#!/usr/bin/env python
#
# A Python script using the OpenAI-compatible API on Duke's AI Gateway to generate a response based on an input prompt.
# Courtesy of Drew Stinnett:https://ai.colab.duke.edu/colab-ai-blog/all-blogs/getting-started-with-dukes-ai-gateway-a-developers-guide
#
# Usage:
# Get your API token from: https://dashboard.ai.duke.edu/api-keys
# Set your API token in a .env file in this directory with the following content:
# LITELLM_TOKEN="your_api_token_here"
# or export the environment variable directly in your shell. Then run,
# ./openai_example.py "Your prompt here"
import os
import sys
from openai import OpenAI
from dotenv import load_dotenv
# Configuration
MODEL = "gpt-5.4"
INSTRUCTIONS = "You are a helpful assistant here to demo the power of AI."
def main():
# Local .env file content
load_dotenv()
# Input arguments
if len(sys.argv) < 2:
print("Usage: ./openai_example.py <prompt>")
sys.exit(1)
token = os.getenv("LITELLM_TOKEN")
if not token:
print("Please set the LITELLM_TOKEN environment variable.")
sys.exit(1)
prompt = sys.argv[1]
# Connect to the OpenAI API
client = OpenAI(
api_key=token,
base_url="https://litellm.oit.duke.edu/v1",
)
response = client.responses.create(
model=MODEL,
instructions=INSTRUCTIONS,
input=prompt,
)
print(response.output[0].content[0].text)
if __name__ == "__main__":
main()
Make it executable,
Run the script with a prompt string,
You should see a text response printed to the terminal.
Run as a Slurm batch job (optional)
For repeated prompts or scheduled runs, submit a batch job. This workload is network-bound rather than compute-bound, so the common partition and a modest resource request are enough; no GPU is needed.
Create openai_batch.slurm,
#!/bin/bash
#SBATCH -J openai-gateway-demo
#SBATCH -p common
#SBATCH -t 00:05:00
#SBATCH --mem=2G
#SBATCH -c 4
#SBATCH -o openai-gateway-demo-%j.out
cd $SLURM_SUBMIT_DIR
# conda activate is undefined in a batch shell until conda.sh has been sourced
source "$(conda info --base)/etc/profile.d/conda.sh"
conda activate openai-gateway
./openai_example.py "Why is the sky blue?"
Submit with,
Check status with squeue -u ${USER} and inspect output in openai-gateway-demo-<jobid>.out.
Note
The script reads .env from the directory it runs in, which is why it starts with cd $SLURM_SUBMIT_DIR. Submit the job from /work/${USER}/openai-gateway-example so that the token is found.
Customize behavior
You can edit these values in openai_example.py:
MODEL: choose a model available to your Duke AI Gateway project.INSTRUCTIONS: define style, tone, or output format.
Example,