Running Inference on Local LLMs with a vLLM Server
This guide explains how to run inference on local LLMs with a vLLM server using the H200 GPUs on NCShare. The vLLM server supports many concurrent client requests with low latency, making it well suited for HPC environments. Additionally, it provides authentication through a secure key so that only authorized users can access the server.
The code for this example is available at: https://github.com/NCShare/examples/tree/main/Running-Inference-on-Local-LLMs-with-a-vLLM-Server.
As large language models (LLMs) continue to grow in popularity, efficient and scalable inference tools are becoming increasingly important in HPC environments. While Ollama is a strong option for smaller-scale inference, vLLM is better suited for larger-scale HPC workloads allowing for higher throughput, lower latency, and more concurrent requests. For a detailed performance comparison, see Ollama vs. vLLM: A deep dive into performance benchmarking.
This guide walks you through setting up and running inference on local LLMs with a vLLM server on the NCShare H200 GPUs. For a guide on running inference directly with the vLLM Python API without a server, see Running Inference on Local LLMs with vLLM. The server-based approach allows for more concurrent requests and enables authentication, while the direct Python API approach is simpler for single-user workloads.
Initial setup
Everything in this section is done once. Once the environment is built you will not repeat any of it; later sessions start at Serving the LLM model.
Assuming you have a Python environment available (see Conda / Python Install if you do not), first create a new environment and activate it.
Then install vLLM and the OpenAI Python API library,
Providing the CUDA runtime
NCShare compute nodes do not provide a system-wide CUDA toolkit, so vLLM has no CUDA_HOME to point at and will fail to start. Supplying it is a one-time step per environment, described in Providing the CUDA runtime in the direct-API guide. Follow that section now before continuing.
If you already built vllm-env for that guide, the activation hook is in place and there is nothing more to do here.
Serving the LLM model
We will serve the Qwen/Qwen2-7B-Instruct model from Hugging Face on a single NVIDIA H200 GPU. We will first need to set up some environment variables to ensure that vLLM can access the model directory and connect to Hugging Face to download it if it is not already cached. The working directory, token file, and cache location below are set up once; the scripts are what you run each session.
First, create a directory in your work directory for this example,
Within this directory, create a .env file to store the environment variables,
echo 'HF_TOKEN=your_hugging_face_api_token' > /work/${USER}/vLLM-server/.env
chmod 600 /work/${USER}/vLLM-server/.env
The 600 permission ensures that only the owner can read and write the file, keeping your Hugging Face API token secure.
Next, add the following to your ~/.bashrc to keep vLLM's caches off your home directory,
export HF_HOME="/work/${USER}/.huggingface"
export FLASHINFER_WORKSPACE_BASE="/work/${USER}"
export VLLM_CACHE_ROOT="/work/${USER}/.cache/vllm"
HF_HOME covers downloaded model weights only. vLLM separately JIT-compiles FlashInfer kernels into $HOME/.cache/flashinfer and writes a torch.compile cache to $HOME/.cache/vllm, so all three need to point at /work to avoid filling your home quota; the symptom otherwise is OSError: [Errno 28] No space left on device. Note that FLASHINFER_WORKSPACE_BASE is a base directory, FlashInfer appends .cache/flashinfer to it itself.
Save the following bash script in the working directory as vllm_server.sh. This script will launch the vLLM server with the specified model and configuration.
#!/bin/bash
# vLLM Server Startup Script
# Usage:
# 1. Set HF_HOME to control where Hugging Face cache is stored.
# export HF_HOME=/path/to/hf_cache
# 2. Set the HF_TOKEN environmental variable in a .env file in the root of the directory.
# 3. Run the script:
# ./vllm_server.sh [ADDITIONAL_VLLM_ARGS...]
#Example:
# ./vllm_server.sh --max-model-len 8192
set -euo pipefail
# Configuration
PORT=8000
MODEL_NAME="Qwen/Qwen2-7B-Instruct"
SERVED_MODEL_NAME="local-vllm"
API_KEY="your-secret-key"
# Setup Hugging Face parameters
export HF_HOME="${HF_HOME:-/work/${USER}/.huggingface}"
# vLLM's JIT caches default to $HOME; keep them off the home quota too
export FLASHINFER_WORKSPACE_BASE="${FLASHINFER_WORKSPACE_BASE:-/work/${USER}}"
export VLLM_CACHE_ROOT="${VLLM_CACHE_ROOT:-/work/${USER}/.cache/vllm}"
if [[ -f ".env" ]]; then
set -a
source .env
set +a
fi
# Determine tensor parallel size based on CUDA_VISIBLE_DEVICES
if [[ -n "${CUDA_VISIBLE_DEVICES:-}" ]] && [[ "$CUDA_VISIBLE_DEVICES" != "NoDevFiles" ]]; then
IFS=',' read -r -a gpu_ids <<< "$CUDA_VISIBLE_DEVICES"
TENSOR_PARALLEL_SIZE="${#gpu_ids[@]}"
else
TENSOR_PARALLEL_SIZE="1"
fi
# Launch vLLM server
vllm serve \
--host 0.0.0.0 \
--port "$PORT" \
--model "$MODEL_NAME" \
--served-model-name "$SERVED_MODEL_NAME" \
--tensor-parallel-size "$TENSOR_PARALLEL_SIZE" \
--api-key "$API_KEY" \
--trust-remote-code \
"$@" \
> "vllm_server.log" 2>&1 &
# Wait for the vLLM server to become ready by polling its models endpoint,
# and exit if the server process dies during startup.
SERVER_PID=$!
until curl -fsS -H "Authorization: Bearer ${API_KEY}" "http://127.0.0.1:${PORT}/v1/models" >/dev/null 2>&1; do
if ! kill -0 "$SERVER_PID" >/dev/null 2>&1; then
echo "Error: vLLM exited during startup. See vllm_server.log"
exit 1
fi
sleep 2
done
NODE_FQDN="$(hostname -f)"
echo ""
echo "vLLM is serving at: http://${NODE_FQDN}:${PORT}"
echo "Model: ${MODEL_NAME}"
echo "Model Alias: ${SERVED_MODEL_NAME}"
echo "Tensor parallel: ${TENSOR_PARALLEL_SIZE}x GPU"
if (( $# > 0 )); then
echo "Extra vLLM args: $*"
fi
echo ""
echo "Export on client shell:"
echo "export VLLM_HOST=http://${NODE_FQDN}:${PORT}"
echo "export VLLM_API_KEY=${API_KEY}"
echo ""
echo "Stop server:"
echo "kill $SERVER_PID && pkill -f VLLM::"
Feel free to experiment with different models by changing the MODEL_NAME variable. The SERVED_MODEL_NAME is an alias to the model and can be used to connect the server to an agentic AI coding tool like OpenCode. We cover this in the guide, Running OpenCode Agentic AI workflows with local LLM models on NCShare with a vLLM Server. The API_KEY is used to authenticate clients that connect to the server, ensuring that only authorized users can access it.
Let's request a Slurm interactive session on a single H200 GPU to serve the model. You may also submit the script as a batch job if you prefer.
Warning
Jobs on the gpu partition may be pre-empted (cancelled and requeued) to accommodate higher-priority jobs, which will take your server down mid-session. For short interactive runs of up to an hour, the interactive-gpu partition is not pre-empted and is a better choice. See the GPU Guide for the differences between the GPU partitions.
Once your allocation is ready, activate the conda environment we created earlier and launch the server,
You should see an output similar to the one below,
vLLM is serving at: http://compute-gpu-03:8000
Model: Qwen/Qwen2-7B-Instruct
Model Alias: local-vllm
Tensor parallel: 1x GPU
Export on client shell:
export VLLM_HOST=http://compute-gpu-03:8000
export VLLM_API_KEY=your-secret-key
Stop server:
kill 567059 && pkill -f VLLM::
Copy the export commands from the output and run them in your client shell to set the VLLM_HOST and VLLM_API_KEY environment variables. These will allow you to connect to the vLLM server from your client applications. The client can be a regular non-GPU node as the inference engine is doing the heavy work on the GPUs of the server node.
With 141 GB of VRAM per H200, a 7B model like this one leaves plenty of headroom. To serve a larger model across multiple GPUs on the same node, simply request more GPUs (e.g., --gres=gpu:h200:2); vllm_server.sh reads CUDA_VISIBLE_DEVICES and sets the tensor parallel size to match.
Important!
If you will be performing computationally intensive inference tasks, ensure that you are on a compute node requested through a Slurm interactive session or submit as a Slurm batch job.
Running inference on the model
Now that the client shell knows how to connect to the vLLM server, we can run inference on the served model. The following Python script demonstrates how to connect to the vLLM server and run inference on the served model using the OpenAI-compatible API. Save this script as vllm_client.py in your working directory.
#!/usr/bin/env python
# vLLM client connecting to the OpenAI-compatible API server.
#
# Usage:
# Once you have a vLLM server running, set the VLLM_HOST environment variable to the server's hostname and port.
# Then launch this script with,
# ./vllm_client.py
import os
from openai import OpenAI
HOST = os.environ.get("VLLM_HOST", "http://127.0.0.1:8000")
API_KEY = os.environ.get("VLLM_API_KEY", "your-secret-key")
# Array of prompts
PROMPTS = [
"Tell me about North Carolina",
"Why is the sky blue?",
"Write a Python code that calculates the Fibonacci sequence up to 15.",
]
def main() -> int:
client = OpenAI(base_url=f"{HOST.rstrip('/')}/v1", api_key=API_KEY)
try:
resp = client.models.list()
if not resp.data:
print(f"No models served at {HOST}.")
return 1
except Exception as e:
print(f"Could not list models from {HOST}: {e}")
return 1
model_obj = resp.data[0]
model_id = model_obj.id
model_name = getattr(model_obj, "root", None)
print(f"Connected to {HOST}")
display_model = model_name or model_id
print(f"vLLM model: {display_model}")
print()
# Send each prompt in turn, streaming the response token by token
for i, prompt in enumerate(PROMPTS, start=1):
print("-" * 60)
print(f"Prompt {i}/{len(PROMPTS)}: {prompt!r}")
print("Output:")
try:
stream = client.chat.completions.create(
model=model_id,
messages=[{"role": "user", "content": prompt}],
stream=True,
)
for chunk in stream:
delta = chunk.choices[0].delta.content
if delta:
print(delta, end="", flush=True)
print()
except Exception as e:
print(f"Request failed: {e}")
return 1
print("-" * 60)
return 0
if __name__ == "__main__":
raise SystemExit(main())
Feel free to change the prompts in the PROMPTS array as you like. To run a single
prompt, simply keep one entry in the array. Each prompt is sent as an independent
request with no shared conversation history, so the model does not remember earlier
prompts. Run the script in your client shell,
Sending the prompts concurrently
The loop above waits for each response before sending the next request. Since a vLLM
server batches many in-flight requests on the GPU, you can instead submit the whole
PROMPTS array at once and let the server interleave them, which is considerably faster
for a large number of prompts,
#!/usr/bin/env python
# vLLM client sending an array of prompts concurrently.
#
# Usage:
# Once you have a vLLM server running, set the VLLM_HOST environment variable to the server's hostname and port.
# Then launch this script with,
# ./vllm_client_batch.py
import os
from concurrent.futures import ThreadPoolExecutor
from openai import OpenAI
HOST = os.environ.get("VLLM_HOST", "http://127.0.0.1:8000")
API_KEY = os.environ.get("VLLM_API_KEY", "your-secret-key")
# Array of prompts
PROMPTS = [
"Tell me about North Carolina",
"Why is the sky blue?",
"Write a Python code that calculates the Fibonacci sequence up to 15.",
]
def main() -> int:
client = OpenAI(base_url=f"{HOST.rstrip('/')}/v1", api_key=API_KEY)
try:
resp = client.models.list()
if not resp.data:
print(f"No models served at {HOST}.")
return 1
except Exception as e:
print(f"Could not list models from {HOST}: {e}")
return 1
model_id = resp.data[0].id
def ask(prompt: str) -> tuple[str, str]:
completion = client.chat.completions.create(
model=model_id,
messages=[{"role": "user", "content": prompt}],
)
return prompt, completion.choices[0].message.content
# Submit all prompts at once and print them in the original order
with ThreadPoolExecutor(max_workers=len(PROMPTS)) as pool:
results = list(pool.map(ask, PROMPTS))
print(f"Connected to {HOST}")
for prompt, answer in results:
print("-" * 60)
print(f"Prompt: {prompt!r}")
print("Output:")
print((answer or "").strip())
print("-" * 60)
return 0
if __name__ == "__main__":
raise SystemExit(main())
Responses are not streamed here, since interleaved token streams from several prompts
would be unreadable. Run it with,
If you would like a chat session instead of hard-coded prompts, use the following script,
#!/usr/bin/env python
# vLLM client chat interface connecting to the OpenAI-compatible API server.
#
# Usage:
# Once you have a vLLM server running, set the VLLM_HOST environment variable to the server's hostname and port.
# Then launch this script with,
# ./vllm_client_chat.py
import os
from openai import OpenAI
HOST = os.environ.get("VLLM_HOST", "http://127.0.0.1:8000")
API_KEY = os.environ.get("VLLM_API_KEY", "your-secret-key")
def main() -> int:
client = OpenAI(base_url=f"{HOST.rstrip('/')}/v1", api_key=API_KEY)
try:
resp = client.models.list()
if not resp.data:
print(f"No models served at {HOST}.")
return 1
except Exception as e:
print(f"Could not list models from {HOST}: {e}")
return 1
model_obj = resp.data[0]
model_id = model_obj.id
model_name = getattr(model_obj, "root", None)
display_model = model_name or model_id
messages = [
{
"role": "system",
"content": "You are a helpful assistant for an HPC user.",
}
]
print(f"Connected to {HOST}")
print(f"vLLM model: {display_model}")
print("Type 'exit' or 'quit' to leave.\n")
while True:
try:
user = input("You: ").strip()
except (EOFError, KeyboardInterrupt):
print("\nBye!")
return 0
if not user:
continue
if user.lower() in {"exit", "quit"}:
print("Bye!")
return 0
messages.append({"role": "user", "content": user})
print("Model:", end=" ", flush=True)
assistant_text = ""
try:
stream = client.chat.completions.create(
model=model_id,
messages=messages,
stream=True,
)
for chunk in stream:
delta = chunk.choices[0].delta.content
if delta:
print(delta, end="", flush=True)
assistant_text += delta
print("\n")
except Exception as e:
print(f"\nRequest failed: {e}\n")
continue
messages.append({"role": "assistant", "content": assistant_text})
if __name__ == "__main__":
raise SystemExit(main())
and run it with,
Once you are done with your inferencing tasks, make sure to stop the server with the kill command provided in the server startup output. E.g.,
Slurm batch job workflow
Instead of serving through an interactive session, you may run the whole workflow — server and client — inside a single Slurm batch job. The startup banner goes to the job's output file rather than your terminal in that case, so the script reads the connection details back out of the log,
#!/bin/bash
#SBATCH -J vllm_batch_job
#SBATCH -p gpu
#SBATCH --gres=gpu:h200:1
#SBATCH --mem=100G
#SBATCH -t 00:30:00
cd $SLURM_SUBMIT_DIR
# conda activate is undefined in a batch shell until conda.sh has been sourced
source "$(conda info --base)/etc/profile.d/conda.sh"
conda activate vllm-env
# Batch jobs do not read ~/.bashrc, so set the cache locations here
export HF_HOME="/work/${USER}/.huggingface"
export FLASHINFER_WORKSPACE_BASE="/work/${USER}"
export VLLM_CACHE_ROOT="/work/${USER}/.cache/vllm"
# Start the server; the script returns once it is ready to accept requests
./vllm_server.sh > server.log
# Extract the connection details printed by the server
eval "$(grep '^export VLLM_HOST=' server.log)"
eval "$(grep '^export VLLM_API_KEY=' server.log)"
# Run inference
./vllm_client.py > output.txt
# Stop the server
pkill -f VLLM::
From your working directory, submit the job with,