Measuring Efficiency
NCShare is a shared resource across North Carolina institutions. Slurm reserves everything a job asks for, whether or not the job uses it, so CPU cores, memory, and GPUs that were requested but left idle are unavailable to everyone else for the life of the job. Right-sizing requests also helps you as smaller requests are easier for the scheduler to place and generally start sooner. On the gpu-hp partition, the monthly institutional GPU quota is charged for every GPU allocated, for as long as the job runs. A job that ends early is charged only for the time it actually ran.
This page covers the tools available on NCShare for measuring what your jobs actually consumed and for turning those numbers into better resource requests.
| Tool | Use for |
|---|---|
seff |
CPU and memory efficiency of a single completed job |
| sacct | Raw accounting fields, including peak memory per job step |
| slurm-report | CPU, memory, and GPU efficiency across many jobs over a date range |
| slurm-stat | Live CPU, memory, and GPU statistics for currently running jobs |
Measuring CPU and RAM efficiency
To check how efficiently a job used its allocated CPU cores and memory, run seff from a login node,
E.g.,
$ seff 724345
Job ID: 724345
Cluster: ncshare
User/Group: uherathmudiyanselage1/uherathmudiyanselage1-unixcluster-group
State: COMPLETED (exit code 0)
Nodes: 1
Cores per node: 64
CPU Utilized: 02:34:02
CPU Efficiency: 92.57% of 02:46:24 core-walltime
Job Wall-clock time: 00:02:36
Memory Utilized: 31.57 GB
Memory Efficiency: 6.31% of 500.00 GB (500.00 GB/node)
CPU Efficiency is the fraction of the allocated core-walltime (cores × wall-clock time) that was actually spent computing. Memory Efficiency is the peak memory used relative to the memory requested. Low values indicate the job requested far more CPU cores or RAM than it needed. Use these numbers to right-size your --cpus-per-task, --ntasks, and --mem requests in future submissions so that CPU and RAM allocations match your job's actual needs.
Warning
seff only reports meaningful values once a job has completed. For a running job, CPU and memory utilization show as 0.
Monitoring running CPU jobs
Use the slurm-stat tool to monitor CPU and memory efficiency of currently running jobs,
E.g.,
$ slurm-stat -u ${USER}
Monitoring 1 jobs...
Processing job 724345 (1/1)...
┌───────────────────────┬────────┬─────────┬──────────┬─────────┬────────┬────────┬────────┬─────────┬───────────┬────────┬─────────────┐
│ User ┆ JobID ┆ State ┆ Elapsed ┆ TimeEff ┆ CPUEff ┆ MemEff ┆ GPUEff ┆ GPUUtil ┆ GPUMemEff ┆ GPUMem ┆ Partition │
╞═══════════════════════╪════════╪═════════╪══════════╪═════════╪════════╪════════╪════════╪═════════╪═══════════╪════════╪═════════════╡
│ uherathmudiyanselage1 ┆ 724345 ┆ RUNNING ┆ 00:01:25 ┆ 2.4% ┆ 86.7% ┆ 6.3% ┆ --- ┆ --- ┆ --- ┆ --- ┆ interactive │
└───────────────────────┴────────┴─────────┴──────────┴─────────┴────────┴────────┴────────┴─────────┴───────────┴────────┴─────────────┘
Important
slurm-stat uses sstat in the backend to sample running averages of CPU and memory usage for job steps. Metrics from completed jobs through seff are more accurate than the running averages reported by slurm-stat.
Querying CPU and memory accounting
For detailed accounting information query the Slurm database directly with sacct. MaxRSS reports peak resident memory and is recorded per job step. TotalCPU is the sum of the SystemCPU and UserCPU time used by the job or job step. The total CPU time of the job may exceed the job's elapsed time for jobs that include multiple job steps or multiple cores.
E.g.,
$ sacct -j 724345 --units=G --format=JobID,JobName%16,State,Elapsed,AllocCPUS,ReqMem,MaxRSS,TotalCPU
JobID JobName State Elapsed AllocCPUS ReqMem MaxRSS TotalCPU
------------ ---------------- ---------- ---------- ---------- ---------- ---------- ----------
724345 GaAs_HSE06+SOC COMPLETED 00:02:36 64 500G 02:34:01
724345.batch batch COMPLETED 00:02:36 64 31.57G 02:34:01
724345.exte+ extern COMPLETED 00:02:36 64 0 00:00.002
Measuring GPU efficiency
The H200 GPUs are the most contended resource on NCShare, and a GPU job that spends most of its time waiting on data loading or on the CPU holds a device that another job could use. The slurm-report tool reports GPU utilization alongside CPU and memory efficiency for a set of completed jobs. We also share weekly usage reports with the goal of helping users better understand their GPU usage patterns and to support more efficient and effective use of the GPU partitions.
From a login node, run the following command to check the efficiency of your jobs over a time range,
E.g.,
$ slurm-report -r gpu-hp -S 2026-09-01 -E 2026-09-05 -u ${USER}
┌───────────────────────┬────────┬──────────────┬──────────┬─────────┬────────┬────────┬────────┬─────────┬───────────┬────────┬───────────┐
│ User ┆ JobID ┆ State ┆ Elapsed ┆ TimeEff ┆ CPUEff ┆ MemEff ┆ GPUEff ┆ GPUUtil ┆ GPUMemEff ┆ GPUMem ┆ Partition │
╞═══════════════════════╪════════╪══════════════╪══════════╪═════════╪════════╪════════╪════════╪═════════╪═══════════╪════════╪═══════════╡
│ uherathmudiyanselage1 ┆ 719314 ┆ TIMEOUT ┆ 00:10:20 ┆ 100.0% ┆ 44.8% ┆ 99.1% ┆ 2.0% ┆ 2% ┆ 21.9% ┆ 30.7G ┆ gpu-hp │
│ uherathmudiyanselage1 ┆ 719319 ┆ COMPLETED ┆ 00:10:25 ┆ 8.7% ┆ 54.6% ┆ 6.0% ┆ 4.0% ┆ 4% ┆ 92.3% ┆ 129.2G ┆ gpu-hp │
│ ┆ ┆ WEIGHTED AVG ┆ 00:20:45 ┆ ┆ 49.7% ┆ 52.4% ┆ 3.0% ┆ 3.0% ┆ 57.2% ┆ --- ┆ │
└───────────────────────┴────────┴──────────────┴──────────┴─────────┴────────┴────────┴────────┴─────────┴───────────┴────────┴───────────┘
For GPU workflows, the quantities that matter the most are GPUEff, GPUMemEff, and TimeEff.
- The GPU efficiency (
GPUEff) is the normalized per-GPU average utilization which represents the percentage of time GPU compute resources were actively engaged as reported bynvidia-smi. - The GPU memory efficiency (
GPUMemEff) represents the percentage of GPU memory (an H200 GPU has a total of 141 GB VRAM) that was actively used during the job. TimeEffis the job's elapsed time as a fraction of the requested walltime.
The lower CPU and memory efficiency values are not necessarily indicative of a problem for GPU jobs. A --- entry means the metric was not sampled by Slurm or that the job did not request GPUs.
Run slurm-report --help for the complete list of options.
Important
Access to the high-priority gpu-hp partition is granted on the basis of demonstrated GPU efficiency, and users who do not sustain efficient GPU usage on gpu-hp may be removed from the partition. See Requesting Access for the current thresholds.
Time-weighted averages
The last row of the output table shows the time-weighted average across all jobs in the specified time range. This quantity is formulated as,
where \(i\) is the index for your jobs, \(GPUEff_i\) is the GPU efficiency of job \(i\), and \(\text{time}_i\) is the elapsed time of job \(i\). The same formula applies for the other efficiency columns. Weighting by elapsed time keeps a handful of very short jobs from dominating the summary.
Monitoring running GPU jobs
slurm-stat reports the same metrics for jobs that are currently running. Use it to catch a misconfigured job early rather than discovering the problem after it has held a GPU for a day,
E.g.,
$ slurm-stat -u ${USER}
Monitoring 1 jobs...
Processing job 724338 (1/1)...
┌───────────────────────┬────────┬─────────┬──────────┬─────────┬────────┬────────┬────────┬─────────┬───────────┬────────┬───────────┐
│ User ┆ JobID ┆ State ┆ Elapsed ┆ TimeEff ┆ CPUEff ┆ MemEff ┆ GPUEff ┆ GPUUtil ┆ GPUMemEff ┆ GPUMem ┆ Partition │
╞═══════════════════════╪════════╪═════════╪══════════╪═════════╪════════╪════════╪════════╪═════════╪═══════════╪════════╪═══════════╡
│ uherathmudiyanselage1 ┆ 724338 ┆ RUNNING ┆ 00:03:59 ┆ 36.2% ┆ 0.5% ┆ 4.4% ┆ 100.0% ┆ 100.0% ┆ 89.7% ┆ 125.6G ┆ gpu-hp │
└───────────────────────┴────────┴─────────┴──────────┴─────────┴────────┴────────┴────────┴─────────┴───────────┴────────┴───────────┘
For a direct look at a GPU while a job runs, attach to the allocation and run nvidia-smi or gpustat, both of which are installed on the GPU nodes,
E.g.,
$ srun --jobid=724338 --overlap --pty nvidia-smi
Sat Sep 5 19:47:02 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.173.02 Driver Version: 580.173.02 CUDA Version: 13.0 |
+-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA H200 On | 00000000:BA:00.0 Off | 0 |
| N/A 56C P0 701W / 700W | 128595MiB / 143771MiB | 100% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 0 N/A N/A 1910376 C /app/gpu_burn 12858... |
+-----------------------------------------------------------------------------------------+
$ srun --jobid=724338 --overlap --pty gpustat
compute-gpu-04 Sat Sep 5 19:47:35 2026 580.173.02
[0] NVIDIA H200 | 56°C, 100 % | 128594 / 143771 MB | uherathmudiyanselage1(128586M)
You may also ssh into the node you are running your job on and issue the nvidia-smi or gpustat commands directly.
Querying GPU accounting
Similar to CPU and memory efficiency, you can query the Slurm database for GPU efficiency metrics with sacct by passing in the TRESUsageInTot field,
E.g.,
$ sacct -j 724338 --units=G --format=JobID,JobName%16,State,AllocTRES%15,TRESUsageInTot%80
JobID JobName State AllocTRES TRESUsageInTot
------------ ---------------- ---------- --------------- --------------------------------------------------------------------------------
724338 gpu_burn-10m COMPLETED billing=1,cpu=+
724338.batch batch COMPLETED cpu=8,gres/gpu+ cpu=00:00:17,energy=0,fs/disk=0.24G,gres/gpumem=125.57G,gres/gpuutil=100,mem=1.+
724338.exte+ extern COMPLETED billing=1,cpu=+ cpu=00:00:00,energy=0,fs/disk=0.00G,gres/gpumem=0,gres/gpuutil=0,mem=0,pages=0,+
724338.0 nvidia-smi COMPLETED cpu=8,gres/gpu+ cpu=00:00:00,energy=0,fs/disk=0,gres/gpumem=0,gres/gpuutil=0,mem=0.02G,pages=0,+
724338.1 gpustat COMPLETED cpu=8,gres/gpu+ cpu=00:00:00,energy=0,fs/disk=0,gres/gpumem=0,gres/gpuutil=0,mem=0.03G,pages=0.+
The quantities gres/gpuutil and gres/gpumem report the GPU utilization and GPU memory usage, respectively, for each job step. Both are summed across all GPUs allocated to the step.
Acknowledgements
The slurm-report and slurm-stat tools deployed on NCShare were developed by Joe Shamblin of Duke CS. slurm-stat and slurm-report are symlinks into the parent slurm-gpu tool.