Success Rate
NORMAL
99.84%
↑ +0.18% compared with prior period
Successful requests are only normal completions where finished_reason="stop".
LLM Inference Observatory
vLLM performance · reliability · scheduler · cache · H200 GPU telemetry
Preiodic Reload · last updated · every 8 seconds
01 · Top-line Summary
Service at a glance
Core availability, scheduler efficiency, active demand, user response speed and KV Cache preemption risk.
Active Requests
LIVE
64requests
↑ +8.4% traffic volume
Current work in the serving engine: requests in Running plus Waiting states.
System Efficiency
NORMAL
89%
Running / (Running + Waiting). Below 70% indicates scheduler overload and queue pressure.
TTFT P99
HEALTHY
0.86sec
↓ -6.2% faster first-token delivery
99th percentile Time To First Token. The user-facing initial-response speed target is under 1 second.
Preemption Rate
CRITICAL
0.19%
Running requests halted due to insufficient KV Cache. Any non-zero value should be investigated immediately.
02 · Latency & User Experience
End-to-end latency and inference behavior
Separate request tail latency from prefill and decode bottlenecks to identify the correct scaling or optimization action.
E2E Request Latency
Total time from request acceptance until response completion. P99 spikes warrant bottleneck analysis.
Latency Percentile Summary
Current E2E latency distribution for active inference traffic.
P50 request latency728 ms
P95 request latency1,260 ms
P99 request latency1,670 ms
If P50 remains stable while P99 rises, investigate queue contention, unusually long prompts, cache pressure, retries and load imbalance.
Inference Stage Breakdown
Prefill is input processing. Decode is output-token generation.
TTFT vs TPOT
High TTFT points to a Prefill bottleneck. High TPOT points to a Decode bottleneck.
03 · Token Throughput & Workload
Token flow, cache savings and prompt distribution
Measure workload shape to understand whether your serving fleet behaves like a RAG, summarization or long-form generation system.
Token Throughput
Prompt input tokens and generated output tokens processed per second.
Token I/O Ratio
RAG-LIKE
1.92input : output
Higher values indicate long prompts with short outputs, typical of RAG and context-heavy workflows.
Prefix Cache Savings
EFFICIENT
55%
Saved Prefill work: (1 − actual computations / total input computations) × 100.
Request Length Heatmap · Prompt Tokens
Darker ranges represent higher request counts at that prompt-token length.
0 tokens
12,288+ tokens
Request Queue Time
Time requests spend waiting for scheduler and GPU execution capacity.
04 · Engine Internal & Cache
KV Cache capacity and scheduler state
Early capacity signals that help prevent queue growth, swapping and disruptive request preemption.
GPU KV Cache Usage
WARNING
84%
Warning above 80%; preemption risk becomes critical when KV Cache use exceeds 95%.
Prefix Cache Hit Rate
EFFICIENT
78%
Hit rate for reusing system prompts, shared few-shot examples and common request prefixes.
Running Requests
GPU ACTIVE
56requests
Requests actively being processed by the GPU inference engine.
Waiting Requests
QUEUE
7requests
Requests waiting in the scheduler queue for GPU capacity and token-generation slots.
Scheduler State
Running requests infer on GPU; Waiting requests queue; Swapped requests move to memory.
Running56
Waiting7
Swapped2
GPU Cache and Process Pressure
KV Cache usage and process RSS memory trends over the selected time range.
05 · System Health & Reliability
Memory behavior and request outcomes
Use these panels to detect Python memory pressure, abnormal completion patterns and request failures.
Python GC & Memory
Frequent Python GC and continuously rising RSS can indicate memory leaks or process pressure.
Finish Reason Distribution
stop is normal EOS; length reached maximum output tokens; abort means a terminated request.
Throughput vs Success
Total requests should closely track successful stop-completed requests. A persistent gap represents failures or abnormal exits.
06 · Infrastructure & Load Balancing
Instance distribution and H200 DCGM telemetry
Validate load-balancer fairness and observe raw GPU utilization and framebuffer allocation across the serving fleet.
Instance Request Distribution · Load Balancer
Live request assignment by serving instance. A skew can create an avoidable bottleneck.
H200 GPU Utilization · DCGM
Raw GPU compute saturation from DCGM telemetry across the active H200 devices.
H200 VRAM Allocation · DCGM_FI_DEV_FB_USED
Framebuffer memory actively consumed by each H200 GPU. Higher VRAM allocation increases KV Cache pressure.
| GPU Device | Serving Instance | DCGM_FI_DEV_FB_USED | VRAM Allocation | Capacity Signal |
|---|