Discrete-Event LLM Serving Scheduler & Gantt Studio

Compare continuous batching FCFS, SJF, Sarathi Chunked Prefill, and Disaggregated serving. Simulate Head-of-Line prefill blocking, inspect Time-to-First-Token (TTFT) and Inter-Token Latency (TPOT), and validate against empirical silicon.

Schedulers

< 4.8%

Hardware Error

165

Serving Traces

Empirical Scheduler Comparison Matrix

Discrete-event serving metrics simulated on LLaMA-3.2-1B under realistic Poisson traffic

Workload Domain:

Scheduler Policy	Workload	Throughput (tok/s)	TTFT P50 (ms)	TTFT P99 (ms)	TPOT P50 (ms)	TPOT P99 (ms)	Preemptions	SLA Goodput %

Interactive Head-of-Line (HOL) Blocking & Chunking Player

Visualizing how long prompt prefill blocks decoding requests across batch iterations

Policy:

Prefill Phase

Decode Step

Chunked Prefill

Starvation / HOL Bubble

Request ID / Flow Timeline (0ms → 800ms)

FCFS Head-of-Line (HOL) Blocking Analysis

In standard continuous batching FCFS, when Request #1 arrives with a 2,048-token prompt, its prefill phase consumes the full GPU compute budget for 120ms. During this entire prefill burst, ongoing decoding requests (Req #2, #3, #4) cannot generate any tokens, causing their inter-token latency (TPOT) to explode from 25ms to over 140ms.

Physical Silicon Validation Matrix

Empirical validation against dual Tesla T4 & RTX 3050 Laptop across concurrency levels

Inference Engine	Model	Concurrency	TTFT Real (ms)	TTFT Sim (ms)	TTFT Error %	TPOT Real (ms)	TPOT Sim (ms)	TPOT Error %

✓ Roofline Latency Precision: All discrete-event execution steps utilize accurate analytical FLOP-bound and memory-bandwidth bound models with cache KV pagination. Simulation errors strictly remain below 4.8% compared to physical PyTorch/vLLM/Ollama silicon telemetry.

Real Empirical Serving Traces (165 Requests)

Trace dataset capturing real LMSYS conversation distributions, RAG lookups, and Code generation

Request ID	Workload Category	Arrival Time (s)	Prompt Tokens	Decode Tokens	Total Tokens	Ratio (P/D)	SLO TTFT (ms)	SLO TPOT (ms)

Python SDK & CLI Usage

Run discrete-event serving simulations from terminal or Python scripts

Installation:

pip install sched-sim

Run Serving Simulation CLI:

# Run Sarathi Chunked Prefill simulation on RAG traces sched-sim run --scheduler chunked --chunk-budget 512 --trace traces/rag_workload.json # Compare continuous batching vs SJF sched-sim compare --schedulers fcfs,sjf,chunked,disaggregated --output results.json

Python API Integration:

from sched_sim.core.engine import EventEngine from sched_sim.schedulers.chunked import ChunkedPrefillScheduler from sched_sim.hardware.gpu import GPUProfile # Configure hardware profile and scheduler gpu = GPUProfile.from_preset("rtx_4090") scheduler = ChunkedPrefillScheduler(chunk_budget_tokens=512) engine = EventEngine(hardware=gpu, scheduler=scheduler) results = engine.simulate(trace_path="traces/lmsys_sample.json") print(f"Throughput: {results.throughput_tok_per_s:.1f} tok/s") print(f"TTFT P99: {results.ttft_p99_ms:.1f} ms | TPOT P99: {results.tpot_p99_ms:.1f} ms")