Speculative decoding · Adaptive compute · vLLM

APEX: Speculate Smarter,
Not Deeper

Adaptive Planning and EXecution for speculative decoding: a learned controller that chooses which speculator to run for each request and how deep to draft at every verification block.

Manvi Jha^1,2* Zach Zhang² Zhichao Xu² Linbo Liu² Sai Muralidhar Jayanthi² Vinayak Arannil²

¹University of Illinois Urbana-Champaign ²AWS AI *Work done during an internship at AWS AI

Route once, then adapt draft depth at every verifier block. Illustrative scheduling; timing not to scale.

5.24×

peak speedup over autoregressive decoding (math reasoning, APEX-S)

4.27×

aggregate speedup across six workloads (APEX-S)

−41.0%

relative wasted-token rate vs. fixed n-gram at k=16 (APEX-B)

7.73

waste-adjusted speedup (WAS), vs. 5.64 for the best fixed configuration

Overview

Speculative decoding speeds up LLM inference by letting a cheap drafter propose tokens that the target model verifies in one parallel pass. But the best drafter and the best draft depth shift from request to request, and from block to block within a request. Any fixed choice either leaves speed on the table or burns compute on tokens that get rejected.

APEX is a learned, two-level controller. APEX-Router picks a speculator (EAGLE-3, n-gram, or a draft model) and an initial depth for each request. APEX-Depth re-selects the draft depth after every verification block, trading speed against wasted tokens.

Integrated into vLLM and evaluated with Qwen3-8B on six workloads, APEX reaches up to 5.24× speedup over autoregressive decoding.

Method

APEX overview: a Prompt-State Builder feeds APEX-Router, which selects one expert among EAGLE-3, n-gram SD and Draft-SD; APEX-Depth then changes the draft depth k per block during generation; drafts are verified by the target model, and the accepted prefix and rejection outcome feed back to both controllers. — The router picks an expert and initial depth from prompt features. After each verifier block, the depth controller uses the updated state and feedback to pick the next depth.

What's new

Choose the drafter, not just the depth

One controller works across different kinds of speculators and picks the right one for each request.

Adapt depth while generating

Draft length is re-chosen after every verification block, using live feedback from the verifier.

Learn where drafts break

Acceptance is modeled as survival: APEX learns how likely a draft is to be rejected at each position, and learns from fully accepted blocks too.

Put a price on waste

A single learned utility balances throughput, accepted progress and rejected tokens when choosing what to do next.

Three operating profiles

APEX-S · Speed APEX-B · Balanced APEX-E · Efficient

The three share one architecture and differ only in the weights of the utility function, which set how much each one favors raw speed over avoiding wasted tokens.

For the survival model, utility, training objectives and full algorithm, read the paper.

Results

Qwen3-8B served in vLLM, six workloads: code (MBPP, HumanEval), conversation (UltraChat-200k), long-chain reasoning (GPQA), long-context completion (The Stack Smol), software engineering (SWE-bench Lite), and math (GSM8K, AIME-style). Baselines are fixed-depth EAGLE-3, n-gram, and draft-model speculation at k ∈ {2, 4, 8, 16}.

Scatter plot of speedup over autoregressive decoding against non-wasted draft tokens. APEX-S (4.27×) and APEX-B (3.27×) sit in the shaded upper-right region, above and to the right of fixed n-gram, EAGLE-3 and Draft-SD configurations. — Speedup vs. token wastage across depths and mechanisms, aggregated over all workloads. Up and to the right is better: APEX-S and APEX-B reach the region no fixed configuration does.

Aggregate performance

Method	Depth	Speedup ↑	Waste % ↓	WAS ↑
EAGLE-3	k=4	2.35×	66.25	3.55
EAGLE-3	k=16	1.37×	97.67	1.41
n-gram	k=4	2.79×	54.14	5.16
n-gram	k=8	3.59×	63.56	5.64
n-gram	k=16	4.22×	79.90	5.28
Draft-SD	k=2	0.96×	27.82	3.44
APEX-S	adaptive	4.27×	55.30	7.73
APEX-B	adaptive	3.27×	47.17	6.93
APEX-E	adaptive	1.83×	63.69	2.87

Selected rows; the paper reports every mechanism at k ∈ {2, 4, 8, 16}. Waste % = rejected ÷ drafted tokens. WAS = speedup ÷ (waste % / 100).

By workload

Speedup / waste % / WAS for APEX-S and APEX-B on each workload. Open a workload to compare against every fixed configuration.

What matters

Ablations on a 120-request diagnostic subset (20 per workload). Removing the survival loss hurts waste most; removing verifier-history features hurts speedup most. The same pattern holds for all three profiles; APEX-B shown.

APEX-B variantWAS  (speedup · waste %)

Full model4.39 (2.40× · 54.62)

w/o cost/TPS objective4.03 (2.33× · 57.83)

w/o entropy features4.03 (2.35× · 58.34)

w/o repetition features3.94 (2.30× · 58.45)

w/o survival loss2.21 (1.53× · 69.38)

w/o verifier-history2.03 (1.14× · 56.22)

Restricting the candidate depths to k ≥ 3 raises speedup slightly but increases waste and lowers WAS for every profile. The shallow actions are doing real work.

Limitations

On code generation, fixed n-gram (k=16) and EAGLE-3 (k=4) beat every APEX profile; routing and utility calibration need work there.
Wall-clock results come from one target model (Qwen3-8B) and one serving stack (vLLM), on single-turn prompts.
The controller is trained on offline traces and is not updated online. The profiles are not guaranteed Pareto-efficient.

Code

[CODE coming soon.]

Citation

@article{jha2026apex,
  title   = {APEX: Speculate Smarter, Not Deeper},
  author  = {Jha, Manvi and Zhang, Zach and Xu, Zhichao and Liu, Linbo
             and Jayanthi, Sai Muralidhar and Arannil, Vinayak},
  journal = {arXiv preprint arXiv:2610.07780},
  year    = {2026}
}