Speculative Decoding System
During each inference cycle, the lightweight draft model rapidly generates a sequence of K candidate tokens autoregressively. The larger, computationally heavy target model evaluates all K tokens simultaneously in a single forward pass by computing their probability distributions. Tokens that satisfy the target model's acceptance criteria are retained, while the first divergent token is corrected, enabling the system to produce multiple valid tokens per target forward pass.
