Self-Speculative Decoding Speeds for Multi-Token LLMs

Figure S10 illustrates the relative throughput and latency improvements of self-speculative decoding with k heads for a 4-token prediction code model across various batch sizes.