LLM text generation improved with memory guided sampling for speed and accuracy

FLEET: From Logits Entropy to Enhanced Trajectories in Text Generation

Machine LearningArtificial IntelligenceComputation and Language

Summary

Language models often produce many repeated or similar answers if you ask them to generate multiple samples, which wastes time and computing power. The authors created FLEET, a method that remembers what was generated before and focuses on exploring new ideas by tracking the uncertainty of each word choice in a smart way. This approach helps the model avoid duplicates and generate better, more accurate outputs faster. It works well on tough coding problems and can be used with minimal changes to current systems.

What this means in practice

  • For software developers: Generate more accurate and diverse code suggestions faster by integrating FLEET into code generation tools.$Commercial implications: Improves coding assistant products by boosting code suggestion quality and speed, making software development more efficient.
  • For chatbot engineers: Reduce repetitive responses and improve the quality of chatbot outputs within a fixed computation budget using memory-infused sampling.

Authors

Oleksii Streltsov, Oleksandra Vitko

Abstract

Solutions based on large language models (LLMs) often rely on temperature sampling to improve accuracy and stability by aggregating multiple samples from the completion distribution. However, this memoryless approach is inherently suboptimal: because it lacks awareness of prior generations and their evaluations, it produces an increasing proportion of semantically duplicate answers as more samples are drawn, leading to diminishing returns. To address this limitation, we introduce FLEET, a novel method that integrates a memory mechanism into the generation process. FLEET represents each generation as a sparse trajectory through states whose entropy exceeds a predefined threshold and uses these trajectories to infer per-token utility scores that adjust the logits. Benchmark evaluations demonstrate that FLEET achieves the same accuracy as the repeated sampling baseline, with a 3x speedup, and substantially improves accuracy on complex coding tasks (LiveCodeBench Pass@32 increases from 59.9% to 66.2%) under the same budget. Furthermore, in the greedy-decoding configuration evaluated here, the approach is deterministic and uses a single calibration pass to derive its principal hyperparameters, requiring only minimal modifications to existing LLM pipelines.