Validation pipeline improves regulated AI inference decisions and audits

Inference Validation for Regulated AI: Building a Decision Pipeline on NVIDIA AIPerf

Software EngineeringMachine LearningPerformance

Summary

Running large language models safely in industries with strict rules needs more than just speed. The paper describes how the researchers built a system called Runway that checks and records how AI is run before it serves customers. Their system separates measuring AI performance from making business and legal decisions, helping companies trust the AI they deploy. They also found that performance rankings between AI engines can change depending on workload and settings, affecting cost and choice.

What this means in practice

  • For ai compliance teams: Provide verifiable evidence that AI inference configurations meet legal and contractual rules before customer use.
  • For cloud infrastructure managers: Optimize GPU resource selection and configuration for AI models by measuring performance under realistic workload conditions.

Authors

Dev Akre, Roman Bobalo, Manoj Dobbali

Abstract

Deploying large language models in regulated industries requires more than picking the fastest inference engine: it requires evidence that a configuration meets contractual and legal obligations before it serves a single customer request. We describe Runway, an internal validation harness built at MasterControl that builds a governed decision pipeline around NVIDIA AIPerf. The key insight is separating concerns: AIPerf provides the trusted measurement, while Runway provides the experiment design that comes before it and the compliance, cost, and audit decisions that come after. We describe the architecture and the storage model that makes caching and audit one mechanism. We report three findings from Runway sweeps: TensorRT-LLM and vLLM rankings invert with concurrency; a newer, pricier GPU instance can be the cheaper one per request; and speculative decoding behaves differently across engines under load, so engine configuration can outweigh engine choice.