Caleta builds trust layer for local large language model inference
Caleta: An Evidence-Oriented Trust Layer for Reproducible Local Large Language Model Inference
Software Engineering
Summary
Running large language models on your own computer can be private and trustworthy, but the terms often get mixed up. The authors created Caleta, a system that works with an existing engine to check that models and outputs are authentic and untampered, separate privacy from technical safeguards, and record everything needed to reproduce results. They tested Caleta on different operating systems and hardware to confirm it works reliably and quickly. This helps users be more confident about the integrity and privacy of local AI model runs.
What this means in practice
- •For ai application developers: Ensure trustworthy and reproducible local LLM inference by integrating Caleta’s verification and evidence mechanisms into their deployment workflow.
- •For data privacy teams: Use Caleta to separate cryptographic integrity guarantees from OS-level isolation for better privacy controls in local AI processing environments.
Authors
Joan-Marc Fisa
Abstract
Local large language model (LLM) inference is often described as private, verified, or reproducible without separating the technical claims behind those terms. This paper presents Caleta, an Apache-2.0 process supervisor and evidence layer around llama.cpp rather than a new inference engine. Caleta combines versioned manifests, exact-size and SHA-256 verification, controlled backend construction, a restricted OpenAI-compatible proxy, prompt-free execution records, benchmark reports, and a small conformance suite. Its contribution is the composition of these mechanisms into explicit contracts that distinguish integrity from authenticated provenance and privacy controls from operating-system isolation. We evaluate the implementation through static checks, 22 unit tests, deterministic end-to-end tests on Ubuntu, macOS, and Windows, and a pinned real-model experiment on Windows CPU using Qwen2.5-0.5B-Instruct Q4_K_M and llama.cpp b11981. The real experiment validates artifact verification, startup, model listing, non-streaming and server-sent-event chat, and conformance. Three independent benchmark invocations produced fifteen successful warm requests with no failures. In this single configuration, median time to first token was 18.066 ms and median decode throughput was 54.394 tokens/s. These descriptive measurements do not establish cross-engine superiority, output quality, energy efficiency, or hardware-general performance. Raw JSON, commands, and environment metadata are published. Remaining challenges include signed manifests, time-of-check/time-of-use replacement, operating-system containment, proxy-overhead measurement, and broader experimental matrices.