Trust challenges and solutions for offline generative AI on Raspberry Pi 5
Trust Mechanisms for Offline Generative AI at the Edge: A Connectivity-Regime Framework and an Empirical Study on a Raspberry Pi 5
Cryptography and SecurityArtificial IntelligenceMachine Learning
Summary
Generative AI usually depends on internet connections to get updates and verify information, but some devices like those in Internet of Things (IoT) settings have poor or no connectivity. The authors study how to trust AI models running entirely offline on a small computer called Raspberry Pi 5. They test various ways to detect when the AI doesn't know enough or gives wrong answers, including checks on its outputs and updates. While some methods reduce errors or catch corrupted updates, none can guarantee the truthfulness of the AI's information without connectivity.
What this means in practice
- •For iot device developers: Use connectivity-regime framework and signed knowledge packs to run and update generative AI models securely on devices with intermittent or no network connection.
- •For edge ai engineers: Implement retrieval-score gating and deterministic output validation to improve trustworthiness of local AI inference on edge devices like Raspberry Pi 5.
Authors
Georgios Feretzakis, Alexandros Papaspyridis
Abstract
Generative AI and large language models (LLMs) are moving into the Internet of Things, yet most deployments assume cloud connectivity, and many IoT settings have intermittent, restricted, or no connectivity. We propose a connectivity-regime framework (C0-C4), a taxonomy of inference-time trust mechanisms, and signed, versioned knowledge packs as the update channel for disconnected regimes, and evaluate them on a Raspberry Pi 5 running fully local retrieval-augmented generation under network isolation, with three Qwen2.5 model sizes, three context sizes, and 300 SQuAD 2.0 questions. An empirically calibrated retrieval-score gate detects knowledge gaps almost perfectly (AUROC 0.97) but in-topic unanswerable questions poorly (0.68). A deterministic output validator converts malformed outputs into explicit abstentions with no observed loss of correct answers, while the NLI-based verifier we tested adds little; for the 7B model, 96% of substantive answers to unanswerable-labelled questions match normalized spans of the retrieved passages. A 3B model achieves the same gold-answer containment rate as a 7B model at less than half the latency and estimated energy, although the 7B model far more often exceeds a token-F1 threshold of 0.5; five retrieved passages lower correctness from 75% to 52% compared with three. Verifying a signed pack takes about 4.5 ms versus about 29 min for embedding its passages on the device; signature and hash checks alone accept six of nine tested update attacks that layered verification rejects. Yet a validly signed poisoned pack passes every pack check: on 30 previously correct questions, correct answers drop to zero and 14 return the injected answer. None of the mechanisms we evaluated establishes the truth of the evidence. We release code and data.