Architecture-Dependent Causal Transfer of Activation States Across Large Language Models
2026-08-17 • Computation and Language
Computation and LanguageMachine Learning
AI summaryⓘ
The authors studied whether the internal signals inside different AI language models can be shared directly, instead of using words, to make them talk to each other faster and cheaper. They tested this by trying to translate activations from one model's 'brain' to another's using a learned projection method. While some models showed shared internal patterns and could partly recognize each other's signals, only one pair actually influenced each other's word choices during text generation. The authors conclude that passing these internal signals only works in some model combinations and depends on their design, not universally.
large language model (LLM)activation statesrepresentational similarityprojection networkdecoder-only modelsencoder-decoder modelsmutual k-nearest-neighbour alignmentcentered kernel alignment (CKA)activation injectioncross-model retrieval
Authors
Fernando Cardenas Piepereit
Abstract
Direct communication between AI systems relies on natural language as an intermediate layer, incurring encoding/decoding overhead, token cost, and latency. We ask whether internal activation states can instead be transferred causally between different large language model (LLM) architectures via a learned projection, evaluated at three levels: representational similarity, cross-model retrieval from projected states, and end-to-end causal transfer via activation injection during generation. Using four architecturally diverse open-weight models (Qwen2-0.5B, Phi-3-mini, Mistral-7B, FLAN-T5-base), we find that representational alignment in trained models exceeds a random-initialization null baseline and is best captured by a rank-based metric (mutual k-nearest-neighbour alignment), more robust to activation-magnitude outliers than centered kernel alignment (CKA) or Procrustes analysis. A learned projection network retrieves the correct target-model representation from a held-out set well above chance for the three causal decoder-only model pairs (45-50% top-1 accuracy vs. 5% chance) but at chance level for the encoder-based FLAN-T5. Injecting projected activations into a target model during generation produces a statistically significant, pre-registered causal effect on retrieval-based output similarity for only one of the three decoder-only pairs (Qwen2-0.5B to Phi-3-mini: 23.3% vs. 0.0% under negative control, p=0.047, FDR-corrected); the two pairs targeting Mistral-7B show no such effect despite comparable representational alignment at the hidden-state level. We interpret these results as evidence for causal transfer of the representational vehicle, not of meaning, and conclude that end-to-end activation-state transfer between LLMs, as currently implemented, is architecture-dependent rather than universal.