Serialization invariant checks improve data integrity verification by agents

Same Data, Different Bytes: Serialization-Invariant Integrity Checks for Agent-Published Datasets

Cryptography and SecurityArtificial IntelligenceMultiagent Systems

Summary

Sometimes the same data can be stored in different ways that look different to computers but actually mean the same thing. This can confuse software agents that check if data they received is the same as what was originally shared. The authors created a method that treats these different but equivalent versions as the same, so agents can verify data more accurately. They showed in tests with multiple data models that their method catches real data changes better and ignores harmless formatting differences. This helps agents trust the data they get without being tricked by simple re-encodings.

What this means in practice

  • For data engineers: Ensure data pipelines verify datasets accurately despite different serialization formats by using serialization-invariant integrity checks.
  • For cloud storage operators: Improve validation of replicated or re-encoded datasets across distributed systems by adopting identifiers that recognize equivalent data encodings.

Authors

Jin Gao, Maura Clark

Abstract

Agent networks require reliable ways for requesting agents to verify where data is located, how it can be accessed, and whether the data received matches what the data owner made available. Checksum mismatches can reflect either altered content or serialization differences that preserve data equivalence under specified rules. Data identification mechanisms for agents, such as DataFacts, describe where data can be accessed and provide checksums for checking received data against published references. We propose a serialization-invariant identification method that assigns the same identifier to data representations equivalent under explicit rules. We use DataFacts as a case study, retaining its existing checksum alongside the proposed identifier. In a controlled agent study, we compare the proposed configuration with ordinary-tool and checksum baselines across equivalent re-encodings, altered content, and unavailable references. The study includes three model tiers from each of two providers. For equivalent re-encodings, the proposed configuration increased expected-outcome rates by 80.4 percentage points over the ordinary-tool baseline and by 70.2 percentage points over the checksum baseline. The proposed configuration also increased evidence-supported rejection of altered content by 3.5 percentage points over the checksum baseline. We contribute a serialization-invariant identification method with explicit equivalence rules, an integration approach illustrated through DataFacts, and an empirical evaluation against ordinary-tool and checksum baselines.