O-Funnel captures data from changing document formats with perfect accuracy
O-Funnel: Lossless Structural Capture and Requirement-Driven Extraction from Drifting, Heterogeneous Documents
DatabasesInformation Retrieval
Summary
Documents often come in many different formats and change over time, making it hard to reliably pull out important information. The authors designed O-Funnel to break down the problem into understanding the document’s layout separately from finding the data you want. It turns many types of documents into a single structured tree and checks every extraction for accuracy. O-Funnel stays accurate even when document formats change or when traditional extractors fail, and it can improve itself by learning new field names from leftover data.
What this means in practice
- •For data engineering teams: Extract fixed fields from diverse and changing data sources without rewriting parsers after every format change.
- •For health informatics teams: Automatically pull structured information from complex, drifting biomedical records such as PubMed entries with guaranteed completeness.
Authors
Osama Mustafa
Abstract
Pulling a fixed set of fields out of documents that arrive in many formats and under drifting schemas is usually done with hand-written byte patterns, which break whenever a key is renamed, a value is reformatted, or a lookalike value appears first. We argue the cause is structural: one pattern must both describe the value and locate it among its surroundings. O-Funnel separates the two. It transcribes any XML, JSON, CSV, HTML or key-value text document into one typed tree over five constructors, gated by an oracle that rejects any capture that does not reconstruct its source. Each needed field is declared in the tree's own terms and located by fusing independent evidence (key, path, value shape, synonym, key spelling, record neighborhood, value profile), so the best-supported node wins and a missing field is reported with a reason. Data no requirement claims becomes residue that a funnel traces back to the requirements to learn new key aliases. On 34,989 real PubMed records, O-Funnel matches a hand-written parser (F1 1.00). After a five-element schema rename, the parser's regular expressions fall to 0.20 while O-Funnel stays at 1.00, with every capture verified complete. On constructed suites that isolate regex failure modes it raises F1 from 0.43 to 1.00, and from 0.80 to 0.94 after self-improvement; on held-out schema-matching instances it is competitive with classical matchers without training. O-Funnel is a dependency-free Python library (pip install ofunnel).