NovaFilter makes data transformation easy for nonprogrammers

NovaFilter - A Rule-Based ETL Staging Framework for Fault-Tolerant Relational Databases

Databases

Summary

Turning messy information from many sources into neat tables can be very hard and usually requires writing computer code. The researchers created NovaFilter, a new way that uses rules and questions to guide users, even those without programming skills, to organize and check data. This approach keeps all the transformation steps clear and reusable, helping people fix problems together with the computer. Because it talks with users through examples, it helps keep data accurate and makes the whole process easier to understand.

What this means in practice

  • For enterprise data engineers: Create clearer and reusable data transformation workflows with less need for scripting by involving domain experts directly.
  • For business analysts: Help nontechnical users guide data transformation processes through example discussions, reducing reliance on programmers.

Authors

Meike Klettke

Abstract

NovaFilter is a completely new approach to deriving structured relational data from heterogeneous input data in different data formats. It combines consistency checks, rule-based data transformation, and a human-in-the-loop example discussion. Using these three components, NovaFilter stores all transformation knowledge in an explicit rule base, which is continuously expanded through the example discussion with the user. The resulting data transformation process is deterministic, transparent, and reusable. As a result, the method can replace conventional ETL scripts and offers a completely new strategy for transforming data between different data models and schemas. Its main advantage is that users without programming skills (e.g. domain experts) can answer the example discussion and thus it is a building block for democratization of ETL script developement.