Authors:
Lisa Ehrlinger
1
;
2
;
Alexander Gindlhumer
1
;
Lisa-Marie Huber
1
and
Wolfram Wöß
1
Affiliations:
1
Johannes Kepler University Linz, Altenberger Straße 69, 4040 Linz, Austria
;
2
Software Competence Center Hagenberg GmbH, Softwarepark 32a, 4232 Hagenberg, Austria
Keyword(s):
Data Quality Monitoring, Data Profiling, Automation, Knowledge Graphs, Heterogeneous Data, Sensor Data.
Abstract:
High data quality (e.g., completeness, accuracy, non-redundancy) is essential to ensure the trustworthiness of AI applications. In such applications, huge amounts of data is integrated from different heterogeneous sources and complete, global domain knowledge is often not available. This scenario has a number of negative effects, in particular, it is difficult to monitor data quality centrally and manual data curation is not feasible. To overcome these problems, we developed DQ-MeeRKat, a data quality tool that implements a new method to automate data quality monitoring. DQ-MeeRKat uses a knowledge graph to represent a global, homogenized view of local data sources. This knowledge graph is annotated with reference data profiles, which serve as quasi-gold-standard to automatically verify the quality of modified data. We evaluated DQ-MeeRKat on six real-world data streams with qualitative feedback from the data owners. In contrast to existing data quality tools, DQ-MeeRKat does not req
uire domain experts to define rules, but can be fully automated.
(More)