Antonello Ceravola, Frank Joublin, Heiko Wersing, Stephan Hasler, Behzad Dariush, Yi-Ting Chen, "A Data Management Infrastructure for Intelligent Systems", Japanese Society of Artificial Intelligence (JSAI), pp. 45-54, 2018.Abstract
In this paper we describe the design principles, implementation choices and general challenges we encountered in the creation of a data management infrastructure for recording data streams from test vehicles, robots and other platforms. The trigger for this data management infrastructure project was twofold: First from the proper setup of new test cars equipped with many sensors, delivering high bandwidth data recordings and second from achieving organized storage of such recordings for the development and testing of intelligent systems operating on the data. After the clearly stated demand of such a data management system from different divisions of our company, we, step by step, conceived it as a very general data management platform targeting different projects with different recording formats and platforms. Recording data from different projects have systematic commonalities, for instance most use time series of data, often from similar type sensors with similar information. However considerable differences exist with respect to data organization in recording sessions, stream formatting or coverage of specific situation/event or environment conditions. Our data management infrastructure targets to support different needs in the data management work-flow. At first we want to centralize storage of recording sessions, avoiding proliferation of copies of recordings in our network. Facilitating visualization, inspection of recordings and annotation of events or entities present in the data is among our main targets. We allow both tool-based manual annotation processes as well as automated processes using AI/deep-learning methods. Subsequently we extract events or meta-information from recordings, storing them in a database. Our system enables then an efficient search over extracted information for exporting relevant recording segments, used for the creation of automotive or robotics intelligent systems.