TY - THES

T1 - A DSMS approach to support surveillance data based services in U-space

AU - Pfisterer, Daniel

N1 - no embargo

PY - 2024

Y1 - 2024

N2 - The market for UAS (Unmanned Aerial System) holds a lot of potential for growth in the near future, both in industrial and consumer applications. With an increasing rate of adoption, standardized regulations and technical developments are crucial to enable safe UAS operations. One initiative that blends both is the definition of a U-space that provides digital services for a safe access to airspace. In order to provide these services in real-time, with 1Hz as reference value, the implementation of a prototype of a DSMS (Data Stream Management System) is proposed.The developed DSMS is specified based on a combination of regulatory and problemspecific requirements. For defining the technical specifications the system is divided into three parts (ingestion, processing, and storing). According to the specifications a DSMS should be fault-tolerant and scalable while providing tools for stateful computations and SQL-compatibility. These specifications serve as basis for selecting the frameworks for the implementation of the prototype. For the data ingestion Apache Kafka is utilized, the stream processing is done with Kafka Streams, and all data is stored in a PostgreSQL database. Based on these frameworks the architecture of the whole system is designed. As Kafka is responsible for most of the data handling, designing the data flow within, from and to Kafka proved to be crucial for a successful implementation. To simplify testing and deployment of the prototype, all frameworks are implemented as containerized applications using Docker. Yet, even as containerized applications, applying these frameworks is not trivial. A smooth data exchange between the di!erent components, is only possible with consistent schema definitions. Coordinating the partitioning logic from Kafka with the stream processors for scalability requires careful adjustment of all parameters. Overcoming these challenges demands a deep understanding of the whole system with all its components and the interactions between them.The finished DSMS prototype implements a selection of U-space services which are tested with a customized simulator. Although results have to be viewed with care, as all tests are conducted in a controlled environment, the results demonstrate the feasibility of using a DSMS to provide U-space services.

AB - The market for UAS (Unmanned Aerial System) holds a lot of potential for growth in the near future, both in industrial and consumer applications. With an increasing rate of adoption, standardized regulations and technical developments are crucial to enable safe UAS operations. One initiative that blends both is the definition of a U-space that provides digital services for a safe access to airspace. In order to provide these services in real-time, with 1Hz as reference value, the implementation of a prototype of a DSMS (Data Stream Management System) is proposed.The developed DSMS is specified based on a combination of regulatory and problemspecific requirements. For defining the technical specifications the system is divided into three parts (ingestion, processing, and storing). According to the specifications a DSMS should be fault-tolerant and scalable while providing tools for stateful computations and SQL-compatibility. These specifications serve as basis for selecting the frameworks for the implementation of the prototype. For the data ingestion Apache Kafka is utilized, the stream processing is done with Kafka Streams, and all data is stored in a PostgreSQL database. Based on these frameworks the architecture of the whole system is designed. As Kafka is responsible for most of the data handling, designing the data flow within, from and to Kafka proved to be crucial for a successful implementation. To simplify testing and deployment of the prototype, all frameworks are implemented as containerized applications using Docker. Yet, even as containerized applications, applying these frameworks is not trivial. A smooth data exchange between the di!erent components, is only possible with consistent schema definitions. Coordinating the partitioning logic from Kafka with the stream processors for scalability requires careful adjustment of all parameters. Overcoming these challenges demands a deep understanding of the whole system with all its components and the interactions between them.The finished DSMS prototype implements a selection of U-space services which are tested with a customized simulator. Although results have to be viewed with care, as all tests are conducted in a controlled environment, the results demonstrate the feasibility of using a DSMS to provide U-space services.

KW - U-space

KW - Apache Kafka

KW - DSMS

KW - SQL

KW - Datenstromverarbeitung

KW - Kafka Streams

KW - data engineering

KW - Docker

KW - Datenstrom

KW - Log-basierter Nachrichten- broker

KW - stateful processing

KW - exactly once

KW - idempotence

KW - PostgreSQL

KW - Kafka Connect

KW - Java

KW - Echtzeitdatenverarbeitung

KW - U-space

KW - Apache Kafka

KW - DSMS

KW - SQL

KW - stream processing

KW - Kafka Streams

KW - data engineering

KW - Docker

KW - unbounded data

KW - log-based message broker

KW - stateful processing

KW - exactly once

KW - idempotence

KW - PostgreSQL

KW - Kafka Connect

KW - Java

KW - real-time processing

U2 - 10.34901/mul.pub.2024.068

DO - 10.34901/mul.pub.2024.068

M3 - Master's Thesis

ER -