Analysis of sensor data streams to support minimally invasive surgical procedures

Minimally invasive heart valve procedures are characterised by reduced pain, shorter hospital stays and a faster recovery. During the operation, all sensor and imaging data – including ultrasound and vital signs – are recorded in order to detect risks such as post-operative delirium or kidney failure at an early stage and to enhance safety.

The project aims to integrate multimodal data streams – including endoscopic videos, ultrasound images and sensor data – in real time. This is intended to enable precise, automated analyses of surgical phases, instruments and relevant anatomy. The use of efficient multi-task deep learning networks to analyse the video data is intended to improve intraoperative navigation, allow realistic modelling of cardiac structures during various states of deformation, and thus enhance surgical safety.

A particular focus is on linking data from before, during and after the operation. The exact positional data from the cameras, combined with ultrasound data, enables improved intraoperative orientation and early problem detection. This contributes significantly to minimising complications and improving real-time decision-making. Overall, the project aims to strengthen clinical application, increase surgical precision and significantly reduce the risk to patients.

The project is funded as an agility project under the BIFOLD programme and is led by Dr Steffen Zeuch, Prof. Dr Volker Markl, Prof. Dr Alexander Meyer and Prof. Dr Anja Hennemuth.