Augmented reality application

The augmented reality application is the main interface of the system. It allows visualizing the virtual 3D models of the mannequins aligned with their physical counterparts. In addition, it incorporates real-time hand tracking, developed with Mixed Reality Toolkit 2 (MRTK 2), and generates a virtual replica of the user's hands in the form of "ghost hands" (Fig. 2). The system allows both the user's movements to be observed and the three-dimensional coordinates of the 25 joints of each hand to be recorded, enabling comparison with the reference postures.

Fig. 2

Virtual ghost hand model created for real-time hand tracking

During training, a second operator is responsible for a number of essential functions, such as freezing the user's hands at each stage of the birthing process, saving the corresponding three-dimensional coordinates for later evaluation, and loading the reference postures when necessary. These reference postures are displayed in purple to clearly distinguish them from the user’s ghost hands that appear in blue. When the resident's hand placement gets close enough to the reference, the ghost hands turn green, providing immediate feedback and allowing for real-time correction (Fig. 3). This dual approach allows residents to focus exclusively on hand placement, while ensuring the systematic recording of performance data for later evaluation.

Fig. 3

Training interface. Left: Loaded reference hand posture. Right: Trainee hand placement turns green when it approaches the reference posture

QR marker

To ensure stable alignment between the real and virtual environments, a custom 3D-printed holder was designed to fix a QR marker at a predefined position and orientation relative to the mother mannequin (Fig. 4). The corresponding rigid transformation between the marker coordinate system and the virtual mannequin model was predefined in the AR application. Importantly, the marker itself does not require any custom definition or configuration, as the HoloLens 2 natively supports QR code detection and pose estimation. This functionality enables the application to consistently anchor all 3D virtual models to their physical counterparts in the correct reference frame once the QR marker has been positioned and detected. Incoming tracking transformations are then applied relative to this frame, enabling automatic registration without manual adjustment.

Fig. 4

Prototype and final result of the QR marker holder

If the marker is temporarily occluded, the system preserves its last valid pose. The marker’s status is continuously monitored in real time. If the marker is not detected for more than two seconds, tracking is considered lost, and pose updates are suspended until the marker is detected again. To provide visual feedback, the QR marker frame turns green when detection is stable and red when tracking is lost.

3D slicer module

A custom 3D Slicer module acts as an intermediate layer between the EM tracking system and the AR application. It facilitates the communication between devices and the registration process by automating tasks to make it easier and faster. The workflow of the module begins by loading the virtual models of the mother and newborn. When the tracking system is connected and receives real-time signals from the sensors, point-based registration is calculated to accurately align the physical mannequins with their virtual models.

Real-time communication between Unity and 3D slicer

Once the virtual models have been uploaded and registered in 3D Slicer, the resulting geometric transformations are exported to Unity using the OpenIGTLink protocol, following the architecture described in [11]. From Unity, this information is transmitted to Microsoft HoloLens 2 through Holographic Remoting (Fig. 5). In this way, the augmented reality glasses receive the updated transformations and apply them to the virtual 3D models, guaranteeing their synchronization in real time with their corresponding physical ones.

Fig. 5

System workflow showing data transfer from Electromagnetic tracking to Microsoft HoloLens 2

Definition of reference postures

To establish the reference positions, a preliminary session was held with the head of the Maternal–Fetal Medicine Service of the Hospital General Universitario Gregorio Marañón. During this session, the correct placement of the dominant and non-dominant hands in each phase of the birth process was demonstrated (Fig. 6). The dominant hand mainly protects the maternal perineum during delivery of the head and shoulders (positions 1, 2, 4 and 6), while the non-dominant hand initially follows and controls the fetal head descent (positions 1 and 2) and later stabilizes it after emergence (position 3 onwards). Both hands then coordinate to facilitate shoulder release (position 5) and finally the dominant hand supports the newborn’s trunk to complete delivery (position 7). These postures were recorded and saved as reference models for the AR application, where they were used both to guide the training and to evaluate the performance of the participants.

Fig. 6

Sequential hand positions defined for vaginal delivery training

System evaluation

To evaluate the complete system, a workshop was organized at Hospital General Universitario Gregorio Marañón, with a total of 11 participants, including gynecology and midwifery residents and final-year medical students. Each participant individually completed three simulations of the vaginal childbirth process:

1.

Unassisted attempt–The seven positions were executed according to their own clinical judgment, without any guidance.

2.

Guided attempt–The sequence of maneuvers was repeated while the reference positions were loaded, as shown in Fig. 3. These reference models were progressively loaded for each stage of the delivery, allowing participants to compare their current hand placement with the corresponding ground truth.

3.

Final attempt–The positions were executed again without guidance, applying the knowledge acquired during the guided phase.

The hand postures of each resident were saved on the first and third attempts in order to compare them with the reference models and evaluate whether there were improvements in performance.

At the end of the practical sessions, participants answered a questionnaire about their experience with the simulator. The survey included several questions related to the usability and educational value of the system.

Analysis of results

The data collected in the workshop was analyzed using different metrics. For each participant and each attempt, performance was summarized as a global success rate. Specifically, two binary variables were defined for every delivery position: whether the hand posture matched the corresponding reference posture and whether the dominant hand was correctly used (values 0/1). These binary outcomes were averaged across all delivery positions to obtain, for each participant and each attempt, a percentage of correct postures and a percentage of correct dominant-hand use. Improvements in postural accuracy and correct use of the dominant hand for each participant were statistically assessed with paired t-tests, comparing the unassisted attempt with the final attempt. Prior to applying the t-tests, the normality of the paired differences was assessed to confirm the assumptions required for parametric testing. Statistical significance was set at 0.05.

In addition, two continuous error metrics were defined. The positional error (expressed in millimeters) quantifies the spatial accuracy of the placement of the hand, aligning the hand of each participant with the reference hand at the wrist joint, normalizing it with respect to the reference hand and calculating the average Euclidean distance in absolute value between the 25 articular points of each hand. In a complementary way, angular error is defined, which evaluates the clinically relevant angles of each posture (e.g., the opening between thumb and index finger in positions 1, 2, 4, and 6, or the extended palm configuration required in positions 3 to 7). Angular error is calculated as the absolute difference between the participant's angle and the reference value, then averaged to obtain a single score per hand and per position.