This study investigated the feasibility of integrating AR HMDs with AI for the precise localization of small, deformable objects, specifically ioECoG grids used during epilepsy surgery. Our approach, combining object detection, super-resolution, 2D pose estimation, and stereo triangulation with an AR HMD (HoloLens 2), has shown promising results in a controlled experimental setting. The super-resolution and 2D pose estimation models were successfully trained on purely synthetic data. The system achieved sub-5 mm accuracy below 60 cm distances. At 40 cm, this accuracy improved to less than 1.9 mm, with an average standard deviation of 0.5 mm. The estimation of individual electrodes did not demonstrate uniform performance, with the top right and bottom left electrodes of the ioECoG grid showing the poorest results.

Positional errors increase with distance, particularly with the low resolution (640x480) and wide-angle lenses of the HoloLens 2 cameras. As distance grows, image details diminish, forcing the super-resolution and 2D pose estimation model to rely more on approximations. While our method cannot match the sub-millimeter accuracy of the NDI Polaris Vega with manual registration, it remains within a clinically acceptable margin of 5 mm for localizing ioECoG grids up to 60 cm away.

As shown in Fig. 8b). This may stem from bias in the model’s pose estimation, though further investigation is needed.

Other localization methods, such as recursive grid partitioning, have demonstrated competitive results, with a reported mean localization error of 2.0 mm [10]. However, this approach is time-consuming and requires specialized expertise. In contrast, our AR-based method offers a more efficient, less operator-dependent solution.

GridLoc, an alternative ioECoG grid localization method based on signal analysis, achieved an overall accuracy of 1.94 mm [2]. However, GridLoc assumes a known and small inter-electrode distance, a large enough grid coverage, is time consuming and relies on pre-surgical angiographic data and radioactive MRI imaging.

Both methods lack real-time feedback and show the ioECoG electrodes in the MRI scan or on a 3D model, without a direct way to translate to the operating field. This limitation may reduce their utility for neurosurgeons, who could benefit from real-time overlays directly onto the patient. The latter is only possible by integrating AR technology. This enhanced visualization, which might also be utilized to visualize the epileptic area, may result in improved workflow, shorter interventions and increased ergonomics for this type of surgery.

Despite promising results, several limitations must be addressed in future work. First, our experiments were conducted in a controlled setting, lacking the complexities present in operating rooms. Factors such as brain deformation, grid obscuration by the skull, and tissue appearance variations could affect accuracy. Additionally, the controlled setup had static ceiling lights with no shadows on the grid, which may differ in the OR. However, we expect adequate lighting in the OR, and since our synthetic data account for various lighting conditions, we anticipate minimal impact from lighting variations.

Second, this study did not evaluate different ioECoG grid deformations, as the grid was kept static to minimize variables. However, the AI models are not restricted to specific deformations, as long as the data are similar to the training set, offering an advantage over methods like grid-fitting. Incorporating more varied deformations in future synthetic data could improve the method’s ability to handle greater variations. Future work should explore these deformations to assess the robustness of the models.

Third, we measured relative accuracy. While this aligns with metrics used in comparable studies [7], additional testing is needed to determine absolute accuracy in a clinical setting. Given the submillimeter accuracy of NDI, we expect any error given compared to the ground truth to be negligible.

Fourth, in our evaluation, everything is considered static. While the ioECoG grid can be assumed to be static in a clinical context due to the head being fixed, some minimal movement of the HMD may occur. Since this method focuses on registration rather than tracking, we expect the impact of this movement to be minimal. Any significant HMD movement can be addressed by filtering out images where the acceleration exceeds a certain threshold.

The method could be improved by adding more sanity checks to minimize false positives. For instance, the depth camera could provide distance estimates in centimeters, serving as a threshold for filtering. Additionally, with the use of an AR HMD, the detected positions can be visualized, enabling the neurosurgeon to quickly and intuitively assess the results.

Finally, the results can be further refined by integrating additional sensors, such as an RGB camera, refining pose detection to the circle centers, or incorporating prior knowledge. For example, future studies could explore integrating pre-surgical imaging (e.g., MRI or cone beam CT) to improve localization accuracy. Refinement of the result can be achieved by combining this method with the MRI-based method of Trotta et al., which can be automated by utilizing the proposed method in this work to get an initial prediction [11].

Accurate real-time localization of ioECoG grids using AR HMDs has significant clinical implications for epilepsy surgery. It could enable more precise mapping of epileptic tissue, improving resection success rates and reducing seizure recurrence. The system’s non-disruptive nature and the ability to directly visualize ioECoG data via the HoloLens 2 could enhance surgical workflows and potentially shorten procedure time. As AR HMDs become more integrated into neurosurgery, leveraging this existing platform allows for smooth implementation. Additionally, this system could be adapted for other surgical applications requiring precise localization of small, deformable objects using synthetic data.