Augmented reality navigation-guided pulmonary nodule localization in a canine model

Introduction

Compared with conventional, preoperative computed tomography (CT)-guided pulmonary nodule localization, intraoperative localization techniques, performed in the operating room under general anesthesia, can benefit patients by shortening the wait time between localization and surgery, reducing complication rates and the need for patient mobilization (1-5). Although the current intraoperative localization techniques have been demonstrated to be safe and effective, most of these procedures require the use of a hybrid operating room (1-3,5), although other techniques have been reported (6-8). Their infrastructure requirement has limited patients’ access to these procedures.

Augmented reality (AR) is a rapidly emerging technology that allows for the superimposition of data and virtual objects in front of a user’s natural vision (9). In addition to surgical planning and medical training applications, researchers have begun to investigate the possibility of using a head-mounted camera, together with AR techniques, to navigate spinal surgery procedures in cadavers (10).

Current intraoperative pulmonary nodule localization techniques mainly include CT-guided localization in a hybrid operating room, intraoperative ultrasonography and electromagnetic navigation bronchoscopy (7). Each technique requires specific instruments, large-scale equipment or well-trained operators, which hinders their widespread application, and may cause extra expense. By using the nature of AR technique, a virtual image of 3D reconstruction of pulmonary nodules, bronchi and lung vessels could be superimposed upon the patient’s thorax, which may enable an easily identification and localization of pulmonary nodule. Different commercially available AR devices could also make this technique accessible and cost-effective.

Here, we present an innovative AR navigation-guided pulmonary localization technique that was applied to a canine lung model using a commercially available head-mounted AR device: HoloLens (Microsoft, Redmond, Washington). Through the real-time tracking of a percutaneous device, combined with patient CT data registration, this head-mounted device can serve as a tradition infrared-based navigation system that is widely used in dedicated neuro- (Curve platform, Brainlab, Munich, German), and orthopedic surgical procedures (Stealth Station, Medtronic, Dublin, Ireland). Guided by the information displayed by the HoloLens device, surgeons can determine the optimal position for a trans-thoracic puncture in real-time and implant lung tissue markers into the parenchymal tissue surrounding the targeted lung lesion. These procedures can be performed in a standard operating room, enabling the widespread application of this intraoperative pulmonary nodule localization technique.

We present the following article in accordance with the ARRIVE reporting checklist (available at https://dx.doi.org/10.21037/tlcr-21-618).

Methods

Experiments were performed under a project license granted by the Animal Care and Use Committee of Silver Snake Medical Center, where all procedures were performed, in compliance with China national and institutional guidelines for the care and use of animals.

AR navigation system

The AR navigation system was developed by Jedicare Medical Co., Ltd., Shanghai China, and consists of a 3-dimensional (3D) patterned, trackable transthoracic puncture device (brand name: Lungbrella, Figure 1A) and a Windows 10-based software program (brand name: JediVision), that is run on both a stand-alone personal computer (PC) and the HoloLens computer. The video stream obtained from the HoloLens camera was used to track the position of the puncture device while 3D virtual navigation information was displayed on the HoloLens screen.

Figure 1 Lungbrella marker device set. The device set includes the following three major components. (A) Puncture needle with 3D-trackable QR code-like pattern; (B) nitinol braided marker; (C) putter. The marker is pre-loaded inside the puncture needle and extruded by the putter after the needle has reached the desired release location. QR code, quick response code.

Before performing the animal study, the spatial accuracy and repeatability of this HoloLens-based AR navigation system were calibrated using a six-axis manipulator (LR Mate, FANUC, Yamanashi, Japan). The maximum spatial accuracy error of the needle tip (3 times the standard deviation of the distance between the software-calculated position and the actual position) was 1.8 mm when simulating the patient registration and device-guided procedures in a lab environment.

During the procedure, a target registration error (TRE)-based registration estimation technique was used to ensure registration accuracy (11).

Lung localization marker

The marker used for lung localization was a braided nitinol marker (Figure 1B,1C), which was compressed and pre-loaded in the Lungbrella apparatus. After deployment inside the lung parenchyma, the marker expands into an umbrella shape, 6–8 mm in diameter, with a tail extending to the lung surface. The diameter of this marker will expand further when the tail is pulled from the lung surface, which engages a self-locking mechanism and may strengthen its hooking capability during the surgical procedure.

Animal model

Peripheral pulmonary lesions were artificially established in a canine model through the transthoracic insertion of a metal tube (0.9 mm ×5 mm), as previously reported (12). Each animal firstly underwent a CT-guided artificial lesion implantation procedure, during which three artificial lesions were established. After the successful implantation of artificial lesions, the marker implantation procedure was performed to implant three corresponding markers. If pneumothorax was absent, according to a CT scan performed immediately after the first marker implantation procedure, the animal was considered a candidate for immediate second or third marker implantation. If pneumothorax was identified, the next implantation procedure would be rescheduled at least 4 weeks later, to allow for lung tissue recovery. Animals were bred normally between each procedure until the last marker removal operation was performed. All artificial lesions and markers were removed by wedge resection or lobectomy during a single video-assisted thoracoscopic surgery (VATS). All procedures were performed under general anesthesia, with airway access via single- or double-lumens tracheal intubation. All animals were euthanized at the end of the last operation.

Navigated transthoracic localization procedure

Before the nodule localization and marker implantation procedures, and according to the locations of the artificial lesions, the operator made a rough estimate of the surface projection and attached skin markers (Jedicare, Shanghai China, Figure 2) to the chest wall after skin preparation. Then, the animal underwent a CT scan during the end-expiration phase, which covered all skin marker points and artificial targets. The acquired CT images were analyzed using the JediVision PC software. Chest bones, lung lobes, and lung arteries were automatically segmented from the CT image. After manually prescribing the lesion target, the desired marker release location, and the skin entry point, the entire procedure plan was loaded into the HoloLens computer through a USB cable.

Figure 2 Skin marker. The skin marker consists of a transparent skin dressing and CT-visible copper markers, with holes in their centers, covered with strength paper to facilitate the taping procedure. The distance between two consecutive copper markers is 2 cm. CT, computed tomography.

When performing the localization procedure, while wearing the HoloLens, the operator placed a Quick Response (QR) label (Jedicare, Shanghai China, Figure 3) on the percutaneous side of the animal to indicate the operating area. After registering the QR codes using the HoloLens camera, the operator was presented with four points, labeled A–D, on the AR screen. By matching the Lungbrella needle tip with the marker point labeled A and holding the handle towards the operator’s eye, the software was able to record the 1st registration location. The same procedure was repeated for all labeled points, based on the indication that appeared on the AR screen. After all four points were recorded, the Hololens computer will automatically calculate the TRE of this procedure. When the TRE value is below 2 mm, the virtual organs generated according to the CT image were registered, merged with the actual animal chest spatial location and displayed on the AR screen. Next, the navigational puncture step was initiated. If the TRE value was greater than 2 mm, due to animal respiration, operator hand movement, or any other reason, the entire registration procedure would be automatically repeated, starting from the QR label recognition step.

Figure 3 Positioning QR code. A board with a printed QR code, together with green head/feet direction indication letters (in Chinese). This board will be placed beside the operating area before navigation start. QR code, quick response code.

After the navigational puncture step was triggered, the operator was able to easily determine the puncture path by aiming the extended virtual line from the needle to the virtual marker release point, avoiding any bones or arteries (Figure 4) on the virtual pathway. During the puncture process, the software was able to continually display the virtual pathway and the remaining distance between the needle tip and the target, by real-time tracking the 3D pattern of puncture device. When the software estimated that the needle tip was within a 3 mm radius of the release point, the indicator became green, and the operator could initiate the release of the Lungbrella marker. A follow-up CT scan was immediately performed to verify the distance between the artificial lesion and the marker. The localization procedure was presented in Video 1. The registration and implantation time was recorded, starting when the operator equipped the HoloLens until completion of the marker implantation procedure.

Figure 4 HoloLens navigation-guided puncture. The operator (Author Chengqiang Li) inserting the second needle into the animal’s lung under HoloLens guidance. The animal was under general anesthesia in the supine position.

CT data analysis

All CT scans were performed using a 16-slice CT scanner (GE Lightspeed, Milwaukee, Wisconsin), with 1-mm slice thickness, at the expiration stage. The distance between the implanted marker and the artificial lesion was measured using a 3D reconstructed CT image. The shortest distance between the artificial lesion and the implanted marker was calculated and recorded.

Statistical analysis

All the statistical analyses were performed using STATA version 15.1 (StataCorp, College Station, TX, USA). Mean ± standard deviation was used for continuous variables with normal distributions, while continuous variables with non-normal distributions were presented as median and range.

Results

From June 2019 to June 2020, 12 peripheral pulmonary lesions have been artificially created in 4 canine models. Among the 12 lesions, 4 were located in upper lobes, 3 in middle lobes, and 5 in lower lobes.

Feasibility and accuracy of marker implantation

Two dogs (D15 and D16) underwent three AR-navigated marker implantations during a single procedure, whereas the other 2 dogs (D11 and D12) required two procedures to achieve all implantations due to pneumothorax occurred after the first or the second marker implantation. All 12 Lungbrella markers were precisely implanted, and the median registration and implantation time was 6 minutes (range, 2–15 minutes). The average measured distance between the marker and the lesion, based on the reconstructed CT image, was 1.9±1.7 mm (Table 1), and the maximum distance was 4.4 mm (Figure 5A). Because all desired marker release locations were intentionally designed to overlap with the lesion target during the procedure planning stage, this CT-measured distance was considered to represent the navigation accuracy.

Figure 5 Post-marker implantation procedures (marker ID 12). After each marker implantation, a CT scan was performed to verify its immediate distance from the artificial lesion. After three markers were implanted, the VATS procedure was executed to evaluate each marker’s anchoring status within the lung tissue. All three implanted markers were resected by wedge resection or lobectomy procedures to determine the actual relative positions of the marker and the artificial lesion. (A) Reconstructed CT image showing the artificial lesion and the implanted marker; (B) pulling the tail of the marker during the VATS operation; (C) sliced tissue sample after wedge resection. CT, computed tomography; VATS, video-assisted thoracoscopic surgery.

The average marker implantation period (from the implantation date to the VATS removal date) was 16.5 days (range, 1–46 days), and the success rate of lesion and marker identification and removal by VATS was 100% (Figure 5B). No marker displacement was identified, and the relative distances were all confirmed in postoperative specimens (Figure 5C).

Safety of marker implantation

According to the immediate follow-up CT scan after each marker implantation procedure, the status of pneumothorax and bleeding were evaluated and graded. Three (25%) moderate pneumothorax and 3 (25%) moderate bleeding were observed after 12 marker implantation procedures, which had no need for further intervention. No severe complication occurred. The complications of each marker implantation are described in Table 1.

Discussion

Several intraoperative pulmonary nodule localization techniques have been introduced in recent years, most of which require a hybrid operating room that is equipped with a CT scanner to guide the hookwire placement or dye injection (5). An electromagnetic navigation bronchoscope-guided approach and near-infrared fluorescence-guided techniques have also been reported (8,13). However, the realization of these techniques requires specific medical equipment or skillful operators, which can restrict their widespread application. Intraoperative ultrasound has also been reported for the detection of pulmonary nodules, which can be limited by air in the pulmonary parenchyma and may be difficult to use for the detection of ground-glass opacities (14).

The application of AR during surgical procedures has increased in recent years, not only because of its ability to provide additional visual information projected onto real images but also because it provides the opportunity to implement advanced computer algorithms, such as deep learning or artificial intelligence (15), depth estimation (16), and fast surface digitalization and registration (17), into clinical practice. The optimal combination of a digital algorithm with a surgical procedure may enable the significant enhancement of operation efficiency and flatten the learning curve for difficult and new procedures. The accuracy and robustness of such procedures may improve rapidly according to the development of corresponding computer algorithms.

If the presented AR navigation-guided pulmonary nodule localization technique becomes integrated into routine clinical practice, patients will simply be required to undergo a standard CT scan, with attached skin markers, prior to surgery. A single thoracic surgeon will be able to perform the transthoracic marker implantation procedure after the patient receives general anesthesia, and then the lung nodule resection can be performed in a standard operating room. The estimated intraoperative nodule localization procedure workflow is as follows (Figure 6).

Figure 6 Estimated workflow of the AR navigation-guided nodule localization. AR, augmented reality; CT, computed tomography.

Day 0: (I) a doctor attaches the skin markers to the patient’s skin in the proposed operating area; (II) the patient undergoes a low-dose CT scan with attached markers; (III) the patient returns to the ward, and nurses remove the skin markers while labeling the location of each marker by marking the patient through the center hole of each marker; (IV) the doctor reviews the CT image and generates the transthoracic puncture plan.

Day 1: (I) the patient receives general anesthesia in the operating room; (II) a doctor executes the transthoracic puncture while wearing the HoloLens device, implanting the localization marker beside the target lesion; (III) a standard VATS operation is performed by locating the tail of the marker and removing the lesion, together with the associated marker.

This AR navigation guided localization procedure can be performed under general anesthesia in a standard operating room. Right after the completion of marker implantation, pulmonary nodule resection can be performed. Due to the significantly reduced wait time between marker implantation and surgery, patients who undergo this intraoperative lung nodule localization procedure are unlikely to experience any additional risk of pneumothorax or bleeding between the localization procedure and lung surgery compared with the risks associated with the conventional 2-stage preoperative CT-guided hookwire or microcoil localization (18). Meanwhile, the preoperative marking could also precede surgery by several days with low risk of dislodgment. As a novel technique, training would be needed for mastering this AR-guided nodule localization procedure. However, with the help of AR navigation, the localization procedure was greatly simplified and the learning curve was shortened. Surgeons could also get trained virtually before performing on the real patient. The marker implantation time was also greatly reduced. The median registration and implantation time was 6 minutes under AR navigation, which was significantly shorter than the time required for intraoperative CT-guided procedures (mean localization time of 21.19 min) (19).

This AR navigation-guided localization may also reduce the need for interventional radiologists or hybrid operating rooms and could eventually reduce the medical financial burdens of patients because of fewer medical professionals and specialized equipment associated with this procedure. Given that no specialized operator or large-scale equipment is involved in the AR navigation-guided localization, and the procedure is able to be adapted on all commercialized head-mounted AR devices other than HoloLens, this newly developed localization method could be more reachable and easily introduced to other hospitals.

This intraoperative pulmonary nodule localization technique also has some limitations and will require some modifications before being applied to human studies. First, the patient position must be identical between the CT scanning and the marker implantation procedure. Methods that are capable of matching the patient respiration level during the CT scanning procedure with that during the AR navigation procedure must also be developed. Further studies are planned to address these issues and to investigate the possibility of applying AR navigation-guided techniques to humans in clinical practice.

In conclusion, the AR navigation-guided pulmonary nodule localization technique was safe and effective in a canine model. The validity and feasibility of applying this technology to patients will be examined further (NCT04211051).

Acknowledgments

The authors thank Fei Sun and Jiatao Ma from Jedicare Medical (Shanghai) Co., Ltd., who designed the LungBrella markers and associated software.

Funding: None.

Footnote

Reporting Checklist: The authors have completed the ARRIVE reporting checklist. Available at https://dx.doi.org/10.21037/tlcr-21-618

Data Sharing Statement: Available at https://dx.doi.org/10.21037/tlcr-21-618

Peer Review File: Available at https://dx.doi.org/10.21037/tlcr-21-618

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at https://dx.doi.org/10.21037/tlcr-21-618). The authors have no conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. Experiments were performed under a project license granted by the Animal Care and Use Committee of Silver Snake Medical Center, where all procedures were performed, in compliance with China national and institutional guidelines for the care and use of animals.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Bolton WD, Howe H 3rd, Stephenson JE. The utility of electromagnetic navigational bronchoscopy as a localization tool for robotic resection of small pulmonary nodules. Ann Thorac Surg 2014;98:471-5; discussion 475-6. [Crossref] [PubMed]
2. Gill RR, Zheng Y, Barlow JS, et al. Image-guided video assisted thoracoscopic surgery (iVATS) - phase I-II clinical trial. J Surg Oncol 2015;112:18-25. [Crossref] [PubMed]
3. Gill RR, Barlow J, Jaklitsch MT, et al. Image-guided video-assisted thoracoscopic resection (iVATS): Translation to clinical practice-real-world experience. J Surg Oncol 2020;121:1225-32. [Crossref] [PubMed]
4. Chen PH, Hsu HH, Yang SM, et al. Preoperative Dye Localization for Thoracoscopic Lung Surgery: Hybrid Versus Computed Tomography Room. Ann Thorac Surg 2018;106:1661-7. [Crossref] [PubMed]
5. Chao YK, Pan KT, Wen CT, et al. A comparison of efficacy and safety of preoperative versus intraoperative computed tomography-guided thoracoscopic lung resection. J Thorac Cardiovasc Surg 2018;156:1974-83.e1. [Crossref] [PubMed]
6. Zhang L, Wang L, Kadeer X, et al. Accuracy of a 3-Dimensionally Printed Navigational Template for Localizing Small Pulmonary Nodules: A Noninferiority Randomized Clinical Trial. JAMA Surg 2019;154:295-303. [Crossref] [PubMed]
7. Keating J, Singhal S. Novel Methods of Intraoperative Localization and Margin Assessment of Pulmonary Nodules. Semin Thorac Cardiovasc Surg 2016;28:127-36. [Crossref] [PubMed]
8. Predina JD, Newton A, Corbett C, et al. Localization of Pulmonary Ground-Glass Opacities with Folate Receptor-Targeted Intraoperative Molecular Imaging. J Thorac Oncol 2018;13:1028-36. [Crossref] [PubMed]
9. Vávra P, Roman J, Zonča P, et al. Recent Development of Augmented Reality in Surgery: A Review. J Healthc Eng 2017;2017:4574172 [Crossref] [PubMed]
10. Molina CA, Theodore N, Ahmed AK, et al. Augmented reality-assisted pedicle screw insertion: a cadaveric proof-of-concept study. J Neurosurg Spine 2019; Epub ahead of print. [Crossref] [PubMed]
11. Fitzpatrick JM, West JB, Maurer CR Jr. Predicting error in rigid-body point-based registration. IEEE Trans Med Imaging 1998;17:694-702. [Crossref] [PubMed]
12. Schwarz Y, Mehta AC, Ernst A, et al. Electromagnetic navigation during flexible bronchoscopy. Respiration 2003;70:516-22. [Crossref] [PubMed]
13. Anayama T, Qiu J, Chan H, et al. Localization of pulmonary nodules using navigation bronchoscope and a near-infrared fluorescence thoracoscope. Ann Thorac Surg 2015;99:224-30. [Crossref] [PubMed]
14. Kondo R, Yoshida K, Hamanaka K, et al. Intraoperative ultrasonographic localization of pulmonary ground-glass opacities. J Thorac Cardiovasc Surg 2009;138:837-42. [Crossref] [PubMed]
15. Pan J, Liu W, Ge P, et al. Real-time segmentation and tracking of excised corneal contour by deep neural networks for DALK surgical navigation. Comput Methods Programs Biomed 2020;197:105679 [Crossref] [PubMed]
16. Luo H, Yin D, Zhang S, et al. Augmented reality navigation for liver resection with a stereoscopic laparoscope. Comput Methods Programs Biomed 2020;187:105099 [Crossref] [PubMed]
17. Dennler C, Jaberg L, Spirig J, et al. Augmented reality-based navigation increases precision of pedicle screw insertion. J Orthop Surg Res 2020;15:174. [Crossref] [PubMed]
18. Park CH, Han K, Hur J, et al. Comparative Effectiveness and Safety of Preoperative Lung Localization for Pulmonary Nodules: A Systematic Review and Meta-analysis. Chest 2017;151:316-28. [Crossref] [PubMed]
19. Chao YK, Wen CT, Fang HY, et al. A single-center experience of 100 image-guided video-assisted thoracoscopic surgery procedures. J Thorac Dis 2018;10:S1624-30. [Crossref] [PubMed]