Visual Tracking in Conventional Minimally Invasive Surgery

Shahram Payandeh

• Publisher: Chapman and Hall/CRC
• Year: 2016
• ISBN:
  9781315350486
• Edition: 1
• Topic: Computer Science
• Subtopic: Programming Games
• Index: Computer Science

Visual Tracking in Conventional Minimally Invasive Surgery introduces the various tools and methodologies that can be used to enhance a conventional surgical setup with some degree of automation. The main focus of this book is on methods for tracking surgical tools and how they can be used to assist the surgeon during the surgical operation. Various notions associated with surgeon–computer interfaces and image-guided navigation are explored, with a range of experimental results.

The book starts with some basic motivations for minimally invasive surgery and states the various distinctions between robotic and non-robotic (conventional) versions of this procedure. Common components of this type of operation are presented with a review of the literature addressing the automation aspects of such a setup. Examples of tracking results are shown for both motion and gesture recognition of surgical tools, which can be used as part of the surgeon–computer interface. In the case of marker-less tracking, where no special visual markers can be added to the surgical tools, the tracking results are divided into two types of methodology, depending on the nature and the estimate of the visual noise. Details of the tracking methods are presented using standard Kalman filters and particle filters.

The last part of the book provides approaches for tracking a region on the surgical scene defined by the surgeon. Examples of how these tracking approaches can be used as part of image-guided navigation are demonstrated. This book is designed for control engineers interested in visual tracking, computer vision researchers and system designers involved with surgical automation, as well as surgeons, biomedical engineers, and robotic researchers.

CHAPTER 1

Overview of Minimally Invasive Surgery

THERE HAVE BEEN MANY definitions associated with the term minimally invasive surgery (MIS). We state here the most common definition, which is also directly related to the theme of this book. MIS is a procedure in which a surgeon can access the surgical site through small incisions or natural openings into the body. In general, through one of these ports of entry the surgeon inserts a long-stem endoscope with an illuminating tip that allows the viewing of the surgical site on an overhead monitor. Through other ports of entry, various long-stem surgical tools are inserted that allow the surgeon to perform manual surgical procedures. In general and from the patient’s perspective, this type of surgical procedure results in a reduction in hospital stay, a faster recovery time, less chance of infection, and a reduction in large tissue scars when compared with open surgery. Currently, MIS is performed in a large number of general surgical procedures for the gallbladder, lungs, heart, spine, and urinary system (Singla et al. 2009). The surgical tools and systems used for a typical MIS procedure in an operating room are very similar (Figure 1.1). For example, cholecystectomy (i.e., gall bladder removal), one of the most common MIS procedures, is performed using laparoscopy (Sages 2016).

Laparoscopic surgery as a minimally invasive technique was successfully introduced for gynecological procedures in the early 1970s (Spaner and Warnock 2009). Nowadays, more and more surgical procedures are likely to be performed using laparoscopic techniques as the advantages of minimally invasive procedures are further realized. In laparoscopic surgery, a surgical operation is performed with an endoscope (laparoscope) and a number of long, thin, rigid instruments (see Figure 1.2) inserted into the body through several small incisions created by the surgeon. As stated, when compared with traditional open surgery, it can provide less pain, faster recovery, and smaller scars from the incision points. During conventional two-handed laparoscopic procedures, the field of view of the surgical site through an endoscope sometimes needs to be adjusted. This adjustment is usually accomplished by the surgeon’s assistant, who holds the endoscope while viewing the surgical site and listening to the verbal commands of the surgeon.

FIGURE 1.1 Placement of the endoscope in various MIS procedures.

A common surgical challenge in conventional laparoscopic surgery is the lack of 3-D perception of the surgical site given in a conventional operating room theater. The surgeon needs to view the surgical site through a single camera (which can have more than one charged-couple device [CCD] sensor for better resolution) that can offer a clear 2-D projection of the surgical site (Figure 1.3). Currently, technologies are being introduced for creating a stereo view of the surgical site (Visionsense 2016). However, in general, the implementation of such tools requires the surgeon to wear a headpiece, which results in a loss of peripheral vision. Besides the main requirement of viewing the surgical site on the overhead monitor, the surgeon also wants to have a clear view of the other tools, equipment, and personnel around the operating table. In addition, the benefits of 3-D imaging in comparison with conventional 2-D imaging are not highly valued as yet in the medical communities (Clinical 2016).

FIGURE 1.2 Visualization of the main instruments used for laparoscopic procedures: (a) placements of the laparoscope and surgical tools; (b) endoscopic view of the surgical site.

Besides MIS having a number of benefits for patients, it also presents additional challenges for surgeons. These include the loss of a direct view of the surgical site, the loss of tactile perception, and unnatural hand–eye coordination. To compound such challenges, a direct view of the surgical site is replaced by a distorted endoscopic view and the surgeon is required to manipulate the surgical site using long-stem surgical instruments. Surgeons have to operate by looking at viewing monitors displaying images of cavities at awkward angles with a very restricted range of views. Moreover, they can no longer touch tissues with their hands, as only long, thin surgical instruments small enough to go through the tiny incision holes are used. Therefore, surgeons have to develop new hand–eye coordination and instrument manipulation skills in order to cope with the less ergonomic operating environment and limited working space.

FIGURE 1.3 Two examples of a conventional operating room setup for MIS.

In performing a conventional MIS procedure (Figure 1.3), a typical surgical theater consists of the surgeon, the assistant to the surgeon, who holds and positions the endoscope, and the supporting nurses, who supply surgical instruments and medical information about the patient. Historically, besides some of the challenges in performing MIS previously stated, the introduction of some degree of automation in such procedures has been desirable. This can lead to a reduction in the number of surgical personnel in the operating room and also may result in a consistent view of the surgical site through the endoscope. Some of the earlier attempts to introduce some degree of automation are presented in Faraz and Payandeh (1998, 2000). Full automation was later introduced through a product by Intuitive Surgical in 2000 (Intuitive 2000). The main idea of this product is to replace the presence of the surgeon and, during various intervals, the presence of supporting personnel in the operating theater, and to have robotic arms holding the endoscope and surgical instruments. These robots are controlled by a surgeon located at a remote workstation (usually in the operating room). By viewing 3-D images of the surgical site on the surgical workstation and by manipulating special hand-held mechanisms, the surgeon can control the viewing angle and motion of the surgical tools at the site.

OVERVIEW OF MINIMALLY INVASIVE SURGERY TRAINING ENVIRONMENTS

Mastering the challenges in performing MIS procedures requires the surgeon to enhance his or her skills by both using training environments and by performing the actual procedures under the supervision of an experienced surgeon. The training environments range from an emulated physical setup consisting of an endoscope and surgical tools inserted inside a training box to a virtual surgical training environment. In the virtual environment, the view of the endoscope and the presence of the surgical tools are represented graphically. The position of the endoscope in the virtual environment can be controlled externally and the trainee can interact with the virtual surgical tools through various user interface devices (Payandeh and Li 2003; Payandeh 2015; LapMentor 2016).

Virtual training environments can offer a variety of learning scenarios with varying degrees of difficulty. Since both the motion of the surgical tools and their interaction with the virtual environment are computed and displayed through various algorithms, it is also possible to collect such data for further analysis and evaluation. As such, various metrics can be developed that can be used to assess the performance of the trainee.

One of the drawbacks of deploying such a system is its initial cost and the further costs associated with its maintenance. Besides offering a platform for mastering the basic hand–eye coordination that is needed for a general class of MIS (Shi and Payandeh 2010), the virtual simulation environment also lacks the necessary realism. In order to increase the realism of the virtual surgical site, numerical methods and associated material properties of tissue and organs need to be collected and modeled.

Surgical training boxes are another alternative for enhancing the basic skills needed for MIS. They offer a physical in vitro setup in which the body cavity is recreated as a box using basic materials such as rubber or plastics. An endoscope is inserted inside the box through an opening and, depending on the type of surgical procedure, surgical tools are inserted through various predefined incisions (Figure 1.4) (LapTrainer 2016; Fls 2016; Seal 2016).

FIGURE 1.4 Examples of physical in vitro training boxes for MIS.

Some of the key advantages of surgical training boxes are the ease of their manufacturability and their cost-effectiveness. They can be assembled using off-the-shelf materials and are being utilized in most surgical curricula around the world (Ma and Ujiki 2015). However, while offering a physical setup that can be used to emulate the actual surgical environment, these types of training boxes do not offer any degree of automation. For example, being able to visually track the surgical tool during a basic surgical procedure can offer a metric for evaluating the performance of the surgical residence. Tracking of the surgical tool combined with the task duration can offer another type of metric that can be used as part of the performance skills assessment. In addition, enhancing such basic surgical training boxes with advanced notions of visual mentoring and guidance can further increase their range of effectiveness. For example, by monitoring the motion of the surgical tools during a specific procedure, graphical images and overlaid information can be displayed that can be used as a mentoring tool during image-guided navigation (Parada and Payandeh 2009). The visual tracking methods presented in this book can be used as one of the main frameworks for the further enhancement of such physical surgical training environments with some degree of automation.

CONVENTIONAL VERSUS ROBOT-ASSISTED MINIMALLY INVASIVE SURGERY

Conventional MIS is performed in most countries around the world regardless of their financial and economic status. Besides the usual costs associated with using an operating theater, the costs related to surgical theater setups for conventional MIS are nominal (JHN 2015). Similar to automating other application areas, introducing some degree of automation to the surgical field and in particular to MIS can offer both advantages and disadvantages. One of the main disadvantages of introducing a high degree of automation through a teleoperation system (Intuitive 2000) is the initial high cost of its deployment and the associated maintenance costs. As a result, such systems are only available to medical centers and institutions around the world in better financial positions and with more means. In addition, since the deployment of robot-assisted surgery, various studies have been conducted to further understand its overall advantages to health care when compared with the conventional approaches to performing MIS.

As a part of the comparative studies, Sarlos et al. (2012) concluded that robot-assisted laparoscopic hysterectomy and conventional laparoscopy compare well in most surgical aspects, but the robotic procedure is associated with longer operating times. The postoperative quality-of-life index was better. However, in the long term, there was no difference, and subjective postoperative parameters such as analgesic use and return to activity showed no significant difference between groups. Another study conducted by Baek et al. (2012) concluded that robot-assisted surgery is more expensive than laparoscopic surgery for rectal cancer. Considering that robotic surgery can be applied more easily for low-lying cancers, the cost-effectiveness of robotic rectal cancer surgery should be assessed based on oncologic outcomes and functional results from future studies. Wright et al. (2013) analyzed the uptake of robotically assisted hysterectomy to determine the association between the use of robotic surgery and rates of abdominal and laparoscopic hysterectomy, and compared the in-house complications of robotically assisted hysterectomy with abdominal and laparoscopic procedures. They concluded that between 2007 and 2010, the use of robotically assisted hysterectomy for benign gynecologic disorders increased substantially. Robotically assisted and laparoscopic hysterectomy had similar morbidity profiles, but the use of robotic technology resulted in substantially higher costs.

In general, the most common drawback to introducing an advanced degree of automation to surgical procedures is the associated cost. Although the advancement and availability of tools and technology can result in an overall cost reduction for deploying surgical robots, there also exist some challenges in reassuring their safe integration, design, and usage. For example, the BBC (2015) reported that robotic surgery had been linked to 144 deaths in the United States.

MOTIVATIONS AND OBJECTIVES OF THIS BOOK

This book explores various approaches in which it is possible to introduce some level of automation to the conventional MIS and surgical training environments. The framework is based on utilizing the existing tools that are present in a conventional operating room, which include an endoscope, surgical tools, and a camera system. The images from the camera used on the viewing monitor are also captured and processed using a conventional computational platform (e.g., a desktop personal computer). As explored in this book, by being able to track the motion and orientation of surgical tools, it is possible to achieve various implementations of the notions related to a surgeon–computer interface (SCI). In this paradigm, similar to the notion of a computer mouse as a human–computer interface, the tracked motion of the surgical tool can be used as a surgical mouse. Using the calibration parameters of the endoscope, this analogy can further be extended to a 3-D mouse paradigm, where the spatially tracked information of the surgical tool can be utilized. This allows the surgeon, as needed, to retrieve and interact with medical information related to the patient through overlaid graphical images (i.e., augmented reality). For example, Figure 1.5 shows a conceptual representation of an operating theater where, by using a surgical tool, the surgeon is making a particular gesture. Through visual recognition of the gesture, it is possible for the computer to retrieve medical images and display the information on an overhead monitor, which can be used to guide the surgeon in a particular surgical procedure. Besides the automatic retrieval of information, the proposed framework of this book offers approaches that can be further extended to allow surgeons to manipulate the overlaid graphical images on their endoscopic display to further register them on the surgical site. Preoperative image registration on the surgical site to assist with image-guided surgery has been a challenging task. The method of this book is to offer approaches in which, using surgical tools, the surgeon can manually position overlaid preoperative graphical images on the surgical site. In addition, it shows that it is possible to introduce the notion of image tracking to track endoscopic features on the surgical site. As such, it is possible to register preoperative graphical information on the endoscopic site and maintain its position regardless of the motion of the endoscope. This allows surgeons to also place virtual overlaid markers on the surgical site using their tools and to anchor them to the tissue regardless of the motion of the endoscope.

FIGURE 1.5 A conceptual operating room, where the gesture of the surgical tool is recognized (left monitor) and then associated with certain medical images (right monitor).

LITERATURE OVERVIEW

In this book, the methods for tracking surgical tools are divided into two main categories. The first category is the tracking methodology, where the surgical tools are designed with specific artificial visual markers and gestures located on the tool. The second category is the general case, where surgical tools do not have any specific markers assigned to them and their shapes are used to accomplish visual tracking results. The last chapter of the book explores and presents some results that can be used for tracking special features on the surgical scene.

Surgical tool gesture recognition is easier than human hand gesture recognition as surgical tools have a fixed color and shape. Their surface colors do not differ from one another, as human skin tones do. Furthermore, surgical tools do not deform, as human hands do. In other words, the number of variations of one marked surgical tool gesture is less than that of a human hand gesture. However, surgical tool gesture recognition presents its own unique challenges, the most difficult being extracting surgical tools from the background. In most human hand gesture recognition literature, the background is often very simple and constant. In our case, the background, which is composed of body tissues and organs, is more complex. Moreover, the background changes with laparoscopic movement and the reflections from body tissues and organs further complicate the surgical tool extraction process.

In order to achieve an accurate position of the surgical tool, a number of challenges must be overcome. One is that the only available measured information is through the use of ...

Table of contents

• Cover
• Half Title
• Title Page
• Copyright Page
• Table of Contents
• Summary
• CHAPTER 1 OVERVIEW OF MINIMALLY INVASIVE SURGERY
• CHAPTER 2 ENDOSCOPE SETUP AND CALIBRATION
• CHAPTER 3 MARKER-BASED TRACKING
• CHAPTER 4 MARKERLESS TRACKING: GAUSSIAN APPROACH
• CHAPTER 5 MARKERLESS TRACKING: NON-GAUSSIAN APPROACH
• CHAPTER 6 REGION-BASED TRACKING
• REFERENCES
• APPENDIX I
• APPENDIX II
• APPENDIX III
• APPENDIX IV
• INDEX