Content:

Abstract

1. Introduction
    1.1 Aim of System
2. State of the Art
3. System Description
    3.1 Camera Model
    3.2 User Interface
    3.3 User Options
4. Implementation
    4.1 AVW-Library
    4.2 Tracking System
    4.3 Tracking Process and Communication
    4.4 Configuration
5. Results and Conclusions
6. Acknowledgements
7. References
 

Abstract

Minimally invasive surgery, developed during the last 15 years becomes a major field of surgical intervention. The patient benefits from little damage around the focus of surgery. Special skills of the surgeon are needed to work in narrow body cavities with little space for the endoscope and surgical tools.

The surgeon’s navigation during surgery can be supported by modern visualisation techniques based on 3D medical image data. A tool for intra-operative navigation was developed providing three orthogonal sections through the image volume, a 3D display of surgical tools relative to sensitive anatomical structures, a virtual endoscopic image together with the video image through the endoscope. The user interface has a simple layout for easy handling to meet the needs in the surgical theatre.

Keywords: virtual endoscopy, volume rendering, computer-assisted surgery
 

1. Introduction

 
In otorhinolaryngological (ORL; ear, nose, and throat) surgery minimally invasive techniques (endoscopic techniques) are widely applied. In comparison to commonly used surgical techniques, minimally invasive procedures have less adverse effects on the patient, but have the drawback that the field of surgery is not directly visible and the field of view is decreased through the endoscope. Besides, the metrics of the optical system differs substantially from the familiar spatial perception. Due to the extreme optics only experienced ORL specialists can judge the distorted anatomic structures. Furthermore the endoscopic view is decreased during the surgery by blood and humidity covering the surface of the lens.

1.1 Aim of the System

To aid minimally invasive surgery an intra-operative surgical navigation system was developed. It is based on multi-modal 3D image data, providing complementary information (e.g., magnetic resonance imaging (MRI) and x-ray computed tomography (CT)) to the surgeon.

The visual information is generated by using all available 3D imaging modalities (Spiral-CT, MRI, single photon emission computed tomography (SPECT) and positron emission tomography (PET)). The reason for using different image modalities lies in the properties of the imaging methods. CT images have a good contrast for bones, MR images show especially soft tissues, SPECT and PET are used for functional analysis.

The combination of volume data sets results from the registration of respective anatomical structures within the data sets. The registration of the structures is completed semiautomatically using algorithms, which fit to the image modality, e.g. region growing, morphological procedures, thresholding operations. Image combination is done by minimizing the distance between the surfaces of respective segmented structures. This is done in an optimized way by chamfer matching [1].

The volume data set is rendered accordingly to the position of the endoscope. The 3D view of the anatomical structures is calculated and displayed using the intrinsic imaging properties of the endoscopic optics. In addition to this view, we show the current position of the endoscope on a 3D view and 3 orthogonal slices of the volume at the position of the tip of the endoscope. We use an optical tracking system (FlashPoint 3D Localizer, Image Guided Technologies, USA) for monitoring the position of the endoscope relative to the patient. This tracking system can be installed in a surgical environment. Our developments provide guidance for the physician and a possibility to use different image views directly within the operating room.
 

 
There are a number of scientific research groups working in the field of visualizing medical volume data. An overview of the computational tasks and diversities of technologies, which support the different applications in the field of medical imaging, was produced by Ezquerra at al. [2].

Especially for virtual endoscopy two approaches, surface rendering [3] and direct volume rendering [4] are used for 3D visualisation.

The surface rendering methods use an intermediate segmentation step to transform the volume data into a mesh of polygons. Due to the large amount of voxels to be processed and the complexity of the scene, most segmentation techniques are semiautomatic. Manual segmentation is time-consuming and may contain operation errors, fully automatic procedures are not generally applicable. The resulting mesh of polygons can be rendered using the usual graphics hardware support. The drawback is often a time-consuming preparation phase and reduced accuracy, since information about the inner parts of the objects is lost. These techniques were implemented by several authors [5, 6, 7].

Direct volume rendering is used in many applications [8, 9, 10, 11]. Two-dimensional views are generated by casting rays from an observation point through the entire volume. No loss of information has to be taken into account and by rendering transparent objects, information about the scene behind the objects is visualised. Limitations are that small, complicated internal structures within a large data set might be difficult to display. In recent time, hardware acceleration [12, 13] made it possible to achieve interactive frame rates and implement new solutions [11].

There are also applications, which use both techniques [14].

Virtual endoscopy systems are used today in a wide field of applications, such as interactive virtual colonoscopy [5, 14], providing an overview of the colonic surface to support the navigation of the endoscope and to avoid penetration of the surface. Others suggest the use for virtual bronchoscopy [6], virtual ventricle endoscopy [7] or endoscopic sinus surgery [15]. Furthermore, it is possible to create interactive fly-throughs. For training purposes, surgery is simulated in a virtual environment for optimal path calculation, orientation, diagnosis and tumor studying. A drawback of most virtual endoscopy systems is that they do not simulate the distorted optical model of a real endoscope.
 

As pointed out Ezquerra et al. [2], there is a number of reasons, which affect adversely the use of diagnostic visualization of 3D medical data. One reason is that clinical specialists are used to interpret or „read" 2D images rather than 3D volumes or 3D models. Another consideration is the discrepancy, sometimes considerable, that has been noted between the information that is available in the volume data and its visualization in 2D formats. It seems strange that many imaging modalities lead to volumetric representations, nonetheless, for diagnosis purposes, radiologists have all but ignored it, while surgeons and radiation oncologists need it. Clinically useful applications of 3D imaging are not restricted to visualization for diagnostic purposes or planning of the surgery. Furthermore intra-operative applications fully exploit information provided by 3D images. It supports the idea of seeing 3D images when acting in a 3D environment.

The motivation for our display system was to add complementary information to the current endoscope systems. In the current surgical equipment only the video image through the endoscope is displayed. A 3D overview of the position of the endoscope relative to a 3D rendering of the patients head, 3 orthogonal slices of the volume data at the position of the endoscope and a perspective rendering using the optical model of the real endoscope are added. Thus useful means of orientation are provided to the surgeon during intervention.

3.1 Camera Model

 
The camera model, which can be seen in figure 1, was developed for simulating the real endoscope.
 

 

Figure 1: Camera model for virtual endoscopy: simple perspective model (left),
perspective camera model with endoscopic distortion (right)




The left section of figure 1 shows a perspective camera model used in computer graphics, the right section shows the fish-eye distortion of a real endoscope as used for endoscopic volume rendering. This effect has been implemented by wrapping the image plane around a sphere. This model was calibrated and verified by Eisenkolb et al. [16].
 
3.2 User Interface

A snapshot of the display system is shown in figure 2 and consists of the following windows:

• three orthogonal slices of the volume
• transversal slice (upper left window)
• sagittal slice (lower left window)
• coronal slice (lower middle window)
• transparent rendering window (upper middle window)
• endoscopic rendering window (upper right window)
• video image (lower right window, future development)

Figure 2: User interface of the display system

The selection of the current slices (transversal, coronal, sagittal) is done either by user input or automatically by the tracking system using a defined position on the endoscope, eg. the tip, as reference. It is possible to map the information about the anatomical structures on the actual slices as shown in figure 2, which shows the brain (violet), the tumor (blue), optical nerve (green) and bone (white). The information about the density is still available by using different shades of the color of the object. The orientation guidance lines, which represent the position of the endoscope at line crossings, are shown. The lines also indicate the position of the orthogonal slices. The contour of the endoscope can also be displayed in these images.

A 3D view (figure 2: transparent rendering window) is generated using a transparent volume rendering algorithm [4]. During the pre-processing each voxel is aligned to an anatomic structure and for each structure color and opacity are defined. A voxelised endoscope is modelled and rendered together with the volume data set. The rendering is the most-time consuming task. To improve the rendering performance we use an image-based method. The position of the endoscope is calculated first. If the endoscope is inside the volume, the corners of the subvolume, where the endoscope is currently positioned, are mapped to the image plane and define those pixels, which are affected by the change of the position of the endoscope. These pixels are called rendermask. Therefore only a small part of the rays has to be recalculated during the rendering process, which allows interactive display speed of the system. The size of the rendermask depends on the size of the endoscope and the part, which is currently within the volume data set.

The size of the image can be customised from the configuration file. It is possible to zoom in and out as well as to rotate the viewpoint to allow a view of the volume data from different angles and to enlarge areas of interest.

The endoscopic rendering at the display system uses the algorithm described Eisenkolb et al. [16]. There are 3 different implementations of this algorithm. The "preview" mode is used for fast image generation with less accuracy, the "detail" version takes more time but produces a more precise output. The "object" mode includes the colors and opacities from the object map.

As a future development the display of the current video image from the endoscope is planned using a video frame grabber.

3.3 User Options

Beside the command-buttons for each window the following options are accessible for the user:

Data management options

It allows the loading of the volume data set and its object information as well as its unloading. It is furthermore possible to get additional information about the volume, e.g. date of scan, information about the patient and the original voxel size.

Display options

There are some functions for manipulating the display of the object map. It is possible to superimpose the grey-scale image of the 3 orthogonal slices with the colored information about the anatomical structures. Display options for each object can be set interactively. This allows to turn on and off the visualization of selected structures, e.g. nerves, vessels, which are of particular interest during surgical intervention. This influences the transparent rendered image as well.

The display of orientation guidance lines on these images can also be selected here. These lines show the position of the tip of the endoscope. Another function allows the superimposition of the contour of the endoscope on these images.

Further options are available for turning on and off all windows on the screen. A screen refresh function is implemented.

Tracking system options

This menu topic provides access to two functions concerning the tracking process. The first is the start of the registration process of the tracking system with the volume data. The second starts and stops the tracking. During tracking the position of the endoscope is shown on the three orthogonal slices and the 3D view.
 

 
The display system has been implemented in an X-Windows environment using C/C++ as programming language running on a Sun UltraSPARC II Workstation using Sun Solaris 2.6. The system architecture is shown in figure 3:



Figure 3: Architecture scheme

The implementation uses the basic X-Library and the X-Toolkit together with the OSF/Motif window manager. As can be seen in figure 3, two platform independent libraries are used, the AVW-Library, described in section 4.1 and the endoscopic view functions, which implement the optical model of the endoscope described in section 3.1.

4.1 AVW-Library

 
The software basis are the volume data types and image processing functions of the AVW-Library, developed by the Biomedical Imaging Resource of the Mayo-Foundation, Rochester, USA [17].

AVW is a comprehensive C-library for manipulation of 3D medical data sets. It allows an extensive analysis of multi-dimensional and multi-modal data sets. The library supports the development and implementation of image-manipulation algorithms and purpose build software solutions. The basics of the procedures are special data types for images and volumes.

The data types for volume data range from binary (1 bit) to 24 bit color images. The information about the objects within the volume is provided by an additional volume of the same size as the medical data set. It allows the definitions of up to 255 objects and maps each voxel of the medical volume to an object. Each object has, amongst other properties, its specific color and opacity, which are used for transparent rendering and display of orthogonal slices. The functions provided by the AVW-Library can be divided into the following groups [17]:
 

• transformation: 2- and 3-dimensional mathematical and spatial transformations
• analysis: extraction of profiles and regions, manipulation and computation for different parameters e.g. size, density, texture within selected regions
• segmentation: functions such as thresholding, automatic detection of borders, creation of objects within the volume, extraction of surfaces
• visualisation: rendering using different parameters (depth shading, gradient shading, maximum intensity projection, summed voxel projection, surface projection, transparency shading), extraction of orthogonal slices, extraction of slices with arbitrary orientation
• manipulation: filtering, changing of the volume resolution (size, depth), statistical functions
• general functions: input / output of volume data sets, object data sets, single images, manipulation of data types


4.2 Tracking System

 
The tracking system is an optical tracking system (FlashPoint 3D Localizer, Image Guided Technologies, Inc., USA) which consists of two components. The first is a static optical sensor assembly with 3 high-resolution linear digital cameras. The second component consists of the endoscope and a tracking sensor attached to it. This tracking sensor uses light emitting diodes (LEDs). The optical tracker determines the position of these LEDs, which it computes as the intersection point of three planes defined by the photoelectrical sensors of the tracker. A control unit fires the LEDs in a sequence so that the camera assembly can identify single LEDs. Placing at least three LEDs at well-defined positions on a tool, e.g. the endoscope, the position and the orientation of the tool can be tracked.

4.3 Tracking Process and Communication

 
The communication between the tracking and the display process consists of two steps: registration of the patient image data and tracking.
 
 

Display Process                                                 Tracking Process



Registration Step
 



Tracking Step

 
The tracking system sends a reference point to the display process at the registration step. This reference point allows the calculation of the endoscope position relative to the volume data set. During tracking, the distance and rotational parameters of the endoscope in relation to the reference point are sent to the display process, which in turn calculates the position of the endoscope tip inside the volume data set.

For software development and testing, a simulation of the tracking system was implemented. It has 6 degrees of freedom for movement of the endoscope and eases the testing of the process communication.

The process communication between the tracking process and the display process is based on the computation time needed for the transparent rendering. To prevent an event overflow of the display process, the tracking process sends approx. two new positions per second to the display process. This is within the interactive speed range we determined necessary for clinical use.

4.4 Configuration

 
A configuration file allows the adaption of the interface to suit user-preferences. It allows the selection of different windows (orthogonal slices, 3D view, endoscopic view, video image) and the position of the windows on the screen. The size and the default parameters (position of camera) of the 3D view are also defined in the configuration file.

It is further possible to disable the communication to the tracking process and switch to a off-line mode, which can be used for training purposes.

The shape of the simulated endoscope is a voxelized object, whose color and size can be defined in the configuration file. The endoscope can be shown on the 3 orthogonal slices and on the transparent rendered image.
 

 
We presented a display system for navigation in ORL-Surgery to be used for intra-operative navigation. An off-line mode (disabled connection to the tracking system) is provided for training purposes.

In surgical applications, vessels and nerves close to the tissue surfaces must not be damaged. Although modern computer graphics hardware supports real-time surface rendering, we used transparent volume rendering for simultaneous visualisation of structures beneath the surface.

The most time-consuming task is the 3D transparent rendering algorithm. It takes about 2 seconds to render the 256 x 256 pixels image shown in figure 2. The volume consists of 320 x 320 x 216 voxels and the resolution is 8 bit. We improved the rendering by applying a rendering mask and therefore we render only selected pixels when tracking the endoscope. The speed-up factor depends on the position of the endoscope (parallel or orthogonal to the viewing direction) and on its size, but the average speed-up is around 3.5, which leads to approximately 2 frames per second. Unfortunately, when selecting a new viewpoint, this advantage is lost. A precomputation of the 3D view from different selected viewpoints was not deemed sensible. There is hardware available for parallel perspective rendering [13], which achieves interactive frame rates. Unfortunately it does not fit to our hardware environment.

Future developments will see the inclusion of the video image from the endoscope using a video grabber. It is further planned to implement stereoscopic rendering for the 3D view window. When using shutter glasses the user will get a 3D impression about the volume data, the position of the anatomical structures and the position of the endoscope.
 

 
The work was supported by Project P12463-MED of the Austrian Science Fund.

The authors wish to thank Dipl. Ing. Monika Eisenkolb for providing the endoscopic view functions. Special thanks to Dr. Monika Cartellieri of the clinic of ORL, Vienna University Hospital, for the access to the endoscope equipment, providing image data and valuable discussion and to Dr. Fritz Vorbeck from the Clinic of Radiology, Vienna University Hospital, for providing MR and CT data.