HapticIO : Haptic Interface-Systems for

Virtual-Reality Training in Minimally-Invasive Surgery

Kühnapfel U (1), Çakmak H (1), Chantier B (1), Maaß H (1), Strauss G (2), Trantakis C (2),
Novatius E (2), Meixensberger J (2), Lehmann K (3), Buhr HJ (3), Lawo M (4), Bretthauer G (1)
(1)
Forschungszentrum Karlsruhe,- IAI-, Postfach 3640, 76021 Karlsruhe, Germany
(2)
Universitätsklinik Leipzig, Liebigstr. 18 a, 04103 Leipzig, Germany
(3)
Klinikum Benjamin Franklin der Freien Universität Berlin, Hindenburgdamm 30, 12200 Berlin
(4)
Select IT VEST Systems AG, Wilhelm-Herbst-Str. 12, 28359 Bremen, Germany

Abstract

In HapticIO basic methods and technologies were developed and verified for a new haptic interface
system as component of "Virtual Reality" simulators for minimally invasive surgery. A modular and
scalable concept was pursued, which can be used, with small adaptations, for various surgical
disciplines. After development of hard and software, the devices were tested and evaluated.
Acceptance trials with clinical partners were performed and the simulators were verified in clinical
training courses with users. Various surgical VR-scenarios were developed by the clinical partners
involved in the project. (visceral-, neuro- and ENT surgery). Additionally, the high medical,
technological and commercial competence of the partners involved in HapticIO guarantees a fast
transfer of results from HapticIO into application.

This overview report describes requirements, conception and realization of the HapticIO developments
after finishing the project (36 month). We present hardware components (mechanics, electronics) of
the haptic devices, and the VR-scenarios (patient models) developed within HapticIO for the different
surgical disciplines involved in the project. Field reports of clinical users are available, the suitability
of VR-based training for surgical education and training has been verified by comparative clinical
trials. Finally, statements are given regarding the future result exploitation into clinical use and the
HapticIO developments are summarised.

This project HapticIO was funded by the German mistery of education and research (BMBF) through
„Projektträger–Informationstechnik“ (PT-IT, DLR) within the funding programme “Virtual and
Augmented Reality” (VR-AR) (Ref: 01IRA03).

1. Introduction
Minimally invasive surgery (MIS) has been established among surgeons as an elective technique
in a number of surgical interventions. Education and training of surgeons therefore is becoming
more and more important, since the risk for patients is minimised, fast patient recovery helps to
reduce healthcare costs and the efficiency of the surgeon is increased. Traditionally, MIS-training
is performed on plastic models (pelvi-trainer), animals or human cadavers (e.g. in ENT-surgery).
These traditional training methods cannot represent highly realistically the “real” living human
patient because of their difference in anatomical features (animal), lack of detail (plastic models)
or different optical or physical material properties (plastic models, cadavers). The ever increasing
demand for higher visual and physical realism and the wish for material, cost and time savings
together with ethical problems using animals or human cadavers, was therefore the trigger for
searching new methods in surgical “hands-on” education and training. Acceptable computing
performance, coupled with “virtual reality” (VR) technology and new algorithmic simulation
methods allowed therefore the development of “VR-based Surgery Training Systems” since
approximately 1990 as an alternative to the more traditional methods as described above.
VR-surgery simulators to be used effectively in the daily clinical training routine, require a
transformation of the reality into a virtual world. This covers, in addition to the visual impression
of the operation situs and the interaction between the objects to be manipulated (organic tissue),
the tactile interaction (haptics) between the surgical instrument and the hands of the surgeon, i.e.
the haptic impression. Realistic models and the simulation of their behaviour (deformation,
dynamics, kinematics, haptics) therefore build the fundamental base for VR-based surgery
simulation.
Within the project HapticIO, methods and technologies for a new haptic user interface system
(mechanics, electronics, software) as a component of VR-based surgery training systems for
minimally-invasive surgery (VR-MIS-Trainer) was developed, tested and evaluated by the
participating scientific and industrial project partners.
A modular and scalable strategy should be observed, so that the haptical interface devices, to be
developed, could be used for the configuration of VR-MIS-Trainers for various surgical
disciplines. As a result of pre-considerations and initial investigations, the decision erupted to
target the first development steps into the surgical application fields of laparoscopy, neuro-
surgery and ear-nose-throat (ENT) surgery. This aim overlapped with a needs assessment done
with potential users.

Fig 1a: „Karlsruhe VR-MIS-Trainer“ (1997) Fig 1b: VEST-VSOne Product (2003)
Based on the preparatory pre-development of the „Karlsruhe VR-MIS-Trainer“ [1], the “Virtual
Endoscopic Surgery Trainer” (VEST) system VSOne was developed to product state within the
framework of an industrial co-operation project (technology transfer), running slightly previos in
time to the BMBF-project HapticIO, between the Karlsruhe research centre and „Select IT VEST
Systems AG“ (Bremen, Germany) (Figs. 1a-b, 2).
In the “zero-series” of VEST VSOne haptic devices of the type „Impulse Engine“ manufactured
by the US-company „Immersion Inc.“ are used as OEM product. These devices unfortunately do
not conform to the required quality regarding robustness and conceptual design for a daily
training routine. Intolerable efforts for maintenance and/or repair would be the consequence.
Additionally these devices cannot transfer gripper-forces and shaft-rotation torque back to the
user.
Resulting from the experience gained during the development of VSOne, it was planned to realise
additional VEST-systems for other surgical disciplines. These systems will differ considerably
from VSOne – apart from the VR patient situs – in their haptic user interface. Haptic (force-
reflecting) devices available on the commercial market cannot be used in this development(s) for
technical reasons (number of degrees-of-freedom, manufacturing quality and robustness,
kinematical structure …) and also because of economic considerations (price, competitive
situation of the manufacturer). Additionally, all commercially available haptic systems are
manufactured abroad (USA). Marketable haptic devices of required size and technology were not
available so far, neither in Germany nor in the whole world. Especially Germany provides ideal
resources for development, manufacturing and marketing of “haptic” devices. Precision
mechanics and (recently) “mechatronic” systems have a long tradition in Germany, numerous
SME’s provide sufficient manufacturing capacity and quality.

Fig. 2: VEST-VSOne „user-interface-box“ with haptic devices (up to 3) and

”video-endoscope” device
The proprietary development of an appropriate, modular and scalable series of haptic devices was
therefore applicable. The new haptic devices should be suitable for integration into the VEST
VSOne system to replace the inadequate devices as used up until now.
2. Realisation
The R&D-work in HapticIO was split thematically into a structure of 5 major tasks:
HapticIO system development (mechanics, electronics, software),
Development of pilot-applications (patient VR-models, tutorials),
medical evaluation of laparoscopic application,
medical evaluation of neuro- and ENT-surgery applications,
pre-marketing and pre-manufacturing studies and explorations

„State-of-the-art“ and Concept

Existing haptic devices as known from research and industries were analysed intensively during
the first project phase. Some of these systems were already available within the HapticIO
consortium. Special consideration was focused on some systems, which are partly already
available commercially:
Laparoscopic Impulse Engine, Immersion Corp., San Jose, CA, USA
(refer to: http://www.immersion.com ).
PHANToM Master, SensAble Technologies, Woburn, MA, USA
(refer to: http://www.sensable.com ).
PantoScope, EPFL-Lausanne, CH and XITACT, Morges, CH
(refer to: http://www.xitact.com )
As a result from these investigations, a requirements specification document was compiled. The
basic systems requirements (development targets) are shown in a table, as follows:

Target parameters Laparoscopic Surgery Head Surgery

Number of Instruments 3 + endoscope 1 (+1)
Actuated degrees-of-freedom 6 + gripper 6
Haptic-Feedback 4 axes (torques) + grip 6 (forces and torques)
Working range 20 x 20 x 20 cm 10 x 10 x 15 cm
Max. Force / continuous load, 24 h 20 N / 3 N 30 N / 6 N
Friction 0.07 N 0.07 N
Stiffness 5 N/mm 5 N/mm
Nominal resolution 0.02 mm / 0.01 Grad 0.01 mm / 0.01 Grad
Inertia < 100 g < 75 g
Mechanical cut-off-frequency > 15 Hz > 25 Hz
Just noticable vibration frequency > 200 Hz > 200 Hz
Cycle-time of controller >= 1000 Hz >= 1000 Hz

Table 1: Target parameters of the haptic devices (mechanics)

Hardware-Developments (Mechanics, Electronics) in HapticIO
Within the task of system developments, various kinematical concepts were studied within a 3D-
CAD system. Two promising designs were carried out and prototypically manufactured at
Research Centre Karlsruhe. Fig. 3a, b shows the mechanical prototype for the laparoscopic user-
interface, the device for “head-surgery” is shown in Fig. 4. All designs were carried out as
“variant-models”. As 3D-CAD system, we used „ProEngineer“ (ProE).

Fig. 3a, b: Prototypic haptic device (mechanics) for laparoscopic surgery application

Fig. 4: Prototypic haptic device (mechanics) for Neuro- and ENT-surgery application

Parallel to the mechanical development, the electronic subsystem was realised (Fig. 5) in as an
experimental set-up. The electronics can be used for both mechanical configurations in HapticIO.
An USB-2 interface to the VR-workstation is used for communications between simulation CPU
and haptic controllers. A circuit design based on a –controller architecture was developed by the
R&D-team. Final layout condensation regarding component density and circuit integration will
be carried out in the final optimisation phase (pre-manufacturing), i.e. in spring 2004.

Fig. 5: Components of the electronic-subsystem (controller-, power-electronics)

Pilot-Applications
Medical image data (CT, MRI), which is visualised with 3D-texture-based, real-time volume
rendering techniques and moveable texture slices, serves as a modelling basis for realistic
modelling of surgical simulation scenarios in HapticIO. Further processing of these data
(DICOM format is used to read the medical datasets) is done using two in-hose developed
modelling software tools, called KisMo and Vesuv. Vesuv is used for data enhancement, filtering,
segmentation and reconstruction of volumetric voxel datasets. KisMo (KISMET-modeller) is
used for interactive 3D-modeling of deformable objects (i.e. “organic tissue”), using 3D-Spline
modelling techniques (interpolated Hermite-splines) to generate interconnected, texture-
enhanced, deformable 3D-objects. In KisMo, 3D medical image data serve as a modelling aid to
approximate the position and shape of individual patient organs. The user has the choice between
two modelling methods in KisMo: (a) modeling with pre-defined and (b) with interactively
shapeable free-form surfaces. With both methods, three basic object types (flat, pipe, ball) are
interactively modified to get the desired object shape. The user can interactively select and
modify start-properties of the objects like size, volumetric-mesh refinement level and object
name.
The simulation kernel software KISMET (another in-house development from Research Centre
Karlsruhe) is used for interactive real-time simulation in the VEST system. A complete VR-
simulation scenario, i.e. the “virtual patient”, is finalised in KisMo with surface textures and
deformation parameters. After adding the simulation data for the surgical instruments to be used
in the “virtual patient” operation and the data for interconnection with the haptics subsystem, the
final model is transferred into the VEST system.
Figure 6 illustrates the typical data-flow between acquisition of the medical datasets and finalised
“virtual patient“ scenario in the VEST system.
 CT / MR Dicom KisMo: Modelling VEST System

Project

Adaption

Deformable Objects
Slice-Images / 3D-Volumes
Vesuv: Segmentation KISMET: Simulation Polygonal Data

Fig 6: Data-flow during modelling of pilot-applications

Pilot-Application „Laparoscopic Surgery“

Testing and medical evaluation of the haptic devices for laparoscopic surgery is done with a
pilot-application of the VEST system for abdominal surgery. As a typical application, the
laparoscopic cholecystectomy (gall-bladder removal) was chosen in HapticIO. A training course
for MIS cholecystectomies, comprising 3 different scenarios, with variable degree of difficulty
(simple, medium, quite difficult) was developed in the pilot-application, together with a tutorial
and a “skills-level acquisition” module.

Fig. 7a,b: Interactive object-selection with Hermite-splines and deformable models of the liver
(right image) using the Visible Human dataset
The simulation scenario starts with a virtual endoscopic view into the abdomen. All relevant
surgical interactions (gripping of tissue, cutting, coagulation, clipping, irrigation and suction,
suturing) are simulated. Additional visual and physiological effects (bleeding, steam and smoke
during coagulation, organ motility) are implemented to enhance simulation reality.
The pilot-application was developed in close co-operation between the project partners university
clinic Benjamin Franklin (Berlin), responsible for the medical part, and Research Centre
Karlsruhe, responsible for the technical part of the development and provider of the modeling
software tools. A typical simulated view of the (virtual) situs is shown in figure 8.

Fig 8: VR simulation view of op-situs during cholecystectomy

Pilot-Application(s) „Head-Surgery“
Two different simulation set-ups for “Head-Surgery” were developed in HapticIO. One for
neurosurgery and another for “ear, nose, throat” (ENT) surgery.
Neurosurgery
A virtual model for endoscopic ventriculostomy was generated based on a MRI dataset of a real
hydrocephalic brain. Different software modules were used for segmentation (Vesuv), modelling
of anatomical structures and tips of surgical instruments (KisMo) and visualization (KISMET).
Characteristic endoscopic phenomena like fluid flow, rinsing, bleeding and pulsation were
simulated. The force feed-back system IO-Master7D offers 7 degrees of freedom and consists of
two coupled force feedback elements. The trokar and the acting instruments (scissor, bipolar
coagulation, balloon catheter) are captured separately. In this way the trokar`s position
determines the view of the endoscopic camera, the access to the target and the possible operating
range of the instruments. The virtual endoscopic camera can be fixed with a mechanical holding
device. A complex hydrocephalic configured ventricular system with realistic proportions and
anatomical structures was modelled. The surface of the elasto-dynamic ventricle walls and the
anatomical structures are close to real anatomy using a texture, that is acquired from intra-
operative photographs. An interactive virtual preparation with force feedback was implemented
coupling real surgical instruments (MINOP) with the haptic system IO-Master7D.
The system was evaluated in pilot series of 40 training procedures in 8 test persons to assess
procedure time, failure rate and kinematics.
The pilot-application for neurosurgery was developed in close co-operation between the project
partners university clinic Leipzig (clinic for neurosurgery), responsible for the medical part, and
Research Centre Karlsruhe, responsible for the technical part of the development and provider of
the modelling software tools. A typical simulated view of the (virtual) situs is shown in figs. 9a-c.

Fig 9a,b,c: Ventriculo-cisternomy-OP (real), Flight through the right ventrikel (model),
perforation of the floor of ventrikel III

ENT-Surgery
Another simulation scenario was developed for ENT surgery. A moulding and drilling interaction
with the mastoid bone is simulated. The VEST system is using voxel-volume visualisation
techniques for rendering in this application. Thus, the virtual patient model combines direct
volume rendering (hard bone tissue) with surface rendering technique (deformable soft tissues -
used for blood vessels). Initially a Phantom device (Sensable Technologies Inc., USA) was used
as haptic interface in this application. The Phantom device is replaced by the IO-Master7D
device developed within the framework of HapticIO.
The ENT surgery pilot-application was developed in close co-operation between the project
partners university clinic Leipzig (ENT clinic), responsible for the medical part, and Research
Centre Karlsruhe, responsible for the technical part of the development and provider of the
modelling software tools. The simulation set-up and the (virtual) situs are shown in figs. 10a-b.

Fig 10a, b: Simulation set-up for mastoid-OP, moulding-simulation at the mastoid bone with
integrated elasto-dynamic soft-tissue objects (blood vessels)
3. Results, Validation
Some subsystems of the HapticIO developments (VSOne, mastoid surgery simulator) were
already presented to a wide audience at trade fairs (e.g. MEDICA’2001+2002, CEBIT-2003) and
at medical scientific conferences (MMVR’02+03, CURAC’02+03, DGCh-Congress’2002) to
potential users. All developments were assessed very positive by the audience, especially from
the potential user group (surgeons).

Fig. 11: Test persons during the comparative study performed at UKBF, Berlin

A number of trials (comparative medical studies) were conducted by the project partner UKBF
with the VEST VSOne system (fig. 11) regarding the learning curve of probands. Simple and
similar exercises (“Basic Skills” tasks) were implemented in the VR-simulator and in the
(traditional) pelvi-trainer. Selected groups of test persons with similar experience level in MIS
surgery had to perform these “basic skills” exercises on both systems, VR-trainer and Pelvi-
trainer. Results of the exercises were measured (execution time, number of mistakes) and
displayed in form of diagrams (fig. 12) showing the learning curves on both systems. The results
were published in medical scientific journals.
Cam era: Reference at Pelvi-Trainer
Camera Skills - mean value by day
140
PT 200
120
VEST
100 PT Exp 150
VEST Exp
Time [sec]

80

60 100

40
50
20

0 14 4 . 3 87.0 49.5 3 9.3 56.1

0
Day 1 Day 2 Day 3 Day 4
Tutorial Day

Fig. 12: Data of the comparative test trials (VR-trainer / Pelvi-trainer)

Similar trials were performed in Leipzig with the simulation set-up for neurosurgery and ENT-
surgery, giving similar results (learning curves). All evaluation trials gave a positive result for the
VR-MIS-trainer concept, showing that on the VR-system similar learning results can be achieved
as with traditional learning methods. But the VR-trainer has the great advantage of a much higher
application spectrum and development potential. The set-up time is much smaller compared to
the pelvi-trainer (if used with animal tissue), thus reducing overhead cost drastically. The VR-
trainer concept, compared to the pelvi-trainer, has the advantage of highly realistic “virtual
patient” models of the OP-situs.

Exploitation
The VEST-systen VSOne is already being marketed as a product by the industrial project partner
in HapticIO, the company „Select-IT VEST Systems AG“, Bremen. License agreements between
Research Centre Karlsruhe and „Select-IT“ have been concluded allowing the commercial
exploitation of the technology. All systems and methods developed within HapticIO will be
commercially exploited in the near future.
Scientific project results from HapticIO have been published in about 20 scientific conference
contributions (technical and medical conferences) and journal articles. The systems were shown
demonstrated on various trade fairs (MEDICA, CEBIT, MMVR) within Europe and the USA.
The VSOne system was very positively evaluated on a closed workshop for invited guests
(decision makers from the US medical healthcare) during the MMVR’2003 in Newport Beach,
USA. It has proven to be a very competitive system in the worldwide VR-simulator competition.
In a comparative study on VR-based simulation, performed in 2003 in Eindhofen, Netherlands,
and published through the electronic journal “Surgical Endoscopy” (Springer) [2], VSOne has
reached the best overall rating of all existing VR-MIS-simulators in the worldwide competition.
HapticIO project results will be demonstrated “hands-on” during the CEBIT’2004 in Hannover
on the BMBF-booth.

Summary
The defined aims within the project HapticIO were fully reached completely and within time. The
co-operation and teamwork in the consortium was exzellent. From the surrounding field of
HapticIO already beginnings for new R&D projects resulted. Projects in the funding programmes
of the German countries Baden-Württemberg (LI-BW, project approved), the BMBF (in
preparation) and the European Union (6th Framework) are either in preparation or already
submitted.

References
(1) Kühnapfel U., Çakmak H. K., Maass H.: Endoscopic Surgery Training using Virtual Reality
and deformable Tissue Simulation. Computers & Graphics 24(2000) 671-682, Elsevier (2000)
(2) Schijven M., Jakimowicz, J.: Virtual reality surgical laparoscopic simulators. Surg Endosc
Vol 17 Number 12, 1943-1950, Springer (2003), DOI: 10.1007/s00464-003-9052-6