Student Projects
Contact Information
University: Ulsan National Institute of Science and Technology (UNIST), Korea
Team Members (with year of graduation): Sungbea Ban (2013), You-jin Ahn (2014), Sung-won Shin (2008), Hyeon-geun Kim (2015)
Faculty Advisers: Prof. Woong-gyu Jung
Email Address: whi506@unist.ac.kr, swshin@unist.ac.kr, spencer8929@gmail.com and wgjung@unist.ac.kr
Submission Language: English
Project Information
Title: Smart surgical microscope for accurate microsurgery
Optical Coherence Tomography for Optimal, Precise, and Ultra-advanced Surgery
Description
Microsurgery is an area of surgical operating in tiny structure of body, which deals with micro-bio structures such as blood vessel and nerve. It requires highly accurate and delicate operation technique, and its performance is varied by the physician’s accumulated experiences which can never be standardized in every individual surgeon. In past several decades, surgical microscope has been used as indispensable tool to observe tiny surgical area in which surgeons had difficulty to getting access to while manipulating the surgical tools. Moreover, it often brought about the medical issue of marathon-operation, resulting to medical accident; because it visualizes only surface of bio-structure with narrow field of view. Another problem is that dynamic monitoring in real time is hard to implement in current fashion of operating tool. Most of surgeon want to watch the surgical area ‘before surgery’, ‘during surgery’, and ‘after surgery’ continuously. So far, nevertheless, it has not been known for the optimal medical imaging tool to overcome these limitations.
For this reasons, there has been substantial demand for new operating protocol including ‘OCTOPUS’ system which is highly advanced surgical imaging tools. OCTOPUS is the abbreviation of “Optical Coherence Tomography for Optimal, Precise, and Ultra-advanced Surgery”, the integrated system of Optical Coherence Tomography (OCT) with Surgical Microscope. The OCT uses near-infrared range of light giving deep penetration depth information. Furthermore, one of big merits of OCT compared with other medical imaging modality is that non-invasive technique of OCT does not give any pain to patient. Besides, OCT provides high resolution image in 2D or 3D by scanning in both axial and transverse way. These distinguished advantages of OCT enable to cover the shortcomings of surgical microscope as mentioned above. However, it is not easy to integrate OCT system with surgical microscope as it sounds. Thus, we tried to face these challenges and came up with an idea to resolve the hardware/software issues that can be fixed by the help of National Instrument Products for development of 3D surgical microscope, OCTOPUS.
Products (NI)
PXIe-1082 (PXIe-8381 MXI express x8 interface), PXIe-1435, PXIe-6363, LabVIEW 2013, Vision Acquisition Software, NI IMAQ
Products (Others)
Surgical Microscope (Carl Zeiss), Basler camera (sprint camera link line scan, 2048, 140 kHz), Beam projector (SK smart beam, 640X480), CCD camera (DINO, 1.3 Mega pixel), Beam splitter (Carl Zeiss, 50/50), HD display monitor (Alphascan),
The Challenge
OCT is an advanced optical imaging technology being widely used in various medical fields such as ophthalmology, dermatology, and dentistry; it realizes real-time cross section image of biological tissue up to the depth of 5mm with highest resolution in 2μm. However, most of commercialized medical OCT products have been used only for diagnosis purpose, not surgical purpose. Thus, we thought that if we could utilize the OCT technology combined with surgical microscopy in real time during surgical operation, it would give a huge help to surgeons. However, there were tremendous challenges to develop OCTOPUS system. First issue was the size. The most bulk portion of OCT is taken up by spectrometer unit. The massive complicated optical design of conventional OCT should be converted into compact size to come in handy with surgical microscope to be attached to OCT. Too hefty tool would not be as much effective when it need to be moved to other surgical room in hospital. In addition, the whole design including the position of displays and eyepiece, movement range of sample unit, and quality of image should be considered to provide surgeons with most convenient service of tool systemically and effectively. Second key factor is about speed issue. In order to keep the surgeons informed with 3D image in real time of targeted surgical region which is very powerful functioning to help doctors make a right decision while operating, it is accordingly required to have fast enough speed in image processing through the whole system. The last issue is to set up a handy and user-friendly interface of UI system. The system should be built up in a convenient way to provide ease of use to surgeons and surgical technician who has never experienced this type of medical imaging tool before.
In summary about challenging factors explained above, we will have to solve out following issues in this project,
1. Reducing the size of OCTPUS system into compact one
2. Optimal way of delivering image in most effective way
3. Realizing 3D image in real time by OCT technology
4. User-friendly UI interface to render ease of use to the first-time user
If we are able to figure out the solution for all these challenging factors, it is expected that OCT would be an ideal partner of surgical microscope. Because the OCT technology using near-infrared light has such a good feature of deep penetration depth, its imaging technique using interference of light scattering coming from different layers in biological samples is very effective on distinction of structural change in volumetric tissue sample. Also, the wavelength of light source that OCT system takes is near infrared range, so that it does not make any negative affection on the surface of surgical area as well as it gives better information to surgeons with cross-sectional and volumetric images in real time that currently being used surgical microscope never offers. In a result, it is highly expected that it will increase the accuracy of operation in dealing with micro regions and reduces the possibility of medical accident occurring often in health facility when surgeons is allowed to have only weak superficial information of aimed area. The OCTOPUS takes an important role to contributing on enhancing the safety and stability of surgical operation dramatically.
The Solution
1. An approach to make the OCTPUS systems smaller.
As mentioned in previous passage, the main function of combining Optical Coherence Tomography with Surgical Microscope is an advanced medical imaging instrument providing the top surface and structural depth image both in real time. This type of collected image is achieved by permitting sample unit of OCT system to share the beam path with surgical microscope. It is demanded that space right under the image-collecting part of whole system in which the sample unit of OCT system is combined with surgical microscope should be achieved, thus it will get the more available room needed for surgical operating. Yet, the OCT system consist of numerous optical components along with complicated optical design. Accordingly, what it takes to have high resolution image is to come from making a very exquisite and accurate alignment of all these optical components. For these reasons, it is essential to make compact size of OCT system for the benefit of surgical operating area under the image-collecting unit. The compactness of OCT would be applied to various medical field because it is portable and small enough to carry with. Not only being used in ophthalmology department, but also it can be moved to different department in hospital or other research facility which need to use compact OCTOPUS for different reason. Hence, it is another important that the integration of bulky OCTOPUS system should be assembled in compact way for this project.
In this project, we redesigned two main part of conventional OCT system for the compactness which would be suitable to OCTOPUS. Above all, we mended adjustable rail structure of optical component to be fixed at certain point inside of spectrometer. Existing OCT spectrometer was generally set up on the broad optic table since it is easier to have more accurate alignment and to adjust in case of reconsidering to change the entire design again, but it causes to increase the size of OCT system and makes it hard to be portable. Through a software enabling reliable optical and illumination software called ZEMAX, we were able to simulate and design the fixed type of spectrometer unit. In case of fixed type of spectrometer, it no longer allow any instant modifying as the misalignment found out, but it does have strong point in terms of more stable against the vibration surrounding the spectrometer, resulting to more applicable to portable system. Additionally, because it needs only small number of optical component compared to conventional system, it capable of decreasing the entire size substantially. Secondly, what we did was to design the core part of OCTOPUS system which was the image-collecting unit integrated with surgical microscope to make ample room under it. More specifically, one of the methods to have enabled to have more spacious room was that we created a lens with long focal length which did not sacrifice the resolution while decreased the volume of spectrometer.
Ultimately, it had to be validated whether the image from newly designed compact OCTPUS system shows same performance with image of the bulky existing OCT system. It followed that we developed imaging-evaluation software by using LabVIEW and Signal processing toolkit. Simply equipped with Data Acquisition Board (DAQ) and frame grabber both from National Instrument, we were able to accomplish to realize the OCT image in output at UI of LabVIEW. In a result, it demonstrated that the image from both compact OCTOPUS and existing OCT system were seen similarly with regard to spatial resolution and penetration depth in image.
2. Delivering OCT image in an effective way using beam projector
As mentioned above, what the combination of conventional OCT system and surgical microscope can give us is the optimal surface image and cross sectional image at the same time, thus there has been research done by several groups over past decades. However, the previous works had to be challenged to generate the image on same spot because microscope was separated from OCT sample unit. Moreover, what causes most inconvenience for operation is to take turn to look at computer monitor and eyepiece of microscope, annoying the surgeons. Thus, it is very welcome to fix this kind of error and offer better alternative with highly advanced technological solution.
Therefore, dual display interface that would be used to provide 3D volumetric image in real time to surgeons while performing surgical procedure is going to be operated with beam projector. Specifically speaking about dual display interface, it is a processing technique that acquires images onto the eyepiece coming from OCT system and surgical microscope at the same time. The key point of this technique is to take acquisitions of real time OCT images and to be processed. Basically, we received raw data over the frame grabber which was followed to be computed by FFT process with the ‘LabVIEW CUDA library’ in the GPU domain. This cycle of processing made it possible to acquire frames in the rate of 200FPS without any loss. This high speed OCT image through all this steps would be processed over LabVIEW and GPU board, and finally be seen in the output of beam projector. This image would be overlapped onto microscope display by employing the beam splitter and be shown in combined image as output.
In a result, we realized Dual Display Interface by obtaining the high speed OCT image. The video clip uploaded above contains the material that demonstrated the ability of our system to see the surface image of microscope and cross sectional structure image of OCT both in real time, which directs to surgeons with important information needed and helps them to make decision intuitively. We predict that it makes substantial contribution to medical safety on surgical procedure. It is promising that it gets surgeons geared with more convenient surgical tool and guides them to perform better with more plentiful information
3. Realizing 3D OCT image in real-time
Recently, it pays attention to the 3D medical imaging realization because it is possible to make sure surgical simulation guiding even before performing surgical operation, and it is more feasible for surgeon to watch the affected area in 3D, which could not be seen by rough eye if the superficial image in 2D was only available. Thus, the feature of stacking these images into 3D volumes will be covering more broad area than conventional 2D image used to offer. What made the OCTOPUS more valuable is that it has a capability to add depth information to the existing 2D image, thus it would give users feeling of reality and dynamical monitoring during operation. It is helpful for surgeons to be more careful, alerted and practice surgery especially when they need to cut out affected area under which the hidden micro-vessels are flowing. However, one of challenging was that it required the system rate of 400MB/s at least so as to process the data for achieving 3D image in real time. Dead time in line scan camera has been a main cause to make system delayed from which it has to be fixed in first priority in this project. Hence, we were supposed to know how the camera capacity can be used in fullest for 3D image of OCT.
The camera we chose in this project is line scan camera made by Basler that can perform in the rate of 140k line/s in max speed. Yet, it was measured to have the half speed of maximum due to the delay of trigger signal. Thus, we had to figure out the alternative to boost up this speed and finally decided to equip the NI-PXI to our system. In the end, we were able to increase the synchronization between frame grabber and DAQ board, and ended up processing the data in the speed of 100k line/s. The figure above has demonstrated the way we calculated for the speed of data-processing in case of one single 3D volume; consequently, we fulfilled 2 volumes per sec in real time. 3D volume rendering could be carried out through GPU control of LabVIEW and a skeleton code based on ‘Open GL’ was recalled to be connected to LabVIEW by the calling-node of DLL library. Since the control system coded based on LabVIEW has processed faster than algorithm based on C language, using the LabVIEW for this project has been considered to be used as main software of our whole system. .With the result we have gotten through this project, it is also expected that our product would be applied to Augmented Reality (AR) as well which is hottest issue in medical imaging field. For further study, we will be aiming to implement AR with our OCTOPUS system, so that it offers another type of imaging of AR as well as 2D/3D image OCTOPUS is already providing.
4. Composing user-friendly interface for non-engineering major people
The main user of OCTOPUS system would be those who had surgical experience. Accordingly, we have had meetings frequently with surgeons who are in the actual field of surgery to test our product as well as to get feedbacks from them. These meetings allowed us to have deep understanding how we have to make the OCTOPUS system to be more suitable and convenient to surgeons to serve them in best service. More specifically, they had more difficulty in handling the interface of system to be controlled by their hand rather than in handling the hardware itself. They wanted to know better how to use most of functions OCTOPUS can provide. On top of that, surgeons usually does not have much of background knowledge about Optical Coherence Tomography, it is essential for the control-interface to be more readily accessible and easier to be handled by them.
The user interface has composed of three different real-time images that is provided by control of LabVIEW, leading to be more user-friendly and easier to be used by surgeon. Since the LabVIEW has a variety of strong functions such as intensity graph, image display control, 2D picture control, etc, it enabled the interface to be more directive and intuitive to users.
Conclusion
The proposed OCTOPUS system will provide high-resolution cross-sectional and volumetric images in intraoperative surgery. Especially, capability to distinguish a clear boundary of tumor, neural and vascular distribution will be the key feature provided by OCTOPUS system. The local imaging based on surgical OCT microscope system will accurately detect the position in the targeted surgical area, resulting to dramatically increase in operational stability such as reduced restoration duration and less side-effect after surgery. Also, we expect that surgical OCT microscope will improve the convenience for surgeons, and it will be applied to various medical fields as a versatile medical imaging instrument.
