Software Networking

Vol: 2016    Issue: 1

Published In:   January 2018

Implementing a Stereo Image Correction Method for 3D Surgery Microscopes

Article No: 9    Page: 159-172    doi: 10.13052/jsn2445-9739.2016.009    

Read other article:
1 2 3 4 5 6 7 8 9 10 11 12

Implementing a Stereo Image Correction
Method for 3D Surgery Microscopes

Jiyoung Yoon1, Nakhoon Baek2,*, Cheolhwan Kim3 and Hwanyong Lee4

  • 1Institute of Advanced Convergence Technology, Kyungpook National University, South Korea
  • 2School of Computer Science and Engineering, Kyungpook National University, South Korea
  • 3School of Electronic Engineering, Kyungpook National University, South Korea
  • 4Ajou University, South Korea

E-mail: {wngus235; oceancru; hwanyong.lee}@gmail.com; kch1@knu.ac.kr

*Corresponding Author

Received 10 November 2016; Accepted 24 November 2016;
Publication 6 December 2016

Abstract

Recently 3D scenes are used in various industrial fields such as medical applications, computer games, surface examinations, biology and others. 3D optical microscopes can show extremely precise details in 3D. To reconstruct the 3D images of an optical microscope, two cameras are mounted on the optical microscope. Incoming images through an object lens of an optical microscope are projected on sensors of mounted cameras by using refraction mirrors. Two cameras capture the left and right images to reconstruct the final 3D images. In this paper, we correct the 3D reconstruction errors with the SURF algorithm. We also design a hardware system to correct wrong mirror positions using servo motors. In addition, we propose the surgical system with HMD and wireless communications. This system would be helpful for doctors due to make doctors comfortable and it can be used to educate surgery procedures.

Keywords

  • Stereo Image
  • Optical Microscope
  • Computer Vison
  • SURF
  • Medical Device

1 Introduction

Recently 3D scenes are used in various industry fields. 3D optical microscope [1] is being developed to see much precise details in 3D. The 3D microscopy is widely used in industrial, medical and biological studies. In the industrial field, this system is used to identify errors of the electronic circuits or detects the very small mechanical components. In the medical areas, it is used for actual medical procedures such as polyp removal. It can be used in hair transplant, ophthalmology and biology to identify such cells, chromosomes in research. A microscope with augmented reality technology is being researched to provide a guide to the surgical region and to show the necessary information [10]. Figure 1 is various use cases of imaging devices in medical procedures [2, 3].

images

Figure 1 Various use cases of imaging devices in medical procedures.

Medical doctors make surgery using input images through cameras. They use an optical microscope or an endoscope equipped with cameras. These systems display the expanded images on a monitor for surgical sites it required. To show 3D images of an optical microscope, two cameras are mounted on the optical microscope. Incoming images through an object lens of an optical microscope are projected on sensors of mounted cameras using refractive mirrors. Those refractive mirrors are used to avoid obscuring a view of ocular lens. Two cameras create the left and right images and reconstruct the 3D images. At this time, it is not an easy task, because of the optical design errors and the deformation mechanism of optical microscopes.

In this paper, we corrected those error with the SURF algorithm and designed the hardware system to correct wrong mirror positions using servo motors. In addition, we propose the surgical system with HMD and wireless communication. Doctors can check surgical position and external circumstance by mounting a camera on the front of the HMD. By sending 3D microscope images through wireless communications, doctors are comfortable with surgery and this system can be used for surgical training.

2 Image Output Error of the Optical Microscope

To show the 3D images from the optical microscopes, two cameras are mounted on an optical microscope. Incoming images through an object lens of an optical microscope are projected on sensors of mounted cameras by using refractive mirrors. Figure 2 shows an example of 3D microscope components.

images

Figure 2 An example of 3D microscope components.

To send the picture to two cameras on the optical microscope, incoming image into the ocular lens is projected on the camera sensors [4] such as CCD or CMOS. The camera sensor sizes depend on the type of an optical microscope lens and we use about 1/3” to 2/3” sensors and the image of an optical microscope is projected by the camera sensor to adjust the focal length. We can look 3D images of 3D optical microscope on various display devices such as 3D monitors, embedded displays, Head Mounted Displays (HMD) [5] and etc. Figure 3 is an example of displaying methods of 3D optical microscope.

images

Figure 3 Example of displaying devices of 3D optical microscope.

We developed the 3D optical microscope. We mounted two 1280 × 720 resolution cameras on the microscope. We used an HMD system with two 1280 × 720 resolution microdisplay modules for testing. However, it had the image output error. The image output error of our optical microscope occurred in the cause of the refractive image on the mirrors [6]. The error was generated the image is not projected on the centre point of the camera’s sensor because of refraction error of mirrors. It made transformed images. This was caused by the optical design errors and deformation mechanism of the optical microscope. This error can occur during the development of 3D optical microscope projecting images using the mirror. If this error occurs in the optical microscope image output, it is difference the real eye view image through ocular lens and the camera image. Then, the user cannot see the desired image and the three-dimensional image. Figure 4 shows the example of stereo image error on optical microscope.

images

Figure 4 Example of stereo image error on optical microscope.

Stereo image error makes that users can have dizziness and nausea. Then users cannot treat patients and the system is not available. So stereo image error should be corrected for 3D optical microscope system.

3 Stereo Image Correction for 3D Optical Microscope

To correct incoming error image through the optical microscope, we used a SURF (Speeded Up Robust Features) algorithm [6, 7, 11]. SURF is a robust local feature detector that can be used in computer vision tasks like object recognition or 3D reconstruction. SURF is a detector and a high-performance descriptor points of interest in an image where the image is transformed into coordinates, using a technique called multi-resolution. We used SURF algorithm to extract key points and descriptors on left and right image. We could detect the almost same key points and descriptors on the left and right image even if the images were transformed. Then we got a homography [8] by key points and descriptors. We used a homography to correct stereo image error.

To correct stereo images for 3D optical microscope processed in the following order. First, shot the checker image by left and right camera. Second, extract key points and descriptors on left and right images using SURF algorithm. Third, transform left and right images to align centre and perpendicular using key points. Fourth, clip missed image area by transforming images. Finally, fit stereo images at a screen. Figure 5 shows to correct stereo images using checker.

images

Figure 5 To correct stereo images using checker.

If there is not the checker image, then it should be used other method [6]. We changed Third step in the method described above. We choose the reference image that is left or right image. Then, transform the other image using a homography by feature matching. The fourth and fifth operations are performed in the same manner. Figure 6 shows to correct stereo images without checker.

images

Figure 6 To correct stereo images without checker.

Using SURF algorithm, it takes about 300∼350ms to obtain a homography and clipping region on i7-5930K CPU. Image correction is executed first once using first left and right image frame. Because the deformation mechanism of the camera connector and mirror part on the optical microscope does not occur frequently. This method made a good stereo images. However, this method had two drawbacks. First, it occurs the loss of images on the clipping step. The loss of image due to clipping depends on a transformation of image using homography. But this is very small area and we could get a sufficient image region. Second, it processed to transform left and right images every frame. But it was not big problem. To solve these drawbacks, we should fix the deformation mechanism on the optical microscope.

4 Experiment and Result

We used the devices that were our optical microscope, two Full HD cameras with coaxial cable, the capture board and the personal computer for display and correcting image. The personal computer system was Window 7 64bit and the development tools is Visual Studio 10 MFC. It is not necessary the high performance PC, because image correction is performed once the first time. However, it is required the PC with enough performance to display the image in real time. Figure 7 is the result of the experiment.

images

Figure 7 The result of the experiment.

5 Design Hardware System to Correct Stereo Image Error for 3D Optical Microscope

We designed the hardware system to correct stereo image error for 3D optical microscope because we didn’t want to loss of image region and to process transforming images every frame. This system attach servo motors to mirror tilting part of 3D optical microscope. Servo motors move and tilt the mirror to make correct stereo images. Servo motors are controlled by PC system that runs SURF algorithm. The control values of servo motors are reference a homography. We get the transform values and control servo motors by matching stereo image. This system will make correct stereo images without loss of image region. Figure 8 shows hardware system to correct stereo image error for 3D optical microscope.

images

Figure 8 Hardware module to correct stereo image error for 3D optical microscope.

6 Surgery System with HMD and Wireless Communication

We propose the surgical system with HMD and wireless communication. Generally, doctors operated using microscope to see magnified images on a monitor. In this case, doctor’s eyes cannot look at the surgical site and it can be unnatural. When doctors watch the monitor installed on the top, he can fell the fatigue of his neck. To solve this problem, a HMD is used instead of a monitor. By using HMD, doctors can freely move and focus on surgery. Figure 9 is the proposed surgery system with HMD.

images

Figure 9 Surgery system with HMD.

If the HMD is wired to the system, this cable may interfere with doctor’s operation. Therefore, it is necessary to transmit images through wireless communication. When using a general HDM, doctor’s views are completely obscured. This problem is solved by using a see-through type HMD and mount a camera on the front of the HMD. By using this, doctors can check the surgical position and the external circumstance. Figure 10 shows wireless HMD system and HMD with front camera.

images

Figure 10 Wireless HMD system and HMD with front camera.

By sending 3D microscope image through wireless communication, doctors are comfortable with surgery. However, if 3D images are transmitted through wireless communication then image output delay may occur. In case of WirelessHD standard, high resolution image could be transmitted at a high speed and at a delay rate of less than 5ms using a broadband of 60 GHz [9]. However, when the wireless communication is interfered, the image is delayed. This can occur a fatal problem for doctors’ surgery. Therefore, it is necessary to solve image output delay. When video output delay is a problem in a wireless communication environment, it can be used as a training system for a resident and a student. Residents and students can directly see the doctor’s surgical procedures and doctor’s scenes outside the operating room. Figure 11 is the proposed surgery and education system with HMD and wireless communication.

images

Figure 11 Surgery and education system with HMD and wireless communication.

7 Conclusion

Medical doctors used optical microscopes that shows just a single image. Users do not mind small transformed image errors because this image is used as secondary role to look at things through the ocular lens. However, if there are errors in those 3D images, then users could have dizziness and nausea. If doctors try to treat using 3D display system with stereo image error, they cannot treat patients and the system is not available. So stereo image error should be corrected for 3D optical microscope system. We implemented the stereo image correction system and designed the hardware system for 3D optical microscope. We used the SURF algorithm and got a homography to correct stereo image error. This method made a good stereo images. However, this method had two drawbacks that are decreasing resolution and transforming images every frame. But it was not a big problem and enough to use the system properly. To solve these drawbacks, we designed the hardware system to correct stereo image error for 3D optical microscope. This system will solve stereo image error completely.

In addition, we propose the surgical system with HMD and wireless communication. Doctors can check surgical position and external circumstance by mounting a camera on the front of the HMD. By sending 3D microscope image through wireless communication, doctors are comfortable with surgery and this system can be used for surgical training.

In the future, we will develop the embedded device for this system. We will develop the hardware system that we designed. Additionally, we will make fast embedded device that use SURF algorithm on FPGA to correct stereo image correction.

Acknowledgment

This research was financially supported by the Ministry of Trade, Industry & Energy (MOTIE), Korea Institute for Advancement of Technology (KIAT) and Gangwon Institute for Regional Program Evaluation (GWIRPE) through the Economic and Regional Cooperation Industry.

References

[1] Wikipedia contributors (2015a). Optical Microscope. Available at: https://en.wikipedia.org/wiki/Optical_microscope

[2] Figl, M., Ede, C., et al. (2005). “A fully automated calibration method for an optical seethrough head mounted operating microscope with variable zoom and focus,” in Transaction on Medical Imaging (Rome: IEEE), 1492–1499.

[3] Tamiya, T., Kawanish, M., and Guo, S. (2011). “Skull base surgery using navigation microscope integration system,” in Complex Medical Engineering, eds J. L. Wu, K. Ito, S. Tobimatsu, T. Nishida, and H. Fukuyama (Berlin: Springer), 185–187.

[4] Wikipedia contributors (2015b). Image Sensor.https://en.wikipedia.org/wiki/Image_sensor

[5] Wikipedia contributors (2016). Head-mounted Display.https://en.wikipedia.org/wiki/Head-mounted_display

[6] Yoon, J., Kim, H., and Kim, C. (2015). “Development of image error correction system for 3D optical microscope”, Computing Technology and Information Management (Rome: IEEE), 105–108.

[7] Bay, H., Tuytelaars, T., and Van Gool, L. (2006). Speed-Up Robust Features(SURF), in Computer Vision – ECCV 2006 Lecture Notes in Computer Science, Vol. 3951, eds A. Leonardis, H. Bischof, and A. Pinz (Berlin: Springer), 404–417.

[8] Wikipedia contributors (2016). “Homography_(computer_vision)”, Wikipedia, The Free Encyclopedia Wikipedia. Web.

[9] WirelessHD (2010). “WirelessHD Specification Version 1.1”. Available at: www.wirelesshd.org

[10] King, A. P., Edwards, P. J., and Maurer, C. R., et al. (2000). “Stereo augmented reality in the surgical microscope”. Presence 9, 360–368.

[11] Rublee, E., Rabaud, V., and Konolige, K., et al. (2011). “An efficient alternative to SIFT or SURF,” in Proceedings of the Computer Vision (ICCV), 2011 IEEE International Conference, Rome.

Biographies

images

J. Yoon received the M.S. and Ph.D. degree in Computer Science from Kyungpook National University, Korea in 2008 and 2016. He is currently a senior researcher in IACT (Institute of Advanced Convergence Technology), Korea. His research interests include computer graphics standard and parallel processing.

images

N. Baek is currently a professor in the School of Computer Science and Engineering at Kyungpook National University, Korea. He received his B.A., M.S., and Ph.D. degrees in Computer Science from Korea Advanced Institute of Science and Technology (KAIST) in 1990, 1992, and 1997, respectively. His research interests include graphics standards, graphics algorithms and real-time rendering. He is now also the Chief Engineer of Dassomey.com Inc., Korea.

images

C. Kim is currently a Ph.D. candidate in the School of Electronic Engineering at Kyungpook National University. His research interests include Computer Vision, 3D Reconstruction and Medical Image processing. He is working in Institute of Advanced Convergence Technology at Kyungpook National University, Korea.

images

H. Lee received B.S. degree in Computer Science at KAIST and M.S. degree in Computer Science at POSTECH, Ph.D. degree in Computer Engineering at KNU. He is currently an industry cooperation professor at Ajou University. His research interests include computer graphics, image processing, virtual reality and augmented reality.

Abstract

Keywords

1 Introduction

images

2 Image Output Error of the Optical Microscope

images

images

images

3 Stereo Image Correction for 3D Optical Microscope

images

images

4 Experiment and Result

images

5 Design Hardware System to Correct Stereo Image Error for 3D Optical Microscope

images

6 Surgery System with HMD and Wireless Communication

images

images

images

7 Conclusion

Acknowledgment

References

Biographies