High-Speed Focal Tracking Projection Based on Liquid Lens / 基于可变焦透镜的动态投影系统

Summary/摘要

Projection mapping (PM) is attractive as a fundamental technology for the advancement of various subjects, such as media art, entertainment, and augmented reality. However, conventional projectors have a shallow depth of field (DOF); therefore, sharp images are only visible in the limited depth range. In the case of dynamic projection mapping (DPM), which can project images on the surface of the moving objects, the shallow DOF limits the permissible motion of the object, because the projected images become blurred when the object is outside the DOF.

Our laboratories have developed a high-speed focal tracking projection system, which includes the technologies of high-speed vision, high-speed projector, and high-speed variable focus optics. In this system, the variation of the object’s distance and posture was captured using the high-speed vision technology that served as immediate feedback to the liquid lens and high-speed projector. As a result, the focal distance is compensated, and the projected images are updated in real-time to fit the moving object. Therefore, a well-focused image projection was achieved even when the motion involved large depth range movement.

This system could ensure that the projected images were sharp and clear at variable distances, while the object was moving dynamically in a large three-dimensional area. Hence, this approach can be effectively applied to applications such as Volume Slicing Display. Furthermore, it can turn any physical surface into an interactive display, and enable the manipulation of their appearance to provide detailed information. Our system provides the essential technology for expanding such applications.

This work is the result of collaborative research between our laboratory, Watanabe Laboratory (Tokyo Institute of Technology, Japan), and Ishikawa Group Laboratory (The University of Tokyo, Japan).

2ddof — **Fig.1 The conventional projectors have a shallow depth of field.**

dynadof — **Fig.2 The proposed technology can project sharp images in large depth range.**

Version 1/第一代原型机（From 2018-）

The first version of the high-speed focal tracking projection system has been developed. In this version, we have developed a high-speed variable focus optics unit, which can change the focus at high-speed. In this system, the object’s distance was captured using the depth sensor that served as feedback to the liquid lens. As a result, the focal distance was compensated. Moreover, by updating the projected images, volume slicing display had been achieved.

**Fig.3 Prototype of dynamic focus projector using a liquid lens. (Ver.1)**

dynamove — **Fig.4 Projection images are kept in-focus, while target (white board) moves from 0.5m to 2.0m.**

Version 2/第二代原型机（From 2019-）

The second version of the high-speed focal tracking projection system has been developed. In this version, we incorporated the technologies of high-speed vision and high-speed projector in order to increase in speed of feedback and update the projected images. Moreover, high-speed marker tracking method was incorporated to estimate the object’s distance and posture. As a result, a well-focused dynamic projection mapping was achieved even when the motion involved large depth range movement.

**Fig.5 Prototype of projection system using a liquid lens. (Ver.2)**

**Fig.7 In-focus projection in the large depth range.**

**Fig.8 Volume Slicing Display: change images based on the distance.**

Video/视频

Ver. 1: High-speed focal tracking projection using a liquid lens
If you want to use the original video, please send an e-mail for copyright permission to wanglihui@gdisit.com

Ver. 2: High-speed focal tracking projection system
If you want to use the original video, please send an e-mail for copyright permission to wanglihui@gdisit.com

Award/获奖

The following three awards were given to the project “High-speed focal tracking projection system based on liquid lens”.

ACM SIGGRAPH Special Prize, (Association for Computing Machinery)
Innovative Technologies 2019, (Digital Content Association of Japan)
Innovative Technologies 2019, Sponsor Award (CGWORLD) 2019, (Digital Content Association of Japan)

Media and News/媒体报道

中国科技网. 液体透镜带来光学镜头革命 [Link] (2019.11) (in Chinese)
日刊工業新聞. 動く物体に絵を投影できるプロジェクション技術、どんな用途に使う？[Link](2020.01)(in Japanese)

Reference/ 参考

Lihui Wang*, Satoshi Tabata, Hongjin Xu, Yunpu Hu, Yoshihiro Watanabe, and Masatoshi Ishikawa, Dynamic depth-of-field projection mapping method based on a variable focus lens and visual feedback, Optics Express, 31(3), pp. 3945-3953 (2023) [DOI:10.1364/OE.478416]
Hongjin Xu, Lihui Wang, Satoshi Tabata, Yoshihiro Watanabe, and Masatoshi Ishikawa, Extended depth-of-field projection method using a high-speed projector with a synchronized oscillating variable-focus lens, Applied Optics, 60(13), pp. 3917-3924 (2021) [DOI: 10.1364/AO.419470]
Lihui Wang, Hongjin Xu, Satoshi Tabata, Yunpu Hu, Yoshihiro Watanabe, and Masatoshi Ishikawa: High-Speed Focal Tracking Projection Based on Liquid Lens, ACM SIGGRAPH 2020 Emerging Technologies (SIGGRAPH ’20) (Virtual Event, USA, 2020.8.24-28) [DOI: 10.1145/3388534.3408333]
Hongjin Xu, Lihui Wang, Satoshi Tabata, Yoshihiro Watanabe, and Masatoshi Ishikawa: An extended depth-of-field projection method using a high-speed projector with a synchronized oscillating variable focus lens, SPIE Photonics West 2020 (San Francisco, 2020.2.5)/Proceedings of SPIE Vol.11304 , Advances in Display Technologies X, 113040T (2020) [DOI: 10.1117/12.2542477]
Lihui Wang, Hongjin Xu, Yunpu Hu, Satoshi Tabata, Masatoshi Ishikawa, Dynamic Depth-of-Field Projection for 3D Projection Mapping, ACM CHI Conference on Human Factors in Computing Systems (CHI’19) (Glasgow, Scotland, UK. 2019.05.05-09) [DOI: 10.1145/3290607.3313246]
Lihui Wang, Yunpu Hu, Hongjin Xu, Masatoshi Ishikawa, Dynamic focal tracker display, SPIE Photonics West 2019 (San Francisco, California, USA. 2019.02.07)/ (Oral Session) [DOI: 10.1117/12.2506958]