Jian-Kai Zhu
I'm a Research Assistant at the Institute of Information Science, Academia Sinica, advised by Dr. Jen-Chun Lin. I received my M.S. degree from National Taiwan University under the supervision of Prof. Yung-Yu Chuang.
I'm looking for junior to mid-level industry positions. I'm especially interested in bringing generative AI into games, both as tools for building game assets and as models that run inside the game in real time. AI-assisted asset development shortens the iteration cycle for scenes, animations, and dialogue. Real-time AI models in games open up many possibilities, such as DLSS, NPCs that react naturally through generative motion, human-like enemy and ally strategies, and neural materials. My goal is to help build next-generation game engines that integrate AI models, while lowering the cost of game development through AI tools. Learn more about me: E-mail | CV
My featured project is a universal, real-time render data extraction pipeline for modern game engines, presented on the Technical demo page. Instead of integrating with each engine separately, it works at the graphics API level: a Graphics API Proxy intercepts D3D12 calls to extract camera pose, metric depth, and object geometric and material information from commercial games such as Where Winds Meet, Assassin's Creed Shadows, and Grand Theft Auto V, at a cost of tens of microseconds per frame. The extracted data is intended for large-scale AI model training. The demo page shows gameplay side by side with the extracted camera trajectories, along with the system architecture and performance measurements.
I'm also building an AI-integrated real-time D3D12 render engine, presented on the AI renderer page. It is a low-latency deferred shading render graph in which heterogeneous AI models run as intermediate nodes through ONNX Runtime and DirectML, aiming at applications such as super resolution, neural materials, particle and fluid prediction, and real-world character reactions. So far, a text-guided streaming motion generation model drives a skinned character in real time, adding about 0.42 ms to the mean frame time at 1080p. The demo page shows the engine in action, along with the render graph architecture, the challenges and insights of integrating AI models into rendering, and performance measurements.
I also share ongoing side projects here, including the Cel-shading testbed (takes time and a capable GPU to load), and my blog, where I write about my side-project experiences. The figures in the Cel-shading testbed are currently static. The goal of this project is a robust motion generation framework that animates them from multi-modal conditions, including text, sound, video, and environmental geometry, so that they can interact with people naturally.
Publications
Jian-Kai Zhu, Wen-Li Wei, Jen-Chun Lin
Computers & Graphics
[Project website] [Paper] [Code]
Wei-Chieh Chung*, Jian-Kai Zhu*, I-Chao Shen, Yu-Ting Wu, Yung-Yu Chuang (*: joint first authors)
British Machine Vision Conference (BMVC) 2022
[Paper][Code][Supplementary materials]
