Xudong Lu

 

Xudong Lu (ι™†εΎδΈœ)

 

Multimedia Laboratory (MMLab)

The Chinese University of Hong Kong

Email: luxudong@link.cuhk.edu.hk, luxudong2001@sjtu.edu.cn

 

          

About Me


I am currently a third-year Ph.D. student at the Multimedia Laboratory (MMLab), The Chinese University of Hong Kong. My research advisor is Prof. Hongsheng Li. I am also an intern at ByteDance Seed (2026.5-). Previously, I had the privilege to intern at Huawei Leibniz Research Center (2025.8-2026.5), vivo AI Lab (2024.6-2025.7), and Shanghai AI Laboratory (2023.2-2023.8). I will be on the 2027 job market and welcome relevant opportunities. Please feel free to reach out! I received the B.E. degree from the Department of Computer Science and Engineering at Shanghai Jiao Tong University in 2023, ranking 2/119 in my major. I was awarded the National Scholarship (top 0.2%) in 2020, 2021, and 2022, and the Hong Kong PhD Fellowship Scheme (HKPFS) in 2023.

During my undergraduate studies, from 2020 to 2022, I worked closely with Prof. Yonglu Li and Prof. Cewu Lu on human activity understanding. From 2022 to 2023, I collaborated with Prof. Xue Yang and Prof. Junchi Yan on AI for Science (AI4Science). My current research interests include model compression, large language models, and multimodal large language models.
Research pipeline

My research focuses on advancing AI hardware and applications across diverse environments, spanning from high-computing power devices like NVIDIA A100 to low-computing power devices like Jetson Orin, including consumer GPUs such as RTX 4090 and NPUs like Qualcomm Snapdragon 8 Elite for smartphones.

News


  • August 2026: AURA is supported by vLLM-Omni for online serving.πŸ”₯πŸ”₯πŸ”₯
  • August 2026: We introduce SeedRealtime, a real-time multimodal interaction model released during my internship at ByteDance Seed.πŸ”₯πŸ”₯πŸ”₯
  • June 2026: Two papers on Streaming VideoLLM (one co-first author) are accepted by ECCV 2026.
  • May 2026: AURA is cited by Thinking Machines Lab in their discussion of interaction models.πŸ”₯πŸ”₯πŸ”₯
  • May 2026: One paper on Streaming VideoLLM (first author) is accepted by ICML 2026.
  • April 2026: We release AURA, an end-to-end streaming visual interaction framework for real-time video understanding and assistance.πŸ”₯πŸ”₯πŸ”₯
  • February 2026: We launch CUHKSat-1 satellite, achieving the world's first on-orbit deployment of the DeepSeek large language model.πŸ”₯πŸ”₯πŸ”₯
  • February 2026: One paper on oriented object detection is accepted by ICLR 2026.
  • January 2026: One paper on remote-sensing multi-task learning is published in Remote Sensing (MDPI).
  • December 2025: Mamba-LCD wins the Best Presentation Award at ISPRS–ISDE 2025 in Nairobi, Kenya.
  • August 2025: One paper on the evaluation of end-side LLM (first author) is accepted by EMNLP 2025.
  • June 2025: One paper on MLLM (first author) is accepted by ICCV 2025.
  • May 2025: One paper on oriented object detection is accepted by IJCV (IF 11.6, 2024).
  • February 2025: Two papers on MLLMs (one first author) are accepted by CVPR 2025.
  • February 2025: Two papers (one on KV Cache pruning, another on autoformalization) are accepted by ICLR 2025 (both Spotlight).
  • November 2024: We are happy to introduce BlueLM-V-3B, an algorithm and system co-design method for multimodal large language models on mobile devices.πŸ”₯πŸ”₯πŸ”₯
  • July 2024: One paper on video generation is accepted by ECCV 2024 (Oral).
  • May 2024: One paper (first author) on MoE LLM pruning is accepted by ACL 2024.
  • May 2024: One paper (first author) on LLaMA-alike LLM pruning is accepted by ICML 2024.
  • February 2024: One paper on Human Action Understanding is accepted by CVPR 2024 (Highlight).
  • August 2023: I am joining the MMLab of CUHK as a Ph.D. student.
  • June 2023: I am honored to have the opportunity to speak as the student representative at the undergraduate commencement ceremony of the college.
  • April 2023: Two papers (one first author) on Quantum Machine Learning are accepted by ICML 2023.
  • March 2023: I am awarded with HKPFS (Hong Kong PhD Fellowship Scheme).

Publications

Streaming & Omnimodal Assistants

GuideMe: Multi-Domain Task Guidance and Intervention in Streaming Video
GuideMe: Multi-Domain Task Guidance and Intervention in Streaming Video
Fang Liu*, Jinpeng Chen*, Ke Xu, Yuhao Liu, Huankang Guan, Xudong Lu, Yang Bo, Gerhard Hancke, Rui Liu, Rynson W.H. Lau
European Conference on Computer Vision (ECCV), 2026
[Paper]
X-Stream: Exploring MLLMs as Multiplexers for Multi-Stream Understanding
X-Stream: Exploring MLLMs as Multiplexers for Multi-Stream Understanding
Peiwen Sun*, Xudong Lu*, Huadai Liu*, Yang Bo, Dongming Wu, Huankang Guan, Minghong Cai, Jinpeng Chen, Xintong Guo, Shuhan Li, Rui Liu, Xiangyu Yue
European Conference on Computer Vision (ECCV), 2026
[Paper]
PhoStream: Benchmarking Real-World Streaming for Omnimodal Assistants in Mobile Scenarios
PhoStream: Benchmarking Real-World Streaming for Omnimodal Assistants in Mobile Scenarios
Xudong Lu, Huankang Guan, Yang Bo, Jinpeng Chen, Xintong Guo, Shuhan Li, Fang Liu, Peiwen Sun, Xueying Li, Wei Zhang, Xue Yang, Rui Liu, Hongsheng Li
International Conference on Machine Learning (ICML), 2026
[Paper]
OmniInteract: Benchmarking Real-World Streaming Interaction for Real-Time Omnimodal Assistants
OmniInteract: Benchmarking Real-World Streaming Interaction for Real-Time Omnimodal Assistants
Xudong Lu*, Xueying Li*, Annan Wang*, Yang Bo, Jinpeng Chen, Zengliang Li, Nianzu Yang, Rui Liu, Xue Yang, Jingwen Hou, Hongsheng Li
Arxiv Preprint 2026
[Paper]
AURA: Always-On Understanding and Real-Time Assistance via Video Streams


Xudong Lu*, Yang Bo*, Jinpeng Chen*, Shuhan Li*, Xintong Guo*, Huankang Guan*, Fang Liu, Dunyuan Xu, Peiwen Sun, Heyang Sun, Rui Liu, Hongsheng Li
Arxiv Preprint 2026
[Paper] [vLLM-Omni] [ζœΊε™¨δΉ‹εΏƒ (Synced China)]

On-Device LLMs and MLLMs

SmartBench: Is Your LLM Truly a Good Chinese Smartphone Assistant?
SmartBench: Is Your LLM Truly a Good Chinese Smartphone Assistant?
Xudong Lu*, Haohao Gao*, Renshou Wu*, Shuai Ren, Xiaoxin Chen, Hongsheng Li, Fangyuan Li
The 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP Main Conference), 2025
[Paper] [ζ–°ζ™Ίε…ƒ (Aiera)]
GenieBlue: Integrating both Linguistic and Multimodal Capabilities for Large Language Models on Mobile Devices


Xudong Lu*, Yinghao Chen*, Renshou Wu*, Haohao Gao, Xi Chen, Xue Yang, Xiangyu Zhao, Aojun Zhou, Fangyuan Li, Yafei Wen, Xiaoxin Chen, Shuai Ren, Hongsheng Li
International Conference on Computer Vision (ICCV), 2025
[Paper] [量子位 (QbitAI)]
Adaptive Markup Language Generation for Contextually-Grounded Visual Document Understanding
Adaptive Markup Language Generation for Contextually-Grounded Visual Document Understanding
Han Xiao, Yina Xie, Guanxin Tan, Yinghao Chen, Rui Hu, Ke Wang, Aojun Zhou, Hao Li, Hao Shao, Xudong Lu, Peng Gao, Yafei Wen, Xiaoxin Chen, Shuai Ren, Hongsheng Li
The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025
[Paper]
BlueLM-V-3B: Algorithm and System Co-Design for Multimodal Large Language Models on Mobile Devices


Xudong Lu*, Yinghao Chen*, Cheng Chen*, Hui Tan*, Boheng Chen, Yina Xie, Rui Hu, Guanxin Tan, Renshou Wu, Yan Hu, Yi Zeng, Lei Wu, Liuyang Bian, Zhaoxiong Wang, Long Liu, Yanzhou Yang, Han Xiao, Aojun Zhou, Yafei Wen, Xiaoxin Chen, Shuai Ren, Hongsheng Li

The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025
[Paper] [ζœΊε™¨δΉ‹εΏƒ (Synced China)]
BlueLM-V-3B: Algorithm and System Co-Design for Multimodal Large Language Models on Mobile Devices

Efficient LLMs

ThinK: Thinner Key Cache by Query-Driven Pruning
ThinK: Thinner Key Cache by Query-Driven Pruning
Yuhui Xu, Zhanming Jie, Hanze Dong, Lei Wang, Xudong Lu, Aojun Zhou, Amrita Saha, Caiming Xiong, Doyen Sahoo
International Conference on Learning Representations (ICLR), 2025, (Spotlight)
[Paper]
Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models
Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models
Xudong Lu*, Qi Liu*, Yuhui Xu, Aojun Zhou, Siyuan Huang, Bo Zhang, Junchi Yan, Hongsheng Li
Annual Meeting of the Association for Computational Linguistics (ACL Main Conference), 2024
[Paper]
SPP: Sparsity-Preserved Parameter-Efficient Fine-Tuning for Large Language Models
SPP: Sparsity-Preserved Parameter-Efficient Fine-Tuning for Large Language Models
Xudong Lu*, Aojun Zhou*, Yuhui Xu*, Renrui Zhang, Peng Gao, Hongsheng Li
International Conference on Machine Learning (ICML), 2024
[Paper]

Generation

Rethinking Video Tokenization: A Conditioned Diffusion-based Approach
Rethinking Video Tokenization: A Conditioned Diffusion-based Approach
Nianzu Yang*, Pandeng Li*, Liming Zhao, Yang Li, Chen-Wei Xie, Yehui Tang, Xudong Lu, Zhihang Liu, Yun Zheng, Yu Liu, Junchi Yan
Arxiv Preprint 2025
[Paper]
ZoLA: Zero-Shot Creative Long Animation Generation with Short Video Model
ZoLA: Zero-Shot Creative Long Animation Generation with Short Video Model
Fu-Yun Wang, Zhaoyang Huang, Qiang Ma, Guanglu Song, Xudong Lu, Weikang Bian, Yijin Li, Yu Liu, Hongsheng Li
European Conference on Computer Vision (ECCV), 2024, (Oral)
[Paper]
TerDiT: Ternary Diffusion Models with Transformers
TerDiT: Ternary Diffusion Models with Transformers
Xudong Lu, Aojun Zhou, Ziyi Lin, Qi Liu, Yuhui Xu, Renrui Zhang, Xue Yang, Junchi Yan, Peng Gao, Hongsheng Li
Arxiv Preprint 2024
[Paper]

Vision & Remote Sensing

Point2RBox-v3: Self-Bootstrapping from Point Annotations via Integrated Pseudo-Label Refinement and Utilization
Point2RBox-v3: Self-Bootstrapping from Point Annotations via Integrated Pseudo-Label Refinement and Utilization
Teng Zhang*, Ziqian Fan*, Mingxin Liu, Xin Zhang, Xudong Lu, Wentong Li, Yue Zhou, Yi Yu, Xiang Li, Junchi Yan, Xue Yang
International Conference on Learning Representations (ICLR), 2026
[Paper]
Co-Training Vision-Language Models for Remote Sensing Multi-Task Learning
Co-Training Vision-Language Models for Remote Sensing Multi-Task Learning
Qingyun Li*, Shuran Ma*, Junwei Luo, Yi Yu, Yue Zhou, Fengxiang Wang, Xudong Lu, Xiaoxing Wang, Xin He, Yushi Chen, Xue Yang
The Remote Sensing (MDPI) Journal, 2026
[Paper]
GLEAM: Learning to Match and Explain in Cross-View Geo-Localization
GLEAM: Learning to Match and Explain in Cross-View Geo-Localization
Xudong Lu*, Zhi Zheng*, Yi Wan, Yongxiang Yao, Annan Wang, Renrui Zhang, Panwang Xia, Qiong Wu, Qingyun Li, Weifeng Lin, Xiangyu Zhao, Peifeng Ma, Xue Yang, Hongsheng Li
Arxiv Preprint 2025
[Paper]
PointOBB-v3: Expanding Performance Boundaries of Single Point-Supervised Oriented Object Detection
PointOBB-v3: Expanding Performance Boundaries of Single Point-Supervised Oriented Object Detection
Peiyuan Zhang*, Junwei Luo*, Xue Yang*, Yi Yu, Qingyun Li, Yue Zhou, Xiaosong Jia, Xudong Lu, Jingdong Chen, Xiang Li, Junchi Yan, Yansheng Li
International Journal of Computer Vision (IJCV), 2025
[Paper]
From isolated islands to pangea: Unifying semantic space for human action understanding
From isolated islands to pangea: Unifying semantic space for human action understanding
Yong-Lu Li*, Xiaoqian Wu*, Xinpeng Liu, Zehao Wang, Yiming Dou, Yikun Ji, Junyi Zhang, Yixing Li, Xudong Lu, Jingru Tan, Cewu Lu
The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, (Highlight)
[Paper]

AI for Science and Quantum Computing

Rethinking and Improving Autoformalization: Towards a Faithful Metric and a Dependency Retrieval-based Approach
Rethinking and Improving Autoformalization: Towards a Faithful Metric and a Dependency Retrieval-based Approach
Qi Liu, Xinhao Zheng, Xudong Lu, Qinxiang Cao, Junchi Yan
International Conference on Learning Representations (ICLR), 2025, (Spotlight)
[Paper]
QuantumDARTS: Differentiable Quantum Architecture Search for Variational Quantum Algorithms
QuantumDARTS: Differentiable Quantum Architecture Search for Variational Quantum Algorithms
Wenjie Wu, Ge Yan, Xudong Lu, Kaisen Pan, Junchi Yan
International Conference on Machine Learning (ICML), 2023
[Paper]
QAS-Bench: Rethinking Quantum Architecture Search and A Benchmark
QAS-Bench: Rethinking Quantum Architecture Search and A Benchmark
Xudong Lu, Kaisen Pan, Ge Yan, Jiaming Shan, Wenjie Wu, Junchi Yan
International Conference on Machine Learning (ICML), 2023
[Paper]

Educational Background



CUHK

Aug. 2023 - Present, Department of Electronic Engineering, the Chinese University of Hong Kong


Ph.D. Student

SJTU

Aug. 2019 - Jun. 2023, Department of Computer Science and Engineering, Shanghai Jiao Tong University


Bachelor


Internship and Cooperation



Shanghai AI Laboratory     vivo AI Lab     Huawei Leibniz Research Center     ByteDance Seed

Academic Service


Journal Reviewer

IEEE TCSVT, IEEE TPAMI, IEEE TGRS


Conference Reviewer

CVPR, NeurIPS, ICLR, ICCV, ICML, AAAI, ECCV

Copyright © Lu Xudong, 2026