Tianyi Zhou

Tianyi Zhou

PhD CSE USC

I am a third-year Ph.D. student in Computer Science at the University of Southern California, by co-advised by Prof. Robin Jia and Prof. Vatsal Sharan. I received my M.S. in Computer Science from the University of California, San Diego.

My research focuses on interpretability and efficiency in large language models. I study how internal mechanisms and learned representations give rise to reasoning and numerical capabilities, and leverage these insights to improve learning dynamics, computational efficiency, and generalization through better model architectures and training objectives.

Research

Machine Learning

  • FoNE: Precise Single-Token Number Embeddings via Fourier Features
    (arxiv link)(project website).
    Tianyi Zhou, Deqing Fu, Mahdi Soltanolkotabi, Robin Jia and Vatsal Sharan

  • Pretrained Large Language Models Use Fourier Features to Compute Addition (arxiv link).
    Tianyi Zhou, Deqing Fu, Vatsal Sharan and Robin Jia
    NeurIPS 2024.

  • H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models (arxiv link).
    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang and Beidi Chen
    NeurIPS 2023.

  • Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time (arxiv link).
    Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Ré, and Beidi Chen
    ICML 2023 Selected as Oral.

Theoretical Computer Science (Author names in alphabetical order)

  • Faster Algorithm for Structured John Ellipsoid Computation (arxiv link).
    Yang Cao, Xiaoyu Li, Zhao Song, Xin Yang and Tianyi Zhou
    NeurIPS, 2025.

  • Fourier Circuits in Neural Networks and Transformers: A Case Study of Modular Arithmetic with Multiple Inputs (arxiv link).
    Chenyang Li, Yingyu Liang, Zhenmei Shi, Zhao Song, and Tianyi Zhou
    AISTAT, 2025.

  • Algorithm and Hardness for Dynamic Attention Maintenance in Large Language Models (arxiv link).
    Jan van den Brand, Zhao Song and Tianyi Zhou
    ICML, 2024.

  • Space-Efficient Interior Point Method, with applications to Linear Programming and Maximum Weight Bipartite Matching (arxiv link).
    Sixue Liu, Zhao Song, Hengjie Zhang, Lichen Zhang and Tianyi Zhou
    ICALP, 2023.

  • The Closeness of In-Context Learning and Weight Shifting for Softmax Regression (arxiv link).
    Shuai Li, Zhao Song, Yu Xia, Tong Yu and Tianyi Zhou
    NeurIPS 2024.