Yutong Yao (Patrick) | AI Systems & Sports-Tech

Projects

These projects combine AI, systems, and real-world constraints, with a particular focus on sports performance and embodied intelligence.

Speak-to-Draw — LLM-Powered Multimodal Querying for Time-Series Data

Built a multimodal time-series querying interface combining natural-language and sketch-based input for human-in-the-loop analysis.
Implemented an LLM-driven pipeline that maps language to structured queries for coarse retrieval, with sketches refining local temporal patterns.

View Details

AutoScout — AI Football Film Analysis

Built an end-to-end computer vision system that converts raw football game footage into structured tactical and play-call insights.
Implemented a single-camera vision pipeline using fine-tuned YOLOv8, homography-based field mapping, and trajectory-level supervision to infer formations, play types, and tactical patterns.

View Details

Hoopers — Dual-Sensor Basketball Shooting Coach

Co-founded Hoopers, building a wearable sleeve with IMUs, IR sensing, and computer vision keypoint tracking to capture arm and full-body shooting mechanics. vision keypoint tracking to capture arm and full-body shooting mechanics.
Compared users’ motion against a diverse dataset of professional players to provide personalized shooting feedback.
Won 1st place at Hong Kong InnoX and secured HKSTP IDEATION funding (HKD 100K)

View Details

Franka Toolbox Bridge

Built a Python-based framework that unifies simulation and real-world control of the Franka Emika Panda through a single, ROS-free API.
The framework has been adopted in course CSC376H5 – Fundamentals of Robotics at the University of Toronto Mississauga to teach real robot motion control.

View Details

XLANG — Vision-Language-Action General Policy

The project targets training generalizable VLA policies that can follow fine-grained manipulation instructions beyond coarse, high-level commands.
I helped build an AI-assisted data pipeline that integrates LLMs and Grounding DINO to generate first-pass fine-grained annotations for 2M+ manipulation trajectories, substantially reducing annotation effort.

View Details

Patky — Personalized Sports Trading Cards

Built an AI-assisted design pipeline that uses SAM2 for automated player segmentation and LLMs for content generation to streamline the creation of personalized sports trading cards.
Launched and operated the product with 200+ customers and sustained HKD 2,000+ in monthly revenue, validating demand for AI-driven creative automation.

View Details

Leadership & Activities

Starting Quarterback & Captain

Hong Kong Cobras

Led an underdog American football team as starting QB, responsible for play-calling, in-game adjustments, and team preparation.
Developed a film-study routine and practice structure to raise team performance under limited resources.
Learned to handle pressure, bounce back from mistakes, and keep teammates aligned on a shared goal.

Vice President, Club Management Department

CSSA HKU

Managed a team of ~30 members and supported 6 sports teams across logistics, communication, and event planning.
Organized large-scale student events and coordinated between committees, venues, and sponsors.
Strengthened my ability to balance operations, communication, and people management.

Education

The University of Hong Kong (HKU)

2022 – Present

Bachelar of Engineering in Computer Science

First Class Honours; Dean’s List 2022–23 & 2023–24.
CGPA: 3.82 out of 4.3 (First Class Honor)
Lee Shau Kee Scholarships for Student Enrichment (HKD 12,000)
Mitacs Globalink Research Internship Award (CAD $8,760)
HKSTP IDEATION Fund (HKD 100K)

University of California, Davis (UC Davis)

Jan – Jun 2025

Undergraduate Exchange, Computer Science

Achieved GPA (3.94/4.0) across 5 core Computer Science courses during exchange study.

Research Experience

I’ve contributed to four research projects spanning embodied AI, human–computer interaction, robotics, and bioinformatics.

XLANG Lab, HKU — Vision-Language-Action General Policy

Advisor: Prof. Tao Yu

Contributing to training a VLA general policy capable of executing detailed natural language commands.
Developed an AI-assisted pipeline to filter open-source manipulation datasets and generate fine-grained trajectory labels at a scale exceeding 2 million samples, targeting ICML 2026 submission.

VIA Lab, UC Davis — Speak-to-Draw Analytics Tool

Advisor: Prof. Dongyu Liu

Built a multimodal time-series analytics platform “Speak to Draw,” enabling users to query data using natural language and sketches, resulting in a paper submitted to EuroVis 2026.
Designed a multimodal retrieval pipeline where LLMs map natural-language queries to high-level representations for an initial retrieval and sketches refine results via temporal similarity constraints.

MEDCVR Lab, University of Toronto — Franka Toolbox Bridge

Advisor: Prof. Lueder Kahrs

Developed a unified Python framework integrating real and simulated control of multiple Franka Emika Panda robots, which was later adopted for teaching in UTM’s course CSC376H5 – Fundamentals of Robotics.
Enabled advanced motion planning, testing, and deployment in a single environment to support surgical robot autonomy.

Bioinformatics Lab, HKU — A-to-I Editing with Nanopore

Advisor: Prof. Ruibang Luo

Benchmarked algorithms for detecting RNA A-to-I editing, contributing to improved genomic modification analysis.
Conducted comparative testing on existing models and datasets to evaluate accuracy and computational efficiency.

About Me

I am a Computer Science student at The University of Hong Kong, focused on building end-to-end AI systems across embodied AI, multimodal interaction, and sports technology. My long-term goal is to develop applied AI systems that translate human intent into reliable, real-time decision support—bridging perception, structured reasoning, and action in both digital and physical environments.

Outside the lab, I am the starting quarterback and captain of the Hong Kong Cobras American football team, and serve as Vice President of CSSA HKU. I enjoy working at the intersection of research and real-world systems—turning ideas into working prototypes, and prototypes into tools that people can depend on.

Contact

Email: u3597462@connect.hku.hk

Phone: +85267914926