Real-World Robotics - A Hands-On Project Class

Abstract

In this hands-on course, students work in teams to build intelligent robotic systems capable of solving real-world dexterous manipulation tasks. The course covers robot design, sensing, control, and state-of-the-art learning techniques. Through lectures, workshops, and a final challenge, students gain practical experience integrating hardware and AI to tackle complex manipulation problems.

Objective

Learning Objective 1: High-Level System Design and System Integration Students will learn to design, plan, and manage the development of a complete robotic manipulation system. This includes identifying requirements, organizing project timelines, integrating hardware and software components, and ensuring that all subsystems work together robustly to solve a real-world dexterous manipulation challenge. Learning Objective 2: Robot Design, Sensing, and Control Students will gain practical experience in designing and modifying robotic hardware (specifically a multi-fingered hand), integrating sensors for perception, and developing control strategies for teleoperation and autonomous behaviors. Emphasis will be placed on understanding the role of sensing in manipulation and implementing effective low-level control. Learning Objective 3: Learning for Robotic Manipulation Students will acquire theoretical and practical knowledge of modern machine learning techniques—such as reinforcement learning (RL), imitation learning (IL), and vision-language-action (VLA) models—for dexterous robotic manipulation. They will train models in simulation, perform sim-to-real transfer, and apply learning-based control strategies on real hardware.

Content

During this course, students will work in teams to tackle a real-world dexterous manipulation challenge. Through a combination of lectures, workshops, and hands-on development, students will acquire both theoretical and practical skills across robot design, fabrication, sensing, control, and learning techniques—like reinforcement learning (RL) and imitation learning (IL). Theoretical concepts will be introduced in lectures, while targeted workshops will guide students in applying these ideas to their team projects. Students will do design iterations to build robotic hands and use learning-based approaches to solve complex manipulation tasks. Throughout the course, the latest topics in robotic manipulation research will be presented to expose students to the current state of the art, through specific seminars. The overall course is structured into six parts: 1) Introduction - Overview of the challenges and real-world impact of robotic manipulation - Presentation of the final challenge - Fundamentals of team organization and project management 2) Robot Design and Fabrication - Assembly and evaluation of a five-fingered robotic hand - Design iteration: propose and implement modifications to improve hardware performance 3) Robot Control and Sensing - Introduction to sensing strategies for robotic manipulation - Development of teleoperation and low-level control for the robotic hand 4) Reinforcement Learning (RL) - Learn RL methods for dexterous in-hand manipulation - Train and evaluate policies in simulation - Explore sim-to-real transfer techniques for deployment on physical hardware 5) Imitation Learning (IL) - Study modern IL approaches including diffusion policies and VLA models - Implement IL pipelines for skill acquisition from demonstrations 6) Final Demo and Presentation - Showcase a novel contribution developed during the course - This could include, but is not limited to: improved hardware, new control strategies, novel manipulation tasks, full learning-based pipelines, or model architecture innovations

Resources