VVZ API is not affiliated with ETH Zurich. Data might be outdated or incorrect. Please view the official ETHZ Vorlesungsverzeichnis for binding information.

262-0201-00L 4 Credits DR , MSC D-INFK , D-BSSE

Multimodal Medical AI

Lecturers & Examiners: Prof. Dr. Michael Moor
VVZ CR n/a

Last Updated: 2026-06-03 00:07:32

Abstract

This lecture explores the transformative role of Artificial Intelligence (AI) in multimodal medical applications. It covers AI fundamentals, foundation models (e.g., LLMs, VLMs), training strategies, LLM agents, tool use, reasoning, and external knowledge integration, such as retrievalaugmented generation (RAG), exemplified in the medical application domain.

Objective

- Understand the fundamentals of AI and ML, including core assumptions, standard notations, and problem paradigms (e.g., supervised vs. unsupervised learning, generative vs. discriminative tasks). - Learn about foundation models (e.g., LLMs, VLMs) and their training strategies, as well as inference strategies. - Explore the use of AI agents and frameworks enabling action, tool use, and interaction in diverse environments. - Evaluate how these technologies are applied to multimodal medical contexts to drive robust decision-making.

Content

Artificial Intelligence (AI) has profound potential in advancing multimodal medical applications. This lecture gives a broad overview to multimodal medical AI, as propelled by recent advances in foundation models. Beginning with a crash course on the fundamentals of AI and machine learning (ML), we will outline core assumptions, introduce standard notations, and define key data types and problem paradigms such as supervised vs. unsupervised learning and generative vs. discriminative tasks, as well as different types of supervision. Building on this, we delve into foundation models (FMs)—large pre-trained AI models such as LLMs (Large Language Models) and VLMs (Vision-Language Models)—covering pre-training strategies, self-supervised learning, and post-training efforts like supervised fine-tuning (SFT) and alignment strategies. With medical use cases in mind, the lecture transitions into the realm of AI agents, focusing on frameworks that enable FMs to "act" in various environments, use tools, and interact with systems of AI agents. We will examine state-of-the-art frameworks, and discuss optimization strategies to make AI agents effective and grounded problem solvers in medical domains. Finally, we tackle the evolving landscape of reasoning in AI, highlighting the shift from direct input-output mappings to complex intermediate representations. This section will cover techniques such as external memory and retrieval-augmented generation (RAG), as well as structured reasoning chains, demonstrating how these advances empower more robust decision-making in multimodal medical contexts. This lecture provides a comprehensive overview of how foundational technologies and novel reasoning approaches are shaping the next generation of AI-driven medical tools and systems.

Resources

Literature

Amidi, A., & Amidi, S. (2023). Super Study Guide: Transformers & Large Language Models. Moor, M., Banerjee, O., Abad, Z.S.H. et al. Foundation models for generalist medical artificial intelligence. Nature 616, 259–265 (2023).

General Information

Language
English
Levels
DR , MSC
Frequency
Yearly recurring

Examination

Type
end-of-semester examination
Mode
written 120 minutes
Aids
None

Course Components

Type Title Time & Place Hours
lecture with exercise Multimodal Medical AI
This lecture will take place in classroom in BASEL. Attention: the lecture will start in the second week of the semester.
No time listed 3 h weekly

Offered In