VVZ API is not affiliated with ETH Zurich. Data might be outdated or incorrect. Please view the official ETHZ Vorlesungsverzeichnis for binding information.
Multimodal Medical AI
Last Updated: 2026-06-01 11:30:44
Abstract
This lecture explores the transformative role of Artificial Intelligence (AI) in multimodal medical applications. It covers AI fundamentals, foundation models (e.g., LLMs, VLMs), training strategies, LLM agents, tool use, reasoning, and external knowledge integration, such as retrievalaugmented generation (RAG), exemplified in the medical application domain.
Objective
- Understand the fundamentals of AI and ML, including core assumptions, standard notations, and problem paradigms (e.g., supervised vs. unsupervised learning, generative vs. discriminative tasks). - Learn about foundation models (e.g., LLMs, VLMs) and their training strategies, as well as inference strategies. - Explore the use of AI agents and frameworks enabling action, tool use, and interaction in diverse environments. - Evaluate how these technologies are applied to multimodal medical contexts to drive robust decision-making.
Content
Artificial Intelligence (AI) has profound potential in advancing multimodal medical applications. This lecture gives a broad overview to multimodal medical AI, as propelled by recent advances in foundation models. Beginning with a crash course on the fundamentals of AI and machine learning (ML), we will outline core assumptions, introduce standard notations, and define key data types and problem paradigms such as supervised vs. unsupervised learning and generative vs. discriminative tasks, as well as different types of supervision. Building on this, we delve into foundation models (FMs)—large pre-trained AI models such as LLMs (Large Language Models) and VLMs (Vision-Language Models)—covering pre-training strategies, self-supervised learning, and post-training efforts like supervised fine-tuning (SFT) and alignment strategies. With medical use cases in mind, the lecture transitions into the realm of AI agents, focusing on frameworks that enable FMs to "act" in various environments, use tools, and interact with systems of AI agents. We will examine state-of-the-art frameworks, and discuss optimization strategies to make AI agents effective and grounded problem solvers in medical domains. Finally, we tackle the evolving landscape of reasoning in AI, highlighting the shift from direct input-output mappings to complex intermediate representations. This section will cover techniques such as external memory and retrieval-augmented generation (RAG), as well as structured reasoning chains, demonstrating how these advances empower more robust decision-making in multimodal medical contexts. This lecture provides a comprehensive overview of how foundational technologies and novel reasoning approaches are shaping the next generation of AI-driven medical tools and systems.
Resources
Literature
Amidi, A., & Amidi, S. (2023). Super Study Guide: Transformers & Large Language Models. Moor, M., Banerjee, O., Abad, Z.S.H. et al. Foundation models for generalist medical artificial intelligence. Nature 616, 259–265 (2023).
General Information
- Language
- English
- Levels
- DR , MSC
- Frequency
- Yearly recurring
Examination
- Type
- end-of-semester examination
- Mode
- written 120 minutes
- Aids
- None
Registration & Places
- Max Places
- 50
Course Components
| Type | Title | Time & Place | Hours |
|---|---|---|---|
| lecture with exercise |
Multimodal Medical AI
This lecture will take place in classroom in BASEL.
Attention: the lecture will start in the second week of the semester.
|
|
3 h weekly |
Offered In
-
Computational Biology and Bioinformatics Master (Weitere Informationen: )
-
Kernfächer (Die Liste der Kernfächer ist eine geschlossene Liste - es können keine anderen Kurse in dieser Kategorie hinzugefügt werden. Die Zuordnung der Kurse zu der jeweiligen Unterkategorie kann nicht geändert werden. Studierende müssen mindestens einen Kurs pro Unterkategorie bestehen. Insgesamt müssen 40 ECTS Kernfächer erworben werden, einschliesslich des obligatorischen CBB-Seminars.)
-
-
Doktorat Biosysteme (Mehr Informationen unter: Für Kurse der Kategorie "Integration in die wissenschaftliche Gemeinschaft" bitte die BSSE Webseite konsultieren: )
-
Biotechnologie Master (Weitere Informationen: )
-
Wahlfächer (Offene Liste - weitere Kurse (ETH oder UNIBAS) können nach Absegnung durch den:die Mentor:in als Wahlfächer gewählt werden.)
-