Generative Visual Models

Abstract

This seminar investigates generative models for image synthesis, which can be controlled via language prompts and visual seeding. The relevant methods will be explained in a few initial classes. Participants will study the research literature and develop project ideas in small groups, which will then be implemented. Presentation of research papers, project ideas, and results is a key component.

Objective

The goal of this class is for participants to find, read, understand and critically assess research literature in order to reach the current state of knowledge in the field. Moreover, the project work aims to enrich these readings by hands-on experience and allows for student to develop creative ideas of their own. This is meant to provide a wholistic research experience in small teams.

Content

Phase 1: Introduction & Background During the first weeks of the semester lectures will provide the technical background to understand visual generative models. This includes a historic overview as well as technical deep dives into specialized topics such as stable diffusion and contrastive learning. There will also be a tutorial on suitable software framework to explore and fine-tune such models. Each participant will do a graded pen & paper exercise in order to check on progress. 20% of the grade, correctness of questions. Phase 2: Reading & Planning In the second phase, participants will split up in teams (ideal size 3) and will perform independent reading and planning towards a project idea. Paper suggestions and project sketches will be distributed to provide guidance and inspiration. During this time, participants are also expected to familiarize themselves with the experimental setup (we will locally host models on our GPU servers) and perform some simple warm-up or proof-of-concept experiments to inform the project definition. Each group will give a 15+5 min project pitch and will give/receive feedback from other teams. 30% of the grade, creativity of the idea, clarity of project articulation, recognition of existing work. Phase 3: Project Execution & Presentation In the third phase, teams will implement their project and run the designed experiments to answer the articulated research questions or goals. Participants will have (limited) access to local GPU servers. Each group will produce a written project report and will deliver a presentation. 50% of the grade, success of the project, quality of the experiments, quality of the slides/presentation.

Resources