VVZ API is not affiliated with ETH Zurich. Data might be outdated or incorrect. Please view the official ETHZ Vorlesungsverzeichnis for binding information.

263-5912-00L 2 Credits DR D-INFK

Empirical Research Methods for Computer Science

Lecturers & Examiners: Prof. Dr. April Wang
The deadline for deregistering expires at the end of the second week of the semester. Students who are still registered after that date, but do not attend the course, will officially fail the course.
VVZ CR n/a

Last Updated: 2026-06-03 00:07:38

Abstract

This course introduces core principles for designing, analyzing, and critically evaluating empirical studies in computer science. It focuses on how to formulate research questions, select appropriate study designs (experiments, quasi-experiments, and observational studies), and apply statistical and qualitative methods to generate valid and interpretable results.

Objective

This course introduces core principles for designing, analyzing, and critically evaluating empirical studies in computer science. It focuses on how to formulate research questions, select appropriate study designs (experiments, quasi-experiments, and observational studies), and apply statistical and qualitative methods to generate valid and interpretable results. Instead of covering a broad range of methods, the course focuses on how to reason about empirical evidence. We look at how to identify threats to validity (internal, external, and construct), how to match research questions with appropriate study designs, and how to make sound methodological decisions. The course draws examples from multiple areas of computer science, such as human-computer interaction, software engineering, and NLP, to highlight both shared principles and domain-specifi c challenges. Throughout the course, students will develop and refi ne a proposal for an empirical study, which they will present at the end. The proposal should clearly articulate the research question, study design, analysis plan, and key validity considerations.

Content

Week 1. Framing Empirical Research in CS • Types of research questions (causal, descriptive, exploratory) • Types of claims (causal vs correlational vs descriptive) • Experiments vs. observational studies • What counts as evidence in different CS domains • Common pitfalls in empirical claims Week 2. Study Design and Causal Inference • Controlled experiments and quasi-experiments • Randomization, control, and confounds • Within-subject vs. between-subject designs • Reasoning about causal claims and internal validity • Common confounds Week 3. Measurement and Construct Validity • Operationalization • Metrics and proxies (e.g., performance, engagement, accuracy) • Instrumentation in CS (logs, benchmarks, human evaluation) • Threats to construct validity Week 4. Quantitative Analysis • Descriptive statistics and distributions • Hypothesis testing (practical interpretation) • Effect sizes and confidence intervals • Variance, instability, and common statistical pitfalls Week 5. Qualitative Methods and Mixed Methods • When numbers are not enough • Error analysis and taxonomy building • Annotation and human judgment • Human evaluation of systems • Interviews, observations, and think-aloud protocols • Mixed methods Week 6. Supporting and interpreting empirical claims • Baselines and fair comparisons • Ablation studies • Reproducibility and reporting practices • Generalization and robustness • Validity trade-offs Week 7. Student Presentations

General Information

Language
English
Levels
DR
Frequency
Yearly recurring

Examination

Type
graded semester performance
- Final project (proposal + poster): 50 %- BYOD Exam 40 % (an A4 paper cheatsheet allowed)- Attendance: 10 %

Registration & Places

Priority: Registration for the course unit is only possible for the primary target group

Course Components

Type Title Time & Place Hours
lecture Empirical Research Methods for Computer Science No time listed 1 h weekly

Offered In