VVZ API is not affiliated with ETH Zurich. Data might be outdated or incorrect. Please view the official ETHZ Vorlesungsverzeichnis for binding information.
Big Data
Last Updated: 2026-02-05 16:16:39
Abstract
The key challenge of the information society is to turn data into information, information into knowledge, knowledge into value. This has become increasingly complex. Data comes in larger volumes, diverse shapes, from different sources. Data is more heterogeneous and less structured than forty years ago. Nevertheless, it still needs to be processed fast, with support for complex operations.
Objective
Do you want to be able to query your own data productively and efficiently in your future semester projects, master thesis, or PhD thesis? Are you looking for something beyond the Python+Pandas hype? This courses teaches you how to do so as well as the dos and don'ts. "Big Data" refers to the case when the amount of data is very large (100 GB and more), or when the data is not completely structured (or messy). The Big Data revolution has led to a completely new way to do business, e.g., develop new products and business models, but also to do science -- which is sometimes referred to as data-driven science or the "fourth paradigm". Unfortunately, the quantity of data produced and available -- now in the Zettabyte range (that's 21 zeros) per year -- keeps growing faster than our ability to process it. Hence, new architectures and approaches for processing it are needed. Harnessing them must involve a deep understanding of data not only in the large, but also in the small. The field of databases evolves at a fast pace. In order to be prepared, to the extent possible, to the (r)evolutions that will take place in the next few decades, the emphasis of the lecture will be on the paradigms and core design ideas, while today's technologies will serve as supporting illustrations thereof. After visiting this lecture, you should have gained an overview and understanding of the Big Data landscape, which is the basis on which one can make informed decisions, i.e., pick and orchestrate the relevant technologies together for addressing each one of your projects efficiently and consistently.
Content
This course gives an overview of database technologies and of the most important database design principles that lay the foundations of the Big Data universe. We take the monolithic, one-machine relational stack from the 1970s, smash it down and rebuild it on top of large clusters: starting with distributed storage, and all the way up to syntax, models, validation, processing, indexing, and querying. A broad range of aspects is covered with a focus on how they fit all together in the big picture of the Big Data ecosystem. No data is harmed during this course, however, please be psychologically prepared that our data may not always be in third normal form. - physical storage: distributed file systems (HDFS), object storage(S3), key-value stores - logical storage: document stores (MongoDB), column stores (HBase), graph databases (neo4j), data warehouses (ROLAP) - data formats and syntaxes (XML, JSON, RDF, Turtle, CSV, XBRL, YAML, protocol buffers, Avro) - data shapes and models (tables, trees, graphs, cubes) - type systems and schemas: atomic types, structured types (arrays, maps), set-based type systems (?, *, +) - an overview of functional, declarative programming languages across data shapes (SQL, XQuery, JSONiq, Cypher, MDX) - the most important query paradigms (selection, projection, joining, grouping, ordering, windowing) - paradigms for parallel processing, two-stage (MapReduce) and DAG-based (Spark) - resource management (YARN) - what a data center is made of and why it matters (racks, nodes, ...) - underlying architectures (internal machinery of HDFS, HBase, Spark, neo4j) - optimization techniques (functional and declarative paradigms, query plans, rewrites, indexing) - applications. Large scale analytics and machine learning are outside of the scope of this course.
Resources
Literature
Course textbook: https://ghislainfourny.github.io/big-data-textbook/ Papers from scientific conferences and journals. References will be given as part of the course material during the semester.
Learning Materials (Links)
- Main link
- Information
- Recording
- All lectures are recorded and made available on YouTube
- Literature
- Big Data: textbook (in progress!)
General Information
- Language
- English
- Levels
- DR , MSC , WBZ
- Frequency
- Yearly recurring
Examination
- Type
- session examination
- Mode
- written 180 minutes
- Aids
- General dictionaries are allowed. This includes general English dictionaries (with word definitions) as well as general bilingual dictionaries (English <-> other language). Specialized dictionaries are not allowed. Dictionaries cannot be annotated by hand.
- Digital
- The exam takes place on devices provided by ETH Zurich.
Registration & Places
Course Components
| Type | Title | Time & Place | Hours |
|---|---|---|---|
| lecture | Big Data |
|
3 h weekly |
| exercise |
Big Data
Groups are selected in myStudies.
|
|
2 h weekly |
| independent project |
Big Data
Individual work to get hands-on experience with the technologies covered, no fixed presence required.
|
No time listed | 4 h weekly |
Offered In
-
-
-
Computational Biology and Bioinformatics Master (More information at: )
-
Advanced Courses (A total of 30 ECTS needs to be acquired in the Advanced Courses category. Thereof at least 16 ECTS in the Theory and at least 10 ECTS in the Biology category. Note that some of the lectures are being recorded: )
-
Theory (At least 16 ECTS need to be acquired in this category.)
-
-
-
-
Doctorate Computer Science (More Information at: )
-
-
-
-
-
-