The course covers the mathematical models and the algorithms that underpin sequential decision making in systems that evolve over time. The emphasis is on optimal control and adaptation, on learning a policy from interaction, and on choosing actions when the state of the world is only partly known.
Practical matters
Lecturer: Petri Ahlgren.
Teaching assistants (TAs): Nina Salomaa (main TA), Aleksi Rautio, Daniel Weiss, Sofia Marchetti, Omar Haddad, Ravi Menon, Lotta Peltola, Ines Duarte, Jonas Kraft, Mira Oksanen, Tomas Novak
Grading runs 0-5. The seven individual assignments carry 55%, the quizzes 15% and the exam 30%.
Join the course chat early in the term: updates land there first, and it is where exercise questions get answered. Register with your Aalto account. Since the answers stay in the channel, read through it before asking something new.
Lectures
Lecture arrangements for the term:
- Location: Hall C3
- Opening lecture: Wednesday 2.9.2026, 14:15 - 16:00
- Tuesdays 14:15-16:00 from week 37 onwards, through Periods I and II
- No lecture on Tuesday 13.10.2026
- Coming along in person gets you the discussion and the questions, but every lecture is recorded and stays available to watch later
| Week | Lecture | Lecture_Date | Reading |
|---|---|---|---|
| W36 | L1 Course Overview | Wed, 2.9 | no readings |
| W37 | L2 Sequential decision models | Tue, 8.9 | Course textbook, Chapters (Ch.) 1-1.4, 2.2-2.6, 3-3.5 |
| W38 | L3 Learning in finite state spaces | Tue, 15.9 | Course textbook, Ch. 4-4.5, 4.8, 5-5.3 |
| W39 | L4 Approximate value representations | Tue, 22.9 | Course textbook, Ch. 7-7.4, 8-8.2 |
| W40 | L5 Direct policy search | Tue, 29.9 | Course textbook, Ch. 11-11.3 |
| W41 | L6 Actor-critic architectures | Tue, 6.10 | Course textbook, Ch. 11.6, 11.9 |
| W42 | No Lecture | Tue, 13.10 | |
| W43 | L7 Learning the system dynamics | Tue, 20.10 | Course textbook, Ch. 6 - 6.3 |
| W44 | L8 Planning with a learned model | Tue, 27.10 | Course textbook, Ch. 6.4 - 6.7 |
| W45 | L9 Balancing exploration and exploitation | Tue, 3.11 | 1) Course textbook, Ch. 2.8, 6.9 - 6.11 and 2) Virtanen, H., Lange, P., Okafor, C., & Bergqvist, S. (2022). Posterior sampling for sequential decisions: a practical tutorial. Reviews in Machine Intelligence, 9(2), 1-84. https://reviews.example.com/posterior-sampling.pdf Section 2, 3, 5 |
| W46 | L10 Guest Lecture: Ravi Menon: Safe exploration under constraints, Mira Oksanen: Hierarchical option discovery | Tue, 10.11 | |
| W47 | L11 Decisions under partial observability | Tue, 17.11 | 1) K. Obermeier, Partially observable models tutorial, https://tutorials.example.com/partially-observable/, steps from “A Quick Refresher on Decision Processes” until “Where Belief States Come From” and 2) Decision Making under Partial Observability in Robotics: A Review. https://preprints.example.com/pdf/2211.40815 Sections II.B, III.A, III.D |
| W48 | No Lecture | Tue, 24.11 |
Quizzes
Quizzes are solved alone, without help from anyone else. Nothing in them goes beyond the lecture and its assigned reading, and each one closes as the following lecture starts.
| Quiz | Release | Deadline (always before the lecture) |
|---|---|---|
| Quiz 1 | 2.9 | 8.9 |
| Quiz 2 | 8.9 | 15.9 |
| Quiz 3 | 15.9 | 22.9 |
| Quiz 4 | 22.9 | 29.9 |
| Quiz 5 | 29.9 | 6.10 |
| Quiz 6 | 6.10 | 20.10 |
| Quiz 7 | 20.10 | 3.11 |
| Quiz 8 | 3.11 | 17.11 |
Exercises
Seven assignments are compulsory, and both they and the quizzes count as individual work. Arguing about algorithms, implementation choices and course concepts with other students is part of learning and nobody discourages it; passing around answers, data or source code is where the line falls. Put briefly—trade intuitions, not solutions.
- Submit on time. The closing page of each assignment instruction lists the files your return has to contain, so check it before uploading — nothing arrives late, neither a whole submission nor a forgotten file.
- Several notebooks take a long while to finish running, so begin well before the deadline. Compute outages and queue backlogs are not grounds for an extension.
- The TAs read the course chat on weekdays and answer there. The channel is swept roughly once a day and there may well be a queue ahead of you, so ask early rather than the night before a deadline.
- Never paste your own solution code into a public channel
- Debugging help is given face to face in the exercise sessions, not over chat.
- Direct a question about an exercise only at the TAs listed for it in the table below
- Every deadline falls at 23:59 on the stated day.
- One exercise session per week; coming along is up to you.
- Location: Lab 4
- Time: 10.15-12.00
- No exercise session during week 42
| Exercise | Release Date | Submission Date | Exercise Session Date | TAs |
|---|---|---|---|---|
| Exercise 1 | 2.9 | 14.9 | 9.9 | Aleksi, Lotta |
| Exercise 2 | 7.9 | 21.9 | 16.9 | Daniel, Lotta |
| Exercise 3 | 14.9 | 28.9 | 23.9 | Sofia, Omar |
| Exercise 4 | 21.9 | 5.10 | 30.9 | Sofia, Omar |
| Exercise 5 | 28.9 | 12.10 | 7.10 | Ines, Jonas |
| Exercise 6 | 5.10 | 26.10 | 21.10 | Ines, Jonas |
| Exercise 7 | 19.10 | 2.11 | 28.10 | Aleksi, Mira |
Exam
You cannot pass the course without passing the exam.
- Exam Passing grade: 45%
- Exam: Friday 27.11.2026, 9.00–12.00, Hall C3
- A single retake is arranged and its date follows later in the term.
- Exam consists of two parts
- Part A: brief written answers. Typical prompts are “state the optimality condition for a finite-horizon problem” or “explain what bootstrapping means in a value update”. One or two of them reach into later material such as planning with a learned model. Either describe the idea in your own words or give the expression behind it.
- Part B: two application problems. Each problem names one algorithm, sets the scene, and then asks a chain of questions about it. These may ask you to justify why the algorithm behaves as it does, to analyse a case where it fails, or to carry a small calculation through by hand.
- Looking for practice questions? The end-of-chapter problems in the course textbook are the closest thing available, and they give a usable sense of the level for anyone who works best from concrete examples. The exam reuses none of them, and its scope is the quizzes, the lectures and the assignments rather than the book’s table of contents.
- You must be able to show a photo ID at the exam hall door; the exam instructions spell out what counts.
- No calculators and no notes of any kind may be brought in.