Optimization Course SS 15 U Stuttgart

Please subscribe to this mailing list.

Optimization is one of the most fundamental tools of modern sciences. Many phenomena -- be it in computer science, artificial intelligence, logistics, physics, finance, or even psychology and neuroscience -- are typically described in terms of optimality principles. The reason is that it is often easier to describe or design an optimality principle or cost function rather than the system itself. However, if systems are described in terms of optimality principles, the computational problem of optimization becomes central to all these sciences.

This lecture aims give an overview and introdution to various approaches to optimization together with practical experience in the exercises. The focus will be on continuous optimization problems and we will cover methods ranging from standard convex optimization and gradient methods to non-linear black box problems (evolutionary algorithms) and optimal global optimization. Students will learn to identify, mathematically formalize, and derive algorithmic solutions to optimization problems as they occur in nearly all disciplines. A preliminary list of topics is:

  • gradient methods, log-barrier, conjugate gradients, Rprop
  • constraints, KKT, primal/dual
  • Linear Programming, simplex algorithm
  • (sequential) Quadratic Programming
  • Markov Chain Monte Carlo methods
  • 2nd order methods, (Gauss-)Newton, (L)BFGS
  • blackbox stochastic search, including a discussion of evolutionary algorithms
Students should bring basic knowledge of linear algebra and analysis.
  • This is the central website of the lecture. Link to slides, exercise sheets, announcements, etc will all be posted here.
  • See the 01-introduction slides for further information.
Schedule, slides & exercises
date topics slides exercises
(due on 'date'+1)
14.4. Introduction 01-introduction
21.4. Unconstrained Opt.(Pt.1) 02-unconstrainedOpt e01-introduction
28.4. Unconstrained Opt.(Pt.2) e02-unconstrainedOpt
06.5. Constrained Opt. 03-constrainedOpt e03-newtonMethods
13.5. e04-constraints
20.5. e05-lagrange
02.6. e06-primaldual
09.6. Convex Opt. 04-convexOpt e07-convexOpt
16.6. Global Bayesian Optimization 05-globalBayesianOptimization e08-ILPrelaxation
23.6. Philipp Hennig's slides on Entropy Search CANCELED
30.6. Blackbox Optimization 06-blackBoxOpt e09-globalOptim
07.7. e10-blackBoxOpt
14.7. SCRIPT 15-Optimization-script e11-blackBoxOpt_2