Optimization Course SS 14 U Stuttgart


Optimization is one of the most fundamental tools of modern sciences. Many phenomena -- be it in computer science, artificial intelligence, logistics, physics, finance, or even psychology and neuroscience -- are typically described in terms of optimality principles. The reason is that it is often easier to describe or design an optimality principle or cost function rather than the system itself. However, if systems are described in terms of optimality principles, the computational problem of optimization becomes central to all these sciences.


This lecture aims give an overview and introdution to various approaches to optimization together with practical experience in the exercises. The focus will be on continuous optimization problems and we will cover methods ranging from standard convex optimization and gradient methods to non-linear black box problems (evolutionary algorithms) and optimal global optimization. Students will learn to identify, mathematically formalize, and derive algorithmic solutions to optimization problems as they occur in nearly all disciplines. A preliminary list of topics is:

  • gradient methods, log-barrier, conjugate gradients, Rprop
  • constraints, KKT, primal/dual
  • Linear Programming, simplex algorithm
  • (sequential) Quadratic Programming
  • Markov Chain Monte Carlo methods
  • 2nd order methods, (Gauss-)Newton, (L)BFGS
  • blackbox stochastic search, including a discussion of evolutionary algorithms
Students should bring basic knowledge of linear algebra and analysis.
Organization
  • This is the central website of the lecture. Link to slides, exercise sheets, announcements, etc will all be posted here.
  • More information to come
Schedule, slides & exercises
date topics slides exercises
(due on 'date'+1)
15.4. Introduction 01-introduction --
22.4. Unconstrained Optimization 02-unconstrainedOpt e01-introduction
29.4. Unconstrained Optimization (cont.) [Nathan did the lecture. Sorry I was ill.] e02-unconstrainedOpt
6.5. Constrained Optimization 03-constrainedOpt e03-newtonMethods
13.5. Constrained Optimization & Convex Problems 04-convexProblems e04-constraints
20.5. Constrained Optimization & Convex Problems (cont.) questions time (optional to attend)
27.5. Blackbox Optimization 05-blackBoxOpt e05-constrainedOpt
3.6. Blackbox Optimization (cont.) e06-convexOpt
10.6. [Pfingstferien]
17.6. Bayesian Optimization 06-globalBayesianOptimization e07-stochasticSearch
24.6. Bayesian Optimization (cont.) [no exercise]
1.7. Consider this to prepare for exam: 14-Optimization-script e08-globalOptim
../data/gp01pred.m ../data/test.m
Literature