Learning in Autonomous Systems

Syllabus

Week	Subject	Notes
1	1 Introduction	This week only the lecture will start late at 11:30 (October 11, 2011)
1	2 EC and the Simple GA
2	3 Representations and Operators
2	4 Fitness Landscapes	Additional notes on hillclimbing Some background on neighbourhoods, local optima, and hillclimbing start on page 40 here
3	5 Issues with EC
none	6 Evolving Neural Networks	We skipped this and it will not be on the test but I'm putting it here in case anyone wants to look at it
3	7 A Framework for Genetics-based Machine Learning	Smarts | Evolved Virtual Creatures | Creatures' Developmental process | Game of life
4	8 Introduction to RL (Ch.1 Sutton & Barto)
4-5	9 Evaluative Feedbback (Ch. 2)
5	10 The RL problem (Ch. 3)
6	11 Dynamic Programming (Ch. 4)	| AIBO playing bandit | TED talk: 7 ways games reward the brain
7	12 Monte Carlo Methods (Ch. 5)
8	13 Temporal Difference Methods (Ch. 6)	Presentations by Long, Barnabas and James on their CVRP solutions.
9	14 Planning and Learning (Ch. 9) and Dimensions of RL (Ch. 10)
10	No lecture	Class test moved to January 23 2012, 12-2pm in 1.68 Queens Building.

Assignments and test

What	Weight	Deadline
CVRP | notes	40%	Friday 11/11/11
Reinforcement Learning | marks	30%	Thursday 15/12/11 Friday 16/12/11
Class test	30%	Monday January 23 2012 between 12 and 2pm in 1.68QB

The CVRP assignment is worth more than the RL assignment because CVRP is probably harder, there's more scope for extra reading and extra work, and you should have more time to work on it. There are usually a lot of assignments due in week 10, and this unit has both a class test and the RL assignment due then, which together are worth 60% of the unit.

Textbooks

There is no text for the EC part of the unit. The handouts are based on different sources and usually indicate what they are. Some are based in part on chapter 2 of:

• C.R. Reeves and J.E. Rowe Genetic Algorithms - Principles and Perspectives: A Guide to GA Theory, 2003. [library] [google preview] [errata]

There are a number of books suitable for background reading, including:

• Chapters 1-3 of: A.E. Eiben and J.E. Smith. Introduction to Evolutionary Computing, 2003. [library] [google preview]
• The introductory chapters of: Melanie Mitchell. An introduction to genetic algorithms, 1998. [library] (Wider coverage than most.)
• The introductory chapters of: David E. Goldberg Genetic algorithms in search, optimization and machine learning, 1989. [library] (Old, but a gentle introduction.)

The textbook for the RL part of this unit is:

• R. Sutton and A. Barto. Reinforcement Learning. An Introduction. The MIT Press, 1998. ISBN: 0-262-19398-1. [library] [HTML and scanned versions] [errata]

Other Resources

To get a quick overview from a different perspective you may want to read the chapters on EC and RL in a more general book e.g.:

• T. Mitchell. Machine Learning. McGraw Hill, 1997. ISBN: 0-07-042807-7. [library]
• S. Russell and P. Norvig. Artificial Intelligence: A Modern Approach. Prentice Hall, 2003. ISBN: 0-13-080302-2. [library]

Or see this journal paper:

• L. Kaelbling, M. Littman and A. Moore. Reinforcement Learning: A Survey Journal of Artificial Intelligence Research, Vol. 4, 1996.

See also:

• Sutton's RL FAQ.
• RL Repository at UMass including a glossary of RL terms.
• An introductory tutorial on reinforcement learning I wrote. It mainly covers material from chapters 1-4 and 6 of Sutton and Barto very quickly.

Lecturer

• Tim Kovacs
• I don't have regular office hours. It's easiest to reach me by email or through the unit forum, or to talk to me after a lecture.
• My contact details.
• If you have any feedback on how to improve the unit I'll be happy to hear it.