Learning in Autonomous Systems

Syllabus

I will bring copies of the slides to the lectures so you do not need to print them. I will also put copies online.

Week	Subject	Slides	Handout
1	1 Introduction	slides	print
1	2 EC and the Simple GA	slides	print
1	3 Representations and Operators	slides	print
2	4 Issues with EC	slides	print
2	5 Evolving Neural Networks	slides	print
2	6 GBML framework	slides	print
3	7 Fitness Landscapes	slides	print
3	8 Introduction to RL (Ch.1)	slides	print
4	9 Evaluative Feedback (Ch.2)	slides	print
5	10 The RL problem (Ch.3)	slides	print
5-6	11 Dynamic Programming (Ch.4)	slides	print
6	12 Monte Carlo Methods (Ch.5)	slides	print
7	Class test	N/A	N/A
8	13 Temporal Difference (Ch.6)	slides	print
8	14 Eligibility Traces (Ch.7)	slides	print
9	15 Generalisation (Ch.8)	slides	N/A
9-10	16 Planning and Learning (Ch.9)	slides	print
10	17 Dimensions of RL (Ch.10)	slides	print

Assignments and test

What	Weight	Deadline
Class test	40%	Nov. 20
MC	25%	Nov. 30
Final	35%	Dec. 21

Textbooks

There is no text for the EC part of the unit. The handouts are based on different sources and usually indicate what they are. Some are based in part on chapter 2 of:

• C.R. Reeves and J.E. Rowe Genetic Algorithms - Principles and Perspectives: A Guide to GA Theory, 2003. [library] [google preview] [errata]

There are a number of books suitable for background reading, including:

• Chapters 1-3 of: A.E. Eiben and J.E. Smith. Introduction to Evolutionary Computing, 2003. [library] [google preview]
• The introductory chapters of: Melanie Mitchell. An introduction to genetic algorithms, 1998. [library] (Wider coverage than most.)
• The introductory chapters of: David E. Goldberg Genetic algorithms in search, optimization and machine learning, 1989. [library] (Old, but a gentle introduction.)

The textbook for the RL part of this unit is:

• R. Sutton and A. Barto. Reinforcement Learning. An Introduction. The MIT Press, 1998. ISBN: 0-262-19398-1. [library] [HTML and scanned versions] [errata]

Other Resources

To get a quick overview from a different perspective you may want to read the chapters on EC and RL in a more general book e.g.:

• T. Mitchell. Machine Learning. McGraw Hill, 1997. ISBN: 0-07-042807-7. [library]
• S. Russell and P. Norvig. Artificial Intelligence: A Modern Approach. Prentice Hall, 2003. ISBN: 0-13-080302-2. [library]

Or see this journal paper:

• L. Kaelbling, M. Littman and A. Moore. Reinforcement Learning: A Survey Journal of Artificial Intelligence Research, Vol. 4, 1996.

See also:

• Sutton's RL FAQ.
• RL Repository at UMass including a glossary of RL terms.
• An introductory tutorial on reinforcement learning I wrote. It mainly covers material from chapters 1-4 and 6 of Sutton and Barto very quickly.

Lecturer

• Tim Kovacs
• I don't have regular office hours. It's easiest to reach me by email or through the unit forum, or to talk to me after a lecture.
• My contact details.
• If you have any feedback on how to improve the unit I'll be happy to hear it.