Historically, inductive machine learning has focused on non-incremental learning tasks, i.e., where the training set can be constructed a priori and learning stops once this set has been duly processed. There are, however, a number of areas, such as agents, where learning tasks are incremental. This paper defines the notion of incrementality for learning tasks and algorithms. It then provides some motivation for incremental learning and argues in favour of the design of incremental learning algorithms for solving incremental learning tasks. A number of issues raised by such systems are outlined and the incremental learner ILA is used for illustration.