This paper addresses automatic interpretation of images of outdoor scenes for image indexing and retrieval from databases. The method allows instances of objects from a number of generic classes to be identified: vegetation, buildings, vehicles, roads, etc., thereby enabling image databases to be queried on scene content. This is achieved using a powerful set of image features which are used to train a neural network classifier. A large database of high-quality colour images of outdoor scenes developed at Bristol University provides a ground-truth interpretation of the images, which has enabled a detailed quantitative analysis of the vision system performance. The design of the feature set has been inspired by psychophysical models of vision. Texture is provided in the form of the magnitude response of a set of isotropic Gabor filters. Colour is represented by luminance, red/green and yellow/blue information. The shape of a region is represented by linear combinations of the principal modes of variation of an approximating polygon. The feature set also includes contextual information from an initial pixel classification. Progressive improvements in pixel classification are demonstrated by the addition of colour and texture information. The image interpretation technique is very successful and correctly labels over 90% of the image area in our database of test images, for a wide range of object classes. Such a high recognition accuracy demonstrates that it is possible to interrogate large image databases by content.