A new fully automatic object tracking and segmentation framework is proposed. The framework consists of a motion based bootstrapping algorithm concurrent to a shape based active contour. The shape based active contour uses a finite shape memory that is automatically and continuously built from both the bootstrap process and the active contour object tracker. A scheme is proposed to ensure the finite shape memory is continuously updated but forgets unnecessary information. Two new ways of automatically extracting shape information from image data given a region of interest are also proposed. Results demonstrate that the bootstrapping stage provides important motion and shape information to the object tracker. This information is found to be essential for good (fully automatic) initialization of the active contour. Further results also demonstrate convergence properties of the content of the finite shape memory and similar object tracking performance in comparison to an object tracker with an unlimited shape memory. Tests with an active contour using a fixed shape prior also demonstrate superior performance for the proposed bootstrapped finite shape memory framework and similar performance when compared with a recently proposed active contour that uses an alternative on-line learning model.