University of Bristol > Visual Information Laboratory > Adeline Paiement > Integrated Registration, Segmentation and Interpolation...


Integrated Registration, Segmentation, and Interpolation of Sparse and Misaligned 3D/4D Data


Adeline Paiement, Majid Mirmehdi

Object modelling from 3D and 4D sparse and misaligned data has important applications in medical imaging, where visualising and characterising the shape of, e.g., an organ or tumor, is often needed to establish a diagnosis or to plan surgery. Two common issues in medical imaging are the presence of large gaps between the 2D image slices that make up a dataset, and misalignments between these slices, due to patient's movements between their respective acquisitions. These gaps and misalignments make the automatic analysis of the data particularly challenging. In particular, they require interpolation and registration in order to recover a complete shape of the object. Our work focuses on the integrated registration, segmentation, and interpolation of such sparse and misaligned data. We developed a framework which is flexible enough to model objects of various shapes, from data having arbitrary spatial configuration and from a variety of imaging modalities (e.g. CT-scan, MRI).

The source code for ISISD and IReSISD can be downloaded at the bottom of this page and here. Previous version are available in section Downloads at the bottom of the page.

ISISD: Integrated Segmentation and Interpolation of Sparse Data

Our modelling method is based on a new level set framework that we developed in order to handle sparse data. In this new framework, the level set implicit function is interpolated by Radial Basis Functions (RBFs), and its interface can propagate in a sparse volume, using data information where available, and RBF based interpolation of its speeds in the gaps. This new level set framework benefits from a better robustness to noise in the images, and can segment sparse volumes by integrating the shape of the objects in the gaps. Several modalities may be processed simultaneously (e.g. T1- and T2-weighted MRIs) thanks to the method interpolating the level set contour rather than the image intensities.

Robustness to noise

Robustness to Noise: Initialisation Robustness to Noise: Chan&Vese Algorithm Robustness to Noise: Narrow-Band Level Set With Piecewise Constant Model Robustness to Noise: ISISD With Piecewise Constant Model Robustness to Noise with gaps: Initialisation Robustness to Noise with gaps: ISISD With Piecewise Constant Model
Initialisation Chan&Vese algorithm Narrow-band level set with piecewise constant model ISISD with piecewise constant model Initialisation ISISD with piecewise constant model

Example 1: Modelling of the ventricles of a neonatal brain from T1- and T2- weighted MRIs simultaneously

Slices T1 T2 Modelled Shape
Initial slices (top left quadrant removed for visualisation) Central T1-weighted slice (red: segmentation) Central T2-weighted slice (red: segmentation) Modelled shape

Example 2: Modelling of the cavity of the left ventricle of a heart from short- and long-axis MRIs simultaneously

Slices Central SA Slice LA Slice Modelled Shape
Initial slices (top left quadrant removed for visualisation) Central short-axis slice (red: segmentation) Long-axis slice (red: segmentation) Modelled shape

ISISD in action

Click on the images to start and stop the animations

ISISD in Action Example 1 ISISD in Action Example 2

Publication and source code

This new RBF interpolated level set framework is described in:

Adeline Paiement, Majid Mirmehdi, Xianghua Xie, Mark Hamilton, Integrated Segmentation and Interpolation of Sparse Data. IEEE Transactions on Image Processing, Vol. 23, Issue 1, pp. 110-125, 2014. [External information]

Its code is available at the bottom of this page.

IReSISD: Integrated Registration, Segmentation, and Interpolation of Sparse Data

A new registration method, also based on level set, has been developed and integrated to the previous RBF interpolated level set framework. Thus, the new framework can correct misalignments in the data, at the same time as it segments and interpolates it. The integration of all three processes of registration, segmentation and interpolation into a same framework allows them to benefit from each others. Notably registration exploits the shape information provided by the segmentation stage, in order to be robust to local minima and to limited intersections between the images of a dataset.

Example 1: Modelling of the brain's ventricles from 3 artificially misaligned MRIs.

Example Misalignment 1 view front Example Misalignment 1 view side Example Registration 1 view front Example Registration 1 view side Example Modelling 1
Misaligned slices: view from the front Misaligned slices: view from the side Registered slices: view from the front Registered slices: view from the side Modelled object

Example 2: Modelling of the left ventricle cavity of the heart from artificially misaligned short- and long-axis MRIs.

Example Misalignment 2 Example Registration 2 Example Modelling 2
Misaligned slices Registered slices Modelled object

IReSISD in action

Click on the images to start and stop the animations

IReSISD in Action Example 2 IReSISD in Action Example 1

Comparative results

The registration method of IReSISD has been compared against the registration method of [1] (denoted as "SR2" in the publication - see section Publication and source code below), implemented with Squared Intensity Difference (SSD), Normalized Cross-Correlation (NCC), and Normalized Mutual Information (NMI). Details on the experimental setup as well as analysis of the results may be found in the publication. The tables in the article are reported in the graphs below:

Stack-wise registration error - Test 1
Stack-wise registration error - Test 2
Slice-wise registration error - Test 1 Slice-wise registration error - Test 2
Slice-wise registration error - Test 3
Slice-wise registration error - Test 4
Jaccard measures

Publication and source code

This new integrated registration, interpolation, and segmentation framework will soon be described in:

Adeline Paiement, Majid Mirmehdi, Xianghua Xie, Mark Hamilton, Registration and Modeling From Spaced and Misaligned Image Volumes. IEEE Transactions on Image Processing, Vol. 25, Issue 9, pp. 4379-4393, 2016. [External information]

Its code is available at the bottom of this page.

Application to 4D Modelling of the Cavity of the Heart's Left Ventricle from cine MRIs

Click on the images to start and stop the animations

Example of LV Cavity Modelling 1 Example of LV Cavity Modelling 2 Example of LV Cavity Modelling 3 Example of LV Cavity Modelling 4

Application to 3D Modelling from RGB-D data

Example 1: Artificial data made with Blender: red monkey head

Example RGB-D 1: RGB Image Example RGB-D 1: Depth Image Example RGB-D 1: Initial Point Clouds Example RGB-D 1: Aligned Point Clouds Example RGB-D 1: Modelled Shape At Low Res Example RGB-D 1: Modelled Shape At High Res
RGB image Depth image Initial point clouds Aligned point clouds Modelled shape at low resolution Modelled shape at high resolution

Example 2: Kinect data: blue plush toy

Example RGB-D 1: RGB Image Example RGB-D 1: Depth Image Example RGB-D 1: Initial Point Clouds Example RGB-D 1: Aligned Point Clouds Example RGB-D 1: Modelled Shape At Low Res Example RGB-D 1: Modelled Shape At High Res
RGB image Depth image Initial point clouds Aligned point clouds Modelled shape at low resolution Modelled shape at high resolution

Downloads

The latest version of the source code for ISISD and IReSISD can be downloaded here (version 1.5).

Previous versions:

  • version 1.4
  • version 1.3
  • version 1.2
  • version 1.1
  • version 1.0
  • IReSISD was my PhD project. My PhD thesis can be downloaded here.

    References

    [1] J. Lotjonen, M. Pollari, S. Kivisto, and K. Lauerma, Correction of Movement Artifacts from 4-D Cardiac Short- and Long-Axis MR Data, in MICCAI, 2004, pp. 405-412.


    updated: 02/05/2016