Digital Image Processing Lab

Department of Radiology, University of Michigan

MIAMI Fuse: Mutual Information for Automatic Multimodality Image Fusion

An Application of MI-based Automated Registration:
Fusion of Three Datasets

This 3D rendering of a patient was performed using three coregistered data sets: a T1-weighted MRI, a FDG PET, and a Thallium-201 SPECT. The MRI was performed after the patient had received radiation therapy treatment for a metastatic tumor. A post-therapy MRI study raised concern that the cancer may be regrowing. A differential diagnosis included the possibility that the suspicious area was simply an enlarged necrotic regon following therapy. An FDG PET was performed to determine the metabolic rate of the suspected region, and a Thallium-201 SPECT was performed to demonstrate the state of the blood-brain barrier. Side-by-side clinical comparison of the three scans left considerable doubt regarding the FDG activity in the suspect lesion due to uncertainty in location. Increased FDG uptake above background would indicate a metabolically active lesion associated with cancer. Only after automatic registration was performed was the clinician certain that the lesion was simply necrotic.

The registrations were performed using our "MIAMI Fuse" (Mutual Information for Automatic Multimodality Image Fusion) software as described fully in Meyer, et al. and illustrated by a set of QuickTime movies.

3D head image Rendering of the three fused datasets (256K color, 512x512 gif)

The rendering of the fused data sets as shown here was performed using AVS's tracer-tracker modules which implement the ray-casting method of volume rendering. The green hue is driven by the MRI signal amplitude, the red hue is driven by the coregistered PET study, and the blue hue is driven by the coregistered SPECT study. Thus white matter tracts are primarily green, since white matter has very low metabolic actiavity and the blood brain barrier is intact. Functioning gray matter is primarily red due to increased metabolic activity and associated FDG uptake, and lower MRI signal amplitude. The cast rays (from the viewer's location) stop when the integrated attenuation reaches a threshold. This rendering was obtained by using the MRI data for the local coefficient of attenuation map and setting the global multiplicative constant to its maximum, such that the cast rays stop near the surface of the data.

(AVS is a trademark of Advanced Visual Systems, Inc., Waltham, MA.)