Generate parts
A shared, side-conditioned hemisphere model and a dedicated CB model capture local anatomical distributions.
AnaDiffusion
Anatomically Compositional Latent Diffusion for Controllable 3D Brain MRI Generation
1Stanford University · 2Texas A&M University · 3University of Minnesota · 4University of Wisconsin-Madison · 5Johns Hopkins University · 6Yale University · 7Stony Brook University
Accepted at
NeurIPS 2026Generate anatomy as explicit parts. Assemble a scaffold. Refine the whole while preserving local control.
Video
AnaDiffusion in one minute.
Abstract
From monolithic volumes to compositional anatomy.
TL;DR
AnaDiffusion builds a 3D brain MRI from anatomical parts — left/right hemispheres and the cerebellum–brainstem — then assembles and globally refines them into one coherent volume, improving regional FID and enabling controllable part editing with no subject-specific segmentation maps at inference.
3D brain MRI generation has made significant advances for medical imaging, simulation, and controllable anatomical analysis. However, existing generative models typically synthesize 3D volumes monolithically, often overlooking regional anatomical structures and limiting local controllability.
- We introduce AnaDiffusion, a compositional latent diffusion framework for controllable 3D brain MRI generation. AnaDiffusion decomposes synthesis into anatomical parts and a whole-brain latent integration process, modeling heterogeneous brain structures more effectively than monolithic whole-volume generation.
- We propose a part-to-whole latent refinement that integrates independently generated anatomical parts into a coherent whole-brain volume. This supports controllable part replacement while keeping non-edited regions stable and part-to-whole anatomy consistent.
- Motivated by the approximate structural bilateral symmetry of the human brain, we design a hemisphere generator shared across both hemispheres by employing left-right canonicalization and side-indicator conditioning.
- We demonstrate that, relative to the compared baselines, AnaDiffusion improves the evaluated synthesis metrics on held-out subjects from a subject-disjoint ADNI test split. It achieves stronger regional distributional fidelity and segmentation-based anatomical alignment, while enabling localized part replacement with high target transfer and limited off-target drift, without requiring subject-specific dense segmentation maps at inference time.
Method
Assemble first. Refine globally.
AnaDiffusion trains frozen part models and a whole-brain model, composes generated regional assets in image space, and injects the re-encoded scaffold into the reverse diffusion trajectory.
The complete part-to-whole pipeline.
High-fidelity regional parts form a whole-brain scaffold. The scaffold is re-encoded into the whole-brain latent space and injected during denoising, enabling local anatomical detail and globally coherent synthesis.
Build a scaffold
Part assets are placed in fixed MNI152-aligned supports while uncertain boundaries and CSF remain open.
Inject the latent
The assembled image is encoded and inserted with 10 DDIM denoising steps remaining at inference.
Refine the whole
The whole-brain denoiser synthesizes missing context and repairs interfaces without discarding the injected anatomy.
Interactive
Explore a generated volume.
One AnaDiffusion sample in 3D. Drag to rotate or move through slices, scroll to zoom — or drop your own .nii.gz onto the viewer. Switch to Assemble parts to see the left and right hemispheres and the cerebellum–brainstem snap into their world-coordinate positions.
Generation results
Regional fidelity and whole-brain coherence.
All methods are evaluated against 291 real held-out ADNI scans. Confidence intervals use 5,000 subject-cluster bootstrap replicates.
Qualitative comparison across three anatomical planes.
Columns from left to right show Real, LDM, ControlNet LDM, Grid-based LDM, Seg. cLDM, MorphLDM, and AnaDiffusion (Ours). Monolithic baselines show localized artifacts and lose fine-scale structure, whereas part-to-whole refinement preserves local anatomy and boundary coherence.
Table 1
Generation and focused morphometric metrics on ADNI.
MedicalNet FID is reported as FID ×104. SynthSeg metrics report absolute Cohen's |d|. Mean [95% confidence interval]. Lower is better.
View the full results table
Best meanSecond-best meanWB: whole brain · CB: cerebellar-brainstem complex
| Method | MedicalNet FID | SynthSeg Cohen's |d| | ||||||
|---|---|---|---|---|---|---|---|---|
| WB | Left Hemi | Right Hemi | CB | Seam | Ventricles | Cerebellum | Brainstem | |
| VAE-GAN Larsen et al., 2016 | 103.54[75.1, 140.0] | 17.21[13.4, 23.1] | 15.35[11.1, 21.4] | 3.73[2.8, 5.5] | 16.79[12.8, 22.3] | – | – | – |
| HA-GAN Sun et al., 2022 | 310.33[262.9, 360.3] | 63.67[55.7, 72.0] | 25.75[22.0, 30.0] | 9.87[8.3, 11.6] | 20.09[17.1, 23.3] | – | – | – |
| LDM Pinaya et al., 2022 | 40.92[26.71, 58.86] | 7.10[4.26, 10.85] | 6.87[4.51, 9.57] | 1.68[1.17, 2.29] | 0.44[0.28, 0.63] | 0.184[0.007, 0.523] | 0.370[0.036, 0.747] | 0.222[0.011, 0.540] |
| Seg. cLDM Dorjsembe et al., 2024 | 59.47[32.72, 90.83] | 14.54[8.31, 21.92] | 9.22[4.79, 14.59] | 1.45[0.81, 2.30] | 0.49[0.30, 0.73] | 0.421[0.111, 0.725] | 0.207[0.010, 0.509] | 0.178[0.008, 0.489] |
| MorphLDM Wang et al., 2025 | 46.65[29.85, 70.74] | 8.56[5.90, 12.98] | 18.05[15.54, 21.27] | 2.23[1.88, 2.62] | 0.99[0.72, 1.35] | 0.131[0.005, 0.374] | 0.202[0.007, 0.510] | 0.357[0.048, 0.662] |
| ControlNet LDM | 44.24[22.72, 71.79] | 8.30[3.87, 14.15] | 7.58[4.06, 12.19] | 1.24[0.70, 2.00] | 0.36[0.20, 0.57] | 0.301[0.022, 0.665] | 1.366[1.026, 1.761] | 1.311[0.951, 1.667] |
| Grid-based LDM | 64.53[46.6, 86.0] | 9.72[7.7, 12.8] | 11.36[8.6, 15.4] | 3.02[2.2, 4.5] | 4.83[3.3, 7.3] | 0.427[0.082, 0.757] | 1.821[1.449, 2.265] | 1.235[0.888, 1.591] |
| Ours AnaDiffusion | 36.16[19.16, 59.61] | 6.21[3.14, 10.56] | 6.67[3.67, 10.72] | 1.06[0.64, 1.64] | 0.29[0.17, 0.46] | 0.144[0.005, 0.416] | 0.169[0.007, 0.482] | 0.185[0.008, 0.483] |
Table 2
Ablation study on ADNI.
Effects of compositional design, latent injection strength, injection timing, and assembly space (image-space assembly with re-encoding, as in AnaDiffusion, versus direct assembly of the part latents). MedicalNet FID ×104. Lower is better.
View the full ablation table
| Configuration | WB FID | Left Hemi FID | Right Hemi FID | CB FID | Seam FID |
|---|---|---|---|---|---|
| Compositional design · rinj = 10 | |||||
| separate hemisphere models | 38.72[16.83, 67.13] | 5.59[2.10, 10.47] | 8.94[4.29, 14.76] | 0.89[0.43, 1.54] | 0.19[0.10, 0.35] |
| Latent injection strength · rinj = 10 | |||||
| w/o latent injection (α = 0) | 52.27[30.69, 79.63] | 9.83[5.40, 15.53] | 8.52[5.23, 12.55] | 1.84[1.12, 2.75] | 0.55[0.35, 0.81] |
| full latent injection (α = 1) · Ours | 36.16[19.16, 59.61] | 6.21[3.14, 10.56] | 6.67[3.67, 10.72] | 1.06[0.64, 1.64] | 0.29[0.17, 0.46] |
| Injection timing | |||||
| rinj = 7 | 53.27[34.21, 76.75] | 9.31[5.88, 13.73] | 9.04[5.88, 12.76] | 1.75[1.22, 2.41] | 0.69[0.51, 0.91] |
| rinj = 15 | 51.66[28.31, 80.82] | 8.96[4.73, 14.63] | 9.52[5.39, 14.82] | 1.43[0.90, 2.10] | 0.31[0.16, 0.51] |
| Assembly space · rinj = 10 | |||||
| w/o re-encoding (latent assembly) | 149.05[104.63, 199.33] | 26.77[16.99, 37.68] | 23.93[15.95, 32.99] | 18.23[16.20, 20.36] | 0.63[0.45, 0.86] |
Controllable editing
Replace locally. Preserve globally.
A donor part can replace a selected generated region. AnaDiffusion reassembles, re-encodes, injects, and refines the modified asset set while preserving the surrounding anatomy.
Localized part editing.
A donor part asset is substituted into a generated recipient brain and refined into an edited whole-brain sample. Seg. cLDM and AnaDiffusion (Ours) are shown side by side for a cerebellum–brainstem edit and two hemisphere edits. Green boxes mark the replaced region.
Figure 4a · readable table
Quantitative editing locality.
Mean paired MS-SSIM [95% confidence interval] from 100 donor-recipient edits and 5,000 subject-cluster bootstrap replicates. Higher is better.
View the quantitative editing table
| Region | Method | Target transfer ↑ | Off-target preservation ↑ | Transfer gain ↑ | Locality contrast ↑ |
|---|---|---|---|---|---|
| Left hemisphere | Ours | 0.9427[0.9411, 0.9441] | 0.9371[0.9359, 0.9383] | 0.1627[0.1565, 0.1688] | 0.1207[0.1151, 0.1261] |
| Segm. cLDM | 0.7952[0.7887, 0.8018] | 0.9487[0.9478, 0.9496] | 0.0152[0.0138, 0.0168] | −0.0078[−0.0090, −0.0065] | |
| Right hemisphere | Ours | 0.9484[0.9471, 0.9495] | 0.9363[0.9350, 0.9376] | 0.1372[0.1313, 0.1428] | 0.1118[0.1064, 0.1172] |
| Segm. cLDM | 0.8343[0.8300, 0.8384] | 0.9447[0.9434, 0.9458] | 0.0231[0.0207, 0.0254] | 0.0056[0.0036, 0.0076] | |
| CB complex | Ours | 0.9618[0.9609, 0.9628] | 0.9496[0.9484, 0.9508] | 0.0752[0.0716, 0.0788] | 0.0313[0.0278, 0.0348] |
| Segm. cLDM | 0.9151[0.9122, 0.9183] | 0.9512[0.9500, 0.9523] | 0.0285[0.0269, 0.0302] | −0.0207[−0.0215, −0.0199] |
Generated samples
Additional generated samples.
Additional samples generated by AnaDiffusion. Choose a whole brain or a generated part to see its sagittal, coronal, and axial slices next to a 3D rendering. Drag to rotate or move through slices, and scroll to zoom.
Whole brain
Left hemisphere
Right hemisphere
Cerebellum–brainstem
Limitations
Scope and limitations.
Multi-stage complexity. The multi-stage design introduces additional training and inference complexity compared with a monolithic LDM.
Tissue-level calibration. The fixed, anatomy-driven factorization improves local part and seam behavior, but does not fully solve broader tissue-level calibration.
Predefined anatomy. The chosen decomposition focuses on predefined anatomical regions and may not capture other biologically meaningful organizations, such as tissue classes, functional networks, or multi-scale anatomical hierarchies.
MNI152 correspondence. AnaDiffusion assumes approximate correspondence with MNI152 and has not been validated for severe mass effect or displaced anatomical boundaries. Such pathology may impair registration and invalidate fixed regional placements. Supporting these cases would require lesion-aware or subject-adaptive localization.
Generalization across datasets. All models were trained and evaluated on ADNI with subject-disjoint splits, so generalization to other cohorts, scanners, and age ranges remains untested.
Future work
Where AnaDiffusion goes next.
Learned, adaptive decompositions. Finer subcortical or lobar units beyond the three coarse parts, compared directly with anatomy-free factorizations.
Assembly in latent space. A shared autoencoder or a learned cross-latent mapping would remove the decode and re-encode step and reduce the cost of the multi-stage pipeline.
Region-dependent refinement. Refinement schedules that spend more denoising steps on parts and seams with complex morphology.
Condition-specific synthesis. A selected part generator trained or fine-tuned with a clinical or morphological condition, such as age, diagnosis, or localized atrophy, and then reconciled with the rest of the brain through whole-brain refinement.
Segmentation and new cohorts. Training segmentation models on generated volumes, and evaluating the approach on other cohorts, scanners, pathologies, resolutions, and imaging contrasts, including younger populations.
BibTeX
@article{han2026anadiffusion,
title={AnaDiffusion: Anatomically Compositional Latent Diffusion for Controllable 3D Brain MRI Generation},
author={Han, Huiwen and Liu, Lulin and Liu, Bangya and Cai, Yuanhao and Chen, Nuo and Wang, Xiaoqing and Xie, Ziqian and You, Chenyu and Ji, Shuiwang and Zhi, Degui and others},
journal={arXiv preprint arXiv:2608.23014},
year={2026}
}
