We present Tyche, a stochastic strategy for in-context medical image segmentation, to both generalize to new tasks and capture uncertainty

Tyche produces multiple candidate segmentations for images from unseen biomedical datasets without retraining fine-tuning

Existing learning-based solutions to medical image segmentation have two important shortcomings.

First, for each new segmentation task, usually a new model has to be trained or fine-tuned. This requires extensive resources and machine-learning expertise, and is therefore often infeasible for medical researchers and clinicians. Second, most existing segmentation methods produce a single deterministic segmentation mask for a given image. However, in practice, there is often considerable uncertainty about what constitutes the correct segmentation, and different expert annotators will often segment the same image differently.

We tackle both of these problems with Tyche, a model that uses a context set to generate stochastic predictions for previously unseen tasks without the need to retrain. Tyche differs from other in-context segmentation methods in two important ways.

(1) We introduce a novel convolution block architecture that enables interactions among predictions. (2) We introduce in-context test-time augmentation, a new mechanism to provide prediction stochasticity.

When combined with appropriate model design and loss functions, Tyche can predict a set of plausible diverse segmentation candidates for new or unseen medical images and segmentation tasks without the need to retrain.

Tyche captures the inherent ambiguity in medical image segmentation, while generalizing to new tasks without the need to retrain.

Tyche uses an input context set of image-segmentation pairs, which define the segmentation task, to know how to segment an input target image. Crucially, it captures uncertainty by outputting a flexible number of plausible predictions.

Tyche accurately captures ambiguity in target images by providing plausible, diverse solutions even for new, previously unseen segmentation domains and targets.

Multi-annotator data examples: