Diagram. On the training domain, tumor and normal feature clusters are well separated. Under distribution shift to an unseen hospital the clusters merge, causing prediction bias. Entropy minimization then leads to model collapse, while DSBR restores well-separated clusters.
Feature clusters merge under distribution shift, which induces prediction bias. Entropy minimization then collapses the model, while DSBR keeps the clusters separated.

Abstract

Entropy minimization (EM) is the dominant objective for test-time adaptation, yet its failure mode, model collapse, remains poorly understood. In this work, we show that distribution shifts can cause feature clusters corresponding to distinct classes in the model’s representation space to merge, while the decision boundary remains fixed. This induces a systematic skew in the predicted class distribution, referred to as prediction bias. Prediction bias refers to a shift in the predicted class distribution, with some classes overrepresented and others suppressed. We show that entropy minimization amplifies this prediction bias by tightening the existing clusters, reinforcing the incorrect groupings until all predictions collapse to a trivial solution. Next, to demonstrate the significance of prediction bias and mitigate it, we further propose Distribution Shift Bias Reduction (DSBR), a bias-correcting objective that specifically targets this failure mode by equalizing the contribution of each predicted class to the unsupervised entropy minimization loss. To study this failure mode, we design suitable adaptation settings using four medical-imaging datasets and additionally evaluate on ImageNet-C. We find that DSBR consistently stabilizes test-time adaptation, prevents model collapse, and matches or outperforms state-of-the-art methods. Moreover, DSBR operates solely at test-time.

Method

DSBR reweights the entropy loss so that every predicted class contributes equally. Samples in overrepresented clusters count for less, and samples in underrepresented clusters count for more, which works against cluster merging. It needs no sample filtering or domain-specific augmentations, and its only hyperparameter is the EMA decay \(\alpha\).

At each step, DSBR updates an exponential moving average of the predicted class frequencies, starting from \(\hat{P}_{0}^{(k)} = 1/K\):

$$\hat{P}_{t}^{(k)} = \alpha\,\hat{P}_{t-1}^{(k)} + (1-\alpha)\,\frac{B^{(k)}_{t}}{B}$$

Here \(B^{(k)}_{t}\) is the number of samples in the batch predicted as class \(k\). The DSBR loss over batch \(\mathcal{B}_{t}\) then weights each sample’s entropy loss \(l(X)\) by the inverse of its predicted class frequency:

$$\hat{\mathcal{L}}^{*}_{t} = \frac{1}{B}\sum_{X\in\mathcal{B}_{t}}\frac{l(X)}{K\cdot\hat{P}_{t}^{(\hat{y}(X))}}$$

Using DSBR with minimal edits

DSBR is a small change to entropy minimization (EM; Wang et al., ICLR 2021). It needs no extra forward pass, sample filter or augmentation, because it only changes how the per-sample entropies are averaged. Only the methods that change are shown; the rest of the class stays the same.

7 lines added, 3 changed
class EM(Algorithm):class DSBR(Algorithm): def __init__(self, input_shape, num_classes, num_domains, hparams, algorithm): super().__init__(input_shape, num_classes, num_domains, hparams) self.base_optimizer = hparams.get('base_optimizer', 'SGD') self.base_optimizer = hparams.get('base_optimizer', 'Adam') self.use_sam = hparams.get('use_sam', False) # Control SAM usage via hparams self.use_sam = False self.use_come = hparams.get('use_come', False) # Control COME usage via hparams if self.use_come: self.entropy_fn = entropy_of_opinion else: self.entropy_fn = softmax_entropy self.num_classes = num_classes self.model, self.optimizer = self.configure_model_optimizer(algorithm, lr=hparams['lr']) self.steps = hparams.get('num_update_steps', 1) assert self.steps > 0, "requires >= 1 step(s) to forward and update" self.episodic = False # note: if the model is never reset, like for continual adaptation, # then skipping the state copy would save memory self.model_state, self.optimizer_state = \ copy_model_and_optimizer(self.model, self.optimizer) self.cluster_distribution = torch.ones(num_classes) / num_classes self.ema_alpha = hparams.get('ema_alpha', 0.9) @torch.enable_grad() # ensure grads in possible no grad context for testing def forward_and_adapt(self, x, model, optimizer): """Forward and adapt model on batch of data. Measure entropy of the model prediction, take gradients, and update params. """ # forward optimizer.zero_grad() outputs = model(x) # adapt entropys = self.entropy_fn(outputs) cluster_ids = outputs.argmax(dim=1) local_cluster_distribution = torch.bincount(cluster_ids, minlength=self.num_classes) / len(cluster_ids) self.cluster_distribution = self.ema_alpha * self.cluster_distribution.to(entropys.device) + (1 - self.ema_alpha) * local_cluster_distribution entropys = entropys / (self.num_classes * self.cluster_distribution[cluster_ids]) loss = entropys.mean(0) loss.backward() optimizer.step() return outputs def reset(self): if self.model_state is None or self.optimizer_state is None: raise Exception("cannot reset without saved model/optimizer state") load_model_and_optimizer(self.model, self.optimizer, self.model_state, self.optimizer_state) self.cluster_distribution = torch.ones_like(self.cluster_distribution) / len(self.cluster_distribution)

Results

Collapse on ImageNet-C

Scatter plot of predicted class against online batch for Tent. Predictions start with an initial bias and concentrate on a single class by the end of the episode, labelled model collapse. Scatter plot of predicted class against online batch for DSBR. An initial bias fades by about batch 400 and predictions stay spread across all 1000 classes.

Predicted classes over one ImageNet-C Fog episode (severity 5). Both methods start with the same bias. Tent amplifies it until the model collapses, while DSBR dampens it until it vanishes by about batch 400.

Medical imaging benchmarks

Balanced accuracy (%), averaged over all source–target configurations. Δ Avg. is the average improvement over the non-adapted model. The best result in each column is in bold.

MethodHistopantumMammoBenchWILDSCamelyonGliomaMRIΔ Avg.
ResNet
No adaptation86.6 ± 0.059.4 ± 0.087.4 ± 0.042.9 ± 0.00.0 ± 0.0
Tent88.5 ± 0.158.8 ± 0.187.9 ± 0.044.3 ± 1.3+0.8 ± 0.4
SAR86.6 ± 0.059.4 ± 0.087.4 ± 0.044.2 ± 1.4+0.3 ± 0.4
ROID87.2 ± 1.756.6 ± 0.594.8 ± 0.644.4 ± 1.3*+1.7 ± 1.0
DeYO76.1 ± 2.860.9 ± 0.292.7 ± 0.5n/a†−0.9 ± 0.9
DSBR (ours)89.9 ± 0.462.7 ± 0.596.5 ± 0.144.0 ± 1.1+4.2 ± 0.5
ViT
No adaptation87.1 ± 0.054.4 ± 0.087.0 ± 0.037.2 ± 0.00.0 ± 0.0
Tent87.4 ± 0.054.4 ± 0.087.4 ± 0.137.4 ± 0.2+0.2 ± 0.1
SAR87.1 ± 0.054.4 ± 0.087.0 ± 0.037.2 ± 0.00.0 ± 0.0
ROID88.0 ± 0.354.5 ± 0.894.3 ± 0.238.2 ± 5.0*+2.3 ± 1.6
DeYO87.9 ± 0.255.2 ± 0.193.5 ± 0.3n/a†+2.0 ± 0.2
DSBR (ours)88.8 ± 0.455.2 ± 0.595.6 ± 0.336.5 ± 0.9+2.6 ± 0.5

* Consistency component disabled because it relies on 2D augmentations. † Not available because the method relies on 2D augmentations.

ImageNet-C

Accuracy (%) at severity 5 in the default setting and three wild settings. The best result in each column is in bold.

MethodDefaultLabel shiftsMixed corruptionsBatch size 1Δ Avg.
ResNet50-GN
No adaptation31.4 ± 0.031.4 ± 0.031.4 ± 0.031.4 ± 0.00.0 ± 0.0
Tent32.2 ± 0.330.3 ± 0.732.1 ± 1.730.8 ± 0.30.0 ± 0.7
SAR36.7 ± 3.037.8 ± 0.538.5 ± 0.135.8 ± 0.3+5.8 ± 1.0
ROID46.8 ± 0.12.7 ± 0.137.6 ± 0.10.1 ± 0.0−9.6 ± 0.1
DeYO44.1 ± 4.944.1 ± 5.639.4 ± 1.143.9 ± 4.3+11.4 ± 4.0
DSBR (α = 0.99)49.7 ± 0.146.8 ± 0.243.0 ± 0.241.6 ± 0.1+13.9 ± 0.1
DSBR (α = 0.9)48.8 ± 0.138.2 ± 0.341.4 ± 0.238.3 ± 0.1+10.2 ± 0.2
ViT-B/16
No adaptation39.8 ± 0.039.8 ± 0.039.8 ± 0.039.8 ± 0.00.0 ± 0.0
Tent51.9 ± 0.553.0 ± 0.333.1 ± 4.551.7 ± 0.3+7.6 ± 1.4
SAR54.2 ± 0.356.2 ± 0.153.7 ± 0.256.2 ± 0.3+15.3 ± 0.2
ROID64.6 ± 0.186.4 ± 4.957.2 ± 0.10.1 ± 0.0+12.2 ± 1.3
DeYO61.8 ± 2.062.4 ± 1.153.6 ± 0.861.8 ± 0.9+20.1 ± 1.2
DSBR (α = 0.99)64.9 ± 0.356.2 ± 0.260.9 ± 0.758.2 ± 0.1+20.2 ± 0.3
DSBR (α = 0.9)64.4 ± 0.242.2 ± 0.259.5 ± 1.055.9 ± 0.4+15.7 ± 0.5

Citation

@inproceedings{nielen2026entropy,
  title         = {Entropy Minimization without Model Collapse: Mitigating Prediction Bias in Medical Imaging},
  author        = {Nielen, Tim and Ambekar, Sameer and Kiechle, Johannes and Lang, Daniel M. and Schnabel, Julia A.},
  booktitle     = {Advances in Neural Information Processing Systems},
  year          = {2026},
  eprint        = {2606.02339},
  archivePrefix = {arXiv},
  primaryClass  = {cs.LG}
}

Acknowledgments

This work is supported by DAAD programme Konrad Zuse Schools of Excellence in Artificial Intelligence, Munich Center for Machine Learning, both sponsored by the Federal Ministry of Research, Technology, Bavaria California Technology Center (BaCaTeC) under BaCaTeC-Förderprojekt Nr. 20 [2026-1], Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – 515279324 / SPP 2177, and the German Federal Ministry of Research, Technology and Space (DECIPHER-M, 01KD2420G). DML and JAS received funding from HELMHOLTZ IMAGING, a platform of the Helmholtz Information and Data Science Incubator.