Pretrained Video Models as Differentiable Physics Simulators for Urban Wind Flows

Janne Perini^*1, Rafael Bischof^*1, Moab Arar², Ayça Duran¹, Michael A. Kraus³, Siddhartha Mishra¹, Bernd Bickel¹

¹ETH Zurich, ²Tel Aviv University, ³TU Darmstadt, * Equal contribution

European Conference on Computer Vision (ECCV 2026)

Abstract

Designing urban spaces that provide pedestrian wind comfort and safety requires time-resolved Computational Fluid Dynamics (CFD) simulations, but their current computational cost makes extensive design exploration impractical. We introduce WinDiNet (Wind Diffusion Network), a pretrained video diffusion model that is repurposed as a fast, differentiable surrogate for this task. Starting from LTX-Video, a 2B-parameter latent video transformer, we fine-tune on 10,000 2D incompressible CFDsimulations over procedurally generated building layouts. A systematic study of training regimes, conditioning mechanisms, and VAE adaptation strategies, including a physics-informed decoder loss, identifies a configuration that outperforms purpose-built neural PDE solvers. The resulting model generates full 112-frame rollouts in under a second. As the surrogate is end-to-end differentiable, it doubles as a physics simulator for gradient-based inverse optimization: given an urban footprint layout, we optimize building positions directly through backpropagation to improve wind safety as well as pedestrian wind comfort. Experiments on singleand multi-inlet layouts show that the optimizer discovers effective layouts even under challenging multi-objective configurations, with all improvements confirmed by ground-truth CFD simulations.

Pipeline

WinDiNet repurposes a 2-billion-parameter video diffusion transformer as a fast, differentiable surrogate for computational fluid dynamics (CFD) simulations of urban wind patterns. Fine-tuned on 10,000 CFD simulations across procedurally generated building layouts, it generates complete 112-frame wind field rollouts in under one second — over 2,000x faster than the ground truth CFD solver.

Physics-informed VAE decoder: Fine-tuned with incompressibility and wall boundary losses for physically consistent velocity field reconstruction
Scalar conditioning: Fourier-feature-encoded inlet speed and domain size replace text prompts, enabling precise physical parametrisation
Differentiable end-to-end: Enables gradient-based inverse design of urban building layouts for pedestrian wind comfort
State-of-the-art accuracy: Outperforms specialised neural operators (FNO, OFormer, Poseidon, U-Net) on vRMSE, spectral divergence, and Wasserstein distance

Figure: (a) Procedurally generated urban layouts are simulated with a 2D incompressible Euler solver to produce training data. (b) A latent diffusion model with a physics-informed VAE is trained to generate wind field sequences conditioned on building footprint, inlet speed u_in, and domain size L. (c) At inference, the model generates horizontal and vertical velocity fields (u, v) and enables gradient-based inverse optimization of building layouts.

VAE Adaptation

The pretrained LTX-Video VAE introduces reconstruction artifacts that degrade physical fidelity. We compare the base VAE against two decoder fine-tuning strategies: a pure MSE loss and a physics-informed loss that additionally penalizes divergence and momentum residuals. The physics-informed variant recovers fine-scale flow structures most faithfully.

Benchmarking

We compare WinDiNet against purpose-built neural PDE solvers (OFormer, RNO, FNO, AFNO, U-Net, Poseidon) as well as our own ablations: LoRA fine-tuning and text-based conditioning. The best variant, Dec. FT Physics (fine-tuned decoder with physics-informed losses, scalar conditioning), outperforms all baselines on pointwise metrics while generating all 112 frames in a single forward pass in under a second.

Inference

At inference, the user provides a building footprint, an inlet speed, and a domain size. WinDiNet generates a full 112-frame velocity rollout in under a second.

Inverse Optimization

Since WinDiNet is end-to-end differentiable, it can serve as a physics simulator for gradient-based inverse optimization of building layouts. Building positions are iteratively updated to minimize a comfort loss that penalizes dangerous gusts, speeds outside the desired comfort range, and stagnant zones in a downstream objective region.

(a) Movable buildings (red) and objective region (green rectangle) are defined. (b) A differentiable rasterizer produces a soft occupancy mask. (c) WinDiNet predicts the wind field. (d) Comfort loss penalizes speeds outside the target band. (e) Gradient descent updates building positions. (f) Optimized layout concentrates speeds within the comfort band.

Initial Wind Flow

Optimized Wind Flow

Wind speed distributions within the objective area before and after inverse optimization. The target band is from 1m/s to 5m/s.

Note that multi-inlet optimizations are possible. This allows the creation of settings in which winds from different directions, with varying inlet velocities, are treated differently during optimization. Refer to the paper for an example that sets the target band for western winds to 1m/s-3m/s, whereas winds from the north should fall into the 3m/s-5m/s band.

Further extensions include reparametrizing the buildings and allowing minor “morphing” of their shapes to improve wind flow. Examples are shown in the paper.

Acknowledgments

We thank Cornelia Kalender for her guidance on selecting appropriate evaluation metrics for the CFDcomparisons. We thank Kai Marti for providing the real-world building footprints used in the generalization experiments. We thank Spencer Folk for providing access to the codebase for the fluid solver used to generate the dataset. This work was supported by a grant from the Swiss AI Initiative, operated by the Swiss National AI Institute (co-founded by ETH Zurich and EPFL). We gratefully acknowledge the Swiss National Supercomputing Centre (CSCS) for providing compute resources on the Alps infrastructure.

Citation

				
					@inproceedings{perini2026pretrained,
  title={Pretrained Video Models as Differentiable Physics Simulators for Urban Wind Flows},
  author={Perini, Janne and Bischof, Rafael and Arar, Moab and Duran, Ay{\c{c}}a and Kraus, Michael A and Mishra, Siddhartha and Bickel, Bernd},
  booktitle={European Conference on Computer Vision},
  pages={209--226},
  year={2026},
  organization={Springer}
}

Links

Project Page