IEEE TPAMI 2026

Urban Architect: Steerable 3D Driving Scene Generation
with Layout Prior

Fan Lu^1,2,3,*, Kwan-Yee Lin^4,*, Yan Xu⁴, Hongsheng Li⁵, Guang Chen^1,2,3, Changjun Jiang⁶

¹Physical Intelligence Lab, Tongji University, Shanghai, China ²Shanghai Innovation Institute, Shanghai, China

³Robotscale AI, Shanghai, China ⁴University of Michigan, Ann Arbor, United States

⁵Department of Electronic Engineering, The Chinese University of Hong Kong, Hong Kong, China ⁶School of Computer Science, Tongji University

^* Equal contribution

Paper arXiv Code

City-style transfer demonstration.

Abstract

Text-to-3D generation has achieved remarkable success via large-scale text-to-image diffusion models. Nevertheless, scaling this methodology to large-scale driving scenes is underexplored. Driving scenes, characterized by numerous elements, intricate arrangement relationships, and vast scale, present a formidable barrier to the interpretability of ambiguous textual descriptions for effective model optimization.

In this work, we surmount the limitations by introducing a compositional 3D layout representation into text-to-3D paradigm, serving as an additional prior. It comprises a set of semantic primitives with simple geometric structures and explicit arrangement relationships, complementing textual descriptions and enabling steerable generation. Specifically, we represent the scene via 2D Gaussian splatting and develop two innovative optimization strategies: (1) Layout-Guided Variational Score Distillation, which addresses model optimization inadequacies by conditioning the score distillation sampling process with geometric and semantic constraints of 3D layouts; (2) Semantic-Depth Score Distillation Sampling, which ensures alignment of semantics, geometry, and appearance by matching the rendered semantic and depth maps with 3D layouts.

Extensive experiments substantiate the capability of our framework to scale text-to-3D generation to large-scale driving scenes that cover over 1000 m driving distance for the first time. We also present various scene editing demonstrations, showing the powers of steerable urban scene generation.

Method Overview

Urban Architect pipeline with Sem2DGS, layout-guided variational score distillation, semantic-depth score distillation, layout-aware refinement, and scene editing

Overview of Urban Architect. We generate large-scale 3D driving scenes from 3D layouts and textual descriptions. Semantic 2D Gaussian Splatting (Sem2DGS) augments 2DGS with fixed semantic and instance labels. The scene is optimized using two complementary distillation strategies: Layout-Guided Variational Score Distillation (LG-VSD) incorporates the geometric and semantic constraints of the desired 3D layout into text-to-image distillation, while Semantic-Depth Score Distillation Sampling (SD-SDS) optimizes Gaussian geometry to align semantics, geometry, and appearance. Layout-Aware Refinement improves local details, with additional depth and surface-normal supervision. Instance-based scene subdivision supports large-scale generation, and fine-tuning enables both object manipulation and global style editing.

Demo Videos

Diverse 3D results

Scene 1

Scene 2

Scene 3

Scene 4

Comparison with other methods

EG3D

CC3D

Ours

Transfer to different weather and time

night time

rainy

in winter, with snow

Transfer to different styles

minecraft style

cartoon style

Vangogh paint

Transfer to different cities

in Kyoto

in Hawaii

in Venice

Object manipulation

Translation

Rotation

Scaling

Stuff editing

Large-scale generation

~600 × 400 m region · > 1000 m driving distance

Quantitative Results

Quantitative comparison on KITTI-360
Method	Text2Room	GANCraft	UrbanGIRAFFE	SceneDreamer	EG3D	CC3D	Ours
FID ↓	134.3	149.7	118.6	122.4	109.3	79.1	52.1
KID ↓	0.116	0.161	0.143	0.113	0.121	0.082	0.041

FID and KID are computed using 5,000 randomly sampled images. Lower is better for both metrics.

Camera-motion fidelity
Method	RPE translation (m) ↓	RPE rotation (°) ↓	ATE RMSE (m) ↓
CC3D	0.65	0.53	2.71
Ours w/o LG-VSD	0.22	0.31	1.94
Ours	0.17	0.22	1.01

Camera poses are estimated with Pi3X and aligned with the input trajectories using Umeyama alignment. RPE: relative pose error; ATE: absolute trajectory error. Lower is better.

Ablation studies
Method	w/o LG-VSD	w/o SD-SDS	w/o refinement	Ours
FID ↓	113.0	60.5	57.1	52.1
KID ↓	0.126	0.049	0.048	0.041

Each variant removes one component of the full method. Lower is better for both metrics.

Citation

If you find this work useful, please cite our paper:

@article{lu2026urbanarchitect,
  title={Urban Architect: Steerable 3D Driving Scene Generation with Layout Prior},
  author={Fan Lu and Kwan-Yee Lin and Yan Xu and Hongsheng Li and Guang Chen and Changjun Jiang},
  journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year={2026}
}