Yilin Wang

Yilin Wang

Ph.D. Candidate @ Purdue | Cooperative Perception, Physical AI, Autonomous Driving

AgentSim: An Agentic Test Engineer for Autonomous-Driving Safety Validation

Overview

AgentSim is an agentic world model for autonomous-driving safety validation. Instead of hand-written test scenarios, a multi-agent framework powered by LLMs and VLMs takes a real logged drive and mutates it into safety-critical corner cases, for example a lead car that brakes hard, a car that cuts in, a red-light runner, or a pedestrian crossing. Each mutation runs in closed loop: every other road user follows its recorded path until the mutation reaches it and then reacts, so the variants stay grounded in the original traffic. A video world model then renders each variant as a front-camera clip from the same first frame as the logged drive, so the clips differ only where the mutation changes the traffic.

Demo: one real drive, five variants

Waymo Open Motion Dataset v1 validation, scene d1988864008fc062. One box, 6 time-synchronized clips: the logged drive and 5 AgentSim variants, all from the same first frame.

  • Base The logged drive: the ego pulls away from a queue and crosses a signalized intersection. clip
  • Var 1 The lead car brakes hard at 3 s; the ego stops behind it. clip
  • Var 2 A car cuts in from the next lane and brakes; the ego brakes. clip
  • Var 3 A red-light runner crosses in front of the ego; the ego brakes. clip
  • Var 4 A pedestrian crosses the road ahead; the ego yields. clip
  • Var 5 The lead car brakes hard, then creeps on; the ego follows. clip

Two more drives, with the same treatment, are on the full demo page.

Milestones

  • Jun 2026: First draft release of AgentSim, utilizing LLMs and VLMs for safety-critical scenario generation.

Platform context: TeraSim

AgentSim is built within TeraSim, the city-scale simulation platform of Inchor (SaferDrive AI) that serves as the foundational infrastructure for agentic world models under the Physical AI framework. As a core developer of TeraSim, Yilin also:

  • refactored CNDE, its behavioral-layer foundation model built on MotionTransformer, from a dual-state design into a stateless prediction pipeline, implementing vehicle state synchronization, history tensor construction, and coordinate transformation for large-scale inference;
  • led the joint project with NVIDIA and the City of San Jose on agent-driven Smart City operation, designing the simulation pipeline (city-scale vehicle injection, dynamic routing, multi-phase demo orchestration) behind the Agentic Incident-to-Operation pipeline that went into active deployment in Mar 2026.

Tech Stack: Python, SUMO, PyTorch, CUDA, Docker, FastAPI

Driving scenes in the demo come from the Waymo Open Motion Dataset (non-commercial license). First frames generated with NVIDIA Cosmos-Transfer1; videos generated with NVIDIA OmniDreams.