AgentSim: Agentic World Model for AV/Physical AI Validation

AgentSim is an agentic world model for autonomous-driving safety validation. Starting from trajectories from real world with a sentence or two describing the risk to add, LLM/VLM agents create safety-critical variants of the drive: a lead car that brakes hard, a car that cuts in, a red-light runner, a pedestrian crossing. Each variant runs in closed loop, so every other road user follows its recorded path until the mutation reaches it and then reacts, and the variant stays grounded in the original traffic. A video world model then renders each variant as a 12-second front-camera clip from the same first frame as the logged drive, so the clips differ only where the mutation changes the traffic. The three drives below, from the Waymo Open Motion Dataset, show each logged drive and its five variants side by side.

How to read this page. Each case is one video box: the logged drive and its variants play side by side, time-synchronized, with a short label in each cell and the full descriptions below. Videos are muted and loop; they start playing when scrolled into view. Each legend entry links to that clip on its own, and Frame strip shows all clips of a case as stills at 0, 3, 6, 9 and 11.8 s.

1. Signalized intersection: the ego pulls away from a queue

Waymo Open Motion Dataset v1 validation, scene d1988864008fc062. One box, 6 time-synchronized clips: the logged drive and 5 AgentSim variants, all from the same first frame.

  • Base The logged drive: the ego pulls away from a queue and crosses a signalized intersection. clip
  • Var 1 The lead car brakes hard at 3 s; the ego stops behind it. clip
  • Var 2 A car cuts in from the next lane and brakes; the ego brakes. clip
  • Var 3 A red-light runner crosses in front of the ego; the ego brakes. clip
  • Var 4 A pedestrian crosses the road ahead; the ego yields. clip
  • Var 5 The lead car brakes hard, then creeps on; the ego follows. clip
Frame strip: all 6 clips at 0, 3, 6, 9 and 11.8 s Frame strip for Signalized intersection: the ego pulls away from a queue: one row per clip, five frames each

2. Busy signalized intersection: the ego accelerates out of stop-and-go traffic

Waymo Open Motion Dataset v1 validation, scene 5c14062cdca66b03. One box, 6 time-synchronized clips: the logged drive and 5 AgentSim variants, all from the same first frame.

  • Base The logged drive: the ego accelerates from 12 to 53 km/h through a busy signalized intersection. clip
  • Var 1 The lead car brakes hard right away and stays stopped; the ego stops behind it. clip
  • Var 2 A car cuts in from the next lane at 2 s and brakes hard; the ego brakes to a stop. clip
  • Var 3 A red-light runner crosses in front of the ego at 4 s; the ego brakes to 9 km/h. clip
  • Var 4 A pedestrian crosses at the crosswalk ahead; the ego stops for them. clip
  • Var 5 A slow car just ahead brakes hard at 3 s and stops; the ego stops behind it. clip
Frame strip: all 6 clips at 0, 3, 6, 9 and 11.8 s Frame strip for Busy signalized intersection: the ego accelerates out of stop-and-go traffic: one row per clip, five frames each

3. Freeway, no junction: the ego cruises in dense traffic at about 50 km/h

Waymo Open Motion Dataset v1 validation, scene 909244b878c6c203. One box, 6 time-synchronized clips: the logged drive and 5 AgentSim variants, all from the same first frame.

  • Base The logged drive: the ego cruises at 43 to 52 km/h in dense freeway traffic while trucks pass on the left. clip
  • Var 1 The car ahead brakes hard at 2 s and stops; the ego slows to 5 km/h behind it. clip
  • Var 2 A car from the left lane cuts in just ahead at 1 s and brakes; the ego slows to 3 km/h. clip
  • Var 3 A car from the right lane cuts in ahead at 1.5 s and brakes; the ego slows to 4 km/h. clip
  • Var 4 A large SUV merges in from the right and keeps a low speed; the ego slows to 19 km/h behind it. clip
  • Var 5 The car two ahead stops in the lane like a broken-down car; the lead car and the ego stop behind it. clip
Frame strip: all 6 clips at 0, 3, 6, 9 and 11.8 s Frame strip for Freeway, no junction: the ego cruises in dense traffic at about 50 km/h: one row per clip, five frames each

Waymo Open Dataset: An autonomous driving dataset. https://www.waymo.com/open, 2019–2025.