Natural language to runnable traffic simulations

Why is this hard?

Teach traffic-scene context from real road data, then split generation by responsibility

Real Seoul traffic data becomes supervised prompt–parameter pairs

flowchart LR
    A["Speed detectors\n31-day hourly avg"] --> D["Match by\nroad name & link ID"]
    B["Volume detectors\nhourly count"] --> D
    C["Node-link SHP\nspeed limit · geometry"] --> D
    D --> E["Group by 7\ntime periods"]
    E --> F["Reverse-estimate\nsigma, tau\n(Greenshields)"]
    F --> G["Generate 5\nprompt variants"]
    G --> H["train JSONL\n2,205 (90%)"]
    G --> I["val JSONL\n245 (10%)"]
          

The strongest gain appears in the structured extraction layer

Prompt to runnable simulation, then back to reusable evidence

The workflow has two connected halves: an online generation path that turns requests into SUMO runs, and a review path that classifies human edits so the system can improve without contaminating its own data.

flowchart TD
    A[User Natural-Language Request] --> B[Fine-Tuned Parameter Extraction]
    B --> C[Structured Scenario Parameters]
    C --> D{Usable real location?}
    D -->|Yes| E[OSM-Based Network Retrieval]
    D -->|No / Failed| F[Geometry LLM — XML Generation]
    E --> G[SUMO Network Build]
    F --> G
    C --> H[Demand / Route / Config Generation]
    G --> I[Runnable SUMO Artifacts]
    H --> I
    I --> J[Scenario Execution]
    J --> K[Validation and Statistics]
    K --> L[User Review]
    L --> M{Intent}
    M -->|Correction| N[Trainable Signal]
    M -->|Tuning| O[Analysis Only]
    N --> P[Correction Export]
    P --> Q[Future Fine-Tuning Data]
            

flowchart LR
    subgraph Frontend
        UI["Web UI\nindex.html"]
        AdminUI["Admin Dashboard"]
        AboutUI["About Page"]
    end

    subgraph Backend["server.py"]
        SSE["SSE Streaming"]
        API["REST API"]
    end

    subgraph LLM["LLM Layer"]
        FT["Fine-tuned Model\ngpt-4.1-mini FT"]
        Base["Base LLM\nGPT / Gemini / Claude"]
        Agent["Tool-Calling Agent\n11 tools"]
    end

    subgraph Tools
        OSM["OSM Network"]
        SUMO["SUMO Generator"]
        TOPIS["TOPIS API"]
        Valid["Validator"]
    end

    subgraph Data
        DB[("SQLite DB")]
        JSONL["Training JSONL"]
    end

    UI -->|"POST /api/simulate"| SSE
    AdminUI -->|"GET /api/admin/*"| API
    SSE --> FT --> Base --> SUMO
    Agent --> Tools
    SUMO --> DB
    DB -->|"export"| JSONL
            

{
  "speed_kmh": 18.5,
  "volume_vph": 2400,
  "lanes": 2,
  "speed_limit_kmh": 30,
  "sigma": 0.72,
  "tau": 0.9,
  "avg_block_m": 120,
  "reasoning": "School zone, 30km/h limit. Morning drop-off congestion, V/C ~0.85."
}

Constrained prompts turned format errors from 15% to zero

The fine-tuned model uses a structured system prompt that enforces strict JSON output, required fields, and value-range constraints. This is not a minor implementation detail — without these constraints the model intermittently returned prose, markdown, or partial JSON, making the downstream pipeline unreliable.

You are a traffic engineering expert and SUMO simulation engineer.
When the user describes a road/traffic situation, return only JSON
with the parameters needed for SUMO simulation.
You must fill all 8 fields below with numbers.
Never use empty values or the string '-'.

Output format:
{"speed_kmh": number, "volume_vph": number,
 "lanes": one-way lane count,
 "speed_limit_kmh": number,
 "sigma": between 0~1, "tau": between 0.5~3,
 "avg_block_m": intersection spacing (m),
 "reasoning": "rationale"}

How each subsystem actually works

Greenshields reverse-estimation for sigma and tau speed → V/C → driver behavior calibration

Driver imperfection (sigma) and desired headway (tau) cannot be directly measured from detector data. They are reverse-estimated from observed speed via the Greenshields model.

The observed speed is divided by free-flow speed to get a speed ratio, which is mapped to a V/C ratio. The V/C ratio determines the congestion band, and sigma and tau are calibrated accordingly.

speed_ratio = v_observed / v_free
V/C ≈ max(0.05, 1.0 − speed_ratio × 0.85)

For example: observed 15 km/h on a 50 km/h limit road → speed_ratio = 0.33 → V/C = 0.72 → congested band → sigma 0.6–0.8, tau 0.8–1.2 s.

V/C range	sigma	tau
> 0.8 (congested)	0.6 – 0.8	0.8 – 1.2 s
0.5 – 0.8 (moderate)	0.4 – 0.6	1.0 – 1.5 s
< 0.5 (free-flow)	0.2 – 0.4	1.5 – 2.5 s

Tool-calling agent — 11-tool orchestration LLM tool-use · autonomous selection

The agent autonomously selects and executes tools based on user requests, using the LLM tool-use feature.

Tool	Description
search_location	Geocode area names to coordinates
build_road_network	Build SUMO network from OSM
get_traffic_stats	Query local Seoul traffic statistics
generate_simulation	Generate SUMO config files
run_sumo	Execute simulation
query_topis_speed	Real-time Seoul traffic API
load_csv_data	Load external traffic data
recommend_road	Suggest similar roads
find_similar_roads	Find roads matching criteria
validate_simulation	Validate simulation output
calibrate_params	Calibrate parameters from results

Example: input "simulate a congested commute road"

1. search_location("commute road") → no location
2. get_traffic_stats("arterial","rush hour") → refs
3. generate_simulation(params) → .net/.rou/.sumocfg
4. run_sumo(config) → avg 22.3 km/h, 1850 veh
5. validate_simulation(results) → grade B, −8.2%

The agent layer uses LLM tool-use. The base LLM is configurable across Claude, GPT, and Gemini.

Role-separated LLM design — 4 components FT extractor · geometry LLM · agent · logging

Each component owns one responsibility. The fine-tuned model does not attempt geometry; the geometry LLM does not attempt structured extraction.

llm_parser.py

Fine-tuned extractor

Parses natural language into structured simulation parameters. Provides the machine-readable target for the rest of the pipeline.

base_llm.py

Geometry LLM

Classifies each user edit as parameter, geometry, or mixed. Handles road layout reasoning and generates fallback XML when OSM fails.

agent.py

Tool-calling agent

11-tool orchestration via LLM tool-use. Autonomous tool selection based on user intent with multi-turn execution.

session_db.py

Logging and export

Stores simulation runs and modification sessions. Separates trainable corrections from non-trainable tuning. Exports retraining JSONL.

Error pattern analysis — directional bias across 30 samples base overpredicts speed +68%, block spacing +165%

Directional bias reveals where each model systematically over- or under-predicts, beyond just the MAPE number.

Field	FT Bias	FT Accurate	Base Bias	Base Accurate
speed_kmh	+0.5% (balanced)	21/30	+68.3% (overpredict)	0/30
volume_vph	+25.9% (over)	13/30	+21.4% (over)	3/30
speed_limit_kmh	−1.7% (balanced)	29/30	+15.7% (over)	11/30
sigma	−0.9% (balanced)	24/30	+9.9% (over)	6/30
tau	+1.6% (balanced)	20/30	+0.9% (over)	3/30
avg_block_m	−0.6% (balanced)	20/30	+165.0% (overpredict)	1/30

Key finding: the base model defaults to free-flow speeds (+68.3% bias, 0/30 accurate) and lacks urban block-structure knowledge (+165% spacing). Fine-tuning corrects both. The remaining weak point is volume_vph (34.8% MAPE) — the most context-dependent field that would benefit from additional training data.

Modification classification — parameter, geometry, or mixed LLM classifier · keyword fallback

When a user requests a change, the geometry LLM first classifies it before routing to the correct handler.

The geometry LLM receives the user's edit request and returns one word: parameter, geometry, or mixed. If the LLM call fails, a keyword heuristic takes over.

User edit → Geometry LLM classifier
  → "parameter" → update speed/volume/sigma/tau
  → "geometry" → regenerate .nod.xml / .edg.xml
  → "mixed" → both paths, then rebuild

Type	Examples
parameter	"lower the speed", "increase volume to 3000", "make it more congested"
geometry	"add an intersection", "bend the road", "make it a 4-way crossing"
mixed	"set speed limit to 70 and make road straight", "add lane and raise volume"

Correction pipeline — human fixes stored in SQLite, exported as retraining data correction vs tuning · trainable flag · JSONL export

Every modification is stored in SQLite with before/after snapshots, modification type, edit intent, and a trainability flag. Only correction-intent records are exported for retraining.

prompt → FT prediction → simulation → user review
  → Correction: stored with trainable=1
  → Tuning: stored with trainable=0

export_corrections_for_training()
  → SELECT WHERE trainable=1 AND intent='correction'
  → sessions_corrections_openai.jsonl
  → merge with train_real_openai.jsonl → re-fine-tune

This separation prevents preference-driven edits from polluting the fine-tuning signal. The admin dashboard at /admin shows correction history, modification breakdowns, and downloadable exports.

SQLite field	Purpose
edit_intent	"correction" or "tuning"
trainable	1 = exportable, 0 = analysis only
modification_type	parameter / geometry / mixed
details_json	Before/after parameter snapshots

Data source	Samples	Ground truth
Real-data (Seoul)	~2,450	Observed speed/volume
Corrections (SQL)	grows over time	Human expert fixes on FT outputs

Runtime parameter wiring — how FT output becomes a SUMO simulation network XML · vType injection · two kinds of "speed"

The FT model predicts eight fields. Some define the physical road, some describe driver behavior, and speed_kmh serves as the validation target for calibration.

FT field	Target	SUMO mechanism
speed_limit_kmh	.net.xml	Rewrite lane/edge speed after netconvert
lanes	.net.xml	Network topology / capacity
avg_block_m	.net.xml	Intersection spacing (generated geometry)
sigma	.rou.xml vType	Krauss driver imperfection (0–1)
tau	.rou.xml vType	Desired headway in seconds
volume_vph	randomTrips.py	Trip generation rate
max_speed	.rou.xml vType	Capped at limit × 1.05
speed_kmh	Nowhere	Validation target only

Two kinds of "speed"

speed_limit_kmh is a legal/physical cap — "cars cannot go faster than this." Written into network XML.

speed_kmh is the predicted average speed under congestion — used for validation and calibration, not written into SUMO files.

Validation error

error = (sim_speed − FT_speed) / FT_speed × 100%

A ≤ 10% · B ≤ 20% · C ≤ 30% · D ≤ 50% · F > 50%

Automatic calibration loop — proportional control with bounded drift volume · sigma · tau · max 3 iterations · ±20% drift cap

When validation error exceeds ±10%, the calibration loop nudges behavioral parameters so the simulated speed converges toward the FT-predicted target.

Algorithm

1. Compute error = (sim − target) / target
2. If |error| ≤ 10% → converged, stop
3. Adjust proportionally:
  volume × (1 + 0.4 × error)
  sigma + 0.15 × error
  tau + 0.25 × error
4. Clamp to drift bounds
5. Re-run SUMO → repeat (max 3)

Gains are derived from SUMO Krauss model sensitivity analysis. If all parameters hit drift caps without converging, the loop stops early — the error signals a geometry mismatch rather than a parameter error.

Parameter	Drift limit	Effect
volume_vph	±20%	Primary congestion lever
sigma	±0.15	Driver imperfection → capacity
tau	±0.3 s	Headway → throughput

Fixed during calibration

speed_limit_kmh, lanes, avg_block_m, and network geometry are never modified. They define the physical road. Calibration only adjusts how vehicles behave on it.

Data separation

Calibrated values are stored in calibrated_params_json — separate from the original FT output. This prevents calibration artifacts from contaminating retraining data.

Generate, correct, and tune — all in one conversation

One chat session covers the full cycle. First, pick which LLM handles geometry reasoning. Then describe a traffic scene — the fine-tuned model extracts structured parameters, the system builds the road network, and SUMO executes the scenario. If the result is wrong, open Correction mode: fix the parameter or geometry error, and the delta becomes retraining data for the next fine-tuning round. If the result is acceptable but you want a variant, use Tuning mode instead — the change is logged for analysis but kept out of the training signal so preference edits never pollute the dataset. An optional Calibrate button auto-adjusts behavioral parameters so the simulated speed converges toward the FT-predicted target.

The browser flow is also a real working interface, not just a demo shell: simulation progress streams live, the generated network is previewed in-chat, the latest SUMO artifact set can be downloaded as a ZIP, and, in local environments, completed scenarios can be reopened in sumo-gui for manual inspection.

1

LLM Selection Base LLM to use for geometry reasoning. Server Gemini, local CLI (claude/gemini/codex), or bring your own API key.

2

Prompt & Run Describe a traffic scenario in natural language. The fine-tuned model extracts parameters, builds network geometry, and runs SUMO.

3

Parameter Correction Review extracted parameters and correct errors. Trainable corrections feed back into the fine-tuning dataset.

4

Geometry Modification Edit road geometry, add intersections, or change lane structure. The base LLM regenerates the network XML.

Monitor corrections, inspect error patterns, and export retraining data

The admin dashboard at /admin closes the feedback loop by surfacing what the model got wrong and making correction data directly exportable. Every correction is traceable end to end: user edit, SQLite log, admin panel, exported JSONL, re-fine-tune.

Why the admin surface matters

Without an inspection layer, corrections disappear into chat history. The admin dashboard makes error patterns visible — which fields drift, which geometry types fail, and how often corrections are trainable — so the next fine-tuning round targets the right gaps.

Three downloadable exports close the loop: Corrections JSONL for re-fine-tuning, Evaluation Report for grade distribution and fidelity, and LLM Evaluation Report for field-level error analysis and parameter deltas.

Dashboard Preview

Correction rates, error bars, and modification logs in one place

Top-level cards show correction rate and fail rate. The LLM evaluation panel breaks down per-field errors and geometry correction types.

Open admin dashboard

Overview

Top cards

Correction rate, mixed fail rate, total simulations, parameter/geometry correction counts at a glance.

Error Analysis

LLM Correction Evaluation

Per-field error bar charts, geometry correction type breakdown, and average parameter deltas.

History

Recent Simulations

Prompt, grade (A/B/C/D), and timestamp for the latest 30 runs.

Modifications

Recent Modifications

Edit intent, modification type, trainable flag, and user input for the latest 50 edits.

What worked, what stayed fragile, and what the project actually proves

The main win is not perfect scenario generation. It is that the system makes evaluation and future improvement structurally possible after deployment.