Race track environment
Built the Unity track used throughout the experiments, combining straight segments and different turning geometries in one driving task.
REINFORCEMENT LEARNING / TEAM PROJECTUCSB CS190A · UNITY / PYTHON
PPO for Autonomous
Car RacingOne turn at a time.
A custom Unity racing environment for studying how actions, observations, and rewards shape an agent’s ability to navigate a challenging track.
01 / My contribution
From environment design
to training experiments.
I built the Unity race track environment and spline-based progress tracking, co-implemented progress rewards and collision penalties, and participated in PPO training and the team’s comparison of five experimental configurations.
Spline-based progress tracking
I mapped the car’s position to the nearest point on the boundary spline, computed progress differences and distance to the boundary, and corrected parameter jumps when crossing the start/finish seam of the closed track.
Training & experiments
I participated in training and comparing PPO agents across reward schemes, control spaces, and observation inputs, examining how each configuration affected turning behavior and track completion.
Team scope: training and evaluation were collaborative. Teammates also handled Gymnasium integration, Python–Unity communication, continuous controls, visual observations, and physics timing.
02 / Environment design
Turning geometry
into a learning signal.
The team used a spline along the left boundary as a reference for the car’s position. Projecting the car onto that spline provides normalized track progress and distance to the boundary.
Let pₖ ∈ [0, 1] be the car’s normalized spline progress at step k, and Δpₖ = pₖ − pₖ₋₁. The continuous progress reward is rₖ = 100 × Δpₖ. Boundary collisions reset the car; episodes also end after two minutes.
The report describes numerical observations of position, heading, track progress, and distance to the left boundary. The continuous action space uses two values for thrust and steering.
Unity simulationgRPCPython / GymnasiumPPO policy
Unity acts as the client and Python as the server, exchanging observations and actions through gRPC.
03 / Team experiments
From checkpoints
to full-track completion.
We compared five configurations, varying reward design, action space, and observations. The outcomes below summarize the course report; the configurations change multiple factors, so they do not isolate the effect of reward design alone.
Configuration 03 rewards each step’s progress increment. Configuration 04 instead rewards accumulated progress, returning −1 whenever the car makes no forward progress.
How the cumulative progress reward works
Ignoring collisions and other events, let A₀ = 0 and update Aₖ = Aₖ₋₁ + 100 × Δpₖ at each step. The output reward follows this rule:
Backward movement reduces Aₖ. Stopping or moving backward overrides the current output with −1; that penalty and collision penalties are not added to Aₖ. The accumulator resets to zero each episode. Progress differences account for crossing the spline’s start/end seam.
“Cumulative” refers to accumulated progress, rather than the sum of all rewards previously output. PPO accounts for future rewards under both configurations.
Discrete rewards
Checkpoint rewards and a time penalty encouraged straight-line progress, but the car failed at the first turn.
Curve bonuses
Increasing rewards around turns improved cornering, but the agent still could not complete the track.
Continuous progress
Each step rewarded the change in spline progress (rₖ = 100 × Δpₖ). Adding distance to the boundary as an observation improved completion, but the agent still did not finish.
Cumulative progress reward
Used accumulated progress as the reward at each step, with −1 for no forward progress. With continuous controls, this configuration completed the full track.
Vision only
A CNN received 64 × 64 overhead images. This configuration did not learn to finish the track.
A harder observation problem
The vision-only experiment replaced numerical features with a 64 × 64 image centered on the car. Even with simplified scene colors, the CNN policy did not complete the track within the reported experiments.
04 / Takeaway
The environment is part
of the learning system.
This project connected my experience building interactive 3D environments with reinforcement learning. Track geometry, progress measurements, and collision feedback all influence the behavior an agent can learn.
Our strongest reported configuration combined continuous controls with cumulative progress rewards and completed the track. The result offers a useful direction for further experiments with more controlled comparisons across reward schemes and observations.