Case study

Engineering Dashboard and AI Checks

The checks on every Unity Explorer pull request and the daily dashboard I built for release and priority reviews, so those decisions started from evidence rather than a weekly scramble for status.

GitHub Actions / scheduled agents / Sentry MCP / Unity Cloud costs / priority reviews / AI-assisted notes

Internal or private project material.All projects

Problem

With frequent releases, information was spread across GitHub, Sentry, Unity Cloud and several reports. I built one view for the questions I kept asking in priority and release reviews. Scheduled agents collect the data and prepare summaries, with links back to the underlying issues and pull requests.

Dashboard summary with a yellow health line, four headline numbers and two cards: what to know today and top risks
The daily view, shown with made-up sample data.

Checks on every pull request

AI review on every PR

Each pull request carries an AI review status check that analyses correctness, performance and error handling and sets pass/fail. It runs before developer and QA approval. For simple fixes and chores, a passing AI review could stand in for the developer review; QA approval was always required.

Visual regression as a comment

A PR comment triggers a pixel-diff suite against committed image baselines and posts the verdict back to the PR; a second command regenerates baselines when a change is intentional. Rendering regressions show up in review instead of depending on whether QA happened to spot them.

Performance signal per PR

Successful builds dispatch frame-time benchmarks on a dedicated machine fleet, and the results come back to the PR as a comment with CPU, GPU and exception data. A performance regression is visible before merge, not after release.

Where AI sits

Automation produces early evidence. People still decide what ships.

A code change produces AI review, test, visual and performance evidence before developers and QA decide whether to release or return to the change.
AI and automated tests shorten the path to useful evidence. Developers and QA remain accountable for interpreting it and approving the release.

Goals

  • One daily view for release and priority reviews.
  • Release, CI, Sentry and team-load data in one place.
  • Unity Cloud costs next to engineering activity.
  • Less time spent collecting status by hand.
  • No new process just to feed the dashboard.

Architecture

Raw activity becomes a decision only after its meaning is defined.

Raw engineering activity becomes traceable evidence after the signal and threshold are defined, then supports a priority decision and an explicit action.
The platform does not treat activity as impact. It fixes definitions, preserves the underlying evidence and closes the loop when a metric produces the wrong decision.

What it computes

Each run collects four groups of information:

  • Flow: lead time to production (median and p75, with sample sizes), PR aging bucketed by reason it waits: on the author, on developer review, on QA, or ready to merge.
  • Quality: escaped defects per release (pre-release vs 14-day escape window), regression rate per release, rework rate with confidence tags, hotfix pressure.
  • Team: review distribution and concentration, QA queue size, bus factor per critical system over a 90-day window, with former-employee attribution flagged as a knowledge gap.
  • Operations: CI health, release deltas, Sentry signals, Unity Cloud build costs.
Four dashboard charts: release cadence, open pull requests by reason they wait, lead time to production and escaped defects per release
Flow and quality charts from the same sample.

Decisions that mattered

  • AI checks run before people, not instead of them. Every pull request still needs developer and QA approval. The AI check raises the floor before people spend attention.
  • Ask for something concrete. A pass or fail, a pixel diff, a frame-time change. Output that repeatedly changed nothing in review was removed.
  • Pin every metric definition. Each metric has a fixed formula, window and exclusions, recorded next to the code. When a definition changes, the next report says so, so a delta is never a quiet redefinition.
  • Link every number to its records. Ratios come with the exact issues and pull requests counted, so anyone can re-derive them.
  • Turn corrections into rules. A lessons file records each mistake as a rule the next run applies: cumulative totals are not trends, duplicates are excluded and named, partial windows are flagged.

Lessons learned

  • The checks that stuck were attached to an existing review or release step.
  • Quality checks were worth more than measuring how much code AI generated.
  • The dashboard became useful once it was part of the priority review. We removed the metrics nobody used.