AI Agent

A failing test, explained and fixed.

Connect Claude or OpenAI and a failing report gets an AI Analysis card. The agent reads the failure, its recent history and a screenshot, tells you whether it's a real bug, a flake, or a mistake in the test itself, and for a test-code bug, opens a real pull request with the fix.

One click, four possible answers

It never just says "failed."

The agent weighs the failure against each scenario's recent pass/fail history before it decides. A re-run is slow and takes a node away from the team, so it only reaches for one when the history alone is inconclusive.

Test code issue

The E2E test itself is wrong: a stale selector, a bad assertion, a missing wait. The only verdict that unlocks Fix it and open a PR.

Real issue

The application under test is genuinely broken. Nothing to fix in the test, so it goes to the team.

Flake

Nondeterministic. The test and the app both check out, so no fix is needed, just noise to ignore.

Inconclusive

Not enough evidence either way. Re-analyze from a different angle, or dig in by hand.

01 · Analyze

Ask why, right on the report.

Every failing report gets an AI Analysis card. One click, and the agent reads the failing scenario, its recent pass/fail history and a screenshot from the moment it failed, then settles on a verdict, a plain-language summary, and the evidence behind it.

recent historyfailure screenshotone verdict

Plum AI Analysis card showing an inconclusive verdict with findings and a recommended next step

02 · Fix

Test code issue? It can fix it.

When the verdict is a test-code bug, Fix it and open a PR appears. The agent clones your repo into an isolated workspace on its own branch, makes the smallest change that addresses the cause, and opens a real pull request, or says it isn't confident and stops, rather than guessing.

isolated clonesmallest diffreal PR

Plum AI Analysis card with a test code issue verdict and a View pull request button

03 · Review

A pull request, not a black box.

Every fix lands as a normal GitHub pull request: what was wrong, what changed, and how to verify it, written by the agent. Nothing reaches your default branch without you reviewing and merging it yourself, same as a PR from anyone else on the team.

A real GitHub pull request opened by Plum's AI agent, describing the problem, root cause and fix

The boundary, enforced not asked

It can open a PR. That's the ceiling.

Every fix runs inside a fresh, isolated clone, never your live project folder, on its own branch, deleted once the PR is pushed or the attempt is abandoned. This is enforced server-side in the tools the agent gets, not something a prompt has to be trusted to honor.

Read the full docs →

agent-permissions

✓ read the failing scenario & its history
✓ work inside its own isolated clone
✓ commit, push its own branch
✓ open a pull request
✕ push to the default branch
✕ merge anything, ever
✕ touch a file outside its workspace

04 · Connect

Two keys, five minutes.

Connect Claude or OpenAI and a GitHub personal access token in Settings → Integrations, owner only, instance-wide. Every key is verified against the provider itself before it's saved, so a bad or expired one is rejected immediately instead of failing silently later. Connect either provider; if both are set, Claude is used.

Settings → Integrations → AISettings → Integrations → GitHub

Settings → Integrations

AI provider keys
  Anthropic API key    sk-ant-…
  OpenAI API key       sk-…

GitHub connection
  Personal access token github_pat_…