Chinese
Uncategorized

GitHub Copilot Evals and Event Streams: Observe the Agent Before You Trust It

GitHub’s new Copilot Evals and Event Streams previews point to the same lesson: agent work needs evidence before automation.

Jacky Wang 5 min read 47 views

Last week I was looking at an agent workflow that edits a branch, waits for CI, and continues after a pull request comment. The first question was not whether it could produce a patch. I wanted to know what I could still explain after it paused. A task marked “done” is not evidence that the dangerous cases were tested. A task with no visible activity may simply be waiting for an event.

On November 17, GitHub put two relevant previews in the Copilot changelog: Copilot Evals and GitHub Event Streams. Evals is aimed at checking Copilot as prompts, models, skills, and tools change. Event Streams is a real-time coordination layer between developers, agents, and automation. An agent can sleep when there is no work and wake when an event arrives. Read together, these releases are more interesting than another model selector because they address the part of agent work that tends to disappear between the demo and production.

What I would put in an eval

I would not judge a coding agent from one impressive session. I would collect recurring repository tasks: fix a bug with a regression test, add a field to an API, or update an image tag in a Kubernetes manifest. Each case gets a fixed input, a known repository revision, and a restricted tool set. The output includes the diff, test results, permission checks, and elapsed time.

Even a tiny case definition should state what must not happen:

{
  "case": "upgrade-image-tag",
  "base": "main@abc123",
  "allowed_tools": ["read", "edit", "test"],
  "checks": ["git diff --check", "npm test"],
  "must_not": ["kubectl apply", "send-email"]
}

That last field is a real acceptance criterion. If the agent changes the code correctly but touches a production cluster, I would fail the run even when every test passes. For a SaaS repository, I would also score whether the agent leaves a reviewable diff. A green result without a diff is hard to audit; a diff without tests is not enough.

GitHub’s Evals preview is a useful starting point, but it cannot define my team’s bar. I would separate hard gates from softer scores. Tests passing, no forbidden tool call, and no secret in the diff are hard gates. Patch size, explanation quality, and human rework time can be scored. That keeps the comparison focused on work output when the underlying model changes, rather than on one unusually good conversation.

Event-driven work needs durable state

Many agent systems fail while waiting. They keep a long-running process alive for a pull request comment, a CI result, or an approval. That consumes runtime and makes recovery vague. Event Streams points toward a different shape: sleep while idle, wake on an event, then resume from stored state.

For a Kubernetes-based service, I would make the state explicit instead of trusting process memory:

task:
  id: pr-482
  phase: waiting_for_ci
  last_event: check_suite.completed
  resume_from: verify.patch
  budget_seconds: 900

After a restart, the worker reads resume_from and checks whether the event ID has already been handled. Without an idempotency key, a wake-up path can post the same comment twice, open duplicate pull requests, or treat an old check result as new. Real-time delivery is not reliability. Reliability comes from durable state, deduplication, and a pause point that a person can take over.

Start with a narrow surface

I would not connect every repository at once. Pick a low-risk service, run ten or twenty fixed cases, and compare the agent with a human baseline. Evals tells me whether quality moved when the prompts or tools changed. Event Streams tells me whether waiting and resuming are controllable. Only after both are boring would I allow automatic pull request creation. Merge and production release would remain behind human approval.

My bet is that coding-agent competition will move from “who writes the most code” to “who can make failure legible.” If a system cannot show what it read, what it changed, and why it stopped, I would rather run a slower agent with a clean recovery path.

I would also keep a human baseline for the same cases. This is not about ranking people against a model. It tells me whether the failures come from the agent or from a flaky test environment. If failures cluster around permission decisions, I would tighten the tool surface before tuning prompts. After each model or harness update, I would rerun the fixed set and retain the input commit, tool version, and result summary. That is the only practical way to tell whether an improvement came from the model, the harness, or a lucky test run.

In the real integration I would also give every task a budget: a maximum number of tool calls, a wall-clock limit, and a file-change limit. Once a budget is exhausted, the task pauses for a person. That is not meant to punish the agent. It stops a repair loop from turning a small test fix into an uncontrolled repository rewrite.

The budget should also be observable. Near the limit, the worker should save its state and alert rather than fail silently. Then I can resume from the last safe step and decide whether the task was too hard or the tool boundary was too broad.

GitHub Copilot Evals public preview, GitHub Event Streams preview, GitHub November 2026 changelog

发表评论