Alert Fatigue at Scale: How GitHub Ate 20,000 Secrets in Nine Months

The Number That Would Make Anyone Panic

Twenty thousand secrets. Fifteen thousand repositories. No centralized ownership. When GitHub Security ran their own secret scanning pilot and got the number back, most teams would have done one of two things: hire a dedicated remediation squad or quietly deprioritize it as someone else's problem.

GitHub did neither. Nine months later they were at zero open alerts.

The story is worth studying because the approach has almost nothing to do with security specifically. It is a case study in operational backlog management applied to a domain where mistakes cost real money. The patterns apply to any SaaS: vulnerability management, compliance findings, tech debt tickets, customer-reported bugs, anything that accumulates faster than individuals can resolve.

Stop the Bleed Before You Triage

Phase one was boring and essential. Enable secret scanning and push protection across every enterprise and organization, enforced at the org level so no team could quietly opt out. Push protection blocks new secrets before they hit the repository. That kept the backlog from growing faster than the team could burn it down.

This is the first mistake most teams make. They jump straight to remediation without fixing the intake valve. You cannot dig a hole while it is still filling.

The infrastructure detail that matters: GitHub Advanced Security lets you enforce settings at the organization level, not repository by repository. That one feature turned a 15,000-repository slog into a few configuration changes. If your tooling requires per-repo or per-service toggling, you will never get full coverage.

18,000 Alerts Disappeared in a Week

When the team broke down the 20,000+ alerts by repository, secret type, and age, something obvious emerged. Five repositories accounted for 18,000 alerts. And every one of those secrets was inactive. Test fixtures. Deactivated credentials. Fake-but-valid-looking tokens in secret scanning test repos. (GitHub builds secret scanning, so naturally they have repos full of legitimate-looking fake secrets.)

They defined criteria for bulk closure. If a secret was in a dedicated test repo, had never been active, and matched a known test pattern, close it. Days later, 90% of the backlog was gone.

The raw alert count is almost never the real scope of work. Triage by source first, not by severity. A handful of high-volume sources will dominate any alert distribution.

Validate Before You Escalate

The remaining 2,000 alerts needed attention. But a credential sitting in a repo could have been rotated years ago or could still unlock production. The team could not prioritize without knowing the difference.

Since secret scanning did not have native validity checking at the time, they built a minimal approach: make a single authenticated request to a low-impact endpoint and check the response. For a GitHub token, a curl to GET /user with a 200 vs 401 told them everything they needed. They treated ambiguous responses (rate limits, 403s) as inconclusive and moved on. The goal was the smallest useful answer, not perfect certainty.

This required coordination with privacy and legal teams. Even a read-only validity check has implications when you do not own the credential you are testing. That cross-functional step is easy to skip and costly to skip.

You Cannot Remediate What You Cannot Route

After validation, the team hit a deeper problem. They knew a credential was live, but they did not know who could rotate it. Not every repository had a clear owner.

For GitHub-issued credentials like personal access tokens, they worked with the product team to surface metadata directly in the alert: who created the token, when, and what scopes it had. That eliminated the need to use the token itself to figure out ownership.

For everything else, ownership was harder. The pain of this cleanup drove a broader initiative to tag every repository with an owner using GitHub Custom Properties, plus a parallel effort to ensure every secret in their credential manager had a named responsible person. You cannot rotate a secret if you cannot find its owner.

The Hard Decisions

The team documented their decision framework for the edge cases that had no clean answer. When a secret lives in an issue, do you edit the body and lose revision history, or preserve the audit trail? When a secret is in a commit, do you rewrite git history? (Anyone who has rewritten git history at scale knows what happens next: force-pushes break open PRs, invalidate commit SHAs, and interrupt developers.)

Their default: rotate or revoke the credential first. Then decide whether the residual risk warrants history rewrite or whether a revoked credential in history can stay. A deleted repository takes its audit trail, so they kept repos intact even when archiving them. The audit trail outlives the convenience of deletion.

Making It Everyone's Problem

The final phase was systematization. Route alerts into the internal vulnerability management platform. Document playbooks by secret type so teams can self-serve. Automate notifications to teams based on repository ownership.

The master stroke: tie secret hygiene to GitHub's Engineering Fundamentals program, the internal standard every team is measured against. When leadership watches the dashboard, teams find time to fix things. Security went from a specialist function that nags to a shared responsibility that is tracked.

What a SaaS Builder Should Steal

The specifics here are about secrets, but the pattern applies to any operational category where alerts outpace capacity. Here is the framework:

1. Stop the intake valve first. Fix the pipeline before cleaning the backlog. No exceptions, no opt-outs.
2. Triage by source, not by severity. Five repos caused 90% of the noise. Find your five repos.
3. Validate before escalating. A raw count is almost never the real workload. Build the smallest useful check.
4. Invest in ownership metadata. Surface who created what and when before someone has to trace a credential.
5. Default to rotation over deletion. Preserve audit trails. Revoke the credential, leave the history.
6. Make it a shared metric. Security cannot remediate alone. Put it in the engineering scorecard.
7. Document the edge case decisions. You will encounter items without clean paths. Write down how you decided so the next person does not re-litigate.

None of this required heroic effort. It required treating alerts as a systems problem rather than a personnel problem. That distinction is what separates teams that stay on top of operational debt from teams that drown in it.