Background recache re-warming URLs that don't exist

[stephenmeehanuk]

On a busy public site, the half measure static cache (ApplicationCacher) records every 404 it renders into its tracked URL set — including the constant stream of scanner/bot probes (SQLi payloads in path segments, .php/.jsp/.env guesses, base64 junk, dead/renamed URLs). Nothing ever checks that a tracked URL maps to real content.

With STATAMIC_BACKGROUND_RECACHE=true and wildcard invalidation rules, this becomes a problem:

• A content save fires a wildcard rule like /rail/*.
• refreshWildcardUrl() prefix-matches the raw tracked .urls set and dispatches a StaticWarmJob (HTTP GET with ?__recache=…) for every match - junk included.
• Each junk URL returns 404, Guzzle throws (StaticWarmJob has http_errors on and $tries = 1), and you get one failed_jobs row per URL.
• On our production site this meant ~97% of the ~20,900 tracked URLs were junk 404s (plus Glide image URLs and bot query strings). Every editor save re-warmed thousands of non-existent URLs, flooding failed_jobs and delaying real edits appearing live (the warm worker churns through junk before reaching real pages).

Example log line (our own recache worker, not the scanner):

production.ERROR: Client error: GET https://example.com/news/categories/@@yMKFc?__recache=… resulted in a 404 Not Found

Root cause (Statamic v6.21)

404s are cached and tracked per-URL.
Cache::shouldBeCached() allows [200, 404] for ApplicationCacher; ApplicationCacher::cachePage() then calls cacheUrl(), which records the URL in the forever-stored .urls set.

Wildcard invalidation trusts that set blindly.
AbstractCacher::refreshWildcardUrl() does getUrls()->filter(Str::startsWith($prefix)) - a prefix match against whatever was ever cached, with no “does this resolve to real content?” check.

Background recache replays it.
DefaultInvalidator::refresh() dispatches a warm job per matched URL; junk 404s throw and fail.

Overview of our fix (workaround)

We swapped in a custom cacher that skips per-URL caching/tracking of 4xx/5xx responses, but keeps the shared-error page — so share_errors still serves a fast cached 404, while junk URLs never enter the tracked set (and so are never re-warmed).

class ErrorAwareApplicationCacher extends ApplicationCacher
{
    public function cachePage(Request $request, $content)
    {
        if ($content instanceof Response
            && $content->getStatusCode() >= 400
            && ! str_contains($this->getUrl($request), '/__shared-errors/')) {
            return; // don't cache or track per-URL error responses
        }
        parent::cachePage($request, $content);
    }
}

Registered by extending the manager’s application driver (via afterResolving(StaticCacheManager::class, …), because the Invalidate subscriber resolves the cacher early in boot).

Result on a local half-measure repro: after hitting several junk 404s + a real page, the tracked set was just ["/", "/about", "/__shared-errors/default/404"] - junk excluded, shared 404 still cached and served.

(We also excluded /img*//assets* and switched query-string handling to an allowed_query_strings allowlist to kill Glide-URL and bot-query-string bloat — but those are separate, config-level.)

Suggested core improvement

Any one of these would remove the footgun without a custom cacher:

1. Don’t track per-URL 404s in .urls (they’re served by the shared-error mechanism when share_errors is on) — or make it a config toggle (e.g. track_errors => false).
2. Make StaticWarmJob self-healing: a warmed URL that returns 404 should forget that URL from the tracked set rather than fail the job — so the set converges to live pages.
3. Have refreshWildcardUrl() skip URLs that no longer resolve, so wildcard invalidation never re-warms dead URLs.

Environment: Statamic 6.21, Laravel 13, half measure static caching, STATAMIC_BACKGROUND_RECACHE=true.