Cannot resolve any accounts from specific remote instance, despite WebFinger working

[deholic]

Steps to reproduce the problem

1. On my Mastodon instance (https://duk.space), go to the search box.
2. Search for any account on the remote instance b-starhire.cc, e.g. @admin@b-starhire.cc, with “Resolve remote account” enabled.
3. Alternatively, call the API:
   • GET /api/v2/search?q=@admin@b-starhire.cc&resolve=true&type=accounts&limit=1
   • or GET /api/v1/accounts/lookup?acct=@admin@b-starhire.cc
4. Observe that:
   • /api/v2/search either returns only accounts from other instances with the same username, but never from b-starhire.cc
   • /api/v1/accounts/lookup always returns 404 for any account on b-starhire.cc
5. On other Mastodon instances (not duk.space), performing the same search for @admin@b-starhire.cc works as expected and the account can be resolved and viewed.

Expected behaviour

My instance should perform a remote lookup to b-starhire.cc, create the remote account, and successfully return it in search and lookup responses.

Actual behaviour

My instance never attempts a remote lookup to b-starhire.cc, returns 404 for any of its accounts.

Detailed description

This looks like my instance has gotten into a state where the remote domain b-starhire.cc is treated
as if it cannot be resolved, but without any record in the usual tables:

• There is no entry for "b-starhire.cc" in domain_blocks.
• There is no entry for "b-starhire.cc" in unavailable_domains.
• There is no entry for "b-starhire.cc" in domain_allows either, and LIMITED_FEDERATION_MODE is not
  configured in a way that should block this domain (other remote domains work fine).

I also checked the database for existing accounts:

• SELECT * FROM accounts WHERE domain = 'b-starhire.cc'
  returns 0 rows, so my instance has never successfully created any remote account records for this domain.

From the OS shell on the Mastodon host, I can successfully fetch WebFinger from b-starhire.cc, for example:

• curl "https://b-starhire.cc/.well-known/webfinger?resource=acct:admin@b-starhire.cc"

This suggests there is no basic network, DNS, or TLS problem between my server and b-starhire.cc.

However, from Mastodon itself:

• Searching with resolve=true or calling /api/v1/accounts/lookup triggers NO Sidekiq jobs for b-starhire.cc.
  It feels like Mastodon is short-circuiting before trying to contact that domain at all, but I cannot
  find any explicit domain block or "unavailable" flag in Postgres.

The administrator of b-starhire.cc also confirmed to me that there is no instance-level block configured
for duk.space on their side, and I am not present in their block lists. Other instances can resolve
accounts from b-starhire.cc normally, so the remote instance appears to be healthy.

I also experimented with Redis (looking for keys containing "b-starhire.cc" and eventually clearing some
cache), and even restarted the whole stack. The behaviour did not change: my instance still never attempts
to resolve accounts from b-starhire.cc, and /api/v1/accounts/lookup always returns 404.

It looks like some kind of persistent negative cache or internal state that prevents any further attempts
to resolve that domain, but that state is not visible in the usual domain_* tables and survives restarts.

Mastodon instance

duk.space

Mastodon version

v4.5.2

Technical details

Relevant log line from web (API lookup):

Nov 27 05:21:19 ip-172-31-33-116 bundle[1464196]:
I, [2025-11-27T05:21:19.119500 #1464196] INFO -- :
[1bbd8ee6-c7aa-488f-a174-9d8d7792171d] method=GET path=/api/v1/accounts/lookup
format=html controller=Api::V1::Accounts::LookupController action=show
status=404 allocations=1677 duration=19.34 view=0.39 db=12.05

Things I have verified:

• No row for "b-starhire.cc" in domain_blocks.
• No row for "b-starhire.cc" in unavailable_domains.
• No row for "b-starhire.cc" in domain_allows, and federation is not limited in a way that should block this domain.
• SELECT * FROM accounts WHERE domain = 'b-starhire.cc' returns 0 rows.
• Sidekiq logs show no jobs related to b-starhire.cc when attempting to resolve any account on that domain.
• curl from the Mastodon host to b-starhire.cc WebFinger succeeds.

Ruby version: 3.4.7p58
Node.js version: 24.11.0
Database: PostgreSQL 15.12
Redis: 8.0.3
Deployment: Ubuntu 22.04