feat: Configurable timeouts + warmup for long-running / large-snapshot actors

[Maya Wang (mayawang)]

Summary

Makes four control-plane parameters configurable (three new env-var knobs) plus
one opt-out, so that heavy, slow-initializing, or large-snapshot actors can
run end-to-end on Substrate.

No default value is changed. This PR is purely additive — behavior is
identical to today unless an operator sets one of the new environment variables.

Why (context)

This comes out of the OpenClaw-on-Substrate integration: an always-on
conversational agent (a multi-process Node.js workload that holds channel
connections — WhatsApp, etc. — and suspends/resumes per conversation). Running
it end-to-end surfaced four places where the current defaults, which are sized
for lightweight, fast-starting actors, are too tight for this class of workload:

1. The golden-snapshot warmup checkpoints a probe-less workload before it has
   finished initializing → the golden captures a not-ready process.
2. A cold restore-on-demand of a large (~60 MiB) gVisor snapshot takes tens of
   seconds and is cancelled by the resume / route / ext_proc timeouts → 504.
3. runsc start -allow-connected-on-save is rejected by some runsc builds.

Rather than change any default, each is made tunable so operators running such
workloads can opt in; everyone else is unaffected.

No defaults changed

Parameter	Component	Default (before and after)	New override
Golden-snapshot warmup	atecontroller	20s	ATE_GOLDEN_WARMUP_SECONDS
Background resume timeout	atenet	15s	ATE_RESUME_TIMEOUT_SECONDS
Forwarded route timeout	atenet	10s	ATE_ROUTE_TIMEOUT_SECONDS
ext_proc timeout (GrpcService + MessageTimeout)	atenet	5s	ATE_EXTPROC_TIMEOUT_SECONDS
runsc start -allow-connected-on-save	ateom-gvisor	passed (on)	ATEOM_RUNSC_ALLOW_CONNECTED_ON_SAVE=false to disable

An unset or invalid env value falls back to the default in every case.

Changes

cmd/atecontroller/.../actortemplate_controller.go — configurable golden warmup

For templates without a readyz probe, the controller waits a fixed 20s after
booting the golden, then checkpoints. A workload with long, probe-less
initialization is still starting up at 20s, so the checkpoint captures a
not-ready golden. Adds ATE_GOLDEN_WARMUP_SECONDS (via defaultGoldenWarmup());
default 20s unchanged. Templates that declare readyz are unaffected (they
already skip the fixed wait).

cmd/atenet/internal/router/timeouts.go (new) — shared, documented knobs

Centralizes the three atenet timeouts, their defaults, and a whole-seconds env
parser in one place, so the resumer.go / xds.go diffs stay minimal and the
rationale + defaults live together.

cmd/atenet/internal/router/resumer.go — configurable background resume timeout

The detached background resume context (15s) is cancelled before a large cold
restore completes, surfacing as a 504 to the caller. Now resumeTimeout()
(ATE_RESUME_TIMEOUT_SECONDS, default 15s).

cmd/atenet/internal/router/xds.go — configurable route + ext_proc timeouts

• Route timeout (10s): a long agent turn (e.g. an LLM streaming a response)
  exceeds it and is cut off mid-turn. Now routeTimeout()
  (ATE_ROUTE_TIMEOUT_SECONDS, default 10s).
• ext_proc timeout (5s, both GrpcService.Timeout and MessageTimeout):
  the ext_proc round-trip can drive a cold restore that takes tens of seconds;
  5s aborts it. Now extProcTimeout() (ATE_EXTPROC_TIMEOUT_SECONDS, default
  5s).

cmd/ateom-gvisor/runsc.go — gate -allow-connected-on-save

The flag lets runsc checkpoint a sandbox with open connected sockets, but some
runsc builds (certain GKE-Sandbox releases) do not define it and reject
runsc start with flag provided but not defined. Extracted
allowConnectedOnSave() (default true, preserving current behavior); set
ATEOM_RUNSC_ALLOW_CONNECTED_ON_SAVE=false for builds that don't support it.

Relationship to #465

The atenet timeout bumps are stopgaps for the connected-socket suspend/restore
problem tracked in #465 (suspend-safe actor networking). Once actor network
traffic survives checkpoint/restore natively, most of the need to raise these
ceilings should go away — this PR just makes the current behavior tunable in the
meantime.

Verification & testing

• Build: go build ./cmd/atenet/... ./cmd/atecontroller/... ./cmd/ateom-gvisor/... — passes.
• Static analysis: go vet ./cmd/atenet/internal/router/ — clean.
• End-to-end on GKE (current OSS Substrate, CRD group ate.dev): a heavy
  OpenClaw actor (~60 MiB golden) validated through the full lifecycle — golden
  capture with an extended warmup (ATE_GOLDEN_WARMUP_SECONDS=120) reaching
  Ready with a live snapshot; resume-on-demand (~18s cold / ~9s warm); long
  LLM turns; gateway-driven suspend; and repeated suspend/resume cycles with live
  WhatsApp round-trips.

Honest caveats on the e2e run:

• The warmup override (ATE_GOLDEN_WARMUP_SECONDS) was exercised as shipped.
• The atenet timeout values were validated e2e as the equivalent hardcoded
  values (120s / 300s / 60s); this PR refactors those exact values behind
  env vars with the defaults unchanged. The refactor is build- and
  vet-verified; the env-read path is a straightforward whole-seconds parse with
  fallback.
• The runsc gate was validated e2e in the disabled state
  (ATEOM_RUNSC_ALLOW_CONNECTED_ON_SAVE=false) on a GKE-Sandbox runsc build that
  rejects the flag. The default (enabled) path is the pre-existing upstream
  behavior and is unchanged.

Out of scope (intentionally excluded)

The gVisor SandboxConfig change that points runsc.url at a specific build is
not included — it is environment-specific (a private bucket + build SHA) and
does not belong upstream. The only generic, upstreamable finding is that a runsc
build which survives a heavy multi-process actor is required; the flag gate above
is the portable part of that.

Fixes #<issue_number_goes_here>

It's a good idea to open an issue first for discussion.

• Tests pass
• Appropriate changes to documentation are included in the PR

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[Dmitry Berkovich (dberkov)]

Maya Wang (@mayawang) - I have recently added readyz to the container template -

substrate/pkg/api/v1alpha1/actortemplate_types.go

Line 139 in 0f6144e

type ContainerReadyz struct {

. Today it is just action and actually timeout defined as a contant here -

substrate/internal/readyz/readyz.go

Line 42 in 0f6144e

OverallTimeout = 30 * time.Second

.

Have you considered to extend the readyz with a timeout via actorTemplate and push it up to the atelet?