Skip to content

Conversation

@seefeldb
Copy link
Contributor

@seefeldb seefeldb commented Dec 12, 2025

Summary

  • Implement pull-based scheduling that collects dirty dependencies upfront and runs them in topological order, avoiding server roundtrip issues from the previous approach
  • Add cycle-aware convergence using Tarjan's SCC algorithm - fast cycles (<16ms) converge synchronously, slow cycles yield between iterations
  • Add debounce/throttle infrastructure for action performance optimization (auto-detects slow actions >50ms)
  • Combine pull mode's conservative work set with push mode's precision via push-triggered filtering - only runs actions triggered by actual storage changes
  • Remove legacy boolean signature from subscribe() in favor of options object format

Key Features

Pull-Based Core:

  • Dirty tracking (dirty Set, markDirty, isDirty, clearDirty)
  • Effect/computation distinction with isEffect option in subscribe()
  • Reverse dependency graph for transitive dirty collection
  • Topological sorting of work queue

Cycle Detection & Convergence:

  • Tarjan's algorithm for strongly connected component detection
  • MAX_CYCLE_ITERATIONS = 20 safeguard
  • FAST_CYCLE_THRESHOLD_MS = 16 for sync vs async decision
  • Action stats tracking for cycle classification

Debounce & Throttle:

  • setDebounce/getDebounce/clearDebounce API
  • setThrottle/getThrottle/clearThrottle API
  • Auto-debounce detection for actions >50ms after 3 runs
  • Throttled actions accept staleness, stay dirty for future pulls

Push-Triggered Filtering (Phase 5):

  • mightWrite WeakMap: accumulated historical writes per action
  • pushTriggered Set: tracks actions triggered by storage changes each cycle
  • Runs intersection of pull work set and push triggers
  • Bypass conditions: scheduleImmediately, first run, actual storage trigger

Test plan

  • All scheduler tests pass (12 test groups, 88 steps)
  • Backwards compatibility with pullMode=false verified via test
  • Cycle convergence behavior covered by automated tests
  • Debounce/throttle behavior covered by automated tests
  • All 144 generated-patterns integration tests pass

🤖 Generated with Claude Code


Summary by cubic

Implements a pull-based scheduler that runs dirty dependencies in topological order, with cycle-aware convergence, parent–child action ordering, and performance controls. Ensures event handler inputs are current in pull mode, adds push-triggered filtering for precision, a Cell.pull() method for demand-driven reads, and replaces the legacy subscribe() boolean with an options object.

  • New Features

    • Pull-based execution with ordered dirty dependency runs and parent-first action ordering.
    • Cycle-aware convergence (fast cycles sync; slow cycles yield).
    • Debounce and throttle APIs, with auto-debounce for slow actions.
    • Push-triggered filtering to run only actions affected by real storage changes.
    • Diagnostics for performance and filtering (stats, mightWrite).
  • Migration

    • subscribe() now takes an options object. Replace the third boolean with { scheduleImmediately?: boolean, isEffect?: boolean }.
    • Pull mode is enabled by default. Prefer cell.pull() over idle()+get for demand-driven reads.
    • In pull mode, add a sink to long-lived outputs you want to keep reactive (e.g., charms).
    • get() now accepts { traverseCells?: boolean } for deep dependency capture when needed.

Written for commit bbe379a. Summary will update automatically on new commits.

Copy link
Contributor

@cubic-dev-ai cubic-dev-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

3 issues found across 7 files

Prompt for AI agents (all 3 issues)

Check if these issues are valid — if so, understand the root cause of each and fix them.


<file name="packages/runner/test/scheduler.test.ts">

<violation number="1" location="packages/runner/test/scheduler.test.ts:1413">
P1: Test claims to verify legacy boolean signature but uses the new object format instead. If the scheduler should support `subscribe(action, deps, true)` for backward compatibility, this test should use that syntax.</violation>

<violation number="2" location="packages/runner/test/scheduler.test.ts:2413">
P2: Assertion `toBeGreaterThanOrEqual(0)` is meaningless - array length is always &gt;= 0. If cycles are expected to be detected, consider using `toBeGreaterThan(0)` or a specific expected count.</violation>
</file>

<file name="docs/specs/pull-based-scheduler/scheduler-implementation-plan.md">

<violation number="1" location="docs/specs/pull-based-scheduler/scheduler-implementation-plan.md:377">
P3: Stale phase reference: These tasks reference &#39;deferred to Phase 5&#39; but Phase 5 is now &#39;Push-Triggered Filtering&#39;. Should reference Phase 6 (Full Migration) instead.</violation>
</file>

Reply to cubic to teach it or ask questions. Re-run a review with @cubic-dev-ai review this PR

@seefeldb seefeldb force-pushed the feat/pull-based-scheduler branch from ba4f45b to 8c35ab3 Compare December 12, 2025 23:59
seefeldb and others added 3 commits December 15, 2025 15:08
In pull mode, .get() returns cached values without triggering
computation of dirty dependencies. Changed the pattern harness to use
.pull() which properly establishes an effect and waits for the
scheduler to compute all transitive dependencies.

This fixes 27/28 failing generated-patterns integration tests.
One test (counter-when-unless-operators) still fails due to a
deeper issue with nested lift computations through references.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <[email protected]>
When a lift function returns undefined on first run, no write is recorded
in the journal. This broke the dependency chain in pull mode because:
1. updateDependents only looked at actual writes
2. topologicalSort only used actual writes for ordering

Fix: Initialize mightWrite from declared writes during subscribe, and use
mightWrite (instead of just log.writes) in both updateDependents and
topologicalSort.

Adds test case that reproduces the nested lift scenario where inner lift
initially returns undefined.

Fixes counter-when-unless-operators test which has nested lifts where
the inner lift initially returns undefined.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <[email protected]>
seefeldb and others added 4 commits December 15, 2025 16:11
- Add `isEffect` field to Module interface
- Add optional `options` parameter to `raw()` with `isEffect` flag
- Pass `isEffect` through to scheduler.subscribe() in instantiateRawNode
- Mark navigateTo as an effect since it triggers navigation side effects

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <[email protected]>
llmDialog is an effect (triggered by user action), not a computation.
In pull mode, it needs to ensure its dependencies are computed before
reading them.

Changes:
- Mark llmDialog as isEffect: true in registration
- Add pulls for inputs, pinnedCells, and their referenced cells
  at the start of startRequest

The helper functions (buildToolCatalog, buildAvailableCellsDocumentation)
remain synchronous so they can be shared with llm/generateText/generateObject
which are computations that need reactive dependency tracking.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <[email protected]>
llmDialog is an effect (triggered by user action), not a computation.
Mark it as isEffect: true in registration.

Note: Explicit pull() calls were removed because they conflict with
the handler's transaction lifecycle. The handler runs synchronously
with a transaction, but pull() is async. A better solution for
pulling dependencies in effect handlers is needed.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <[email protected]>
- Remove tx parameter from startRequest, use pull() and editWithRetry()
  instead to support pull-based scheduling
- Pull inputs, pinnedCells, and individual context/pinned cells at start
  of startRequest to ensure they're computed in pull mode
- Use editWithRetry() for the write to result.pinnedCells
- Remove isEffect marking since llmDialog is event-handler driven, not
  an effect

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <[email protected]>
seefeldb and others added 3 commits December 15, 2025 18:03
Design spec for ensuring event handler inputs are current in pull mode.

Key decisions:
- Treat handlers as one-time actions in the unified scheduler loop
- Topological sort naturally orders handler inputs before handlers
- Per-stream event queues preserve FIFO ordering
- Dependency re-validation handles dynamic references
- Event handlers get priority among actions at same topo level
- Throttled/debounced deps allowed to be stale

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <[email protected]>
Implements the userland handler pull design to ensure event handlers
don't run until their dependencies are computed.

Changes:
- Add traverseCells flag to validateAndTransform() and Cell.get() to
  recursively read into nested Cells and capture all dependencies
- Add populateDependencies callback to addEventHandler() for schema-based
  dependency discovery without coupling scheduler to schema module
- In pull mode, check if handler dependencies are dirty before running;
  if so, re-queue the event and process pending actions first
- Use global FIFO ordering for events across all streams

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <[email protected]>
Add tests verifying that in pull mode, event handlers wait for their
computed dependencies to be pulled before running:

- Handler as only reader of computed output
- Multiple dirty dependencies pulled before handler runs
- Chained dependencies (source -> computed1 -> computed2 -> handler)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant