-
Notifications
You must be signed in to change notification settings - Fork 11
feat(scheduler): implement pull-based scheduler with cycle-aware convergence #2270
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
seefeldb
wants to merge
40
commits into
main
Choose a base branch
from
feat/pull-based-scheduler
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
3 issues found across 7 files
Prompt for AI agents (all 3 issues)
Check if these issues are valid — if so, understand the root cause of each and fix them.
<file name="packages/runner/test/scheduler.test.ts">
<violation number="1" location="packages/runner/test/scheduler.test.ts:1413">
P1: Test claims to verify legacy boolean signature but uses the new object format instead. If the scheduler should support `subscribe(action, deps, true)` for backward compatibility, this test should use that syntax.</violation>
<violation number="2" location="packages/runner/test/scheduler.test.ts:2413">
P2: Assertion `toBeGreaterThanOrEqual(0)` is meaningless - array length is always >= 0. If cycles are expected to be detected, consider using `toBeGreaterThan(0)` or a specific expected count.</violation>
</file>
<file name="docs/specs/pull-based-scheduler/scheduler-implementation-plan.md">
<violation number="1" location="docs/specs/pull-based-scheduler/scheduler-implementation-plan.md:377">
P3: Stale phase reference: These tasks reference 'deferred to Phase 5' but Phase 5 is now 'Push-Triggered Filtering'. Should reference Phase 6 (Full Migration) instead.</violation>
</file>
Reply to cubic to teach it or ask questions. Re-run a review with @cubic-dev-ai review this PR
docs/specs/pull-based-scheduler/scheduler-implementation-plan.md
Outdated
Show resolved
Hide resolved
ba4f45b to
8c35ab3
Compare
In pull mode, .get() returns cached values without triggering computation of dirty dependencies. Changed the pattern harness to use .pull() which properly establishes an effect and waits for the scheduler to compute all transitive dependencies. This fixes 27/28 failing generated-patterns integration tests. One test (counter-when-unless-operators) still fails due to a deeper issue with nested lift computations through references. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]>
When a lift function returns undefined on first run, no write is recorded in the journal. This broke the dependency chain in pull mode because: 1. updateDependents only looked at actual writes 2. topologicalSort only used actual writes for ordering Fix: Initialize mightWrite from declared writes during subscribe, and use mightWrite (instead of just log.writes) in both updateDependents and topologicalSort. Adds test case that reproduces the nested lift scenario where inner lift initially returns undefined. Fixes counter-when-unless-operators test which has nested lifts where the inner lift initially returns undefined. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]>
75d7226 to
a344414
Compare
- Add `isEffect` field to Module interface - Add optional `options` parameter to `raw()` with `isEffect` flag - Pass `isEffect` through to scheduler.subscribe() in instantiateRawNode - Mark navigateTo as an effect since it triggers navigation side effects 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]>
llmDialog is an effect (triggered by user action), not a computation. In pull mode, it needs to ensure its dependencies are computed before reading them. Changes: - Mark llmDialog as isEffect: true in registration - Add pulls for inputs, pinnedCells, and their referenced cells at the start of startRequest The helper functions (buildToolCatalog, buildAvailableCellsDocumentation) remain synchronous so they can be shared with llm/generateText/generateObject which are computations that need reactive dependency tracking. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]>
llmDialog is an effect (triggered by user action), not a computation. Mark it as isEffect: true in registration. Note: Explicit pull() calls were removed because they conflict with the handler's transaction lifecycle. The handler runs synchronously with a transaction, but pull() is async. A better solution for pulling dependencies in effect handlers is needed. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]>
- Remove tx parameter from startRequest, use pull() and editWithRetry() instead to support pull-based scheduling - Pull inputs, pinnedCells, and individual context/pinned cells at start of startRequest to ensure they're computed in pull mode - Use editWithRetry() for the write to result.pinnedCells - Remove isEffect marking since llmDialog is event-handler driven, not an effect 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]>
1808b63 to
7a2bea1
Compare
Design spec for ensuring event handler inputs are current in pull mode. Key decisions: - Treat handlers as one-time actions in the unified scheduler loop - Topological sort naturally orders handler inputs before handlers - Per-stream event queues preserve FIFO ordering - Dependency re-validation handles dynamic references - Event handlers get priority among actions at same topo level - Throttled/debounced deps allowed to be stale 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]>
Implements the userland handler pull design to ensure event handlers don't run until their dependencies are computed. Changes: - Add traverseCells flag to validateAndTransform() and Cell.get() to recursively read into nested Cells and capture all dependencies - Add populateDependencies callback to addEventHandler() for schema-based dependency discovery without coupling scheduler to schema module - In pull mode, check if handler dependencies are dirty before running; if so, re-queue the event and process pending actions first - Use global FIFO ordering for events across all streams 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]>
Add tests verifying that in pull mode, event handlers wait for their computed dependencies to be pulled before running: - Handler as only reader of computed output - Multiple dirty dependencies pulled before handler runs - Chained dependencies (source -> computed1 -> computed2 -> handler) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
subscribe()in favor of options object formatKey Features
Pull-Based Core:
dirtySet,markDirty,isDirty,clearDirty)isEffectoption insubscribe()Cycle Detection & Convergence:
MAX_CYCLE_ITERATIONS = 20safeguardFAST_CYCLE_THRESHOLD_MS = 16for sync vs async decisionDebounce & Throttle:
setDebounce/getDebounce/clearDebounceAPIsetThrottle/getThrottle/clearThrottleAPIPush-Triggered Filtering (Phase 5):
mightWriteWeakMap: accumulated historical writes per actionpushTriggeredSet: tracks actions triggered by storage changes each cyclescheduleImmediately, first run, actual storage triggerTest plan
pullMode=falseverified via test🤖 Generated with Claude Code
Summary by cubic
Implements a pull-based scheduler that runs dirty dependencies in topological order, with cycle-aware convergence, parent–child action ordering, and performance controls. Ensures event handler inputs are current in pull mode, adds push-triggered filtering for precision, a Cell.pull() method for demand-driven reads, and replaces the legacy subscribe() boolean with an options object.
New Features
Migration
Written for commit bbe379a. Summary will update automatically on new commits.