[PROTOTYPE/RFC] URL-based pattern loading with automatic dependency resolution

[jkomoros]

⚠️ PROTOTYPE FOR DISCUSSION - NOT INTENDED TO LAND AS-IS

This is a prototype/proof-of-concept implementation to demonstrate URL-based pattern loading. This PR is for discussion and feedback only, not ready to merge.

Problem Statement

Patterns in the recipes repo need to reference patterns in the labs repo, but currently this requires:

• Brittle symlinks between repositories
• Manual file copying
• No way to version or pin dependencies

Proposed Solution

Enable patterns to be loaded directly from URLs (e.g., GitHub raw URLs) with automatic resolution of relative imports.

What This Prototype Demonstrates

✅ Core Functionality Working

1. Direct URL Loading

   const charm = await addRecipeFromUrl(
     charmManager,
     "https://raw.githubusercontent.com/commontoolsinc/labs/main/packages/patterns/aside.tsx",
     "Pattern Name",
     {}
   );

2. Automatic Relative Import Resolution

   • Pattern imports ./counter-handlers.ts
   • System automatically resolves to full URL: https://.../counter-handlers.ts
   • Both files fetched and compiled correctly

3. Cross-Repository References

   • Recipes repo can reference labs repo patterns via URL
   • No symlinks or file copying needed

🧪 Tests Passing

All tests pass successfully:

• Unit tests: packages/charm/test/url-loading.test.ts (5/5 passing)
• Integration test: test-url-integration.ts (all scenarios working)
• Real patterns tested: aside.tsx, counter.tsx with dependencies

# Run tests
deno test packages/charm/test/url-loading.test.ts --allow-net --allow-read --allow-write --allow-env --allow-ffi --unstable-ffi --no-check
deno run --allow-all --no-check test-url-integration.ts

Technical Implementation

Files Modified (7 core files)

1. packages/js-runtime/program.ts - HTTP program resolver with cache busting
2. packages/js-runtime/typescript/resolver.ts - URL-based relative resolution
3. packages/js-runtime/typescript/compiler.ts - URL-aware module resolution
4. packages/js-runtime/typescript/diagnostics/checker.ts - Filter URL module errors
5. packages/js-runtime/typescript/bundler/bundle.ts - Preserve URLs in AMD output
6. packages/runner/src/harness/pretransform.ts - Skip hash prefix for URLs
7. packages/charm/src/commands.ts - New addRecipeFromUrl() API

Key Technical Challenges Solved

1. URL Corruption: Standard path operations (path.normalize()) corrupt URLs (https:// → https:/)
2. AMD Module Names: TypeScript strips URLs to short names; fixed with post-processing
3. Relative Imports: Path resolution doesn't work for URLs; use URL class instead
4. Type Checking: TypeScript rejects URL imports; filter specific diagnostic codes

Questions for Discussion

🤔 Architecture & Design

1. Caching Strategy: Currently fetches on-demand. Should we add local caching?
2. Version Pinning: How should we handle pattern versioning? Commit SHAs in URLs?
3. Security: Any concerns with fetching arbitrary URLs? Allowlist domains?
4. Error Handling: Should URL fetch failures be fatal or fallback to cached versions?

🤔 API Design

1. Function Naming: Is addRecipeFromUrl() clear enough? Alternative names?
2. Parameters: Should cache busting be configurable? Default on/off?
3. Return Type: Current behavior OK or need additional metadata?

🤔 Integration & Deployment

1. Performance: Network latency for pattern loading acceptable?
2. Offline Support: How important is offline development?
3. CDN/Proxy: Should we proxy through our infrastructure?
4. npm Packages: Should we support loading from npm instead of raw URLs?

🤔 User Experience

1. Error Messages: Are error messages clear when URL loading fails?
2. Discovery: How do users find available URL-loadable patterns?
3. Documentation: What examples/guides do we need?

Benefits of This Approach

✅ No Symlinks: Eliminates brittle cross-repo symlink management
✅ No Copying: No need to duplicate pattern files
✅ Versioning: Can pin to specific commits via URL
✅ Distribution: Share patterns via simple URLs
✅ Works Today: Functional prototype with passing tests

Known Limitations & TODOs

⚠️ Not Production Ready:

• No local caching layer
• Limited error handling
• No retry logic for failed fetches
• No allowlist/security controls
• Unclosed fetch response in tests (resource leak)

📋 Would Need Before Landing:

• Design review and consensus on approach
• Security review (arbitrary URL fetching)
• Performance testing with multiple patterns
• Caching strategy decision
• Error handling improvements
• Integration with existing pattern discovery
• Documentation for end users
• Fix resource leaks in tests

Documentation

• Full documentation: docs/URL_PATTERN_LOADING.md
• Implementation summary: URL_PATTERN_LOADING_SUMMARY.md
• Research findings: docs/FINDINGS_URL_PATTERN_LOADING.md

How to Test

# Unit tests
deno test packages/charm/test/url-loading.test.ts --allow-net --allow-read --allow-write --allow-env --allow-ffi --unstable-ffi --no-check

# Integration test
deno run --allow-all --no-check test-url-integration.ts

Request for Feedback

Please review and provide feedback on:

1. Is this the right approach? Are there better alternatives?
2. Architecture concerns? Any issues with the implementation strategy?
3. Security implications? Concerns about fetching from arbitrary URLs?
4. API design? Is the interface intuitive?
5. What's missing? What would need to change before this could land?

---

Again: This is a prototype for discussion. Not ready to merge.

Please feel free to experiment with it, break it, and provide honest feedback!

🤖 Generated with Claude Code

---

Summary by cubic

Prototype adds URL-based pattern loading with automatic resolution of relative imports so patterns can be loaded directly from URLs (e.g., GitHub raw). This enables cross-repo references without symlinks or file copying and includes tests and docs.

• New Features
  • Added addRecipeFromUrl API to load and run patterns from HTTP/HTTPS URLs with optional cache busting.
  • Enhanced HttpProgramResolver to fetch the main file and resolve relative imports via URL; includes canonical URL caching.
  • Updated TypeScript resolver/compiler to support URL modules and avoid path normalization issues; AMD bundler preserves full URLs in define/require.
  • Filtered “module not found” diagnostics for URL modules; pretransform skips hash prefixing for URL-based programs.
  • Added unit and integration tests for simple and dependency-based patterns, plus documentation outlining usage and architecture.

^{Written for commit 893890e. Summary will update automatically on new commits.}