refactor: rename datasource command group to webscraper#184
Merged
Conversation
WalkthroughThe CLI and docs rename the Apify integration from ChangesWeb scraper CLI and Apify integration rename
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Possibly related PRs
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Greptile SummaryThis PR renames the
Confidence Score: 5/5Safe to merge once the backend PR renaming the route mount from /datasources to /webscraper is coordinated and lands together. The change is a consistent rename across all touched files with no logic regressions. The route-level 404 guard, the not_connected vs. route-miss disambiguation, and the auth bridge error flow are all preserved correctly. The one new piece of logic (substring-matching a bridge error message to select the user-facing warning) works correctly for the current error set; it is a maintenance concern, not a current defect. No files require special attention beyond ensuring the backend route rename lands in the same deploy window. Important Files Changed
Sequence Diagram%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
participant User
participant CLI as insforge webscraper apify
participant OSS as OSS Backend /api/webscraper
participant Apify as Apify API
User->>CLI: connect
CLI->>OSS: POST /api/webscraper/apify/cli-start
OSS-->>CLI: "{ type: "authorize", authorizeUrl }"
CLI->>User: Open browser to authorizeUrl
User->>Apify: OAuth authorize
CLI->>OSS: Poll /api/webscraper/apify/connection
OSS-->>CLI: "{ kind: "connected", connection }"
CLI->>OSS: GET /api/webscraper/apify/token
OSS-->>CLI: "{ accessToken }"
CLI->>Apify: apify login --token token
Apify-->>CLI: Logged in
CLI->>User: Apify connected
User->>CLI: login
CLI->>OSS: GET /api/webscraper/apify/token
OSS-->>CLI: "{ accessToken }"
CLI->>Apify: apify login --token token
Apify-->>CLI: Logged in
CLI->>User: Authenticated
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
participant User
participant CLI as insforge webscraper apify
participant OSS as OSS Backend /api/webscraper
participant Apify as Apify API
User->>CLI: connect
CLI->>OSS: POST /api/webscraper/apify/cli-start
OSS-->>CLI: "{ type: "authorize", authorizeUrl }"
CLI->>User: Open browser to authorizeUrl
User->>Apify: OAuth authorize
CLI->>OSS: Poll /api/webscraper/apify/connection
OSS-->>CLI: "{ kind: "connected", connection }"
CLI->>OSS: GET /api/webscraper/apify/token
OSS-->>CLI: "{ accessToken }"
CLI->>Apify: apify login --token token
Apify-->>CLI: Logged in
CLI->>User: Apify connected
User->>CLI: login
CLI->>OSS: GET /api/webscraper/apify/token
OSS-->>CLI: "{ accessToken }"
CLI->>Apify: apify login --token token
Apify-->>CLI: Logged in
CLI->>User: Authenticated
Reviews (2): Last reviewed commit: "Preserve bridge remediation on connect f..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/commands/webscraper/apify/connect.ts`:
- Around line 115-119: The catch in connect() is collapsing every
runApifyAuthBridge() failure into the generic “run apify login” warning, which
loses the bridge-specific next step. Update the catch around runApifyAuthBridge
so it distinguishes the malformed-token error coming from the Apify bridge in
apify-bridge.ts and shows the correct follow-up for connect, while keeping the
existing generic message for other failures. Use the existing connect() flow and
clack.log.warn path to preserve the appropriate remediation text.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 218368bc-7554-46e9-9141-1185073fa2e0
📒 Files selected for processing (10)
README.mdsrc/commands/webscraper/apify/connect.tssrc/commands/webscraper/apify/login.tssrc/commands/webscraper/index.tssrc/index.tssrc/lib/api/apify-token.test.tssrc/lib/api/apify-token.tssrc/lib/api/apify.tssrc/lib/api/oss.tssrc/lib/apify-bridge.ts
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
2 issues found across 10 files
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Renames the CLI command group
datasource→webscraper(follow-up to the merged #179). The integration is web-scraping-only, sowebscraperreads truer than the genericdatasource.Changes
insforge datasource apify connect/login→insforge webscraper apify connect/login(provider sub-levelapifykept for future providers e.g. Firecrawl)src/commands/datasource/→src/commands/webscraper/,registerWebscraperCommands/api/datasources/apify/*→/api/webscraper/apify/*Verification
npm run build✓,npx tsc --noEmitclean for changed filesapify,apify-bridge,apify-token,skills)insforge webscraper apify --helpworks; olddatasourcecommand removedCross-repo dependency
Depends on the InsForge OSS backend renaming the route mount
/datasources→/webscraper(PR forthcoming) and the insforge-skills web-scraper rename. Merge together — the CLI now calls/api/webscraper/apify/token, which the renamed backend serves.🤖 Generated with Claude Code
Summary by cubic
Renamed the CLI
datasourcegroup towebscraperand updated API paths, docs, and messages. The connect flow now preserves detailed remediation from the auth bridge on failure.Refactors
insforge webscraper apify connectand... login; removed thedatasourcegroup.src/commands/datasource/*tosrc/commands/webscraper/*;registerWebscraperCommandsreplacesregisterDatasourceCommands./api/webscraper/apify/*.Migration
insforge webscraper apify connectandinsforge webscraper apify logingoing forward./api/webscraper/apify/*; older/self-hosted backends will show “web scraper not available.”Written for commit 2341f82. Summary will update on new commits.
Note
Rename
datasourceCLI command group towebscraperdatasourcetowebscraperin src/index.ts, so all Apify commands now live underinsforge webscraper apify./api/datasources/apify/tokento/api/webscraper/apify/token./api/webscraperpaths, with a new error message referencing "web scraper".webscraperinstead ofdatasource./api/datasources/apify/tokenendpoint anddatasourceCLI commands are no longer referenced; clients using the old path or command will not get the updated error handling.Macroscope summarized 2341f82.
Summary by CodeRabbit
webscrapercommand namespace (Apify first).webscraper apify connect/logincommands.