Skip to content

feat(extract): opt-in --html-as-code to route .html through the AST path (#1230)#2096

Open
Yyunozor wants to merge 1 commit into
Graphify-Labs:v8from
Yyunozor:feat/html-as-code
Open

feat(extract): opt-in --html-as-code to route .html through the AST path (#1230)#2096
Yyunozor wants to merge 1 commit into
Graphify-Labs:v8from
Yyunozor:feat/html-as-code

Conversation

@Yyunozor

Copy link
Copy Markdown
Contributor

Closes #1230.

Problem

.html is always classified as DOCUMENT (detect.py's DOC_EXTENSIONS), so it only goes through the semantic (LLM) extraction path — never the local AST path, even under --code-only, where it's simply skipped. For a project whose .html carries real logic (Angular templates, generated dashboards, anything with meaningful inline <script>), that logic is invisible to the graph:

Without the html templates, the graph loses a lot of its value. (@mfallouh-TS)

The workaround suggested in-thread (.graphifyignore the .html files, @LuisPeregrina) just removes them from the graph entirely rather than making them useful.

Fix

Opt-in --html-as-code flag on graphify extract, off by default (zero behavior change for existing users):

  • detect.classify_file() takes an html_as_code param; when set, .html classifies as CODE instead of DOCUMENT, checked before the DOC_EXTENSIONS/paper-heuristic branch so an opted-in file isn't second-guessed by the paper sniffer.
  • New extract_html() mirrors the existing extract_vue/extract_svelte/extract_astro extractors: mask everything outside inline <script> bodies (preserving newlines, so source_location stays correct in the host file), then feed the masked source to the plain JS grammar. No new dependency — reuses the masking technique already shipping for three other markup+JS host languages, instead of adding tree-sitter-html.
  • <script src="..."> (external, no local body) and non-JS type="..." blocks (e.g. application/json) are blanked, not parsed — they contribute nothing rather than producing garbage nodes.
  • The flag persists into update/watch/hook rebuilds via the existing .graphify_build.json mechanism (same pattern as --exclude/--no-gitignore, --exclude patterns only apply to the initial scan — update/watch/hook rebuilds silently re-include excluded paths #1886/Allow extracting git-ignored code: add --no-gitignore opt-out #1971), and is threaded so .html's registered extractor can't leak into the unrelated doc-AST-fallback path in watch.py when the flag is off.

Validation

  • New tests: masking edge cases (no <script>, external src=, non-JS type=, multiple blocks, call-graph + line numbers), the opt-in contract (off → unchanged, on → extracted), persistence across update.
  • Full suite, branch vs. parent HEAD (e32c9f4): 3395/3381 passed, 1/1 failed (same pre-existing, unrelated: missing optional build dev dependency), 137/137 skipped — delta is exactly the 14 added tests. ruff check clean.
  • Real-world smoke test: a ~4,100-line HTML template with 113 embedded functions (canvas rendering + UI logic) produced 0 nodes by default and 113 correctly-attributed function nodes plus their calls edges under --html-as-code — e.g. updateCamera() calling matPerspective() — with line numbers matching the source.

Scope

Deliberately minimal: JS extraction, not full HTML/DOM structure. The CSS-out-of-scope position from discussion #1625 is untouched. Structural HTML parsing is a natural follow-up on this foundation, not bundled here.

…ath (Graphify-Labs#1230)

.html is always classified as DOCUMENT (detect.DOC_EXTENSIONS), so it only
goes through the semantic (LLM) path, and is simply skipped under --code-only.
For a project whose .html carries real logic (Angular templates, generated
dashboards, meaningful inline <script>), that logic never enters the graph.

Adds a --html-as-code flag to `graphify extract`, off by default:

- detect.classify_file() takes an html_as_code param; when set, .html
  classifies as CODE instead of DOCUMENT.
- New extract_html() mirrors extract_vue/extract_svelte/extract_astro: mask
  everything outside inline <script> bodies (preserving newlines so
  source_location stays correct in the host file), then parse the masked
  source with the plain JS grammar. No new dependency.
- <script src="..."> (external) and non-JS type="..." blocks are blanked,
  not parsed.
- The flag persists into update/watch/hook rebuilds via the existing
  .graphify_build.json mechanism, with .html explicitly excluded from the
  unrelated doc-AST-fallback path in watch.py when the flag is off (that
  path's `_get_extractor(p) is not None` check would otherwise silently pick
  up .html once it has a registered extractor, regardless of the flag).

Full suite: 3395/3381 passed vs. parent (delta = the 14 new tests), same
single pre-existing unrelated failure both sides. ruff clean.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Option to parse html files as code

1 participant