Skip to content

Emit a bare file node for deliberately-skipped data .json so it stays discoverable (without reintroducing #1224) #2108

Description

@dmitryvostryakov

Summary

Data-shaped .json is deliberately skipped by extract_json (#1224 — AST-walking it exploded into orphan key-nodes), which is the right call for structure. But the current skip is total: the file contributes no node at all, so the path is absent from graph.json entirely and is unreachable from query / explain / affected.

Requesting a middle setting: emit a single file node for skipped data JSON — no key traversal, no children, no edges — so the file is discoverable without reintroducing #1224's node explosion.

Why this matters (concrete case)

Our repo runs two implementations of the same underwriting engine (Python and a Java port) and pins their agreed behaviour in shared parity corpora:

  • benchmarks/stateful_corpus.json
  • benchmarks/stateless_corpus.json

These are the contract between the two implementations — CI has dedicated parity, parity-java and parity-java-stateful jobs that consume them. They are arguably the most decision-relevant files in the repo for any "do these two implementations agree?" question.

Neither has ever appeared in our graph. Our project convention (and our CLAUDE.md) routes codebase questions through graphify query first, so the practical result is that a parity question returns a confident, well-formed answer assembled entirely from the implementations, with no indication that the file defining the agreed behaviour exists at all.

That is the part worth weighing: the failure is not an empty result, which would prompt a fallback to grep. It is a plausible result with a silent hole. Same class of concern as the "if someone uses the warning's silence as evidence their corpus is fully covered" point raised in #1666.

The same applies to other data JSON that is meaningful at file granularity — i18n catalogues (portal/src/locales/{en,ru}.json), Grafana dashboards, Postman collections.

Proposal

For a .json that fails manifest recognition, return exactly one node:

{
  "id": "<path_qualified_id>",
  "label": "stateful_corpus.json",
  "file_type": "data",
  "source_file": "benchmarks/stateful_corpus.json",
  "source_location": "L1"
}

No key nodes, no nested traversal, no edges — the file node only. On our repo that is 8 nodes total, against the 561 orphan nodes #1224 was closed to prevent.

Benefits:

  • query / explain can surface that the file exists and where it lives.
  • Co-location edges (if any are derived elsewhere from directory structure) put it near the code that consumes it.
  • It composes with Ruby: stable subset of files yields zero nodes in full-repo runs (0.9.6) — each extracts fine in isolation #1666's warning: a file node is exactly the "every extractable file produces at least a file node" invariant quoted in that thread, so these files stop being anomalous and the warning stops firing on them — which would resolve the false-positive I filed alongside this.

If a blanket file node is too broad, gating it behind a flag (--index-data-json) or a size ceiling would still cover the case, though the value is highest when it is on by default — the whole problem is that the gap is invisible unless you already know to look for it.

Alternative considered

Adding a probe key to our corpora (e.g. $schema) to make them manifest-recognised. Rejected: it would pull them down the manifest path and produce misleading dependency-shaped structure, and it asks every repo to mangle its data files to be visible to the tool.

Environment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions