fix(app): make the Streamlit front end run again

[izzet]

Why

streamlit_app.py has not run since f102ef8 (2025-10-21, 162 commits ago). It fails before rendering a single widget, and nothing in the suite imported it, so the rot was invisible.

Every one of these is a hard failure on develop:

Breakage	Detail
dask_jobqueue signal handler	dask_jobqueue/runner.py calls signal.signal(SIGINT, ...) at import. Streamlit runs scripts off the main thread, where that raises ValueError: signal only works in main thread. import dftracer.analyzer fails in any embedder that isn't the main thread.
from dftracer.analyzer.rules import KnownCharacteristics	rules.py no longer exists
constants.XFER_SIZE_BIN_LABELS	renamed to SIZE_BIN_LABELS
result.characteristics	AnalysisResult has no such field
threshold= hydra override	no longer a config key
assets/logo.png	never tracked in git
no preset override	AnalyzerConfig.preset is MISSING, so the run could not have started
swapped labels	file_count shown as "# of Processes", proc_count as "# of Files"
bottlenecks	never assigned; the tab rendered None
undeclared deps	streamlit and altair appear nowhere in packaging

What

cluster.py — make the dask_jobqueue import typing-only. The three classes were used solely in the ClusterType union; Hydra instantiates them from their _target_ strings, so they were never needed at import time. This is the fix that makes the package importable off the main thread; it also drops a heavy import from every dftracer.analyzer load.

streamlit_app.py — rewritten against the current API. It now renders the JSON output payload (output=json), the same structure tests/test_e2e.py::test_json_output_file already asserts, so the UI tracks a tested schema instead of reaching into internals that move underneath it.

• uploader accepts .pfw.gz only — uncompressed .pfw parses to nothing (fix(tests): gzip the trace data inside the stale dftracer fixtures #63), so offering it invites a silent empty result
• preset selector (posix, dlio, dlio-prev, generic) — previously absent, and required
• results persist in st.session_state, so they survive a widget interaction
• failures surface via st.error instead of a blank page
• warns explicitly when a trace loads with zero events
• worker count pinned to 1 by default, with the reason in the help text: multi-worker reads can silently truncate (Multi-worker read silently drops events: per-worker iter_arrow_dfanalyzer_all() returns an empty/truncated batch set #69), and understated numbers in a UI are worse than a slower run

pyproject.toml — new web extra: pip install ".[web]".

tests/test_streamlit_app.py — a smoke render test (3.2s, catches every import breakage above) and a full test that uploads a real trace and asserts the counts.

Removed, deliberately

• Bottlenecks tab — built on result.characteristics, which no longer exists. The nearest current equivalent is result.analysis_facts, but that is gated behind facts.enabled and has its own schema; wiring it up is a follow-up, not a silent substitution.
• Transfer-size distribution charts — depended on XFER_SIZE_BIN_LABELS. I checked a live run: flat views for these presets contain no size_bin columns, so there is no data to plot. Better absent than fabricated.
• Threshold slider — threshold is not a config key anymore.
• Logo — assets/logo.png was never in the repo.

Verification

Driven headlessly with AppTest, and served for real with streamlit run --server.headless:

Check	Result
First render	0 exceptions; title, uploader, preset selector, Analyze button present
Real server	/_stcore/health → ok, page → HTTP 200, no errors in the server log
Single file (dftracer-posix)	2,056 events / 1 process — matches tests/test_e2e.py
8 files (dftracer-dlio-prev, dlio-prev preset)	18,039 events / 8 processes, runtime 56.70s — matches the e2e ground truth and the README's Job Time 56.695s
Layer breakdown	renders for both perspectives
Existing suite	pytest -m smoke green on this branch
CLI	dfanalyzer --help and both entrypoints still resolve; cluster=slurm|lsf|pbs targets still resolve via Hydra

Resource limits for hosted deployment

Streamlit Community Cloud guarantees 690 MB of memory (2.7 GB ceiling) and 0.078-2 CPU cores. The app did not fit, and nothing stopped a user submitting a trace that could not complete. Three limits, measured rather than guessed:

One in-process worker. n_workers was already pinned to 1 (for #69); this adds processes=False so the worker stops being a separate process re-importing pyarrow/pandas/dftracer. With 2 cores at most it bought no parallelism. Peak RSS, identical settings otherwise:

fixture	before	after	reduction
dftracer-posix (2,056 events)	489 MB	280 MB	1.7x
dftracer-dlio-prev (18,039 events)	601 MB	443 MB	1.4x

Both clear the guaranteed allocation once the ~54 MB Streamlit server is added.

An explicit worker memory_limit. dask otherwise derives one from the host's RAM and core count, which in a container is the host's numbers — on a 94 GiB / 40-core machine it autodetects 2.35 GiB against a 690 MB allocation, so the worker had no ceiling at all. Verified it does not backfire: no pause, spill or restart warnings on either fixture, golden values unchanged (2,056 events / 1 process; 18,039 / 8).

An upload cap in a new .streamlit/config.toml (there was none, so Streamlit's 200 MB default applied) and again across a whole submission in the app, because maxUploadSize is enforced per file while the uploader accepts several. Over the cap is an error naming the number and pointing at the CLI, not an OOM with no explanation. A test asserts the two numbers stay in agreement.

The cap is 2 MB. Peak RSS is ~290 MB plus 11-17 MB per 1,000 events, so ~450 KB fits the guaranteed allocation and ~2.7 MB needs the best case; a 3.8 MB trace measured 3,499 MB. The hard cap sits at the best case with a warning at 450 KB: refuse what cannot complete, warn about what merely might not. (The sibling wisio app uses 16 MB; its read path is far lighter, so the number does not carry across.)

All three paths verified: 3.8 MB blocked with 0 metrics rendered, 981 KB warned and analyzed, 326 KB clean. pytest -m smoke green (157 passed).

Known weaknesses, documented rather than silently accepted: a byte cap is a poor proxy for a gzipped trace, since compressibility decides how many events those bytes become — bounding event count would be accurate but requires reading the trace first. And CPU will likely bite before memory does; these timings are from a 40-core host.

Interaction with #68

Merges clean into develop today. If this lands first, #68 will conflict on two files. Both resolutions are mechanical:

• streamlit_app.py — keep the rewrite. It already satisfies the review point on chore!: drop Meson and build as a pure Python package #68 (.pfw.gz only), so chore!: drop Meson and build as a pure Python package #68's one-line uploader change is redundant.
• .github/workflows/ci.yml — combine: chore!: drop Meson and build as a pure Python package #68 drops the meson -Csetup-args, this PR adds the [web] extra, so the merged line is pip install ".[web]" with no setup-args.

If #68 lands first instead, this branch rebases onto it the same way.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 30.26%. Comparing base (b4d0299) to head (23fabad).
⚠️ Report is 7 commits behind head on develop.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop      #71      +/-   ##
===========================================
+ Coverage    29.91%   30.26%   +0.35%     
===========================================
  Files           30       27       -3     
  Lines         3771     3522     -249     
===========================================
- Hits          1128     1066      -62     
+ Misses        2643     2456     -187     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.