perf(sync-service): Use ETS tables instead of maps in Filter to reduce GC pressure

[robacourt]

Note: This is currently in draft because it's entirely AI generated and I need to check it :)

Summary

Migrates the Filter module and its namespace (WhereCondition, EqualityIndex, InclusionIndex) from
Elixir maps to ETS tables to reduce garbage collection pressure with large numbers of shapes
(200K+).

Problem

With 200K shapes being added and removed frequently, the map-based implementation causes large GC
delays due to copying immutable map data structures on every add/remove operation.

Solution

Store filter data in ETS tables (outside the process heap) instead of nested Elixir maps:

• 5 private ETS tables per filter: shapes_table, tables_table, where_cond_table, eq_index_table,
  incl_index_table
• Same algorithmic complexity: O(1) for equality lookups, O(tree depth) for inclusion index
• API unchanged: All existing tests pass without modification

Performance Results (200K shapes)

Scenario	ETS Implementation	Original (maps)	Target
Equality index lookup	5.75µs	4.41µs	<100µs ✅
Multiple matches (20)	13.82µs	9.07µs	<100µs ✅
Post-churn performance	9.49µs	4.69µs	<100µs ✅

The ETS implementation meets the <100µs target for affected_shapes with 200K equality-indexed
shapes.

Key Changes

1. filter.ex: Changed struct to hold ETS table references; operations mutate ETS in-place
2. where_condition.ex: WhereConditions now identified by refs and stored in ETS
3. index.ex: Changed from protocol dispatch to direct function dispatch
4. equality_index.ex: Stores entries in ETS with keys {where_cond_id, field, value}
5. inclusion_index.ex: Stores tree nodes in ETS with keys {where_cond_id, field, path}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 58.88%. Comparing base (accd2a0) to head (1ec78ae).
⚠️ Report is 1 commits behind head on main.

❗ There is a different number of reports uploaded between BASE (accd2a0) and HEAD (1ec78ae). Click for more details.

HEAD has 6 uploads less than BASE

Flag	BASE (accd2a0)	HEAD (1ec78ae)
unit-tests	6	4
packages/experimental	1	0
typescript	4	2
packages/typescript-client	1	0

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #3547       +/-   ##
===========================================
- Coverage   75.21%   58.88%   -16.34%     
===========================================
  Files          51       37       -14     
  Lines        2744     1396     -1348     
  Branches      404       68      -336     
===========================================
- Hits         2064      822     -1242     
+ Misses        678      574      -104     
+ Partials        2        0        -2     

Flag	Coverage Δ
electric-telemetry	22.71% <ø> (ø)
elixir	57.74% <ø> (+0.35%)	⬆️
elixir-client	74.47% <ø> (+0.52%)	⬆️
packages/experimental	?
packages/react-hooks	86.48% <ø> (ø)
packages/typescript-client	?
packages/y-electric	55.12% <ø> (ø)
typescript	63.44% <ø> (-24.03%)	⬇️
unit-tests	58.88% <ø> (-16.34%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[robacourt]

benchmark this

[magnetised]

Great stuff!

I'm trusting the tests to validate the functionality -- haven't reviewed the algorithm itself. Can we cull the value-less comments and docs though pls.

[msfstef]

This looks pretty good but a bit hard to review - in the meantime might be worth running a benchmark on it as well?

[robacourt]

benchmark this

[github-actions]

Benchmark results, triggered for 7fc5c

• write fanout completed

• unrelated shapes one client latency completed

• many shapes one client latency completed

• concurrent shape creation completed

• diverse shape fanout completed

[netlify]

✅ Deploy Preview for electric-next ready!

Name	Link
🔨 Latest commit	1ec78ae
🔍 Latest deploy log	https://app.netlify.com/projects/electric-next/deploys/69370b216da66b00086b4ef7
😎 Deploy Preview	https://deploy-preview-3547--electric-next.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.