perf(sync-service): Use ETS tables instead of maps in Filter to reduce GC pressure #3547
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #3547 +/- ##
==========================================
- Coverage 75.24% 75.20% -0.04%
==========================================
Files 51 51
Lines 2743 2743
Branches 404 405 +1
==========================================
- Hits 2064 2063 -1
- Misses 677 678 +1
Partials 2 2
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
|
benchmark this |
magnetised
left a comment
magnetised
left a comment
There was a problem hiding this comment.
Great stuff!
I'm trusting the tests to validate the functionality -- haven't reviewed the algorithm itself. Can we cull the value-less comments and docs though pls.
| @env Env.new() | ||
|
|
||
| @doc """ | ||
| Check if the index for a field is empty. |
There was a problem hiding this comment.
Smells like claude. Do we need the "empty? function checks if empty" type docs?
| meta_key = {where_cond_id, field, :meta} | ||
| :ets.insert(table, {meta_key, {type}}) | ||
|
|
||
| # Sort and deduplicate the array |
|
|
||
| defp optimise_where(%Expr{eval: eval}), do: optimise_where(eval) | ||
| @doc false | ||
| def optimise_where(%Expr{eval: eval}), do: optimise_where(eval) |
There was a problem hiding this comment.
why did you make this public?
|
This looks pretty good but a bit hard to review - in the meantime might be worth running a benchmark on it as well? |
|
benchmark this |
|
Benchmark results, triggered for 7fc5c
|
Note: This is currently in draft because it's entirely AI generated and I need to check it :)
Summary
Migrates the Filter module and its namespace (WhereCondition, EqualityIndex, InclusionIndex) from
Elixir maps to ETS tables to reduce garbage collection pressure with large numbers of shapes
(200K+).
Problem
With 200K shapes being added and removed frequently, the map-based implementation causes large GC
delays due to copying immutable map data structures on every add/remove operation.
Solution
Store filter data in ETS tables (outside the process heap) instead of nested Elixir maps:
incl_index_table
Performance Results (200K shapes)
The ETS implementation meets the <100µs target for affected_shapes with 200K equality-indexed
shapes.
Key Changes