Re-institute async mode

[joshleblanc]

This is less of a "this is ready to be merged", and more of an effort to reinvigorate conversation around this feature. I cobbled this together as a proof of concept because I simply couldn't use the multi-process approach on the server I'm trying to deploy to. This is related to #343, and tries to naively reimplement this

I originally stumbled on this feature because I was testing Kamal on a 1GB VPS. The initial deploy would work, however there wasn't enough resource headroom for the second deploy. I narrowed this down to SQ using ~100MB per process, essentially taking up half the available memory on the system.

After implementing this change, I saw the application memory drop from ~700MB to ~270MB.

The usage is simply placing this in puma.rb

if ENV["SOLID_QUEUE_IN_PUMA"]
  plugin :solid_queue
  solid_queue_mode :async
end

I currently have this deployed to a 1GB droplet - the application is running quickly, background jobs are running, and redeploys are succeeding

[ka8725]

@joshleblanc thanks for this amazing job! I installed the fork in an app hosted on Heroku that servers webhooks from a live application. I've run it as a separate process though (as a worker). My Procfile:

web: bin/rails server -p $PORT -e $RAILS_ENV
worker: bin/jobs --mode=async

So far it looks very good:

I used to run the worker before without recurring jobs because it would consume all RAM. Now, even with recurring jobs it takes ~2x less RAM! Thanks for sharing this work!

[headius]

I'd like to help get this finished up, so that non-forking implementations like JRuby can use Solid Queue. Is it complete enough that I could try to run on JRuby now and see what's missing? Do you have an example I could try?

[headius]

See #679 for my recent request to have a non-forking implementation.

[CufeHaco]

Just give me a bit.more time on the JEP-380 socket PR. I can talk to Cruby, C, or any other language using jruby as the layer via Linux Sockets.

Separation of concerns. Let both interpreters run as independent processes and communicate via unix sockets and message pack.

I should having it ready by this evening/tomorrow morning.

[CufeHaco]

https://github.com/CufeHaco/solid-queue-jruby
https://github.com/CufeHaco/solid-queue-jruby/blob/main/BENCHMARKS.md

[rosa]

Hey folks! Going to work on this one soon. Thanks for the help so far.

[rosa]

Hey @joshleblanc, @ka8725, I've pushed a bunch of changes to bring this to the finish line. If you have time, could you try running this in your apps to see if there are any problems? I'll do the same on Friday with one of our apps, as I've refactored a bunch of things that could also affect the regular fork mode. Then, if things are looking good, I'll merge this and release a new version.

No rush! Just whenever you have time.

[ka8725]

Upgraded the gem in the app, it's not under load now. So need time to collect data. Will respond tomorrow.

[ka8725]

Hey @joshleblanc, @ka8725, I've pushed a bunch of changes to bring this to the finish line. If you have time, could you try running this in your apps to see if there are any problems? I'll do the same on Friday with one of our apps, as I've refactored a bunch of things that could also affect the regular fork mode. Then, if things are looking good, I'll merge this and release a new version.

No rush! Just whenever you have time.

It looks great: