By default, the number of worker subprocesses for map/reduce is based on the number of available cores. This can lead to OOM in two ways that I've found so far:
- The process that gathers all the finished pages outputs them in order, 1, 2, 3, ... N, .... If the reducer is waiting on page N, the mapper keeps putting more pages on the queue and the workers keep processing them, and the reducer adds them to the buffer N+1, N+2, ... for as many pages as get produced. If page N takes long enough, this will OOM - I have seen this happen for some configurations of memory & CPU capacity on EN wiki, for example. Naturally, once the reducer OOMs, the extraction stops forever.
A fix is for the mapper to stop providing more jobs once the reducer has a certain threshold of jobs already prepared, for example reusing the same maxsize that limits the size of output_queue. (Note that ordering_buffer in reduce_process is NOT limited by maxsize, nor can it be, as the reducer needs to accept all pages from the workers until it sees page N.)
- Each worker starts off as a clean, empty process. As it processes pages, it maps the templates in that page into its own memory. Those templates are the same as owned by the parent process, but - key fact here is that these are subprocesses, not threads - they represent a real cost in memory for each subprocess. Eventually it is possible for a long running extraction to have enough memory charged to each subprocess that the OS kills one of them for being OOM. This also stalls the extraction, since if the worker was working on page N, that page will now never be completed and the reducer will wait forever for it to come back.
I can't think of any good way for the workers to free up their memory, and neither can Claude. So the simplest fix is to have the reducers stop working after some number of pages, for example 1000, and the mapper creates new reducers to replace the old ones as they cycle out. This resets the memory usage of each reducer so that the entire extraction doesn't get bloated and stalled.
Any thoughts on either of these changes? I think they are necessary in order to fix occasional stalling issues that people have observed over the years, including myself in recent days while trying to use this in my own work. Alternate suggestions are also welcome.
By default, the number of worker subprocesses for map/reduce is based on the number of available cores. This can lead to OOM in two ways that I've found so far:
A fix is for the mapper to stop providing more jobs once the reducer has a certain threshold of jobs already prepared, for example reusing the same
maxsizethat limits the size ofoutput_queue. (Note thatordering_bufferinreduce_processis NOT limited bymaxsize, nor can it be, as the reducer needs to accept all pages from the workers until it sees page N.)I can't think of any good way for the workers to free up their memory, and neither can Claude. So the simplest fix is to have the reducers stop working after some number of pages, for example 1000, and the mapper creates new reducers to replace the old ones as they cycle out. This resets the memory usage of each reducer so that the entire extraction doesn't get bloated and stalled.
Any thoughts on either of these changes? I think they are necessary in order to fix occasional stalling issues that people have observed over the years, including myself in recent days while trying to use this in my own work. Alternate suggestions are also welcome.