Merged server capacity logic

[foodprocessor]

• Handle intermittent size updates by calculating and storing an offset when the size changes, and adding that offset to the tracked size in the interim.
• Handle overuse and fairness using a state machine and a dynamic estimate of the number of servers sharing the total capacity.

There are three states:

• Normal:
  • Set utilization target at 90% of bucket capacity / server count.
  • If bucket is > 92% full, change to Overuse state.
• Overuse:
  • Reduce utilization target to 86% of bucket capacity / server count.
  • If bucket is < 88% full, change back to Normal state.
  • If bucket is > 97% full, change to Emergency state and increment server count estimate.
• Emergency:
  • Display actual bucket usage.
  • Data is evicted aggressively.
  • New servers may fail to write due to insufficient available capacity.
  • If bucket is < 88% full, change back to Normal state.

[jfantinhardesty]

Is this still possibly a data loss issue if the bucket stats don't update in time, which you may have noticed recently?

[foodprocessor]

Since we're saving an offset, rather than the bucket usage value itself, the changes tracked by size_tracker will update the responses to StatFs. So, we shouldn't get into runaway eviction.