This is because the dataspace coordinates change on limit change, which causes a change in limits. We could define `tightlimits` or so but I don't think it would change the fundamental issue - something has to cycle back to the axis/scene.