Just checking that it's not empty can mean that filling it is in
progress, so we need to enter the critical block if that's the case.
This shouldn't have much (any?) performance implication as size(),
like empty(), takes constant time.