Interesting point, fella.
My instinct is telling me that this is probably related to our tendency to break time and space into discrete chunks, whereas in reality time and space doesn't work that way.
You can count the minutes and measure the miles and the litres in a container, but these numbers don't actually exist outside of our arbitrary definitions of those units, whereas physically distinct objects can be counted, and thus, can be fewer.
"I have 9 coins. You have 10. Therefore, I have fewer coins, and less money."
I blame Zeno of Elea. It's all Greek to me...