if the movement is at the hub then that's too much.
If it's at the rim then it depends whether you are holding it as in pushing it, or moving the bike from side to side. I can move my front rim 2mm (which is where the brake is currently set) by gripping it firmly, but the rim is completely worn out, and, in anycase, it doesn't move more than half a millimetre in when in motion.