Stems in the "normal" ranges ~80-120mm, the handling difference is essentially not noticeable if you make smallish changes i.e. 10-20mm IME. Once you start putting silly <80mm stems or 130mm+ stem's on or jumping 40mm in length in one go, I agree with your observations, you start to get a twitchy or sluggish feeling to the bikes handling depending on which way you go (relatively speaking). With regards to OP, unless the stem supplied on the 50 cm frame was right on the limit of "normal", a +/- 20 mm stem would be unlikely to produce a significant steering difference IMO. Especially not to the degree is aids someone transition to riding road bikes.
This is just my experience though, like you said, you have a different experience.