How much force are you applying to create that movement? (I assume that you are holding the front wheel between your legs to stop it moving sideways?) Can you get that movement easily by pulling one-handed on one end of the bars? That should NOT be possible. If you have to use both hands, one pushing hard on one end of the bar, and the other pulling hard on the other end, that doesn't sound worryingly loose since that is way more force than you would ever exert when actually riding the bike.
Make sure that the circular cap on the top is pushing down on the top of the stem NOT on the top of the steerer tube. It is not there to hold the stem on, but to control how hard the stem is pressing against the headset so that the bearings are loaded properly. As he pointed out, you need to make sure that both of the bolts on the stem are below the top of the steerer but that might mean that the top cap isn't pushing down on the stem itself. If that is the case, add a short spacer above the stem which the cap will then push down onto the stem.