Cool
Yep, the caliper should not swing from side to side, there is usually a single nut and bolt that runs through the front fork and through the bracket between the rear stays. Try giving that a quick tighten.
Brakes seem daunting but there are two tricks to be aware of.
1. On the top of the caliper, just to one side, will be a tiny screw hole called the centre adjustment screw...it may take an alen key or a cross head driver. This adjusts how central the calipers are...tighten it up and the arms move one way, loosen it and they move the other. If you find one arm closes on the rim faster than the other, turn this until both arms are kind of centred.
2. When you fit your brake blocks, you can often get a squeal from them under heavy braking. This is caused by a thing called "toe in"...it may be spelt tow in I'm not an expert. Toe in is the amount of "angle" that the blocks have. Making the leading or trailing end of the block angle in, towards the rim, will solve the squealing.
Somebody will be along in a moment to tell you that the blocks should be parallel and I am sure, in an ideal world that should be. However these tiny tweaks will make the brakes feel like new and are worth "fettling" with each time you clean the bike.
Oh and there is a trick to getting the blocks close to the rim when adjusting them so that they don't rub and so that your brake levers don't have to travel loads to have good effect.