Weight loss is a seriously expensive game!
+1, and potentially addictive...
However, imho the best and most economical way to do it is to start with the wheels and tyres. The reason is unless you pay serious (£2500+) money you will not get top quality light wheels on a stock bike - so getting good wheels can be an investment that will last through bike upgrades. Separately tyres and tubes are the best bang per buck weight saving components.
If you have V brakes
these are super light, bomb proof wheels (at ~1570g excluding skewers), and OUTSTANDING value at £245 a pair - I don't expect this price to last for much longer. They are also UST wheels, so you can run it either with tube (no rim tape needed) or tubeless (which would be handy if you want real light tyres because adding 70g of sealant per wheel should stop most punctures for light xc). I would have bought them myself if I didn't have a pair of Crossmax already.
As Cubist said Schwalbe's Nobby Nic is a light tyre. Actually imho Schwalbe has the best selection of light mtb tyres. It might also be worth looking up Furious Fred, Rocket Ron and Racing Ralph since they have different characteristics. Note they all come in different varaints - wired, folding, tubeless etc. At around 300g each Furious Fred 2.0s and Maxxis Maxxlite 1.95 are amongst the lightest mtb racing tyres money can buy. It is not unusual to find stock mtb bike tyres weighing 700g a piece.
If you don't want the mess of running tubeless, you can get tubes at half a normal tube's weight. They need a bit of care mounting, but personally I am not a believer that thick tubes do anything to defend against puncture in use - if a thorn is through the tyre no tube is going to stop it (bar sealant protection - but which is best used tubeless anyway to eliminate the weight of the tubes altogether). Maxxis Flyweight is probably the lightest tube commonly available on the market, I have some weighing 90g.
Most low/medium price mtbs will come with wheelsets weighing 1.5kg or more on top of the above. I am pretty sure most people will notice the difference.