Start simply. A pad of A4 cartridge paper, a few graphite pencils of varying grade, a good putty eraser (it won't tear holes in the paper) and something to use as a board - a wooden tea tray will do if you have one. That's the beauty of art compared to a lot of other hobbies - you really don't need much to get started.
Do you want to stick with pencils? Or try watercolours, acrylics, pastels? I would suggest avoid being tempted by the big selection boxes for now. Start small, with a few basic colours, and gradually build up from there, whatever medium you choose. Either way, don't stint on your equipment - stay away from the very cheap stuff (brushes, paint, and, in some circumstances, paper) as that will be an exercise in frustration. I learnt that the hard way...
Find a subject matter that appeals to you. Landscapes, bowls of fruit, portraits, even racing cars. Whatever floats your boat. It's much more fun drawing and painting something you enjoy than busting your gut painting something you don't. But do pay attention to the basics. There are a lot of good "how to" books knocking around - I really like the Alwyn Crawshaw ones, even though they're a bit long-in-the-tooth these days - but the bottom line is, there's no right way or wrong way of doing things.
And don't dismiss the technique of "squaring up". It really does help.