I wear a headband made of fleece, it is larger where the ears are. Also, I use a triangular bit of fleece material too help keep the cold out of my crotch on really cold rides.
That's what I do.Skull cap for the head and ears - and a buff for the neck area, and pull it up if your nose is cold.