This isn't a yes or no. Prevention is better than cure - being physically fit, good nutrition etc will help prevent colds - plus avoiding those sneezing etc and hand washing good hygiene. Before a longish ride (>3 hours) I usually take 500 mg of vitamin C - previously I found I started sneezing after long/intensive cycles and frequently got colds - the excessive exercise impairs your immune response for sa short period. If you have a cold avoid excessive exercise training but some gentle exercise is probably beneficial. If you have flu - rest is probably what's needed.
If you have 'flu you will be in bed shivering then sweating fit to bust and will certainly not feel like exercising.
We have a regime of washing hands as soon as we enter our house and we get very few colds. When you are out in pubic you pick up viruses from door handles etc, which you will transfer to surfaces in your home. You then rub your eyes or pick your nose and the viruses are in and multiplying in the damp warm environment of your mucous membranes. If someone sneezes over you the best advice is to breathe through your mouth because saliva is geared up to dealing with foreign viruses and bacteria.
The worst cold I've had in years was kindly given to me by an optician fitting me for contact lenses; he didn't wash his hands, just swished them under the tap then stuck them in my eyes. Bingo.
Fitness probably does help; they say you get around ten colds a year but don't notice most of them because your immune system deals with the virus. Losing the habit of rubbing eyes and picking noses is the best way to avoid colds though.