If you have checked the wheels spin easily, that when the chain is off the chainset that this spins, then as you say just shorts and legwarmers may be your problem, that would affect me.
Whilst we have some folk on here who would happily ride naked all year round, the cold will be getting to your thighs and body if you haven't got sufficient clothing. I'm running shorts and bib tights with windproof thighs and front. I also have windproof soft shell over a merino base layer. Buff around neck, skull cap, gloves and overshoes.
What are you wearing overall?