I wear tights when it's under 9 degrees C and knee warmers between about 10 and 18 degrees C. The knees only come out when it's 19 plus with little or no wind chill. I have had knee pain in the past, mostly due to tight IT band. All muscles are looser and more flexible (and therefore less painful/less risk of injury) when they are warm so I try to keep them that way. Muscles surrounding the knee are small and seem to be more prone to injury through repetitive use/lack of stretching, than larger muscles, with the result that you are more likely to get knee pain, so it seems a good idea to keep them warm and to stretch your legs after riding.
You may not actually feel the cold that much on the legs, because the skin there is not particularly sensitive to cold, but the loss of power/speed when the leg muscles are cold is very noticeable to me. If you push hard or try to go at your usual pace when the muscles are cold they are more likely to ache afterwards from greater tearing of the fibres.