Obviously one needs to stay warm, but I notice that my body is less flexible and not as fast in cold weather. This accounts for some loss of distance. (By cold I am speaking of 0*C or 32*F) This is probably due to both the additional clothing and the temperature.

Precept has some excellent info on the effects of wind and temperature on distance here:

http://www.preceptgolf.com/science/index.htm

Everything put together I have noticed distance losses as great as 20 yards compared to summer distances.