A light weight Goretex shell garment should work well with a fleece inner jacket/sweater or pullover. I wear a Sunmountain goretex zipup jacket over a fleece pullover myself on chilly/wet/windy days and it's great. As long as it fits comfortably and doesn't restrict your body movement, go for it.