I was doing the same thing for a while. In my case I was getting way too much body turn and lean in the back swing, then when I came down on the swing I wasn't fully turning back to center. Try staying centered on top of the ball, it may feel like you are only taking a short backswing, but in reality you are not.