The best way I know of to get the message across is to wait until the next time he starts talking like that... just back off, and stand there and stare at him until he gets the point.

I don't care if someone talks while I swing, as long as he just keeps on talking. What will disturb me is when he either starts or stops talking in the middle of my swing.