I heard that joke too, but the version I heard, a man was golfing with his wife for their 40th anniversary. First tee, he confesses that he had an affair briefly near the beginning of the marriage. Around the 16th tee, the wife feels she has to confess something too. She's a man, and she had an operation done about a year before they met. Yadda yadda yadda, "You've been playing from the red tees?!?!?!" and that's the joke as I heard it.