Some courses allow groups of five to play. Others only accept foursomes. If they weren't keeping up with the group in front of them, they definitely should have waived you in front.