It all stems from a photo of the notoriously glum golfer dozing off and zoning out during an appearance at a school last April.
Dufner is taking it all in stride, admitting his laidback personality makes him an easy target.
"Guys make fun of me all the time for my weight and my hair and all this stuff; my aloof attitude. But, you know, I thought it was funny. You know, that's what every kid does in school," he said.
The 36-year-old says he's fine with being identified with Dufnering, but after his first major championship, he will likely become as well known for his golf game.