Rousey won the honor despite Mayweather's historic victory in May over Manny Pacquiao.
Rousey, in accepting the award, said she wondered how Mayweather felt about getting "beat by a woman for once."
Mayweather, of course, has drawn headlines outside the ring for years over multiple domestic-violence incidents and arrests.
"I'd like to see you pretend not to know who I am now," Rousey said, referring to a controversial interview in which Mayweather, when asked if he could beat her in a fight, said he wasn't familiar with Rousey.
Rousey (11-0), who is fighting the undefeated Brazlian Bethe Correia on Aug. 1 in Rio de Janeiro, is coming off a 14-second submission win against Cat Zingano at UFC 184 on Feb. 28.
It was Rousey's fifth title defense and the quickest finish ever in a UFC title fight.
NBA MVP and Golden State Warriors star Stephen Curry won the ESPY for Best NBA Player and was also presented the award before the show. Curry was nominated in three other categories.
Victor Espinoza, who won the Triple Crown aboard American Pharoah, won the Best Jockey ESPY.
Fan voting for the 23rd annual ESPYS was conducted online and based on performances spanning the past 12 months.
The ESPYS was hosted by actor Joel McHale as top celebrities from sports and entertainment gathered to commemorate the past year in sports.
Rousey: 'I wonder how Floyd feels being beat by a woman for once'
UFC fighter Ronda Rousey accepts ESPYS award for Best Fighter, beating out Floyd Mayweather.