The move came a day after the Saints allowed 47 points in a loss to the Washington Redskins.
Payton said the team arrived at the decision Monday afternoon. Senior defensive assistant Dennis Allen will take over Ryan's duties.
"He'll be handling everything from A to Z," Payton said.
Ryan was hired prior to the 2013 season with the hope of pairing a potent Drew Brees-led offense with at least a competent Saints defense, but the results have not been there.
"I'm disappointed for Rob it didn't work out," Payton said. "I'll tell you one thing now: He's a fantastic staff guy, respected greatly, not only in the locker room but by his peers. And we just had to look closely at some point to at least give this an option because the direction we were heading wasn't good. We really struggled with substitutions, getting lined up, getting guys on the field and being able to function."
New Orleans (4-6) ranks last in total defense and yards allowed per game (424.7) and is second-to-last in passing yards allowed per game (293.1). The 424.7 yards allowed per game would be the second-most in NFL history, behind the 2012 Saints.
Redskins quarterback Kirk Cousins torched the Saints' secondary for 324 yards and four touchdowns Sunday en route to a perfect 158.3 passer rating and a 90.1 Total QBR. New Orleans is on pace to shatter the NFL record for the highest opponents' passer rating allowed (116.5).
The Saints have allowed 14 touchdown passes the past three weeks. According to the Elias Sports Bureau, the Saints became the first team to allow at least four TD passes without an interception in three straight games.
Sunday's loss marked the second time New Orleans allowed more than 45 points in a game this season and the fourth time it allowed at least 30. The Saints are the only team in the league to average more than 30 points per game allowed (31.5).
Ryan is the first assistant coach Payton has fired in the middle of a season, and the decision comes as the team heads into its bye week. The Saints return to action Nov. 29 with a road game against the Houston Texans.
"Dennis and I have spoken at length and will continue to speak," Payton said. "I recognize it's a tough challenge mid-stride."
Defensive end Cameron Jordan responded to the move via Twitter:
Earlier Monday, Payton had shot down a report that Ryan had been fired and said no such decision had been made, but he didn't rule it out as a possibility.
Ryan, 52, had immediate success when he was hired as the Saints' defensive coordinator in 2013. He lifted the Saints to fourth in the NFL in total defense that year as they went 11-5 and won a playoff game.
But the Saints' defense fell back to 31st in 2014, and the team went 7-9. It was almost a surprise when Payton didn't fire Ryan after the season.
Payton said he liked Ryan's passion and work ethic, but he chafed at the constant assignment and alignment errors, which have led to several sideline shouting matches.
Instead of firing Ryan, Payton demanded the defensive coordinator simplify his complex playbook and weekly schemes. Also, Payton brought in Allen to help lead the secondary.
The Saints overhauled much of their defensive roster, which includes three rookie starters, but the defense has been plagued by injuries.
Over the past 12 years, Ryan, the son of former NFL coach Buddy Ryan and the twin brother of Buffalo Bills coach Rex Ryan, has been a defensive coordinator with the Oakland Raiders, Cleveland Browns, Dallas Cowboys and Saints. He was fired after two years in Dallas with similar complaints about his playbook being too complicated.
ESPN Saints reporter Mike Triplett contributed to this report.
Saints fire DC Rob Ryan
ESPN Saints reporter Mike Triplett explains why the team decided to part ways with defensive coordinator Rob Ryan and what the future holds for Ryan.