"Maybe this is just me being oversensitive, but doggone it, if you're going to go, that's really tough, but leave us something," Madden told SiriusXM NFL Radio's Hall of Fame show hosts Joe Horrigan, James Lofton and Howard Balzer on Wednesday night.
"Leave us something here in Oakland. Please."
Madden coached the Raiders from 1969 to 1978 and won the franchise's first Super Bowl before embarking on a broadcasting career.
Raiders owner Mark Davis plans on moving the Raiders to a $1.9 billion, 65,000-seat dome stadium with natural grass in 2020, while playing in Oakland for at least the next two seasons with 2019 still in question.
Madden, who turned 81 on Monday, said he was "surprised" by how quickly the vote came to fruition.
"I'm not sure they have that whole deal together yet," Madden said. "I'm not sure that they even know exactly what the stadium is, how many, where it's going to be and all those things.
"For some reason, they jumped into that thing quickly."
Davis announced his intentions to move to Southern Nevada last April, and there are 750 million reasons why the league did not let the deal die on the vine. The $750 million via a hotel tax is the largest public subsidy ever for a stadium.
Still, as Madden indicated, while the Raiders have identified a site, a 62-acre plot on Russell Road, west of Interstate 15 and the Mandalay Bay Resort and Casino on the south end of Las Vegas, there is still no lease, design plans or FAA approval, considering the relative proximity of the site to McCarran International Airport. The league wants to see progress on each of those topics, as well as where the team plans on playing in 2019.
Plus, Madden was confused by the league's apparent double-standard in approving relocation to Las Vegas but preparing to fine players who appeared at an arm-wresting competition in a casino there last weekend.
"I would hate to be a coach to take a team in there," Madden said. "I would hate to have my team in Las Vegas on Saturday night before the game ... every team has a bottom 10."
Madden was referring to the "bottom 10" players on the roster in terms of behavior. "I think there's going to be a lot of problems like that," he said. "I'd stay in Mohave [Arizona] or something."
Proponents of Las Vegas have pointed out the trappings of such existing NFL markets as New Orleans, Miami, New York and Los Angeles.
Madden said the Raiders' pending move to Las Vegas hurts more than when they went to Los Angeles in 1982 "because of the finality of it." The thought was when the Raiders departed then, he said, the Oakland Coliseum was still relevant enough to attract another team.
"With the [state of the] stadium now, when they move out, that's going to be torn down, and it's going to be a high rise or some doggone thing. There'll be no more Oakland Raiders," he said. "There will be no more history of the Oakland Raiders, and that really bothers me.
"Boom, it just goes away."
A team Hall of Fame and accompanying museum are planned for the Las Vegas complex. And Lofton, who played for the L.A. Raiders in 1987 and 1988, told Madden that fans would still follow the team.
"Yeah, but if you still live here [in the Bay Area] and you have to drive by that area, which, by the way, I felt was the best location of a stadium in the NFL, and you have to drive by there and you see something there like a shopping mall and a high-rise apartment, it's going to make you sick, I think."