"The new McCarthy looks a lot like the old McCarthy," Swalwell told reporters at a press conference Wednesday morning. "The old McCarthy abused political power to punish and demean and smear his political opponents, and the new McCarthy in Washington today is doing the same."
Swalwell, who served as a manager on the second impeachment trial of former president Donald Trump, was joined by Congressmembers Adam Schiff (D-CA) and Ilhan Omar (D-MN), who McCarthy also stripped of their committee assignments.
"The three of us have decided to stick together because this isn't about any one committee assignment. This is about an institution where the Speaker of the House is using his power to go after his political opponents and to pick them off the field," Swalwell said.
MORE: Kevin McCarthy clinches votes for House speaker, breaking through GOP opposition on 15th ballot
McCarthy elected House speaker in rowdy post-midnight vote
Schiff also played a prominent role in the impeachment of former President Trump. He told reporters McCarthy's decision was political payback against him and Democrats, who removed Republican congress members Marjorie Taylor Greene and Paul Gosar from their committee assignments in 2021 after embracing politically violent rhetoric on social media.
McCarthy had vowed retribution should Republicans win control of the House.
"In a bitter bargain to get the speakership, he is slowly undermining each part of the work of this institution," Schiff said.
Speaker McCarthy announced that he was planning to remove Swalwell from the House Intelligence Committee during a press conference Tuesday night, citing concerns about national security.
"Let me be very clear, this is not anything political," McCarthy said.
McCarthy alluded to a 2020 report in Axios that reported Swalwell was targeted by a Chinese spy as part of an influence campaign in 2014, before he served on the intelligence committee. The report said that after Swalwell was alerted to the woman's true identity, he cut off ties. But Republicans have continued to raise concerns about the association.
"I'm sorry. Swalwell, you all know, does not have, or should have the responsibility to serve on the Intel Committee," McCarthy said.
Swalwell responded to those allegations saying he did nothing wrong.
"The FBI said three times, all I did was two things: I helped them over and over, and I was never suspected of wrongdoing," Swalwell said. "It's purely political vengeance."
ABC News political director Rick Klein says after McCarthy's narrow battle for speaker, he likely had to make good on a promise to his party.
"Call it retribution, call it payback, or call it the promise he made when running, I think he had to go and do this," Klein said of McCarthy.
But he said doing so could have a lasting impact.
"I think this is going to be something in the future that is set as a precedent," Klein said. "And if Republicans are in control they're going to deny some Democrats seats. And if the Democrats are in control they're going to deny Republicans some seats."