It's the first action taken against the councilmember since a sexual misconduct investigation began.
The City Council, except Councilmember Bien Doan, voted to excuse at least one of those absences from Oct. 8 after Torres said he missed because he was sick.
He has said he does not intend to resign, but city leaders now taking action against him.
"As a father that has children, I believe that somewhere we need to make a strong stance that is appropriate to our community and our constituents out there," Councilmember Doan said.
RELATED: SJ Councilmember Omar Torres responds to being served warrant for child sex misconduct
Torres is now removed from his board and commission appointments and the Mayor has temporarily appointed other councilmembers to fill those roles until January.
Torres serves in many different capacities as a councilmember, including as a liaison for the San Jose Library Commission.
The chairs of the commission wrote a letter supporting Torres' removal, saying the allegations are deeply concerning - particularly for someone serving on a commission which supports hundreds of thousands of children and families.
Tuesday's action from the council is about accountability for the community.
"I believe Councilmember Torres is holding his seat hostage and is denying representation to the 100,000 residents of District 3," Mayor Matt Mahan said during the council meeting. "We've all called on him to resign, he should resign."
Torres is being investigated for alleged sexual activities involving minors.
A search warrant affidavit revealed graphic text messages between Torres and a young man, in which Torres discussed sexual acts with minors.
RELATED: SJ community reacts to Councilmember Torres being investigated in alleged sexual misconduct
Torres said in a statement through his attorney, that the messages were just fantasy and role play and denied allegations of legal wrongdoings.
The council says they intend to let the legal investigation play out, but Doan believes how Torres is viewed in the court of public opinion makes him unfit to continue to serve the City of San Jose.
It's absolutely affecting all of us," Doan said. "And, again, as a parent, I'm appalled and disgusted reading the content. And I think it's important that, somewhere we're going to have to stand our ground and do the right thing."
The council cannot vote to remove Torres from his position as city councilmember despite the allegations.
However, a recall election could be held to remove him. Another option is, if he misses five consecutive city council meetings his seat can be considered vacant.
ABC7 News has previously reached out to Torres multiple times since the news of the investigation broke, but he has not responded.
His attorney spoke to ABC7's media partner, the Mercury News, saying he "has no plans to resign."