O'Neill had been deliberating whether to accept a disciplinary judge's recommendation that Officer Daniel Pantaleo be fired for using a banned chokehold.
"None of us can take back our decisions, especially when they lead to the death of another human being," O'Neill said, adding it was clear that Pantaleo "can no longer effectively serve as a New York City police officer."
Related: Reaction swift on both sides after officer fired in Eric Garner's death
Asked whether Mayor Bill de Blasio forced his hand, O'Neill said the dismissal was his choice.
"This is the decision that the police commissioner makes," he said, calling Garner's death an "irreversible tragedy" that "must have a consequence."
An emotional O'Neill noted he was a beat cop for 34 years and said it could easily have been him in that position. He said Pantaleo used justifiable force when Garner resisted but erred when he kept Garner in the chokehold once the two men tumbled to the ground.
Watch Police Commissioner James O'Neill's full announcement here:
The dismissal means Pantaleo can collect the money he put into his pension but loses other benefits he would have retained upon retirement.
Police Benevolent Association President Pat Lynch spent much of his fiery press conference blasting Mayor de Blasio and NYPD Commissioner O'Neill saying that the job of being a police officer is "dead." Lynch said O'Neill "ran to the corner with the loudest crowd."
The PBA chief called for no confidence vote in the mayor and NYPD commissioner, saying, "The leadership has abandoned ship."
Pantaleo's lawyer Stu London said he plans to file appeal of the decision to fire him.
The NYPD flag on the wall seen during the PBA press conference was upturned because the union says police are in "distress."
The scathing 46-page opinion issued by Judge Rosemarie Maldonado in the officer's disciplinary trial has been released, and she blamed Pantaleo for a "glaring dereliction of responsibility that precipitated a tragic outcome."
She also said Pantaleo was "untruthful" and "self-serving" during the course of the investigation.
The report sent to O'Neill recommended that Pantaleo be dismissed from the department, calling his accounts of what happened that day implausible.
"There is overwhelming evidence that Respondent used a prohibited chokehold, as defined by the 2014 Patrol Guide," Maldonado wrote. "Respondent's use of a prohibited chokehold was reckless and constituted a gross deviation from the standard of conduct established for a New York City police officer."
As his decision neared, ranking officials in the police department told ABC News that O'Neill was acutely aware of the national significance of his choice. A street cop at heart, O'Neill was a rare apolitical commissioner serving at the pleasure of a mayor running for president and promising "justice" for the Garner family.
O'Neill had reportedly come to view Pantaleo as decent person and officer who made a very big mistake that caused public outcry. O'Neill had been hoping to find a middle ground that punished but didn't professionally sacrifice Pantaleo for the sake of making up for a DA who didn't charge.
O'Neill's decision to terminate Pantaleo came as a surprise even to key members of the department's top brass, who watched the news as it was announced on TV, several police sources told ABC News.
O'Neill kept his own counsel and the decision, as was clear, was his and something he worked through largely alone.
Video of what happened that day in July of 2014 played an important role in the judge's report.
The judge not only considered the arguments from Internal Affairs but the defense, saying that Pantaleo was disingenuous when he viewed the video and denied using a chokehold prohibited by NYPD policy.
She said Pantaleo "knew of the grave risks associated with prohibited chokeholds, and that by using one under these circumstances, he engaged in a gross deviation from the standard of conduct established for a New York City police officer."
While the chokehold itself did not kill Garner, the decision found the "chokehold was a significant factor in triggering the asthma attack that contributed to Mr. Garner's death."
However, Pantaleo's conduct lacked intent, the departmental judge found.
"Even though (Pantaleo) recklessly used a prohibited chokehold, the evidence was insufficient to prove that he did it with the intent of obstructing Mr. Garner's breathing," she wrote.
Still, she said Garner was not blameless.
"Whatever Mr. Garner's objective, it is clear that for at least four minutes into the recorded portion of the encounter, and at least one minute prior to that, he refused to cooperate with the arrest and comply with lawful orders," Maldonado's decision said. "From the outset, Mr. Garner was non-compliant and argumentative."
She wrote that while officers are permitted to use reasonable force to take an uncooperative suspect into custody, they are not allowed to use a chokehold.
"The totality of circumstances favors a finding that even though Respondent recklessly used a prohibited chokehold, the evidence was insufficient to prove that he did it with the intent of obstructing Mr. Garner's breathing," she wrote. "There is only one appropriate penalty for the grave misconduct that yielded an equally grave result, (Pantaleo) can no longer remain a New York City police officer."
* Follow us on YouTube
* More local news
* Send us a news tip
* Download the abc7NY app for breaking news alerts