Holder was assigned to the New York City Police Department's PSA 5, which polices the city's public housing developments.
Holder and his partner responded to a report of shots being fired at around 8:30 p.m. near a public housing development in East Harlem. The officers encountered a man with a stolen bicycle on a pedestrian overpass who fired a single shot, police said. Holder, 33, was struck in the head. His partner fired on the susect, who had ditched the bike and fled down the river path near the highway, police said. He was caught several blocks away with a gunshot wound to his leg, NYPD Commissioner Bratton said.
VIDEO: NYPD names Tyrone Howard as suspect in shooting death of officer
He called the officer an "extraordinary individual" and said in a news conference that his heart broke for his father.
"Our hearts are heavy and we offer our thoughts and prayers to his family who are experiencing unimaginable pain as we saw when we met with them earlier this evening," Mayor Bill de Blasio said.
Holder was a native of Guyana. He worked in the NYPD division that polices the city's public housing developments. His father and grandfather were police officers in Guyana, Bratton said.
De Blasio said Holder, who joined the force in July 2010, had an "exemplary record" as a police officer.
VIDEO: Suspect in custody after NYPD officer shot, killed
In honor of Holder, all flags were lowered to half-staff immediately until the day of Holder's interment, by order of de Blasio.
Dozens of Holder's fellow officers stood outside the hospital early Wednesday morning and saluted as the ambulance carrying their fallen comrade left. Afterward, many embraced one another.
"Tonight, he did what every other officer in the NYPD does when the call comes - he ran toward danger," Bratton said. "It was the last time he will respond to that call."
PHOTOS: NYPD officer shot, killed in East Harlem
The shooting in Manhattan's East Harlem neighborhood happened while Holder and his partners were responding to a call of shots fired.
"He is the fourth New York City police officer murdered in this city in the last 11 months," New York City Police Department Commissioner William Bratton said during a press conference at Harlem hospital where Holder was pronounced dead Tuesday night. "That's about as bad as it gets.
Nationwide, 100 officers have died in the line of duty so far in 2015. NYPD Officer Brian Moore, 25, was killed on May 9 during a patrol. A suspect was charged with murder. And at the end of 2014, on Dec. 20, officers Wenjian Liu, 32, and Rafael Ramos, 40 were ambushed and killed by a man who said he wanted to kill some cops in Brooklyn. The suspect killed himself in a nearby subway station.
The New York Police and Fire Widows' and Children's Benefit Fund - Answer the Call announced Wednesday that $25,000 will be donated to Holder's family. "Our thoughts and prayers are with his family and the families of all the men and women of the NYPD who put their lives on the line every day for us," said Stephen Dannhauser, chairman of the New York Police and Fire Widows' and Children's Benefit Fund.
Click here for full coverage on the fatal shooting of NYPD Officer Randolph Holder.