Police used water cannon when scuffles broke out in front of the Bourse, which has become a symbolic rallying point for people to pay their respects to those who died in Tuesday's attacks. Black clad men carrying an anti-Islamic State group banner with an expletive on it trampled parts of the shrine, shouting Nazi slogans. Ten were arrested and two police officers injured.
"We had 340 hooligans from different football clubs who came to Brussels and we knew for sure that they would create some trouble," Police Commissioner Christian De Coninck said. "It was a very difficult police operation because lots of families with kids were here."
PHOTOS: Explosions rock Brussels
Brussels mayor Yvan Mayeur expressed his disgust, with Belgium still in mourning over the suicide bombings at Brussels airport and subway, which killed at least 31 people and injured some 270.
"The police were not deployed to protect people from these hooligans but a whole other threat," said Mayeur told RTL television.
People trying to pay their respects were also dismayed.
"It was important for us to be here symbolically," said Samia Orosemane, a 35-year-old comedian. But, she added, "there were lots of men who were here and doing the Nazi salute, shouting 'death to Arabs' and so we weren't able to get through."
"We are all here today for peace, and for the brotherhood among peoples. Not for right-wing ideas. It's neither the time nor the place," said Theophile Mouange, 52.
RELATED: Officials say Paris bombmaker among dead in Brussels attack
Federal prosecutors, meanwhile, said Sunday morning's raids were linked a "federal case regarding terrorism" but did not specify whether it had any links to the March 22 attacks.
Thirteen raids were launched in the capital and the northern cities of Mechelen and Duffel. An investigating judge was to decide later whether to keep the four in custody. Five were released after questioning.
Tuesday's bomb attacks are also tearing at the fabric of the government, justice system and police, and Belgium's interior minister sought Sunday to contain the growing criticism of the government's handling of the tragedy.
Interior Minister Jan Jambon conceded Sunday that decades of neglect had hampered the government's response to violent extremism. He said the government has invested 600 million euros ($670 million) into police and security services over the past two years but that Belgium's justice system and security services are still lagging behind.
Jambon, whose offer to resign Thursday was declined by the prime minister, also acknowledged some shortcomings prior to the attacks.
"There have been errors," he said on VRT television.
Jambon said it takes time to hire anti-terror specialists and specialized equipment and insisted that the government's new investments need time before they become visible to the public.
As international pressure on Belgium has mounted for serving as an unwitting rear-base for extremist fighters who launched the Nov. 13 massacres that left 130 dead in Paris, the government has felt forced to defend its choices and the actions of investigators. Lawmakers, meanwhile, are demanding an inquiry.
Belgian police and the army have been deployed, sometimes around the clock, at major buildings and sites in the capital in increasing numbers since November, when Brussels went into lockdown over fears that top Paris suspect Salah Abdeslam had returned and was hiding there.
As it turned out, Abdeslam did return, but police did not find and arrest him until March 18, four days before suspects from his network exploded suicide bombs in Brussels.
Belgian investigators have been slammed for not questioning Abdeslam long enough or hard enough after he was shot in the leg during his arrest. Police have also been criticized for taking too long to get to Zaventem airport on Tuesday morning after two suicide bombers blew themselves up there - and left an even bigger third suitcase full of explosives that did not go off.
Jambon and Justice Minister Keen Goens were grilled by lawmakers Friday over how authorities failed to arrest suicide bomber Ibrahim El Bakraoui before he blew himself in the packed departure hall at Brussels Airport.
Turkey has said that Bakraoui - whose brother Khalid was the suicide bomber at the Maelbeek subway station on Tuesday - was caught near Turkey's border with Syria in 2015 and Ankara had warned Brussels and the Netherlands that he was "a foreign terrorist fighter." Belgian authorities said they did not know he was suspected of terror-related activities until after he was deported to the Netherlands.
Jambon also said the Brussels subway network had been told to shut off services around 20 minutes before the attack at the subway station, which is close to both the European Union headquarters and the U.S. embassy. He did not fully explain why it was not closed in time, raising more questions about the efficiency of Belgium's security services.
On Sunday, Italian police in the southern city of Salerno said they had arrested an Algerian wanted in Belgium for an alleged false ID crime ring. Djamal Eddine Ouali was arrested Saturday in the town of Bellizzi, said Luigi Amato, the head of Salerno police's anti-terrorism squad. Ouali, 40, was being held in jail while authorities expect extradition procedures to soon begin.
Belgian prosecutors said Sunday that the man is thought to have made false documents for some of the attackers in the Nov. 13 massacre in Paris, including top suspect Salah Abdeslam. Investigators are trying to establish whether the same false ID ring provided papers for the March 22 attackers.
Meanwhile, Dutch prosecutors say police have arrested a 32-year-old Frenchman in the port city of Rotterdam at the request of French authorities who suspect him of "involvement in planning a terror attack."
Prosecutors said in a statement Sunday that French officials requested the arrest Friday after the suspect emerged during an investigation.
The suspect, whose identity was not released, is expected to be extradited to France soon.
Click here for full coverage on the deadly bombings in Brussels.