It comes just a few days after a judge ruled the lawsuit involving the Robert E. Lee statue could move forward.
RELATED: Obama's Charlottesville message becomes most liked tweet ever
Police say no incidents happened at the rally.
Mayor Mike Signer tweeted at the protesters, saying "Another despicable video by Neo-Nazi cowards. You're not welcome here! Go home!"
The rally lasted under 10 minutes.
Police officers from the City of Charlottesville then escorted a tour bus carrying the protesters out of the city to ensure they left the area.
RELATED: A timeline of the deadly weekend in Charlottesville, Virginia
Timeline of the deadly weekend in Charlottesville