For decades Miller ran what looked to be a successful real estate investment business. He came from a prominent family, was the treasurer of the local Boy Scout troop in Orinda, but when the real estate market turned south, Miller's investors found out their securities were anything but.
Miller lives in a mansion in Orinda. He took expensive vacations to Africa and Bali and sent his kids to private schools.
"The whole thing was a lie. This whole business was a lie," said Alan Weiss.
Weiss a former options trader on the Pacific Stock Exchange is too embarrassed to say how much he lost.
"Had I not met Mr. Miller, I would've been a lot better off… put it that way," said Weiss.
Weiss says he knew his money was gone in October of 2009 when miller sent him an email saying he took Weiss' money out of desperation, only intending to keep it for a week. "Apologies aren't enough" wrote Miller.
Weiss took his paperwork to the district attorney and others started showing up.
When asked what he thought he lost, photographer Doug Kohen said, "Oh, me personally, and my family? Millions… lots of money."
Kohen was another victim of the 175 investors. He says Miller's children went to expensive private schools, at the expensive of his own kids.
"Their college fund was torn up, all the monies that we had put aside for their college fund is gone," said Kohen.
On Monday, Miller's attorney Paul Wolf said, "Carl Miller is going to do everything reasonably within his power to make things right."
"I think it's too little too late," said Kohen.
The deputy district attorney prosecuting the case says there isn't any money left; a lot of it was spent on the house and a lavish lifestyle.
"It hard it's different when you lose money in the market, but when it's basically taken from you, stolen, it hurts. It's like having a car stolen… you feel violated," said Weiss.
Miller is in jail and a bail hearing is set for Tuesday morning. His in-laws are trying to put up real estate to get him out.
The deputy district attorney that is handling the case said Miller took out money out of the Orinda Boy Scout troop's account for no good reason, but he put the money back in. He took out $10,000 one time and another $6,000 on another occasion, but then put the money back in. By the way, that is still embezzlement, even if you don't take it out permanently.