Now, a new report released by the city's budget and legislative analyst estimates that there are more than 40,000 units sitting vacant.
[Ads /]
It's a finding that's caused some lawmakers to look into a vacancy tax as a possible solution to the housing shortage.
"They passed an empty homes tax in Vancouver and saw almost immediately within the first year of implementing it over 20 percent drop in the number of vacant units," said Supervisor Dean Preston.
Preston's office is drafting legislation for such a tax in order to get more people into those 40,000 homes - many of which, Preston says, are owned by wealthy corporations or investors.
"They're just buying it, holding on to it until the market goes up and selling it for profit. They have no interest in providing homes for folks," Preston said.
[Ads /]
But some housing experts say that 40,000 number is misleading, and that the real amount is actually much lower.
"We see about 8,000 units are long-term not occupied that are problematic," said Todd David, the executive director of the Housing Action Coalition.
VIDEO: SF metro area remains most expensive place to live in US
SF metro area remains most expensive place to live in US
David says of those 40,000 units, about 32,000 of them fall into categories that would most likely be exempt from a vacancy tax.
For example, properties that have been recently bought or sold and the new tenants haven't moved into yet, or ones where landlords are currently doing renovations or upgrades.
RELATED: Wealthy Bay Area city pushes back against new housing proposal
"It will not move the needle on our affordability or displacement crisis in any meaningful way," he said.
David says what the city really needs to do is build more housing.
"In order to keep the cost of housing to just be increasing at the same rate as inflation, you need to be adding about two percent to our housing stock per year," he said.
[Ads /]
David says that comes out to about 5,000 new units a year.
But for the past 30, we've only built an average of 1,800 annually, leaving us in a 100,000 unit deficit.
A problem, that David says, won't be fixed overnight.
"This is a 20-year plan. But we need the will. We need to start doing it," David said.