The flag in question is not an American flag, it's the big rainbow flag that flies over the corner of Market and Castro streets.
Community activist Bill Wilson calls it a disgrace.
"I think it's disrespectful to the request that the president made to honor all 9/11 people who were killed," he said.
Wilson and blogger Michael Petrelis are taking on the merchants association that owns and controls the flag.
Petrelis didn't want to talk on camera, but he's been going after the merchants of upper Market and Castro over this issue for years. This time, the merchants association said the request wasn't submitted early enough to be considered.
"It doesn't explain the other problems we've had any time we've asked the flag to be lowered in honor of community members, in honor of international days," Wilson said.
The president of the merchant's association is the general manager of Cliff's Variety. She did not return ABC7 News' calls for comment. Another board member directed ABC7 News to the flag policy is on the association's website. It reads, "Consistent with the principles established by the flag's creators and with broad community support, the Flag flies almost always at full staff with very rare exceptions."
Last year, on the 10th anniversary of the attacks, the flag was lowered. This year it was not.
San Francisco resident George Woyames admitted he hadn't noticed the flag's position.
"I'm glad you called it to my attention to it; I think it would be a good idea to fly it at half staff," he said.
Others were more adamant about not lowering it.
"If we let people control the up and down of that flag, that flag would be up and down every other day for all different purposes; it's a crazy notion that that flag should go down," San Francisco resident Bill Eadie said.
Others didn't care.
"I don't think it matters; I still have full respect for all the people who have been lost in 9/11 but, I mean, it's just a flag," San Diego resident Irene Sweeney said.
"It really doesn't matter much to me; I'm not a flag waver," San Francisco resident Barry Venditti said.
The president's request was for federal government offices and agencies to display the stars and stripes at half staff Tuesday. Some businesses in San Francisco flew their flags at half staff, some did not.