Panetta is from Monterey and spoke to the Commonwealth Club in San Francisco, where he was peppered with questions about the CIA's predator attacks in Pakistan.
The Pentagon operates unmanned drones above battle fields in Afghanistan. But the CIA has its own predator program, credited with targeting terrorists in Pakistan, where we are not at war.
On Friday, members of the Commonwealth Club asked Leon Panetta to explain.
"Well obviously I'm not going to talk about the operations we conduct," said Panetta.
But he said when the CIA uses predators to launch hell fire missiles, the targets are enemies of the U.S.
"And we have deliberately made sure that only those that represent those kinds of targets are the ones we're going to go after," said Panetta
Panetta denied Jane Mayers report in the New Yorker Magazine that said only six of 41 CIA drone strikes in Pakistan have targeted Al Qaeda.
"All I can tell you is it's not true," he said.
Security officers then asked the woman who had been shouting the questions to leave and then escorted out.
After the program ABC7 asked Panetta about reconciling the predator attacks with a long held U.S. policy against targeted assassinations.
"You know it's a war, we're at war with Al Qaeda, and they basically struck at our country on 9/11 and killed over 3,000 of our citizens. So we're at war with them, we're fighting them there, we're fighting them on the battlefield elsewhere. I think that's what this country has to do if we're going to defend ourselves," said Panetta. "I think we have to do what's necessary in order to defend ourselves and that's what we're doing."
In Berkeley, former CIA field officer Robert Baer says it's clear the policy's changed.
"They are committing assassinations, there is no other way to put it," said Baer.
And Baer said, hell fire missiles aren't precise and neither is the intelligence on the targets.
"I mean the Israeli's got into the same problems when they assassinated the wrong people, and that was on the ground with a pistol," said Baer.
Baer is talking about a hit on a suspected terrorist from the Munich Olympics, who turned out instead to be a Moroccan waiter.
"And the question is what is necessary, when do you become a target and when don't you. Is it because you have a bad thought or is it because you actually have American blood on your hands. And that is a question no administration has answered," said Baer.
Referring to the CIA controversy over water boarding, Leon Panetta said again if we get into an argument between the present and the past, we'll lose the future.
Baer says the military holds tribunals, even during wars, and says if the director is sincere about fostering trust in the agency, he should start with what happened to the missing video tapes of the water boarding interrogations.