Officers responded to a 7:30 p.m. call from a neighbor in the 500 block of West Ninth Street who said the pit bull was "killing" another dog, Gilroy police said.
The dogs were in the backyard of a home, but the homeowner, when contacted, denied owning the dogs or knowing where they came from, police said.
Police sprayed the pit bull with pepper spray, but it did not stop the attack.
Concerned for the safety of children and bystanders in the area, one of the officers shot the pit bull, police said. The dog died at the scene.
The smaller dog, which appeared to be seriously injured, was taken to a veterinarian. Police did not have an update on its condition.
The owner of the pit bull was later identified and cited for failure to restrain the dog.