The incident occurred at about 6 p.m. in the area of 450 E. Calaveras Blvd., Milpitas police Sgt. Raj Maharaj said.
The AC Transit driver had pulled up to a bus stop and a woman there appeared to be trying to board the bus, Maharaj said.
However, a man stopped her and forcefully pulled her away toward the back of a nearby business complex, according to Maharaj. No weapon was seen.
The bus driver called 911 and police responded and searched for the pair but didn't find them.
Investigators obtained video footage from surveillance cameras on the bus, which backed up the driver's account of what happened, Maharaj said.
The woman is described as Asian and in her 20s, wearing light-colored clothing. The man is also described as Asian, and is believed to be in his 30s. He was wearing a black jacket, jeans and a dark-colored baseball cap, police said.
Police are still investigating the circumstances of the incident and are asking anyone with information on the case to call them at (408) 586-2400.