The boy's mother told police that at about 5 p.m. that day, she was holding the boy's hand while in the Sporters team apparel shop with her husband, police said.
She then felt a tug from the boy and let go of his hand, thinking that her husband had taken the boy, police said.
The mother then checked on her family members and saw the suspect walking out of the store holding the boy's left hand, police said.
She yelled at the suspect, who was only about 10 feet away from her, and he let go of the boy's hand and fled into the mall, police said.
The boy was unhurt, police said.
All police officers on duty in Milpitas were sent to the Great Mall and assisted mall security with setting up a perimeter at all entrances to the mall, police said.
Police officers walked through the mall for an hour but did not find the suspect, police said.
The suspect is described as an Asian male, aged 50 to 60 years old, 5 feet 4 inches to 5 feet 6 inches tall and weighing about 200 pounds, police said.
Police have released a composite sketch of the suspect.
"We're looking for anyone at the Great Mall who was there or who recognizes him to give us a call," police spokesman Sgt. Raj Maharaj said.
Anyone with information about the incident may call the Milpitas Police Department at (408) 586-2400, or give information anonymously at (408) 586-2500.