Someone on the shore reported a swimmer in distress at 3:43 p.m., according to a fire dispatcher.
An employee who answered the phone at the Cliff House said diners seated near the windows noticed a man in swim trunks on the rocks and that restaurant staff contacted the Coast Guard.
The fire dispatcher said personnel have arrived at the scene and confirmed that someone in the water needs assistance.
The Coast Guard is also responding with a helicopter and a 47-foot motor lifeboat.