This was very similar to what happened in Hawaii over the weekend.
TIMELINE: Hawaii false alarm missile scare
Residents in Japan with the NHK app got a message on their phones telling them to evacuate inside a building or go underground.
Coincidentally today the US and Canada plan to host talks in Vancouver over the North Korean nuclear threat. South Korea and Japan are expected to participate.