Saleh Mohammed al-Awfi was among at least four people killed during a series of raids carried out in Medina and the capital Riyadh.
Just days earlier, Western governments warned of fresh attacks in the gulf state.
The US diplomatic missions were shut down for two days this month amid fears of an imminent threat to American government buildings.
Both Britain and Australia said they had been informed of attacks being planned.
Al-Awfi was slain along with another militant during a shootout with security forces who came upon them while searching seven locations in the holy city.
A third man was wounded during the gun battle and later arrested.
Simultaneously, police in Riyadh moved in on a house in the northern al-Massef district, killing two wanted gunmen and capturing another.
Officials said the remains of one or more people were found inside who appeared to have been killed “from a blast at close range.”
In total, nine arrests were made.
Al-Awfi, a former prison guard, is the latest of Saudi Arabia’s leading al-Qaeda operatives to be killed.
He was reported to have fought in Chechnya and travelled to Afghanistan to join al-Qaeda shortly before the September 11, 2001 attacks in the US.
Al-Awfi reportedly replaced Abdul Aziz al-Muqrin, the notorious militant chief who was shot dead in Riyadh in June 2004 shortly after his group posted gruesome images on the internet of beheaded American engineer Paul Johnson.
Mr Johnson’s execution marked the climax of a string of bloody attacks against Westerners in the kingdom.
Al-Muqrin was thought to have been succeeded by Saud al-Otaibi, one of 15 militants killed during a three-day gun battle with security forces in Al-Qassim, north of Riyadh, in April.
Moroccan-born Yunis Mohammed Ibrahim al-Hayari, who then took over, shared the same fate on July 3 after clashing with security forces in the capital.
Still at large and on Saudi Arabia’s most wanted list is Talib Saud Abdullah al-Talib.