Speaking to journalists in Sydney about the economic crisis, the presenter reportedly called the prime minister a "one-eyed Scottish idiot".
Gordon Banks, Labour MP for Ochil and South Perthshire, said the comment was "unforgivable".
Number 10 would only say that Clarkson "is entitled to his own interpretation of the economic circumstances".
The spokesperson declined to comment on the specific insult about Mr Brown, who lost the sight in one eye after an accident as a teenager.
Mr Banks called on the BBC to make their presenter apologise for the remarks. Scottish Labour leader Iain Gray joined Mr Banks in criticising Clarkson.
Clarkson is in Australia to host Top Gear Live, a stage version of the hit BBC show.
'Lying'
The controversial presenter compared Mr Brown to Australian Prime Minister Kevin Rudd, shortly after Mr Rudd had addressed the country on the severity of the global financial crisis.
According to The Australian newspaper, Clarkson said: "He [Rudd] genuinely looked terrified. The poor man, he's actually seen the books.
"[In the UK] we've got this one-eyed Scottish idiot.
"He keeps telling us everything's fine and he's saved the world and we know he's lying, but he's smooth at telling us."
The comments provoked anger in Scotland.
Mr Gray said: "Such a comment is really a reflection on Jeremy Clarkson and speaks for itself.
"Most people here are proud that the prime minister is a Scot and believe him to be the right person to get the UK through this global economic crisis."
I'm here because of stewart lee
2k7johnnyboy 2 months ago 100
Hammond should take clarkson's dick out of his mouth before he talks
MrOuseburn72 3 weeks ago 17