Education Minister Christopher Pyne has declined six times to back Treasurer Joe Hockey's comments that the poor "don't have cars or actually don't drive very far''.

Mr Pyne insisted on Friday that he "wasn't going to cop any criticism of Joe Hockey" and said that the Treasurer had the government's ''full support'' .

He then refused - six times - to back the Treasurer's comments on the fuel excise's impact on poor people during an interview with the Nine network.

Fairfax Media revealed on Friday that Mr Hockey's colleagues and political allies have rounded on the Treasurer, questioning his judgment and the quality of advice he is receiving after the disastrous comments.

Mr Hockey has lost three key staff from his 17-person office in the nine months since the federal election, including the second and third most senior people in his office.

On Friday, Mr Pyne said Mr Hockey had been ''standing up for low income workers; and middle income workers''.

"The simple fact is that this is the guy who has driven the abolition of the carbon tax, the cost of which fell heaviest on low-income households,'' he said, adding that Mr Hockey had also opposed Labor's move to scrap a self-education tax deduction and the changes to fringe benefit tax changes for cars.

"What he is talking about is the fuel excise changes, which are a 40 cents a week impost on households on average,'' Mr Pyne said.

But Labor transport spokesman Anthony Albanese swiftly pointed out that Mr Pyne had "had six opportunities to support what he said and you refuse to do so".

"What he said was rubbish, if you are in an outer suburb or a regional community you have no choice but to drive and this is the guy who cut all funding for public transport in the budget,'' Mr Albanese said on Friday.

"He [Mr Hockey] should go to western Sydney today and have a look at people driving all the way in to the city, or all the way into their workplace in any capital city from the outer metropolitan areas, they spend a higher proportion of their income than high income earners."

Opposition Leader Bill Shorten said on Friday that Mr Hockey's comments and the backing from ministers such as Mr Pyne showed that the Abbott government was out of touch with ordinary people.

Mr Hockey's gaffe has capped a difficult three months in which the Treasurer has been under fire for puffing on a cigar and, days later, dancing with his son on budget night before delivering wide-ranging budget cuts, taking a holiday to Fiji during a key parliamentary sitting, complaining that everyone in the media was against him and co-operating with a biography that revealed he wanted an even tougher budget.

Fairfax Media contacted more than a dozen of Mr Hockey's supporters in the ministry and party machine; political staff; and staff to former treasurer Peter Costello on Thursday about this week's problem. All expressed surprise, concern and dismay about Mr Hockey's remarks.

Furious ministerial colleagues turned on the Treasurer over comments they variously described as "stupid and wrong", a "bad example of how to make a point" and "loose language".

Several people close to Mr Hockey's office, who asked not to be named, said he was taking advice from an increasingly small circle of advisers - particularly chief of staff Grant Lovett and press secretary Mike Willesee.

The loss of the experienced trio of deputy chief of staff Creina Chapman, who moved to the corporate sector soon after overseeing the first budget; economics adviser Tony Pearson, who took a sabbatical; and media adviser Tony Ritchie, who joined the NSW Police media team, suggested power was increasingly concentrated in the hands of too few people.

One observer suggested the office "lacked direction", and that Mr Hockey lacked an adviser with "a hard political edge".

"Joe is freelancing, being dumb with his words, though it's not Rudd office dysfunction," the source said.

A second observer said Mr Hockey "could probably strengthen his office", adding the Treasurer needed a political adviser in the mould of David Gazzard, the hard-nosed former political adviser to Mr Costello.

Follow us on Twitter