Outgoing disability discrimination commissioner Graeme Innes has fired a barrage of parting shots at the Abbott government, declaring it may be time for him to move on because he has ''no stomach to advocate for the rights of bigots''.

Mr Innes' remarks, made during an address to the National Press Club, are a direct rebuke to Attorney-General George Brandis, who while speaking on the Coalition's proposed changes to race hate laws earlier this year, said: ''people do have a right to be bigots, you know.''

Mr Innes said on Wednesday he could not understand why at a time when the nation was seeking to recognise indigenous Australians in the constitution, it would be changing laws to reduce their protection from racial vilification.

He also rejected Senator Brandis' criticism of the Australian Human Rights Commission, that it had been too narrowly focused on its anti-discrimination work and had not done enough to promote ''liberal rights'' such as freedom of expression, freedom of association and freedom of religion.

''What the Attorney doesn't take into account when he makes that comment is that all of the legislation which the commission administers is anti-discrimination legislation... the best way frankly for the Attorney to provide the commission with the greater capacity to deal with the freedoms he talks about would be to put forward legislation for a charter of rights,'' he said.

Mr Innes rejected as ''facile'' the Abbott government's use of the ''Ming Dynasty concept'' of ''lifters'' and ''leaners'' to describe those who contribute to the economy and those who depend upon the state.''We all move from one role to the other dozens of times a day,'' he said.

He said many people with disabilities wanted to contribute more but were kept out of the workforce by discrimination.

''In terms of the government's mantra that the best form of welfare is a job, they are absolutely right. Absolutely right. The problem is though that the mantra is not backed up with a jobs plan. What it's backed up with is a welfare plan.''

Mr Innes singled out the Commonwealth health department and Westpac bank as organisations which had excelled in employing people with disability, and suggested Australia emulate the United States practice of providing every politician with an extra staff member if they employ a person with a disability.

Mr Innes, who was born blind and trained as a lawyer, joined the commission as a deputy commissioner in 1999 and was appointed disability discrimination commissioner and human rights commissioner by the Howard government in 2005. He also served as race discrimination commissioner from 2009 until 2011.

He will finish in his role on Friday and will not be replaced by a full time disability discrimination commissioner because of budget cuts.

Senator Brandis announced on Tuesday the aged discrimination commissioner, Susan Ryan, would add disability discrimination to her existing responsibilities.

Mr Innes questioned the logic of the government deciding in December to appoint former Institute of Public Affairs policy director Tim Wilson to the position of human rights commissioner – a role Labor had proposed to merge with that of the president of the commission – at a time when it was contemplating cutting the organisation's budget.

Mr Innes said the commission's capacity to advocate for people with disability would be diminished, not simply through his departure, but because of a long term decline in the real value of its funding.

''The commission will do its best with the hand it is dealt, but that is becoming a weaker and weaker hand,'' he said.

''When I began as deputy commissioner in 1999 there were four policy staff dedicated to disability issues...

''The downgrading of the disability discrimination commissioner's position... will mean that there is only one person in the policy section with significant disability expertise, and she is moving to another role.''

Senator Brandis has rejected the suggestion that Mr Innes' role has been downgraded, saying that it was common practice for commissioners to hold dual portfolios.

In a statement released on Tuesday, he said for all but three of the last 15 years, the disability discrimination commissioner had also been responsible for another portfolio.

But Mr Innes said it was ironic that in the same week his role was disappearing as a standalone position, in the environment portfolio the government was appointing the nation's first threatened species commissioner.

He said given 45 per cent of people with disability lived in poverty, and rates of employment for disabled Australians were 30 per cent lower than those for their counterparts with disability, ''I could mount an argument that people with disabilities are a threatened species''.

He praised the Abbott government for its commitment to the National Disability Insurance Scheme, but rejected the suggestion by the minister responsible for the scheme, Mitch Fifield, that the Gillard government's decision to launch the scheme last July, 12 months earlier than the Productivity Commission had recommended, had compromised planning.

Senator Fifield  said this week it would have been ''optimal'' to wait until this year to give the National Disability Insurance Agency more time to prepare.

''My question would be optimal to whom? Because it certainly wouldn't have been optimal to people with disabilities,'' Mr Innes said.

''In my view, the scheme was ready to start when it started... Now of course it needs some tinkering around the edges. Any new scheme does, but as any good software developer will tell you, the best way to get the optimal result out of a piece of software is put it out broken and let the market fix it. That's what software developers do and that's, to an extent, what happened with the NDIS and we're reaping the results of that.''

He criticised tabloid newspapers for portraying people with disability as ''shirkers and rorters'' and retail chains ''who think it's OK to sell t-shirts with 'retarde' across the front, when 'nigger' or 'slut' would not pass muster''.

One of his new roles will be to chair the board of the newly-established Attitude Foundation, which will try to change attitudes about people with disabilities using television and film. It hopes to raise $200,000 by September to fund programming to be shown on the ABC.

''We are viewed as either victims or heroes, when we should be viewed as agents of our own destiny. The soft bigotry of low expectations limits what we can achieve,'' he said.

Follow us on Twitter