Barnaby Joyce replies to allegations of hiding from public