Mainlanders like to joke about the Tasmanian weather. Over the past few days it got its own back, playing an extended prank on the thousands visiting for Hobart's annual MONA FOMA (or MOFO, for short) festival. Even by Tasmanian standards, the unlikely mix of gale-force winds, biting cold, rain and high UV levels made a challenging backdrop for a four-day festival, giving organisers a headache and punters a rare chance to get sunburnt while rugged up under three layers. Some outdoor events had to be cancelled, and Exxopolis – a giant inflatable erected on the Hobart waterfront – was closed for two days because of safety concerns.

Despite the setbacks, Violent Femmes bassist Brian Ritchie continues to curate Australia's most mind-expanding and value-for-money mix of music and art.

MOFO follows a simple enough pattern: installations and events dotted across the city through the day, with the festival converging on Princes Wharf No 1, a giant converted shed on the waterfront, each night.

There is no headline, but if there were it would have been Paul Kelly. His Friday night gig under the banner the Merri Soul Sessions brought together a chorus of giant voices: Vika and Linda Bull, Dan Sultan, Clair Browne and Kira Puru alternating on soul covers and re-imagined takes from Kelly's catalogue. The band was tight, the singing often extraordinary – more than 25 years on, the Bull sisters somehow remain a revelation – and the song choices sound. It made for the weekend's fail-safe gig: smooth and professional and, at a festival that often aims to expose its audience to fresh things, ultimately a little tame.

The other quasi-headline, Swans, leapt from the other end of the scale. Their Saturday night set started with a warning: "Swans play really, really loud. Hearing protection needed. Caution!" They don't lie. Sounding louder with earplugs in than the rest of the bill sounds without them, the New York post-punks deliver a form of brutal hypnosis: rhythmic, droning trance-scapes of guitar, bass and percussion stretching to 20 minutes and beyond. The sound builds through repetition, layer added on layer, all tension, no release.

Their set continued a loose post-punk theme on Saturday. Earlier, from the quieter end of that era, former Soft Boys frontman Robyn Hitchcock played his 1984 solo acoustic record I Often Dream of Trains in full, though not in order. With a set that was equal parts wintry melody, wry humour and sharp observation, Hitchcock won over a crowd largely unfamiliar with his work. He brought out MOFO favourite Amanda Palmer on a few songs to augment his stripped-back trio.

Palmer returned the favour the following day as she played nearby Federation Hall with the Tasmanian Symphony Orchestra. As an artist not unfamiliar with melodrama, Palmer's pop-cabaret is an obvious candidate for the full orchestral treatment, though it's her most intimate moments that work best. The Bed Song, a ballad about lovers distant through their lives, was a standout. A wry duet with Hitchcock on his Queen Elvis also worked a treat, though trading stage verses on a cover of Lou Reed's Perfect Day with husband and author Neil Gaiman was less successful. By the encore Palmer was singing about her Map of Tasmania and stripping near-naked, but even amid the bombast and forced jokes the strings sounded sublime.

Two very different artists provided the glue holding the festival together. Jim Moginie, a former Midnight Oil linchpin, brought his electric guitar orchestra to the big shed on Thursday, playing a late-afternoon set while artists painted a giant wheel of bright colours behind them. Chiming in restraint as often as they were loud and fuzzy, the sextet – mostly drawn from Sydney indie acts – also turned up regularly to play a cominside the catacomb-like Exxopolis, created by British company Architects of Air. Watching with my two-year-old son as Moginie band picked a lullaby-like section of a commissioned composition, the sun transforming the sculpture's blue and red ceiling into a plastic stained-glass window, was a festival highlight.

The other unifying thread came from artist-in-residence Ava Mendoza, a New York guitarist and singer who arrived in town on Monday, met local bassist Hamish Houston and drummer Alf Jackson and rehearsed with the pair in open public sessions through the week, before playing a virtuosic Friday night set. Set aside any reservations about an artist billed as avant-jazz – she is one to seek out.

The installations around the city largely took a back seat to the music, but brought their own diversions. German artist Johannes S Sistermanns connected the pillars holding up Salamanca Arts Centre's Long Gallery with cling film and showed it would conduct sounds through the room, creating its own spooky echo of a language.

On Friday night at Princes Wharf No 1, Beijing's Li Binyuan exhausted himself as he sat on the floor and, over more than an hour, broke 250 small hammers by hitting them against each other. Given the time spent on the task and the rapturous response he received from the large crowd when he smashed his final tool, the title of his performance – Deathless Love – made some sense.

Somewhere between music and art was Omar Souleyman, a heavily moustached Syrian wedding-singer-turned-western-music-festival-favourite who sings in Arabic and Kurdish. He had the crowd jumping with his four-to-the-floor Middle Eastern club music but, with the oud and mijwiz triggered by a keyboard button, the live experience was less than the sum of the sound.

On the music front, the best was saved for last. On Sunday night - after US composer Alvin Curran had conducted an orchestra of mostly local brass and pipe bands playing on barges off the wharf, and Gotye had made an appearance with the slow-breeze electronic outfit Zammuto - all-female Japanese punk band Shonen Knife played a blinding set of high-energy, high-spirited pop blasts about shopping, cats and how much they like green tea. Only guitarist/vocalist Naoko Yamano remains from the line-up that won notice in US alternative rock circles in the 1990s, but the newer members have been well chosen. Singing drummer Emi Morimoto is a powerhouse of rhythm, audacious fills and rockstar poses. Their debt to the Ramones and Motorhead is obvious and intended, but those bands were never this polite.

They were followed by The Clean, part of the first wave of guitar bands that emerged from Dunedin on the Flying Nun label in the 1980s. Revitalised in 2015, they sounded like Joy Division playing surf music, and were as good as that suggests.

For those not heading to Faux Mo, the nightly club across town at the Odeon Theatre, the night finished with a livewire electronic set by US composer Dan Deacon, who spent most of his time exhorting the crowd into rock eisteddfod-like contortions: dancing like dinosaurs; West Side Story face-offs between the two sides of the room, each gyrating in formation with a randomly chosen leader; forming human rivers that flowed in and out of the shed's giant side doors. The music was sometimes forgettable, but this was about performance. With the crowd as willing participants, that worked just fine.