When the Coalition Government appointed Ian Callinan to the High Court of Australia, they knew they were in for controversy.