'Move on!' Former PMs pile in ... Note