Warren Gatland started having doubts about his decision to return to coaching Wales within weeks of his appointment.