The billionaire still owns another home in the town.