The Doctor made up the group a long time ago in the year 100,000,000,000,000. One day, they decided to form the group - with themselves. Or, rather, this version of them.
"Hermits United. We meet up every 10 years. Swap stories about caves. It's good fun! For a hermit."