I believe this is a tradition that has been grandfathered into voice chat apps. Previous ones like Ventrilo have you actually connect to a private server (that someone pays for / hosts).
I second this explanation. Additionally, from a UX perspective, if a user wanted to join your server you had to give them IP/port/pw and they'd type it in manually. So even the users were accustomed to addressing servers. "Server" entered the community lexicon as "the place we chat".
This is most likely the case. Discord's introduction was mainly to pull in the gaming audience using Skype while providing the quality behind Mumble and Ventrillo
