Great designers come from everywhere
I've met excellent designers with backgrounds in writing, theatre, various sciences, various artistic media, the culinary arts, the military, visual design, music, production, martial arts, computer science, psychology, engineering, education, medicine, philosophy, construction, automotive, transportation, and on and on.
I used to think this was because game design was a relatively new career and no specific degrees used to exist for it, but I still seem to meet at least as many good game designers that have backgrounds in something wildly different from games as those with game design degrees.
I believe game design really is one of those fields where the wider your past experiences have been, the better you can become at it. As with fiction writers and actors, almost any experience that you've had can help you to be better at your craft. There are so many ways to learn something about psychology and motivation, why our brains think something is fun, how to collaborate with people, how to create an immersive fiction, how to methodically diagnose a problem, etc.
That's not to say that you shouldn't actively pursue game design as a hobby while you're doing all those other things. People who are constantly thinking about game design (or anything) are much more likely to recognize subtle connections and have epiphanies related to that topic, no matter how else they spend their time. Even slinging burgers or mopping floors will teach you something about the world or yourself that's useful in game design if you keep your mind primed and open to realizations.
Great designers stay curious
Once you've gotten a job as a game designer, don't see that as the signal to become complacent and stop looking to other areas for inspiration: play lots of games outside of your specialty; pursue interesting hobbies inside and outside of game development; watch your coworkers in other disciplines closely for lessons that are applicable to your job; read books across the whole spectrum of human knowledge.
Games genres are splintering and intermixing all the time, and there's no telling when your interesting side project could make you the perfect designer for a new type of game. I'm sure there more than a few designers have found surprising uses for their "unrelated" background or hobby (such as music, biology, car repair, parkour, economics, or ancient egyptian anthropology). A unique background makes you more qualified, not less.