Its fine to do world building for its own sake, if you enjoy it. But if you intend to use your world for anything, then it’s not so much “when is it enough” as “what is it for”. It’s fun to develop, say, a tavern full of interesting characters. But are the players going to go there?
My suggestion would be to develop the rough sketches of the world at large, and then focus more on the players’ surroundings, and do a bit between sessions as you anticipate what the players are interested in pursuing. Even if you let them go where they want, they probably aren’t going to go anywhere.
Also, if you let the players develop the rough sketches of where they want to go during a session (“I want to find a rowdy tavern in the seedy part of town!” / “My character is a veteran, maybe he has an army buddy living around here!”) their prompts will help you improvise, and they’ll be more invested as well. That will do for the first session that the place/character appears, and if it feels like something that’ll reoccur, you can build upon it between sessions to make it come alive more for subsequent sessions.