You need people who can actually do work to take care of all the old people & sustain human society. "Less People" is not by-default "more sustainable" especially not if it happens all at once; that was in fact a huge problem with cyclical famines & political turmoil in the days before mechanized agriculture.

If some asshole went around raiding hamlets for plunder, or whatever reason, yeah that would mean fewer mouths to feed in that particular area, but it also means fewer hands to bring food to harvest. Which means other regions have to contribute larger proportions of their own food-stock to sustain the needed intake of urban centers. Which means that they have less food to eat for themselves, and less to replant for the next harvest. Which pushes people on the margins of the the agricultural economy into banditry to sustain themselves, which causes us to return to the beginning of our story.

Eventually this cycle of regional depopulation leading to productivity shortfalls, leading to further regional depopulation becomes self-reinforcing & before you know it you have a country-wide catastrophe on your hand & the total implosion of existing society.

Now we aren't dependent on mass manual agriculture these days, so famine specifically is an unlikely cause of cyclical societal collapse, but the modern world still requires that a shitload of manual physical labor get done in order to maintain the basic infrastructure that gets everything from where it is, to where it needs to be in order for us to all not die. If you don't have people to fill those positions, then that's work that needs to get done, that isn't being done.