Probably not for a long time. Essentially, 1.12.2 is the last stable Minecraft build for one map servers. They are starting to work on it though, so there is a chance that in 1.17-1.18 it may be optimized enough to be usable in a one map server.
From what I’ve heard from Noobcrew, he regrets even updating it to 1.12.2. Of all of the newer versions, 1.8-1.8.9 are the most stable and least buggy- hence the reason most big servers are based on 1.8.9. As Caleb said above, it would take 1-2 years from new releases for the developers to optimise it.
What Tog and Caleb have written are probably the most accurate information you're going to get. To port all current 1.12.2 optimisation plugins for lighting, redstone, terrain generation, anti-exploit, entity movement and even event plugins to a newer version of Paper will take well over 6 months of solid work - with multiple devs - on a platform that will most likely be less stable than it currently is. In all honesty, some of the optimisation mods for Fabric 1.16 look really promising, some, when put together, boosting your FPS from <60 to >400. If only they could be integrated into a server environment....
I dont think so. Newer versions decrease server performance dramatically, and the server still runs on fairly low TPS. Imagine skyblock at 5-7 tps. Doesn't sound so fun, eh?
Updating the game seems very unlik I don't think an update is coming, but if it does its going to be a long time away
There's a possibility that Mojang might make 1000 players in one lobby possible in 2 years or more, We might not get a version beyond 1.12.2 into the server until Mojang fixes this, this isn't the only thing they have on hold they have the Combat Update on hold but we don't know both of these updates will into 1.17 or 1.18. Noobcrew would have to do something about the mob arena, grinders, and more that cause lag, not just the player count.