Dealing With Low TPS

To begin with, I know nothing about how this server is actually hosted. I have seen in other threads that the server costs around $1250/month to maintain, which makes me believe that the server is being run through a rented out box or virtual machine. If anyone could give me further insight into what the server is actually being run on, that would help a good bit with my input as to how to improve performance.

For context about who I am, I used to play this server back in 2013 and 2014 when I was in high school, and have recently returned a month ago, now being halfway through college, working on a Computer Engineering degree. The last year, I have done significant amounts of work with running servers, managing game servers, and handling overall performance optimization of said servers. I personally own a Dell PowerEdge R710 which I've sunk like $2000 into at this point. This server I own runs from my home, and is used to host ARK and ATLAS servers (along with various personal programs and servers). At my workplace, I've spent a good amount of the last year working with our server racks, and installing server racks at our remote locations. This involved settings up various workstations through VMware, vSphere, ESXI, and other VM related products in that family.

That out of the way, lets get into my thoughts on dealing with the lag...

As a community:
Short term wise, there's a lot we can do as players to help reduce on lag. One quick fix I've been going about lately is disabling intricate redstone contraptions when they aren't in use.

My main focus here is a popular stone generator that I built myself after seeing a whole bunch of players using this generator on their islands. The generator in question can be seen here:
This generator uses a 1.12 functionality of the observer, in which it fires when it detects the random-tick change of lava blocks when they begin to flow. This detection method is very useful, as it prevents the lava from ever flowing down and turning your water into cobblestone, clogging up the system. However, the first thing I noticed when using this generator is the massive amount of lag you get when near it, and even greater when using it. I could see this becoming an issue when many people have this same generator, always running, always firing off for those random ticks. I have seen at least 10 people with this same generator, some having 2-3 on one island. Fearing that this could be doing a good bit of damage to the server, I've begun tearing about the redstone on it when I'm not using it, as to keep it from lagging anything up. I believe other players should follow (if you do, make sure you wait until all pistons are done firing, otherwise you risk some water sources become cobblestone).

The other big contributor to lag I've seen has been disturbingly large cactus, sugar cane, melon, and pumpkin farms. Some people have farms using more blocks than I probably have placed on my entire island, with redstone circuitry run throughout all of the farm. I understand these are nice to have, but I don't think they're needed. I'd be mindful of how large of farms you're creating, and try to keep them at a reasonable size.

Software side:
I know the developer team is already working on this, but I believe getting us to 1.13 as soon as possible will help a lot. For one, players joining the server from a 1.13 client will no longer have to deal with the bugged out chests, doors, and fences. More importantly, however, this will change the behavior of observers. Observers will no longer be able to detect random ticks in lava, thus breaking the aforementioned stone generators that rely on this functionality. Players will instead have to switch to other forms of looping mechanisms, all of which I've seen being much less laggy. Even a hopper loop timed to fire before the lava can spread would be less laggy than the current observer loops.

I understand switching to 1.13 opens a whole new world of issues, and is easier said than done, though I do see there being a whole lot of potential benefit in doing so, if nothing else than just to break observer-lava loops.

There's further options that could be configured in server startup and Spigot/Paper/whatever the server uses, though I'm sure most of those options have already been tried, or are in the process of being tried.

Hardware side:
Don't worry, I won't say we need more RAM. I know we're already more than set on that, as seen by the numbers in /lag. This whole area is where my lack of knowledge about the server prevents me from saying too much. So I'll speculate a bit.

1. Upgrade CPU(s). Redstone circuitry can get very CPU intensive, and if we don't have the power to handle that, that would be greatly hurting us. From what I've seen in personal experience, better processors generally translates into higher TPS. If the server is being rented, though, and already costing $1250/month, I can see why this wouldn't be an option.
2. Better network switch/and or network card. When I first got my server, I was using a dinky network card, and a $30 D-Link switch. Upon upgrading from these to professional grade equipment, I saw TPS in my servers skyrocket. The switch was most likely the main issue, as it didn't like to handle the heavy amounts of traffic, but improving upon the network card couldn't hurt (if it does need any upgrades to begin with).
3. Higher-tier business grade internet. This was another issue from my personal experience. Running a server off of consumer-grade 20Mb/s internet just wasn't cutting it. Once I got around 50-75 players across all of my servers, ping would start to skyrocket. Upgrading to a true business grade line allowed me to eliminate these ping issues.

Further speculation:
I feel that my points 2 and 3 above may be useless, as I remember times where the server was effectively handling hundreds of players with no issue. That's what leads me to believe either something has changed on the networking side, our CPU(s) are now throttling us, or we have something that's changed in the software.

Also, correct me if I'm wrong, but I don't recall any major issues with lag prior to the Valentine's day update. I only played on the server for a week and a half leading up to that, so I very easily could've missed that lag. If the lag has only been reaching these levels of bad in the last two weeks, there may be value looking into what all was changed software-wise in that update.

TLDR;
Players need to disable redstone intensive generators when not in use, and downsize any ungodly large farms they may have. Upgrading to 1.13 could yield some benefits, especially with new players and eliminating laggy generators, but may ultimately have no net gain. Some configurations may have been changed that are hurting us, though that can be hard to track down. Server potentially needs better CPU(s) and/or networking equipment/networking speed.

Feel free to correct me if anything I said was wrong, or if I come off as sounding very misinformed. Most of my information is based off the little I've read in the forums and in chat, along with my personal experiences.

As someone who runs their own servers, I understand how much the developers and noobcrew hate people trying to tell them how to run their server. I still think there's some value in throwing out suggestions, as 100 bad suggestions could at least inspire one good idea for someone reading.

 

Sorry for the huge wall of text. I've tried to include a TLDR for those who aren't fully interested in everything, and tried to highlight my main points.

 

Low tps is what causes issues. We want high tps

 

I realize now I mistitled the post. I was fighting between calling it "Dealing with Low TPS" or "How to Achieve High TPS". Accidentally did " Dealing With High TPS". If you look at the content of my post, I referred to TPS properly, and how to improve it.

 

Do you know any more about the data center it's hosted from? Because depending on what kind of setup it is, a lot of the blame could be on the host provider instead of anyone on the Skyblock team.

This isn't a very good approach for overall numbers, though. A lot of players want to play on the weekends, but are turned away because of the connection. Other players just don't have the option or free time to play any time other than on the weekends. I've seen a significant number of players join the server, ask about the lag, and then leave because it's nearly unplayable for them, or extremely frustrating.

I'd have to do more research into that. Though if that is the case, it would be better to opt for alternative software/hardware upgrades. My focus was purely on observer-lava circuits, which are significantly laggy, and not an option in 1.13.

Based on what I'm being told, server is run from a data center. If it's a physical data center they can visit, then there's may be more they can do. But if it's a data center that they just remote into, there's a lot of variables completely out of their control that could be causing the lag.

 

That would be another option I think would help. It's nice getting people in who are using 1.13 clients, but a lot of them get annoyed by the performance/glitchiness they experience, and may just leave all together instead of asking in chat to learn they need to use the 1.12 client.

The question would center more around whether you get more dedicated players from letting people join on 1.13 and later switch to 1.12, or if you'd get more dedicated players just by only letting 1.12 and lower join from the start.

Another thing I've seen a lot of lately is just straight up being kicked from the server, even when player count is low and TPS is high. This morning we had 60 people on /skyblock, running at 20 TPS, when randomly have the lobby got completely kicked from the server. These boots seem to deal with something other than TPS, as they still consistently happen to me and friends, even on high TPS times.

Something with it seems client dependent. I can be on with three friends, where myself and one friend is on my WiFi, the other two friends on another WiFi. I was never getting kicked back to the server hub, my friend on my WiFi and one of my friends on the other WiFi were getting occasionally kicked, while one friend on the other WiFi was getting kicked back to the server hub almost ever 3 minutes. TPS was around 12-15 at the time.

All of these kicks were the errors were the "transferring you to a fallback server" message, with the occasional "ConnectionTimedOut" kicking you back to pick a server, usually requiring you to restart your client.

These server failures could be an even bigger issue than TPS, because you can at least play and deal with some lag, but getting booted out of the server and sent back to your spawn point ever 5-10 minutes makes the game legitimately unplayable.