Regarding the recent performance

General announcements
1

As a lot of you probably have experienced, we’ve seen a much lower level of performance than we’re generally used to. This came shortly after our migration to newer hardware within our vendor’s datacenter. We’ve done quite a bit to try and address the issues as the overall cause wasn’t immediately obvious. Partly because there was a certain expectation that came from being with the same provider, and partly due to several attacks against our services giving us a false perspective on the situation.

After some extensive monitoring after the attacks, and back-and-forth communication with our vendor, it has come to our attention that the new server is being thermally throttled under heavy loads causing a substantial decline in overall performance and general stability.

Obviously, this isn’t ideal. We’re currently doing everything in our power to try and make the situation a lot more tenable, but there’s sadly only so much we can do at this time. We are in active talks with our provider to get the situation rectified. But, that may take a bit of time.

I humbly ask for your patience and understanding as we do our best to improve the situation. In order to provide an overall better experience to our players, we will be temporarily reducing the population caps for our servers down to 40 players (donators will still be allowed to bypass this restriction like normal). Once the situation is rectified, this reduced cap will be reevaluated.

Again, I want to express sincere apologies for the overall poor performance as of late. Things will improve, it’s just going to take a little longer.

Update - May 12th, 2021

We’ve gotten the old server upgraded to be in-line with what we have now and have confirmed that the thermal throttling issue is no longer present on that hardware. We are currently awaiting additional network assignments from our datacenter before we can begin moving things to that environment.

Current projected date of completion: Friday, May 14th.

Update - May 14th, 2021

We have migrated to the new-new environment complete with better cooling and proper turbo-boosting. We’ll be monitoring things from here to see if we continue to get the expected performance.

image

Update - May 15th, 2021

I semi-secretly lifted the pop caps on Sage/Golden to see how things would perform. While it has been generally fine, we’ve observed some minor hiccups. Nothing overly concerning during the higher population tests. This has been somewhat counteracted by a DoS attack targeting one of our internal procs, we have implemented a temporary solution to help mitigate the problem and will seek a better, long-term, solution soon.

We are continuing to monitor the situation.

Update - May 19th, 2021

At this time, we’ve returned to a state of normalcy. The issue with abnormally high time dilation was resolved on the 17th, and we haven’t observed any further issues since. I’ve done several polls in-game and the respondents all seem to agree that we’re in a good spot once again.

This will likely be my last update on this issue. Thanks again to everyone for providing their support and for being patient whilst we worked on improving things.

10 Likes
3

we can buy some bags of ice

18 Likes
5

Host moment
202121212

1 Like
6

Seems kinda strange that the provider would, to some degree. Basically throw customer under the buss. I’m sure that its unexpected on there end too it or would be a massive bruh moment

Considering how little power SS13 needs, and for effects to be this noticeable must be pretty extreme

7

This is actually false. SS13 needs a stupid amount of power. Our CPU is currently operating between 3.8-4.4GHz under heavy load due to the throttling. For comparison’s sake, the old server would idle at 4.7GHz using the same CPU.

4 Likes
8

It kinda sounds like they moved you to a datacenter that’s not fully operational yet…

9

It’s just a different environment all together. The older datacenter that we were in has midi servers and cooling, the new one utilizes ES server housings.

10

Without cold aisles?

11

I can’t say I have a comprehensive understanding of what cooling methods they use in our section of that specific building. Only that the cooling provided to us is configured to guarantee base frequencies, not boosted ones. I do know that they offer cold aisles as well as climate control through raised floors, I am assuming the latter is what we currently have.

12

You should ask for pictures :smiley:
Jokes aside though, I hope they get the thermal issues under control soon, such issues could end up very costly for them if they have any super critical systems among their customers…

13

They offer a virtual tour if you want it. :wink:

We are in Falkenstein.

I have a hunch that there aren’t very many clients who are entirely incapable of using parallel processing like we are. I doubt this is much of an issue for them, to be honest.

14

mail the server could use a new heater durring the siberian winds.
all jokes aside i hope the situation imprves. and the issue gets fixed
you are doing a great job. :slightly_smiling_face:

1 Like
15

That’s a sexy looking datacenter, but is bee co-located or rented?

16

you are underestimating how much of an unoptimised trashmound of code SS13 is

6 Likes
17

Oh my, iam genuinly impressed that can even happen considering how dated the engine is

18

Rented. Colocation with Hetzner would require a bigger investment.

1 Like
19

Just looking at the Github makes my head spin…this is so true.

20

On the contrary, it’s precisely for that reason :stuck_out_tongue:

Running SS13 is almost equivalent in performance to running a batch script containing a million lines that endlessly loops around, it’s a horrible platform performance-wise, it will eat precisely 1 core and it will eat it up as much as it possibly can, the more it gets, the faster the game runs.

(Kind of, there are timing mechanics to keep it running at a certain speed and to make it avoid getting stuck at a certain task.)

21

SS13 was not designed w/optimization in mind…all the background calculations doesnt make it any better Cough cough Fastmos.

But thats part of the reason its fun.