Posts made by Prometheus

Fixed in version a212

Fixed in version a212

All bugs reported fixed in version a212

Fixed in version a212

Fixed in version a212

Fixed in version a212

Fixed in version a212

Fixed in a212

@Cotrabotieso said in December 2019 Stress Test Review:

I'm one of the guys who didn't get to enter at all, I haven't purchased anything at the moment since I am debating between beta 1 & 2 and wanted to try out the game to determine that, and as I've read the next open alpha will be open only to people who have already purchased the game, so I'm wondering if you will do a similar test on the Beta 2 phase but open for everyone. Thanks for the info and the good job, keep on working and you'll get far!

No, I think you misunderstood. We're going to run another stress test open to everyone - there just may be another one for backers only before that

Hi everyone!

Our stress test weekend has ended, and it's been a great... mess! Not uncommon for a stress test, but perhaps a little worse than expected

So, what happened during Fractured's first explosive* event of this kind? Let's find out!

Read the full article

Stay tuned and... fingers crossed for the text open test!

(* servers exploded)

@Aristeaus said in Stress Test Server Issues:

@Prometheus

What does the worker metrics say? How is your bridge to engine latency?

Oh it's nice to see other devs using spatial in the Fractured community!

Runtime to server worker latency is always low and stable (30ms), before and after the issue takes place. There is nothing anomalous in the metrics, not in latency nor in the number of commands, component updates, CPU and RAM usage, etc. No workers are being killed and recreated - workers metrics are great, all server workers are stable at their intended framerate (20 FPS) and with a large margin to receive more pressure.

The only anomalous metric is memory usage by the runtime which about 20-30 minutes after the issue starts manifesting jumps up and eats up all the RAM of the deployment. That happens when the deployment is already FUBAR though, likely because of the crazy amount of messages queued up.

@yvz5 said in Stress Test Server Issues:

Well I never had such a problem with spatial os. You are probably doing something wrong. Debugging such a problem is a pain in the butt. I wish you good luck and patience

Did you ever have 300+ players connected in a 60km2 world with both high entity count and high entity density (500,000+ resource nodes and 25,000+ creatures in total, which means 8000+ resources and 400+ creatures per km2), or something comparable to that? If so, PM me!

Hi everyone,

as you likely know already, we've been experiencing serious server issues since we launched our stress test. The issues are not on our side, but within the SpatialOS runtime.

To give you some context, the current game world is simulated by several different instances of Unity (called "workers"), each controlling a portion of the map. The runtime is the process that stores data and allows for communication between workers, both servers and clients. In short, it's the process that stitches everything together.

To say it in very non-technical terms, after some time the deployment has been running, the runtime just goes nuts. Some bottleneck is born within it that causes a long queue for one type of message (called "component updates"). You cut a tree, the logs stay piled for 30 seconds, then fall? That's because there was a 30-seconds queue for position updates for the logs. If there are more players connected it happens faster, but even if we cap the number of concurrent users to a low value (like last night - 250), it eventually happens anyway.

So, what is causing it? We have no idea (the runtime is out of our control) and neither do the Improbable engineers so far. It might be a bug in the runtime, or more likely there is something wrong we're doing with it we're not aware of. We've already tried two different fixes which didn't make a real difference. Today we're hard at work at another larger server patch, we'll let you know when servers are back online.

We're so sorry for the mess. Yes, this is what a stress test is about, but we did expect it to be a lot better. The only good metric we've had so far is that server workers are never overloaded, even if there are a lot of players fighting monsters and such. This is important, but with the runtime failing, irrelevant.

A word of warning: we'll likely have to wipe the game world before dropping the patch.

Thank you for your patience.

Best,
Jacopo

Hi, fellow gamer and MMO enthusiast!

We hinted to it before Alpha 2 started, but didn't have enough time to actually make it happen. We haven't forgotten about it though, so here it comes: the Open Alpha 2 Weekend is bringing 3 days of free access to Fractured for everyone!

Read the full announcement

Is it all? Nope! Due to the stress test, we’ve decided to let everyone who has access to Alpha 2 play until Monday, December 16!

Can't be bothered to read what's new in Alpha 2 - Test 1? New features trailer to the rescue!

Excited? Spread the word - and see you soon in game!

Fixed in v.a.2.1.1

@Tamdemonium this is intended if you don't get enough damage against the creature compared to what other players did

Fixed in v.a.2.1.1

Fixed in v.a.2.1.1

Fixed in v.a.2.1.X

Fixed in v.a.2.1.1