Hurtworld Update #20

Spencer

Welcome to the 20th Hurtworld Dev Blog! Not a whole lot to show from me this week as the Shigi Tools SDK integration will take a while to get things happening.

I did a bit of work this week on fixing the Vehicle performance issues caused by vehicle parts being more common. This took a little longer than expected, but should now make vehicles that aren’t currently being driven consume negligible amounts of CPU (and they were using a lot). This change will go in with this weeks wipe on Friday.

This was part of a major server performance pass cow_trx and I have been doing over the last week. These changes will hopefully give servers a much longer performance life, if not indefinitely.

Vehicle Lag

Something that might not be immediately obvious is why vehicles are so jerky when everything else is running fine.

We have very complex lag compensation systems in place for player movement. This is by no means out of the ordinary for an FPS, its called client side prediction. This (among other things) allows the server to have a CPU spike for whatever reason, catch up on things and keep running without the client showing you anything was wrong. The only time you see lag on the client movement is when the server is seriously struggling or your ping is higher than the allowed prediction amount.

Vehicles on the other hand use a physics simulation that is much harder to predict on the client, its not impossible but currently we are working with a simpler model: run everything server side then stream positions to the client (Players other than yourself steam positions also as predicting humans isn’t a good idea).

Due to this, if a client stops getting positional updates for 500ms the game has no idea where to move the vehicle and will stop in its tracks waiting for the next update from the server. Once the server gets out of its CPU spike and catches up the car starts moving again.

In short, the vehicle lag is caused by any CPU spike on the server. It could be a new player being sent heaps of construction data, a complex pathing calculation due to a creature moving through a massive base or the Unity garbage collector picking up memory that a plugin we use generated. Whatever it is, you will only really notice it while in a vehicle.

These spikes are called jitter. Hurtworld is more sensitive to jitter than other survival / MMOs games out of choice. We believe once we get our server performance is tight it will allow us to deliver a much tighter gunplay experience than is possible in games that are designed to mask large latency spikes. We think we are pretty close to nailing server performance, hopefully this patch will make some good progress.

Jitter can be hidden by increasing how far behind the current time we run our interpolation of the vehicles, the downside of this is it will greatly increase latency of driving input controls.

The only 2 solutions here are: Get servers running CPU spike free or Rewrite the vehicle network model. Right now we’re going for option A as it benefits all aspects of the game and should be an achievable goal.

Why are you working on insects when you could be working on gameplay feature X?

I get a lot of comments on our dev blogs and updates misunderstanding the way that a game development team works so I will try to clear things up here.

Our team is made up of many different roles, while it might seem that we are doing the same thing, each of our skill sets are very different.

• Programmers: Fix bugs, implement features, change gameplay, balance things
• Artists: Paint concept art, sculpt 3d geometry, paint textures, sometimes animate things
• Battlemu1e: Started as community manager, now master of servers and infrastructure. Currently making sure your server stays online.

Since launch I’ve been prioritizing game stability over token features. Most of the content that has gone in so far has been art that slots into existing code systems as not to drag the programming team away from critical stability issues. We will have stability under control in the next few weeks, and can soon go back to the normal dev cycle of  concept to implementation utilizing the whole team. This is why most of the output of the art team doesn’t have a big impact on gameplay.

Mils

Did someone say RAID?

We have been getting requests for different ways to raid a base, which is nice timing because I have been working on just that. We wanted to create a method of raiding which would make the task more fun and challenging. Nothing is set in stone at the moment, but vaguely this device will require some supervision and material consumption. This will mean the person(s) pulling off the raid will need to be there and will have to tend to the device.

Strike fear into the hearts of all noobs that cower in their bases.

The device makes a fair amount of noise, thereby attracting the attention of anyone within a certain range. This, we hope, should cause interactions between the raiders, their victims and any opportunists that pass by. The upside is the fact that it can be used more often but at the cost of the fuel materials it needs to run. (and maybe your lives). This won’t be in the game for a few weeks yet.

Cow_Trix

I’ve been working on some fairly big changes to ownership stakes and cars, putting a form of upkeep into both. Along side a server major performance pass to get things running smooth again even late into the wipe. All the following changes will be going in with the wipe on Friday.

Changes to Vehicles

Every player can now own one vehicle. You can do this by opening up it’s inventory, and clicking the CLAIM button up the top right. Anyone can drive a claimed car, and modify it’s inventory. Claimed cars won’t despawn if left alone.
Unclaimed cars WILL destroy themselves if left alone. The default time this takes is 3 hours.
You can also manually disassemble unclaimed cars with a new item – the Wrench. Disassembled cars will drop loot!
This should mean cars recycle, meaning one clan doesn’t end up with all of the coolest chassis, while everyone else has to coast around with that one Sharkweek bumper.

Changes to Ownership Stakes

You can place a maximum of 5 amber in a stake, that will be very slowly consumed. The default time per amber is 24 hours, although this is configurable by server owners. This means that, by default, an unattended ownership stake will deauthorise in 5 days. Slow servers should probably increase this time, and fast servers decrease it.
Once the stake is low on amber, it will create a plume of smoke, visible from far away. Once the stake has run out of amber, it will deauthorise everyone, do a slow countdown to destruction and eventually destroy itself, freeing up the land.
This creates player-driven decay, as you can go around clearing old abandoned bases for easy resources. Be wary though – it could be a trap! We think the cost is reasonable enough that even solo players should be able to maintain a few bases fairly easily, if they play every couple of days. It should hopefully also create some areas of contention around large bases that become abandoned and unclaimed.

This should make both bases and vehicles be more heavily contested and quickly recycled, which will extend server lifetimes a fair bit. The Wrench provides a new source of farm by allowing players to harvest old chassis that had a bad loot roll or are just unneeded. This will make the best chassis reachable by even a late-entry or solo player. I like the idea of claiming one vehicle and feeling that it’s yours and a personal possession, rather than your clan having every chassis in the game in their garage.

I’m also looking forwards to seeing how stake decay impacts the meta. We’ll be sure to be looking at feedback for it, but it’s a definite step towards servers that need wiping far less often, and means that valuable stashes of resources will pop up once in a while for others to contend over. Depending on how efficient players turn out to be at removing old structures, we may implement some more structural decay mechanics in the future for unclaimed bases.

Optimization

One of the systems we have in place to ensure the server continues to run smoothly once there are thousands of objects in the world is our load balancer. This system ensures that things that aren’t mission critical never consume too much CPU time in a single frame and cause a spike. Things like storage crates checking for owrongs that need to go moldy, and campfires checking for new objects nearby to apply heat to.

This system works pretty well for the most part, once we reach the budget for a frame we wait until next frame to do the next task in the queue. Once the server gets a bit deeper into the wipe, the number of things in the queue gets so high that the cycle rate of load balancer (how long it takes to go through the entire list of tasks once) gets so low that some tasks need to wait 15 seconds or so to get a turn. This is what causes the crazy delay on things like a campfire kicking in.

I spent some time going through all our loadbalancer items with some of our biggest savegames and tuned the costs of a few of the items, this should ensure the load balancer keeps on top of its queue for much longer into the wipe.

Gavku

Hi all, I have just gotten back from a couple of weeks off, catching up with friends and family so not a great deal from me today.Currently I am working on a CNC machine, which will give the players some more crafting options 😉 Both the high and low poly are done and I am currently working on baking it out and painting up the texture maps.

 

Dazzler

I have rounded out the Dart bug insect animation and stateflow, Cowtrix will be working on AI implementation in coming weeks. They should be a challenging creature, ranged like a Tokar but can stealth/pseudo-teleport by burrowing. I am now working on the wasp animations.