Press Kit Wiki

Hurtworld Update #20

Spencer

Welcome to the 20th Hurtworld Dev Blog! Not a whole lot to show from me this week as the Shigi Tools SDK integration will take a while to get things happening.

I did a bit of work this week on fixing the Vehicle performance issues caused by vehicle parts being more common. This took a little longer than expected, but should now make vehicles that aren’t currently being driven consume negligible amounts of CPU (and they were using a lot). This change will go in with this weeks wipe on Friday.

This was part of a major server performance pass cow_trx and I have been doing over the last week. These changes will hopefully give servers a much longer performance life, if not indefinitely.

Vehicle Lag

Something that might not be immediately obvious is why vehicles are so jerky when everything else is running fine.

We have very complex lag compensation systems in place for player movement. This is by no means out of the ordinary for an FPS, its called client side prediction. This (among other things) allows the server to have a CPU spike for whatever reason, catch up on things and keep running without the client showing you anything was wrong. The only time you see lag on the client movement is when the server is seriously struggling or your ping is higher than the allowed prediction amount.

Vehicles on the other hand use a physics simulation that is much harder to predict on the client, its not impossible but currently we are working with a simpler model: run everything server side then stream positions to the client (Players other than yourself steam positions also as predicting humans isn’t a good idea).

Due to this, if a client stops getting positional updates for 500ms the game has no idea where to move the vehicle and will stop in its tracks waiting for the next update from the server. Once the server gets out of its CPU spike and catches up the car starts moving again.

In short, the vehicle lag is caused by any CPU spike on the server. It could be a new player being sent heaps of construction data, a complex pathing calculation due to a creature moving through a massive base or the Unity garbage collector picking up memory that a plugin we use generated. Whatever it is, you will only really notice it while in a vehicle.

These spikes are called jitter. Hurtworld is more sensitive to jitter than other survival / MMOs games out of choice. We believe once we get our server performance is tight it will allow us to deliver a much tighter gunplay experience than is possible in games that are designed to mask large latency spikes. We think we are pretty close to nailing server performance, hopefully this patch will make some good progress.

Jitter can be hidden by increasing how far behind the current time we run our interpolation of the vehicles, the downside of this is it will greatly increase latency of driving input controls.

The only 2 solutions here are: Get servers running CPU spike free or Rewrite the vehicle network model. Right now we’re going for option A as it benefits all aspects of the game and should be an achievable goal.

Why are you working on insects when you could be working on gameplay feature X?

I get a lot of comments on our dev blogs and updates misunderstanding the way that a game development team works so I will try to clear things up here.

Our team is made up of many different roles, while it might seem that we are doing the same thing, each of our skill sets are very different.

  • Programmers: Fix bugs, implement features, change gameplay, balance things
  • Artists: Paint concept art, sculpt 3d geometry, paint textures, sometimes animate things
  • Battlemu1e: Started as community manager, now master of servers and infrastructure. Currently making sure your server stays online.

Since launch I’ve been prioritizing game stability over token features. Most of the content that has gone in so far has been art that slots into existing code systems as not to drag the programming team away from critical stability issues. We will have stability under control in the next few weeks, and can soon go back to the normal dev cycle of  concept to implementation utilizing the whole team. This is why most of the output of the art team doesn’t have a big impact on gameplay.

Mils

Did someone say RAID?

We have been getting requests for different ways to raid a base, which is nice timing because I have been working on just that. We wanted to create a method of raiding which would make the task more fun and challenging. Nothing is set in stone at the moment, but vaguely this device will require some supervision and material consumption. This will mean the person(s) pulling off the raid will need to be there and will have to tend to the device.

Strike fear into the hearts of all noobs that cower in their bases.

Strike fear into the hearts of all noobs that cower in their bases.

The device makes a fair amount of noise, thereby attracting the attention of anyone within a certain range. This, we hope, should cause interactions between the raiders, their victims and any opportunists that pass by. The upside is the fact that it can be used more often but at the cost of the fuel materials it needs to run. (and maybe your lives). This won’t be in the game for a few weeks yet.

RaidDrill02

Cow_Trix

I’ve been working on some fairly big changes to ownership stakes and cars, putting a form of upkeep into both. Along side a server major performance pass to get things running smooth again even late into the wipe. All the following changes will be going in with the wipe on Friday.

Changes to Vehicles

Every player can now own one vehicle. You can do this by opening up it’s inventory, and clicking the CLAIM button up the top right. Anyone can drive a claimed car, and modify it’s inventory. Claimed cars won’t despawn if left alone.
Unclaimed cars WILL destroy themselves if left alone. The default time this takes is 3 hours.
You can also manually disassemble unclaimed cars with a new item – the Wrench. Disassembled cars will drop loot!
This should mean cars recycle, meaning one clan doesn’t end up with all of the coolest chassis, while everyone else has to coast around with that one Sharkweek bumper.

Changes to Ownership Stakes

You can place a maximum of 5 amber in a stake, that will be very slowly consumed. The default time per amber is 24 hours, although this is configurable by server owners. This means that, by default, an unattended ownership stake will deauthorise in 5 days. Slow servers should probably increase this time, and fast servers decrease it.
Once the stake is low on amber, it will create a plume of smoke, visible from far away. Once the stake has run out of amber, it will deauthorise everyone, do a slow countdown to destruction and eventually destroy itself, freeing up the land.
This creates player-driven decay, as you can go around clearing old abandoned bases for easy resources. Be wary though – it could be a trap! We think the cost is reasonable enough that even solo players should be able to maintain a few bases fairly easily, if they play every couple of days. It should hopefully also create some areas of contention around large bases that become abandoned and unclaimed.

This should make both bases and vehicles be more heavily contested and quickly recycled, which will extend server lifetimes a fair bit. The Wrench provides a new source of farm by allowing players to harvest old chassis that had a bad loot roll or are just unneeded. This will make the best chassis reachable by even a late-entry or solo player. I like the idea of claiming one vehicle and feeling that it’s yours and a personal possession, rather than your clan having every chassis in the game in their garage.

I’m also looking forwards to seeing how stake decay impacts the meta. We’ll be sure to be looking at feedback for it, but it’s a definite step towards servers that need wiping far less often, and means that valuable stashes of resources will pop up once in a while for others to contend over. Depending on how efficient players turn out to be at removing old structures, we may implement some more structural decay mechanics in the future for unclaimed bases.

Optimization

One of the systems we have in place to ensure the server continues to run smoothly once there are thousands of objects in the world is our load balancer. This system ensures that things that aren’t mission critical never consume too much CPU time in a single frame and cause a spike. Things like storage crates checking for owrongs that need to go moldy, and campfires checking for new objects nearby to apply heat to.

This system works pretty well for the most part, once we reach the budget for a frame we wait until next frame to do the next task in the queue. Once the server gets a bit deeper into the wipe, the number of things in the queue gets so high that the cycle rate of load balancer (how long it takes to go through the entire list of tasks once) gets so low that some tasks need to wait 15 seconds or so to get a turn. This is what causes the crazy delay on things like a campfire kicking in.

I spent some time going through all our loadbalancer items with some of our biggest savegames and tuned the costs of a few of the items, this should ensure the load balancer keeps on top of its queue for much longer into the wipe.

Gavku

Hi all, I have just gotten back from a couple of weeks off, catching up with friends and family so not a great deal from me today.Currently I am working on a CNC machine, which will give the players some more crafting options 😉 Both the high and low poly are done and I am currently working on baking it out and painting up the texture maps.

Below is the initial concept.
 CNC_Concept2
And this is a screen grab of the high poly
CNC_HMAX

Battlemu1e

I have been working with providers around the world to get the best protection we can from DDoS attacks. For some insight into this we have been getting attacks varying from 1-5Gbps (often even larger) daily on a large percentage of our current servers. This kind of traffic is a heavy burden for any provider to deal with and requires specific network requirements in order to mitigate. Much of the time this requires providers who are experienced in providing services for game servers and working with the kind of traffic we have been seeing. Couple this with the extreme CPU overhead we need to cover per server and our server and network hardware requirements quickly escalate beyond the typical dedicated server.

This process is nearly completed, I am sure you have seen me migrating servers as new hardware goes live. At the time of writing the only servers that have not been moved onto new hardware are all Hong Kong servers, all Singapore servers, NY 1, 2 + 7 and Montreal but I am fairly confident these servers will be on new hardware by the time the wipes occur (if anything happens that causes this to not be the case I will make sure it is noted on Steam and Reddit.
It might be worth holding off starting in these regions until we can assure you that you won’t have to deal with downtime due to DDoS). Between these changes and the server performance improvements we will be implementing we should be at the end of our DDoS downtime and rubber banding woes just in time for the next wipe.

On the subject of wipes I will be introducing some changes to wipe schedules, server names and the amount of servers following the next wipe. In response to concerns that the current wipe schedule is too long, starting from the next wipe, we will be running 7 day and 14 day vanilla servers. Further to this we will also be offering Full Loot servers with a wipe schedule of 21 days, and due to significant server performance improvements, we will also be trialling ‘infini wipe’ servers. These ‘infini wipe’ servers will not have a scheduled wipe and will only be wiped when a patch requires it.
We will also be doing away with individual country servers and have servers divided by region instead (so EU Fast Wipe, EU Infini Wipe etc as opposed to Amsterdam 1, Amsterdam 2…). Keep an eye out on Reddit for a full list of what servers will be available following the wipe.

I will also be starting these servers at 18:00 local time Friday for each region. This should give everyone a better chance at starting a fresh wipe on equal footing (no more advantages for night owls). Due to this not aligning with the end of the current wipe schedule I will be slightly extending the current wipe. The way this will work is all servers will go down at approx 17:00 – 18:00 AEDT on Friday and as we get to 18:00 in each region that regions servers will come back up.
To make sure this process is smooth I will be manually starting and monitoring these servers as they go live. Any changes or updates to this I will be posting to Steam and Reddit so feel free to drop by and have a chat while waiting for your server to come online!

Dazzler

I have rounded out the Dart bug insect animation and stateflow, Cowtrix will be working on AI implementation in coming weeks. They should be a challenging creature, ranged like a Tokar but can stealth/pseudo-teleport by burrowing. I am now working on the wasp animations.

Hurtworld Update #19

Spencer

Hey guys,

It’s been a pretty chaotic couple of weeks due to the DDOS fairy visiting us more and more frequently. Battlemu1e has been hard at work moving all our servers to new providers that have rock solid DDOS protection, which should fix all the downtime. Battlemu1e also left without adding his devblog update describing the exact state of all the servers so you’ll have to head to reddit for that.

Since the content patch we have been experiencing some CPU issues which appears to be caused by the fact that every man and his dog has a car now. Cars with wheels cost a fair bit of CPU, and since the patch pretty much all chassis on populated servers are drivable in somebody’s base chewing up our megahertz. We will be putting out a patch in the next couple of days to hopefully address this issue along with getting kicked out of vehicles bug.

MAP SDK

This week I’ve started work on getting the first pieces of the map / gamemode SDK up and running.
I am still very much in R&D phase, testing out what is possible, finding ways we can leverage Unity to do the heavy lifting for us.
The first step is to allow content creators to author maps using a free install of the Unity editor running out Mapping SDK “ShigiTools”.
ShigiTools is a set of helpers that currently supports:
  • Terrain heightmap generation
  • Procedural detail, tree & grass scattering
  • Procedural texture painting,
  • Rock placement,
  • Biome definitions,
  • Loot spawn locations
  • Occlusion analysis
The learning curve on these will be steep, so don’t expect a point and click level editor like GTA 5. They will however be powerful.

shigitools

The challenges I am currently working through are how we will package and distribute map files.
Ideally, the process will work like this:
  1. You download and install a free version Unity 5
  2. You load up our SDK project
  3. Edit map until satisfied
  4. Run command in ShigiTools to compile and export a Unity asset bundle containing everything your map needs.
  5. Put asset bundle in folder on server and client
  6. Host game on new map
Distribution of asset bundles could be done lots of different ways. Likely we will use Steam Workshop to serve up files. In an ideal world it will work in engine just like Dota2.
Our game framework already supports multiple maps, which will make integrating maps just a matter of loading them into the engine. What we don’t support currently is multiple configurations for all the things inside the map like Biome stats and visualization, Creature Stats, Player metabolism, Machine behavior, loot tables, items etc.
Step 2 of this process will be taking all this configuration data which is currently in Fluent C# code and moving it to configuration scripts that are loaded at runtime (XML or something similar)
There will be some challenges here but nothing we didn’t plan for.

Why are you working on this before you’ve implemented feature X?

The sooner we do this, the sooner we reap the benefits of the content the community will create, hopefully that will give people a more sustainable flow of content while we work on engine systems and new features. Also because I feel like I’ve been on a bug fixing treadmill and need to get into some juicy engine system code.

Mils

What started out as a simple skyscraper ruin, has become a bit of an epic idea. I made one floor of a square shaped Skyscraper/Office Building with an inbuilt stairwell, which you then stack. This instantly gives you a rough concrete multi storey building with a fairly low texture and polygon overhead. It looks like we are going to break these into smaller pieces and create large format building prefabs.

Cities03

We hope to be able to create buildable cities with this method. Possibly, you will be able to take hold of the whole thing by locking off the stairwell and adding windows or walls to the outside. Whether these are built from scratch within the game by players or pre-built in the map, or both, is unclear as yet.

Cities02

I have added Elevated Highway sections, some signs and lights, which we would like to add also. These are built conservatively so as to keep the polycounts and texture counts down also. Performance is always taken into consideration.

Cities01

Cow_Trix

Recently, I’ve had fun implementing all the new traps, putting in a few fixes for chat (hopefully the next patch will be the last of those tags becoming visible), and implementing some new features. We’re going to try ownership stakes having some form of upkeep – nothing too strenuous or annoying, but something that means that if someone isn’t online for days, their ownership over a base can be contested. Of course we’ll make how long totems last configurable, for casual players on slower servers.

I’ve also been working on the Wrench, a new tool that can disassemble cars into loot. Hopefully this should allow people to finally trash all those crappy chassis.

Finally, I’ll also be looking at drills, and rebalancing them to be slightly less overpowered than they were pre-nerf. We’ve got lots of feedback about what you guys think the drills should play like, and what their place in the meta is, so I’ll be trying to take that all into account.

Dazzler

I have been working on the new creatures. They are all rigged up  and being animated (For the layperson, rigging is where you create all the controls and skinning to animate the creature). Here is a sampler of the yet-to-be-named dart bug. It get aggressive and shakes like a rattle snake before flinging ranged darts at you from it’s back. The weak point is the end of the tail, the rest will be pretty well-armored.


Ant
Wasp

Hurtworld Update #18

TrussNoOne

Happy New Year Hurtworldians!

Welcome to our 18th Devblog, the first of many for 2016.

Spencer

Over the last week I’ve been flat out fixing bugs and exploits, while sorting out our new server infrastructure. While I should have probably got someone else to do it for me, roping people in and bringing them up to speed over XMas / New Years is easier said than done. The servers still need work, but all of the administration work has been now automated.

The only reason official servers will go down from now on is due to DDOS attacks. We are being attacked roughly 4 times per day at different data centers around the world. We can freely move our game servers to different machines with a click of a button now, so as we find better providers that can tank the attacks we will be moving to them. Until then all we can do is wait for the attackers to get bored.

The good news is with most of the admin work and major exploits out of the way, I can start implementing everything the Art crew have been producing over the last month, and can start thinking about working on some more drastic gameplay changes like raiding and loot mechanics.

The wipe schedule for most of the official servers is coming up soon (You can see the time left in the server browser). Like last time, we will provide the savegames for anyone that wants to continue to play their progression in their own server.

This week I will be working on more small features like respawn location, then start looking at endgame as a whole.

All the content you’ve seen so far will likely go out sometime next week. We will have more concrete release schedules once we get into a rhythm, for now we still have to deal with things as they come up.

Dazzler

I am back from a Christmas break, finishing up pistol in next day or so. Have implemented all the First person work, the stateflow for the 3rd person animations is a bit more comprehensive, but almost there. Timings will be tuned up a bit yet, but it is feeling pretty fun. Here is a quick snippet of run and gunning. Take that tree!

After the pistol I am onto rigging Gavku’s wasp and making the bastard buzz! 😀

Mils

I have been busy creating a few more items which will cause your enemies untold despair. A new suite of traps are in the works. These items will give players the capability to turn terrain into pain.

We have a Landmine, Poison Snare and a type of bush that can cause lacerations. It will be interesting to see how cleverly people use these to affect the meta game.

The concept art for the Poison trap explaining (the fairly obvious) method with which it’s triggered.

PoisonTrap

The Poison Trap itself, screenshot in-game.

PoisonTrap

The Landcrab Mine, in all it’s deathly glory. Designed to be fairly visible, so it acts as a deterrent as well as packs a punch.

Landmine

The previously mentioned plant which cuts the player combined with these, should give a comprehensive suite of base protection or ambush creation tools.

Gavku

I have been continuing with player gear, specifically head gear, rounding out the last of the creature based masks with two stitched together Yeti masks and a set of nailed Tokar beaked masks.

yetimasks

tokarmasks_max_1024

We figure that the player needed something a bit more stylish than the stock beanie to help keep their head warm, so I made a Hurtworld styled Ushanka.

WinterHatUNITY2

I now get to move on to doing more creatures ( woohoo! ) We have no shortage of mammals in the game, so these next couple are to be insect based. The first one, shown finished below, is a design based loosely on a flying ant and modified to bring some pain to the player 😉

flyingantmax

Keeping with an insectoid theme, I am now in the process of modelling up a desert biome dwelling creature who will attack the player by flicking thorns from its tail as it curls and uncurls. The very tip of its tail being a weak point which can be see in the concept below.

Bug4

Dev Blog