Hurtworld Update #22

18 02, 2016

Spencer

Greetings Hurtworldians!

Welcome to another episode of the goings on around the studio. In this week’s (late) edition, we are mostly tied up with Map SDK and Unity upgrade fun. As you can probably tell I’m pretty excited about it… I may have just had too much sugar this morning. We’ve got some more stuff to kill you, cow is up to his usual trix and mils has been making more progress on cities. At this rate you ‘ll be able to build a poor mans GTA5 in our level editor.

Due to the surgery state of the project, the next patch might be a week or two away. We could push out the micro changes and keep what we are doing in a separate branch but that would just slow down the awesome stuff, and who doesn’t like awesome stuff!?

Map SDK

I finally made some good ground on the map SDK this week. Previously we had one single project, with references all over the place between map prefabs / utility scripts / assets and engine code / runtime dynamic prefabs. To ensure the map framework is rock solid we needed to make sure we could build Diemensland using it. Which meant completely decoupling all our map assets and code from our main project.

This was a much needed cleanup that completely isolates everything map related from our core game engine, making the core much more lightweight and easy to enhance. The other benefit of this, is now that Diemensland is isolated, we can release the entire source project used to create it!

This will give map makers an end to end example of how the pipeline works, along with all our source assets like rocks, textures, trees and set pieces to use however you please. Think an area of the map would be better slightly different? Change it and branch off using Diemensland as a base!

When do I get it?

Right now everything builds, and can load custom built maps into the game at runtime. We still need to integrate this with Steam Workshop so users can ensure they have the latest version of custom maps simply by joining a server running it, without having to leave the game. This will take another week or so, but for now I will release the base version of the SDK for people to start playing with. For now you will only be able to load maps into the engine by placing them in the maps folder like the old days.

Our only blocker at the moment is that we are waiting on some confirmation of legal issues with Unity in regards to distribution of assets acquired from different sources, these won’t stop what we are doing but might change the way we let you use our assets.

This integration also touched almost every file in the project, so we need to go through some serious regression testing to ensure all the references match up still. When we’ve gotten confirmation from Unity I will throw whatever state the SDK branch is in onto a Steam beta.

If you want to get an head start, you can download a free version of Unity 5.3 here:

Download Unity

Start playing with the terrain tools, importing models / textures (there are heaps of free ones on the asset store) and start designing your masterpiece. Copying it across to the SDK later will be trivial.

Anything you can create in Unity besides scripts will be importable into Hurtworld!

Cow_Trix

UI and Input

It’s been a busy week! The upgrade to 5.3 meant I had to fix a bit of regression in the UI, as Unity did some breaking changes between 5.1 and 5.3. While I was at it, I made UI perform significantly better. The biggest quick win was changing how we hid UI that was being unused. Previously, we hid a lot of things by just setting the object to transparent and keeping it active. Now we’ve got it so windows are completely deactivated when not being used. Another big win was changing how we activate and deactivate the Unity EventSystem, the system which figures out things like where your mouse is on the UI – which should give performance gains across the board.

Something that’s been broken since day one has been mouse remapping. Finally got around to giving that some love, and it’s working well. This is still an experimental feature, so please report any bugs you can find with it.

I’ve also been working on putting in some more feedback into the UI. Firstly, mining a resource will now show the health go down on the object as you hit it, as an overlay on your crosshair. Secondly, Binary Effects (think buffs, debuffs, i.e. “Broken Leg”) will now display above your hotbar in standard MMO style, with more information about exactly what they’re doing to you. I think this will give us a lot more legs on a system we haven’t really explored much. Think items that buff/debuff you, diseases, potions – all that cool stuff!

WIP programmer UI design, but you get the idea

Ownership Stake and Vehicle Claiming Tweaks

There’s also been a few tweaks to the recent Ownership Stake/Vehicle claiming features that were implemented in the last patch. We released that 12 hours of tooting before a stake deauthorises is… probably too much. This is now set to 20 minutes before deauthorisation. Hopefully this will stop the toot spam a bit!

There have been two changes to vehicle claiming. You can now only claim drivable cars – that is, cars that have an engine/gearbox/wheels. This should help stop freshies claiming every chassis they come across. The second change is that you can now disassemble claimed cars, but it will take much longer than a unclaimed one.

Signs

Something else that I’ve implemented recently is signs! You can place these on pretty much anything. Label your chests, make a welcome mat for your base, warn people of hazards, all that kinda stuff. Can’t wait for you peeps to show us what you do with these.

Finally, there’s more reason than ever to fear the cold. Blizzards are coming – and when they hit, you don’t want to be out in the open. Take cover and avoid the chill.

Gavku

I have begun work on some new creatures! Yay! This time I’ll be doing some big cats. Pretty much before starting any new creature I will browse for similar animals, or animals that may display traits that I think might be helpful, usually saving a few out as reference taking into consideration body shape and colour. For example I may save out some images of birds if their plumage has a certain colour range I like, even if the intended creature more resembles a rat.

If I am pretty sure what direction I want to take it when I am sketching up a reference sheet, I will colour one up, as I often then cut up that concept and paste it onto the uv’s as a texturing starting point.

This guys obvious influences are lion/sabre-tooth/ tiger/ caracal , as well as the African wild dog and hyenas.

Mils

So I’m trying to build more interesting parts to the city buildings, the parts that give it that believable human touch.

This is tricky because I have to balance the logical repetitive design of all the bits whilst making them look good and not having things in the wrong places.

It’s a kind of sadistic but enjoyable mental Rubics Cube. The texture (below) file so far has around 130 layers in it and is 4096 x 4096 resolution. This is in the upper limit of texture size. The whole city is built on the one texture, this is to save on ‘draw calls’

battlemu1e

This week I have been monitoring server performance and population. The Official 14 Day and Infini servers have retained a decent population and will continue in their present form. 7 Day Servers have seen fairly low population and will be dropped from the official servers at the end of their current wipe schedule. This will free up some room to potentially introduce a full loot server to regions that currently don’t have this type of server (looking at you Australia). I will be looking at introducing this to coincide with the Full Loot scheduled wipe next week and it should leave each region with 1 Official, 1 Full Loot and 1 Infini Wipe server moving forward.

On the back of some reports I have also spent a lot of time on official servers tracking down lag issues. This is an ongoing problem but we are making strides in isolating some of the causes. A big shout out to Theeotown of O Town NA E Infini Wipe as well as [OG]bigfukinmegaboat and the boys from [OG] on NA W Infini Wipe for helping me reproduce some of these problems. If you have any information or want to report any issue related to lag on official servers send me a message on Reddit or Steam. I’ll also be doing the rounds on the official servers over the coming weeks so feel free to say hi if you see me around. Now that DDoS issues are, for the most part, behind us sorting out lag is a priority.

On the subject of DDoS I expect that CHINA, SEA and SA servers will be migrated to new hardware over the next couple of days which signifies the end of server migrations as well as the stability issues these servers have had over the last week. For players on SA servers this change will also drastically reduce the latency problems you have been seeing the last few days.

Hurtworld Update #21

10 02, 2016

Spencer

This week I’ve been working more on getting loading external maps into the engine at runtime. One of the things we are reliant on to do this is the AssetBundle feature of Unity which has undergone a lot of work in the last couple of patches. We need these changes before we can start loading in maps the way we want, so I’ve spent part of this week working on upgrading to 5.3/5.4.

This is a bit of a juggling act of weighting up the regressions with the improvements, its currently looking like 5.3 is stable enough for us to upgrade. Initial reports also show a good improvement in client side FPS.

I’ve also been working with Mils to ensure his city objects are as re-usable as possible and will even fit into our build system. These prefabs will be re-usable to anyone making a map, and we’re optimizing them so they can be used to create large scale cities. These will go hand in hand with the map tools, I’m looking forward to seeing what game modes people create with them.

Not much flash to reveal today, but much needed progress on mapping SDK work.

Mils

This week I have been creating the outer shells for the urban buildings that I posted about earlier. This takes a bit of nutting out, so that everything works modularly.

These may be autoplaced by the game mode or built into the buildings by players. The internals are still 100% explorable, and will have different internal floor layouts.

The modularity of these should allow map creators the flexibility to vary the cities a lot.

I think some signs and adornments should add a bit more a of human touch to this giant lego set.

Cow_trix

I’ve been working on some new storm effects, implementing the C&C machine and fixing some issues around write only item slots.

In the last patch we fixed a bug that allowed swapping of write only item slots (vehicle parts). The ability to swap them wasn’t intentional but now that we’ve fixed it we realize its pretty crappy not to be able to improve a chassis.

Our ideal scenario is allow people to trash items on vehicles but not get them back. Something we’ve been thinking about for a while is context menus on items, and given we have no clean UI way to delete something without being able to move it, this is a good opportunity to add this infrastructure to our UI. This will add much more freedom to interactions you can have with items, things like unloading weapons of bullets, eating food from the UI etc.

While looking at the item system I am going to try to get some sort of asynchronous synchronization of item transitions to the client so we can drive a UI tooltip that shows progress towards things like food going rotten, smelting, freezing etc.

On the storms front, I am looking at adding a blizzard, and a meteor shower. Blizzards will make using a campfire in the snow futile and make it always a gamble to head into the snow biome without the proper gear. Meteor showers will yield rare ores, but also kill anything in its path. This may include buildings if we can figure out a way for it not to suck, maybe a buildable roof item that is meteor resistant for harsher biomes.

Gavku

Hey all, well I managed to finish up the CNC machine, baking out/painting its texture/setting up its material in Unity, then hand-balled it over to the coders 😉 Should be a fun little addition when it gets in game

Next up for me was to start work on a nest or hive for the flying-ant/wasp critters to spawn from. Initially I drew up a couple of ideas mainly focusing on trying to find an interesting shape and something that we think will work well with our environments.

This hive will be the first of our tier 2 PVE encounters. This object will be a random spawn location and will continuously spew out a large number of wasps. Killing the wasps won’t yield loot, but mining the hive will give you a much sort after resource. Getting close enough to mine it will be the challenge.

We decided to go with the middle design, which will stand about 4 player heights tall. The low poly has been built and unwrapped, and the sculpt done. Currently I am in the process of baking out the maps.

Hurtworld Update #20

03 02, 2016

Spencer

Welcome to the 20th Hurtworld Dev Blog! Not a whole lot to show from me this week as the Shigi Tools SDK integration will take a while to get things happening.

I did a bit of work this week on fixing the Vehicle performance issues caused by vehicle parts being more common. This took a little longer than expected, but should now make vehicles that aren’t currently being driven consume negligible amounts of CPU (and they were using a lot). This change will go in with this weeks wipe on Friday.

This was part of a major server performance pass cow_trx and I have been doing over the last week. These changes will hopefully give servers a much longer performance life, if not indefinitely.

Vehicle Lag

Something that might not be immediately obvious is why vehicles are so jerky when everything else is running fine.

We have very complex lag compensation systems in place for player movement. This is by no means out of the ordinary for an FPS, its called client side prediction. This (among other things) allows the server to have a CPU spike for whatever reason, catch up on things and keep running without the client showing you anything was wrong. The only time you see lag on the client movement is when the server is seriously struggling or your ping is higher than the allowed prediction amount.

Vehicles on the other hand use a physics simulation that is much harder to predict on the client, its not impossible but currently we are working with a simpler model: run everything server side then stream positions to the client (Players other than yourself steam positions also as predicting humans isn’t a good idea).

Due to this, if a client stops getting positional updates for 500ms the game has no idea where to move the vehicle and will stop in its tracks waiting for the next update from the server. Once the server gets out of its CPU spike and catches up the car starts moving again.

In short, the vehicle lag is caused by any CPU spike on the server. It could be a new player being sent heaps of construction data, a complex pathing calculation due to a creature moving through a massive base or the Unity garbage collector picking up memory that a plugin we use generated. Whatever it is, you will only really notice it while in a vehicle.

These spikes are called jitter. Hurtworld is more sensitive to jitter than other survival / MMOs games out of choice. We believe once we get our server performance is tight it will allow us to deliver a much tighter gunplay experience than is possible in games that are designed to mask large latency spikes. We think we are pretty close to nailing server performance, hopefully this patch will make some good progress.

Jitter can be hidden by increasing how far behind the current time we run our interpolation of the vehicles, the downside of this is it will greatly increase latency of driving input controls.

The only 2 solutions here are: Get servers running CPU spike free or Rewrite the vehicle network model. Right now we’re going for option A as it benefits all aspects of the game and should be an achievable goal.

Why are you working on insects when you could be working on gameplay feature X?

I get a lot of comments on our dev blogs and updates misunderstanding the way that a game development team works so I will try to clear things up here.

Our team is made up of many different roles, while it might seem that we are doing the same thing, each of our skill sets are very different.

Programmers: Fix bugs, implement features, change gameplay, balance things
Artists: Paint concept art, sculpt 3d geometry, paint textures, sometimes animate things
Battlemu1e: Started as community manager, now master of servers and infrastructure. Currently making sure your server stays online.

Since launch I’ve been prioritizing game stability over token features. Most of the content that has gone in so far has been art that slots into existing code systems as not to drag the programming team away from critical stability issues. We will have stability under control in the next few weeks, and can soon go back to the normal dev cycle of concept to implementation utilizing the whole team. This is why most of the output of the art team doesn’t have a big impact on gameplay.

Mils

Did someone say RAID?

We have been getting requests for different ways to raid a base, which is nice timing because I have been working on just that. We wanted to create a method of raiding which would make the task more fun and challenging. Nothing is set in stone at the moment, but vaguely this device will require some supervision and material consumption. This will mean the person(s) pulling off the raid will need to be there and will have to tend to the device.

Strike fear into the hearts of all noobs that cower in their bases.

The device makes a fair amount of noise, thereby attracting the attention of anyone within a certain range. This, we hope, should cause interactions between the raiders, their victims and any opportunists that pass by. The upside is the fact that it can be used more often but at the cost of the fuel materials it needs to run. (and maybe your lives). This won’t be in the game for a few weeks yet.

Cow_Trix

I’ve been working on some fairly big changes to ownership stakes and cars, putting a form of upkeep into both. Along side a server major performance pass to get things running smooth again even late into the wipe. All the following changes will be going in with the wipe on Friday.

Changes to Vehicles

Every player can now own one vehicle. You can do this by opening up it’s inventory, and clicking the CLAIM button up the top right. Anyone can drive a claimed car, and modify it’s inventory. Claimed cars won’t despawn if left alone.
Unclaimed cars WILL destroy themselves if left alone. The default time this takes is 3 hours.
You can also manually disassemble unclaimed cars with a new item – the Wrench. Disassembled cars will drop loot!
This should mean cars recycle, meaning one clan doesn’t end up with all of the coolest chassis, while everyone else has to coast around with that one Sharkweek bumper.

Changes to Ownership Stakes

You can place a maximum of 5 amber in a stake, that will be very slowly consumed. The default time per amber is 24 hours, although this is configurable by server owners. This means that, by default, an unattended ownership stake will deauthorise in 5 days. Slow servers should probably increase this time, and fast servers decrease it.
Once the stake is low on amber, it will create a plume of smoke, visible from far away. Once the stake has run out of amber, it will deauthorise everyone, do a slow countdown to destruction and eventually destroy itself, freeing up the land.
This creates player-driven decay, as you can go around clearing old abandoned bases for easy resources. Be wary though – it could be a trap! We think the cost is reasonable enough that even solo players should be able to maintain a few bases fairly easily, if they play every couple of days. It should hopefully also create some areas of contention around large bases that become abandoned and unclaimed.

This should make both bases and vehicles be more heavily contested and quickly recycled, which will extend server lifetimes a fair bit. The Wrench provides a new source of farm by allowing players to harvest old chassis that had a bad loot roll or are just unneeded. This will make the best chassis reachable by even a late-entry or solo player. I like the idea of claiming one vehicle and feeling that it’s yours and a personal possession, rather than your clan having every chassis in the game in their garage.

I’m also looking forwards to seeing how stake decay impacts the meta. We’ll be sure to be looking at feedback for it, but it’s a definite step towards servers that need wiping far less often, and means that valuable stashes of resources will pop up once in a while for others to contend over. Depending on how efficient players turn out to be at removing old structures, we may implement some more structural decay mechanics in the future for unclaimed bases.

Optimization

One of the systems we have in place to ensure the server continues to run smoothly once there are thousands of objects in the world is our load balancer. This system ensures that things that aren’t mission critical never consume too much CPU time in a single frame and cause a spike. Things like storage crates checking for owrongs that need to go moldy, and campfires checking for new objects nearby to apply heat to.

This system works pretty well for the most part, once we reach the budget for a frame we wait until next frame to do the next task in the queue. Once the server gets a bit deeper into the wipe, the number of things in the queue gets so high that the cycle rate of load balancer (how long it takes to go through the entire list of tasks once) gets so low that some tasks need to wait 15 seconds or so to get a turn. This is what causes the crazy delay on things like a campfire kicking in.

I spent some time going through all our loadbalancer items with some of our biggest savegames and tuned the costs of a few of the items, this should ensure the load balancer keeps on top of its queue for much longer into the wipe.

Gavku

Hi all, I have just gotten back from a couple of weeks off, catching up with friends and family so not a great deal from me today.Currently I am working on a CNC machine, which will give the players some more crafting options 😉 Both the high and low poly are done and I am currently working on baking it out and painting up the texture maps.

Below is the initial concept.

And this is a screen grab of the high poly

Battlemu1e

I have been working with providers around the world to get the best protection we can from DDoS attacks. For some insight into this we have been getting attacks varying from 1-5Gbps (often even larger) daily on a large percentage of our current servers. This kind of traffic is a heavy burden for any provider to deal with and requires specific network requirements in order to mitigate. Much of the time this requires providers who are experienced in providing services for game servers and working with the kind of traffic we have been seeing. Couple this with the extreme CPU overhead we need to cover per server and our server and network hardware requirements quickly escalate beyond the typical dedicated server.

This process is nearly completed, I am sure you have seen me migrating servers as new hardware goes live. At the time of writing the only servers that have not been moved onto new hardware are all Hong Kong servers, all Singapore servers, NY 1, 2 + 7 and Montreal but I am fairly confident these servers will be on new hardware by the time the wipes occur (if anything happens that causes this to not be the case I will make sure it is noted on Steam and Reddit.

It might be worth holding off starting in these regions until we can assure you that you won’t have to deal with downtime due to DDoS). Between these changes and the server performance improvements we will be implementing we should be at the end of our DDoS downtime and rubber banding woes just in time for the next wipe.

On the subject of wipes I will be introducing some changes to wipe schedules, server names and the amount of servers following the next wipe. In response to concerns that the current wipe schedule is too long, starting from the next wipe, we will be running 7 day and 14 day vanilla servers. Further to this we will also be offering Full Loot servers with a wipe schedule of 21 days, and due to significant server performance improvements, we will also be trialling ‘infini wipe’ servers. These ‘infini wipe’ servers will not have a scheduled wipe and will only be wiped when a patch requires it.

We will also be doing away with individual country servers and have servers divided by region instead (so EU Fast Wipe, EU Infini Wipe etc as opposed to Amsterdam 1, Amsterdam 2…). Keep an eye out on Reddit for a full list of what servers will be available following the wipe.

I will also be starting these servers at 18:00 local time Friday for each region. This should give everyone a better chance at starting a fresh wipe on equal footing (no more advantages for night owls). Due to this not aligning with the end of the current wipe schedule I will be slightly extending the current wipe. The way this will work is all servers will go down at approx 17:00 – 18:00 AEDT on Friday and as we get to 18:00 in each region that regions servers will come back up.

To make sure this process is smooth I will be manually starting and monitoring these servers as they go live. Any changes or updates to this I will be posting to Steam and Reddit so feel free to drop by and have a chat while waiting for your server to come online!

Dazzler

I have rounded out the Dart bug insect animation and stateflow, Cowtrix will be working on AI implementation in coming weeks. They should be a challenging creature, ranged like a Tokar but can stealth/pseudo-teleport by burrowing. I am now working on the wasp animations.