This week I’ve been working more on getting loading external maps into the engine at runtime. One of the things we are reliant on to do this is the AssetBundle feature of Unity which has undergone a lot of work in the last couple of patches. We need these changes before we can start loading in maps the way we want, so I’ve spent part of this week working on upgrading to 5.3/5.4.
This is a bit of a juggling act of weighting up the regressions with the improvements, its currently looking like 5.3 is stable enough for us to upgrade. Initial reports also show a good improvement in client side FPS.
I’ve also been working with Mils to ensure his city objects are as re-usable as possible and will even fit into our build system. These prefabs will be re-usable to anyone making a map, and we’re optimizing them so they can be used to create large scale cities. These will go hand in hand with the map tools, I’m looking forward to seeing what game modes people create with them.
Not much flash to reveal today, but much needed progress on mapping SDK work.
This week I have been creating the outer shells for the urban buildings that I posted about earlier. This takes a bit of nutting out, so that everything works modularly.
These may be autoplaced by the game mode or built into the buildings by players. The internals are still 100% explorable, and will have different internal floor layouts.
The modularity of these should allow map creators the flexibility to vary the cities a lot.
I think some signs and adornments should add a bit more a of human touch to this giant lego set.
I’ve been working on some new storm effects, implementing the C&C machine and fixing some issues around write only item slots.
In the last patch we fixed a bug that allowed swapping of write only item slots (vehicle parts). The ability to swap them wasn’t intentional but now that we’ve fixed it we realize its pretty crappy not to be able to improve a chassis.
Our ideal scenario is allow people to trash items on vehicles but not get them back. Something we’ve been thinking about for a while is context menus on items, and given we have no clean UI way to delete something without being able to move it, this is a good opportunity to add this infrastructure to our UI. This will add much more freedom to interactions you can have with items, things like unloading weapons of bullets, eating food from the UI etc.
While looking at the item system I am going to try to get some sort of asynchronous synchronization of item transitions to the client so we can drive a UI tooltip that shows progress towards things like food going rotten, smelting, freezing etc.
On the storms front, I am looking at adding a blizzard, and a meteor shower. Blizzards will make using a campfire in the snow futile and make it always a gamble to head into the snow biome without the proper gear. Meteor showers will yield rare ores, but also kill anything in its path. This may include buildings if we can figure out a way for it not to suck, maybe a buildable roof item that is meteor resistant for harsher biomes.
Hey all, well I managed to finish up the CNC machine, baking out/painting its texture/setting up its material in Unity, then hand-balled it over to the coders 😉 Should be a fun little addition when it gets in game
Next up for me was to start work on a nest or hive for the flying-ant/wasp critters to spawn from. Initially I drew up a couple of ideas mainly focusing on trying to find an interesting shape and something that we think will work well with our environments.
This hive will be the first of our tier 2 PVE encounters. This object will be a random spawn location and will continuously spew out a large number of wasps. Killing the wasps won’t yield loot, but mining the hive will give you a much sort after resource. Getting close enough to mine it will be the challenge.
We decided to go with the middle design, which will stand about 4 player heights tall. The low poly has been built and unwrapped, and the sculpt done. Currently I am in the process of baking out the maps.
Hurtworld Update #20
03 02, 2016
Welcome to the 20th Hurtworld Dev Blog! Not a whole lot to show from me this week as the Shigi Tools SDK integration will take a while to get things happening.
I did a bit of work this week on fixing the Vehicle performance issues caused by vehicle parts being more common. This took a little longer than expected, but should now make vehicles that aren’t currently being driven consume negligible amounts of CPU (and they were using a lot). This change will go in with this weeks wipe on Friday.
This was part of a major server performance pass cow_trx and I have been doing over the last week. These changes will hopefully give servers a much longer performance life, if not indefinitely.
Something that might not be immediately obvious is why vehicles are so jerky when everything else is running fine.
We have very complex lag compensation systems in place for player movement. This is by no means out of the ordinary for an FPS, its called client side prediction. This (among other things) allows the server to have a CPU spike for whatever reason, catch up on things and keep running without the client showing you anything was wrong. The only time you see lag on the client movement is when the server is seriously struggling or your ping is higher than the allowed prediction amount.
Vehicles on the other hand use a physics simulation that is much harder to predict on the client, its not impossible but currently we are working with a simpler model: run everything server side then stream positions to the client (Players other than yourself steam positions also as predicting humans isn’t a good idea).
Due to this, if a client stops getting positional updates for 500ms the game has no idea where to move the vehicle and will stop in its tracks waiting for the next update from the server. Once the server gets out of its CPU spike and catches up the car starts moving again.
In short, the vehicle lag is caused by any CPU spike on the server. It could be a new player being sent heaps of construction data, a complex pathing calculation due to a creature moving through a massive base or the Unity garbage collector picking up memory that a plugin we use generated. Whatever it is, you will only really notice it while in a vehicle.
These spikes are called jitter. Hurtworld is more sensitive to jitter than other survival / MMOs games out of choice. We believe once we get our server performance is tight it will allow us to deliver a much tighter gunplay experience than is possible in games that are designed to mask large latency spikes. We think we are pretty close to nailing server performance, hopefully this patch will make some good progress.
Jitter can be hidden by increasing how far behind the current time we run our interpolation of the vehicles, the downside of this is it will greatly increase latency of driving input controls.
The only 2 solutions here are: Get servers running CPU spike free or Rewrite the vehicle network model. Right now we’re going for option A as it benefits all aspects of the game and should be an achievable goal.
Why are you working on insects when you could be working on gameplay feature X?
I get a lot of comments on our dev blogs and updates misunderstanding the way that a game development team works so I will try to clear things up here.
Our team is made up of many different roles, while it might seem that we are doing the same thing, each of our skill sets are very different.
Programmers: Fix bugs, implement features, change gameplay, balance things
Artists: Paint concept art, sculpt 3d geometry, paint textures, sometimes animate things
Battlemu1e: Started as community manager, now master of servers and infrastructure. Currently making sure your server stays online.
Since launch I’ve been prioritizing game stability over token features. Most of the content that has gone in so far has been art that slots into existing code systems as not to drag the programming team away from critical stability issues. We will have stability under control in the next few weeks, and can soon go back to the normal dev cycle of concept to implementation utilizing the whole team. This is why most of the output of the art team doesn’t have a big impact on gameplay.
Did someone say RAID?
We have been getting requests for different ways to raid a base, which is nice timing because I have been working on just that. We wanted to create a method of raiding which would make the task more fun and challenging. Nothing is set in stone at the moment, but vaguely this device will require some supervision and material consumption. This will mean the person(s) pulling off the raid will need to be there and will have to tend to the device.
Strike fear into the hearts of all noobs that cower in their bases.
The device makes a fair amount of noise, thereby attracting the attention of anyone within a certain range. This, we hope, should cause interactions between the raiders, their victims and any opportunists that pass by. The upside is the fact that it can be used more often but at the cost of the fuel materials it needs to run. (and maybe your lives). This won’t be in the game for a few weeks yet.
I’ve been working on some fairly big changes to ownership stakes and cars, putting a form of upkeep into both. Along side a server major performance pass to get things running smooth again even late into the wipe. All the following changes will be going in with the wipe on Friday.
Changes to Vehicles
Every player can now own one vehicle. You can do this by opening up it’s inventory, and clicking the CLAIM button up the top right. Anyone can drive a claimed car, and modify it’s inventory. Claimed cars won’t despawn if left alone. Unclaimed cars WILL destroy themselves if left alone. The default time this takes is 3 hours. You can also manually disassemble unclaimed cars with a new item – the Wrench. Disassembled cars will drop loot! This should mean cars recycle, meaning one clan doesn’t end up with all of the coolest chassis, while everyone else has to coast around with that one Sharkweek bumper.
Changes to Ownership Stakes
You can place a maximum of 5 amber in a stake, that will be very slowly consumed. The default time per amber is 24 hours, although this is configurable by server owners. This means that, by default, an unattended ownership stake will deauthorise in 5 days. Slow servers should probably increase this time, and fast servers decrease it. Once the stake is low on amber, it will create a plume of smoke, visible from far away. Once the stake has run out of amber, it will deauthorise everyone, do a slow countdown to destruction and eventually destroy itself, freeing up the land. This creates player-driven decay, as you can go around clearing old abandoned bases for easy resources. Be wary though – it could be a trap! We think the cost is reasonable enough that even solo players should be able to maintain a few bases fairly easily, if they play every couple of days. It should hopefully also create some areas of contention around large bases that become abandoned and unclaimed.
This should make both bases and vehicles be more heavily contested and quickly recycled, which will extend server lifetimes a fair bit. The Wrench provides a new source of farm by allowing players to harvest old chassis that had a bad loot roll or are just unneeded. This will make the best chassis reachable by even a late-entry or solo player. I like the idea of claiming one vehicle and feeling that it’s yours and a personal possession, rather than your clan having every chassis in the game in their garage.
I’m also looking forwards to seeing how stake decay impacts the meta. We’ll be sure to be looking at feedback for it, but it’s a definite step towards servers that need wiping far less often, and means that valuable stashes of resources will pop up once in a while for others to contend over. Depending on how efficient players turn out to be at removing old structures, we may implement some more structural decay mechanics in the future for unclaimed bases.
One of the systems we have in place to ensure the server continues to run smoothly once there are thousands of objects in the world is our load balancer. This system ensures that things that aren’t mission critical never consume too much CPU time in a single frame and cause a spike. Things like storage crates checking for owrongs that need to go moldy, and campfires checking for new objects nearby to apply heat to.
This system works pretty well for the most part, once we reach the budget for a frame we wait until next frame to do the next task in the queue. Once the server gets a bit deeper into the wipe, the number of things in the queue gets so high that the cycle rate of load balancer (how long it takes to go through the entire list of tasks once) gets so low that some tasks need to wait 15 seconds or so to get a turn. This is what causes the crazy delay on things like a campfire kicking in.
I spent some time going through all our loadbalancer items with some of our biggest savegames and tuned the costs of a few of the items, this should ensure the load balancer keeps on top of its queue for much longer into the wipe.
Hi all, I have just gotten back from a couple of weeks off, catching up with friends and family so not a great deal from me today.Currently I am working on a CNC machine, which will give the players some more crafting options 😉 Both the high and low poly are done and I am currently working on baking it out and painting up the texture maps.
Below is the initial concept.
And this is a screen grab of the high poly
I have been working with providers around the world to get the best protection we can from DDoS attacks. For some insight into this we have been getting attacks varying from 1-5Gbps (often even larger) daily on a large percentage of our current servers. This kind of traffic is a heavy burden for any provider to deal with and requires specific network requirements in order to mitigate. Much of the time this requires providers who are experienced in providing services for game servers and working with the kind of traffic we have been seeing. Couple this with the extreme CPU overhead we need to cover per server and our server and network hardware requirements quickly escalate beyond the typical dedicated server.
This process is nearly completed, I am sure you have seen me migrating servers as new hardware goes live. At the time of writing the only servers that have not been moved onto new hardware are all Hong Kong servers, all Singapore servers, NY 1, 2 + 7 and Montreal but I am fairly confident these servers will be on new hardware by the time the wipes occur (if anything happens that causes this to not be the case I will make sure it is noted on Steam and Reddit.
It might be worth holding off starting in these regions until we can assure you that you won’t have to deal with downtime due to DDoS). Between these changes and the server performance improvements we will be implementing we should be at the end of our DDoS downtime and rubber banding woes just in time for the next wipe.
On the subject of wipes I will be introducing some changes to wipe schedules, server names and the amount of servers following the next wipe. In response to concerns that the current wipe schedule is too long, starting from the next wipe, we will be running 7 day and 14 day vanilla servers. Further to this we will also be offering Full Loot servers with a wipe schedule of 21 days, and due to significant server performance improvements, we will also be trialling ‘infini wipe’ servers. These ‘infini wipe’ servers will not have a scheduled wipe and will only be wiped when a patch requires it.
We will also be doing away with individual country servers and have servers divided by region instead (so EU Fast Wipe, EU Infini Wipe etc as opposed to Amsterdam 1, Amsterdam 2…). Keep an eye out on Reddit for a full list of what servers will be available following the wipe.
I will also be starting these servers at 18:00 local time Friday for each region. This should give everyone a better chance at starting a fresh wipe on equal footing (no more advantages for night owls). Due to this not aligning with the end of the current wipe schedule I will be slightly extending the current wipe. The way this will work is all servers will go down at approx 17:00 – 18:00 AEDT on Friday and as we get to 18:00 in each region that regions servers will come back up.
To make sure this process is smooth I will be manually starting and monitoring these servers as they go live. Any changes or updates to this I will be posting to Steam and Reddit so feel free to drop by and have a chat while waiting for your server to come online!
I have rounded out the Dart bug insect animation and stateflow, Cowtrix will be working on AI implementation in coming weeks. They should be a challenging creature, ranged like a Tokar but can stealth/pseudo-teleport by burrowing. I am now working on the wasp animations.
Hurtworld Update #19
27 01, 2016
It’s been a pretty chaotic couple of weeks due to the DDOS fairy visiting us more and more frequently. Battlemu1e has been hard at work moving all our servers to new providers that have rock solid DDOS protection, which should fix all the downtime. Battlemu1e also left without adding his devblog update describing the exact state of all the servers so you’ll have to head to reddit for that.
Since the content patch we have been experiencing some CPU issues which appears to be caused by the fact that every man and his dog has a car now. Cars with wheels cost a fair bit of CPU, and since the patch pretty much all chassis on populated servers are drivable in somebody’s base chewing up our megahertz. We will be putting out a patch in the next couple of days to hopefully address this issue along with getting kicked out of vehicles bug.
This week I’ve started work on getting the first pieces of the map / gamemode SDK up and running.
I am still very much in R&D phase, testing out what is possible, finding ways we can leverage Unity to do the heavy lifting for us.
The first step is to allow content creators to author maps using a free install of the Unity editor running out Mapping SDK “ShigiTools”.
ShigiTools is a set of helpers that currently supports:
Terrain heightmap generation
Procedural detail, tree & grass scattering
Procedural texture painting,
Loot spawn locations
The learning curve on these will be steep, so don’t expect a point and click level editor like GTA 5. They will however be powerful.
The challenges I am currently working through are how we will package and distribute map files.
Ideally, the process will work like this:
You download and install a free version Unity 5
You load up our SDK project
Edit map until satisfied
Run command in ShigiTools to compile and export a Unity asset bundle containing everything your map needs.
Put asset bundle in folder on server and client
Host game on new map
Distribution of asset bundles could be done lots of different ways. Likely we will use Steam Workshop to serve up files. In an ideal world it will work in engine just like Dota2.
Our game framework already supports multiple maps, which will make integrating maps just a matter of loading them into the engine. What we don’t support currently is multiple configurations for all the things inside the map like Biome stats and visualization, Creature Stats, Player metabolism, Machine behavior, loot tables, items etc.
Step 2 of this process will be taking all this configuration data which is currently in Fluent C# code and moving it to configuration scripts that are loaded at runtime (XML or something similar)
There will be some challenges here but nothing we didn’t plan for.
Why are you working on this before you’ve implemented feature X?
The sooner we do this, the sooner we reap the benefits of the content the community will create, hopefully that will give people a more sustainable flow of content while we work on engine systems and new features. Also because I feel like I’ve been on a bug fixing treadmill and need to get into some juicy engine system code.
What started out as a simple skyscraper ruin, has become a bit of an epic idea. I made one floor of a square shaped Skyscraper/Office Building with an inbuilt stairwell, which you then stack. This instantly gives you a rough concrete multi storey building with a fairly low texture and polygon overhead. It looks like we are going to break these into smaller pieces and create large format building prefabs.
We hope to be able to create buildable cities with this method. Possibly, you will be able to take hold of the whole thing by locking off the stairwell and adding windows or walls to the outside. Whether these are built from scratch within the game by players or pre-built in the map, or both, is unclear as yet.
I have added Elevated Highway sections, some signs and lights, which we would like to add also. These are built conservatively so as to keep the polycounts and texture counts down also. Performance is always taken into consideration.
Recently, I’ve had fun implementing all the new traps, putting in a few fixes for chat (hopefully the next patch will be the last of those tags becoming visible), and implementing some new features. We’re going to try ownership stakes having some form of upkeep – nothing too strenuous or annoying, but something that means that if someone isn’t online for days, their ownership over a base can be contested. Of course we’ll make how long totems last configurable, for casual players on slower servers.
I’ve also been working on the Wrench, a new tool that can disassemble cars into loot. Hopefully this should allow people to finally trash all those crappy chassis.
Finally, I’ll also be looking at drills, and rebalancing them to be slightly less overpowered than they were pre-nerf. We’ve got lots of feedback about what you guys think the drills should play like, and what their place in the meta is, so I’ll be trying to take that all into account.
I have been working on the new creatures. They are all rigged up and being animated (For the layperson, rigging is where you create all the controls and skinning to animate the creature). Here is a sampler of the yet-to-be-named dart bug. It get aggressive and shakes like a rattle snake before flinging ranged darts at you from it’s back. The weak point is the end of the tail, the rest will be pretty well-armored.