Spiral Knights - Analyzing the excessive amount of data uploaded to server that causes lag

Analyzing the excessive amount of data uploaded to server that causes lag

31 replies [Last post]

Sat, 05/07/2011 - 09:41

Everblunt

In dungeons the game will sometimes start sending a lot of data upstream to the server.

There are numerous reports of this such as:

http://forums.spiralknights.com/en/node/4917
http://forums.spiralknights.com/en/node/5175
http://forums.spiralknights.com/en/node/4565

Sometimes I can play a few levels with a reasonable amount of upload data sent, but almost always the problem comes back. This is especially troublesome since most ISP's cap upload speeds much lower then download speeds.

I used a protocol analyzer to see what is going on at the packet level. In dungeons under normal conditions, I am sending about 20-25 UDP packets per second with a size ranging from 90 bytes to 218 bytes. On the low end that is only 1800 bytes/sec (14kbps) and on the high end that could reach 5450 bytes/sec (42kbps), but in real-life usage it settles around the average of 3542 bytes/sec (~27kbps).

Under the above described excessive upload conditions, the client is still only sending about 20-25 packets per second but they grow in size until they reach a max size of 1434 bytes! That's 32982 bytes/sec or 257kbps which happens to be close to my capped upload speed. The data of these packets
seems very redundant with about 90% of the packet repeating which may indicate these large UDP packets are probably fragmented by the stack. If any of the fragments become missing or corrupted, the whole datagram is discarded. For that reason, it's not a good idea to use large UDP packets.

Without knowing exactly what's going on under the hood I can only speculate why the client is sending such large amounts of redundant data back to the server. A few ideas are that:

One client is chosen at random to be a "Main" which keeps track of mobs and such and sends this info to the server, which then relays it to the other players.
High latency causing the client to retransmit lots of data to catch up
It could just be a bug

Nobody really knows except for the devs or anyone that wants to try to reverse-engineer the netcode.

Any other thoughts or investigations would be appreciated.

Sat, 05/07/2011 - 23:22

Stardrinker

Legacy Username

This game uses UDP? I'm

This game uses UDP? I'm surprised people would still go to that protocol in an online action game where latency can vary significantly. It would explain the packet inflation and redundant packets being sent which would lead to inefficient use of bandwidth.

One client is chosen at random to be a "Main" which keeps track of mobs and such and sends this info to the server, which then relays it to the other players.

I don't think that's what's happening as I'm pretty sure all players are seeing a high amount of upload usage.

What I think is happening is, because UDP is an unreliable protocol, 3R basically decided to overcome this by getting the client to spam packets (20-25 per second? Good Looordy!) to the main server (hence the high upload usage). All players' clients would essentially be doing this and the server would be tasked with sorting out which one is the correct information and relaying it back to the client and telling it 'This is the correct location and state of everyone'.

So, I'm sure you can see what happens when there comes a time that there's lots and lots of enemies on the screen and your upload bandwidth can't keep up with the spam (Jelly King, I'm looking at you). The point of UDP is that it's fast and has low a overhead so long as you don't care about reliability, but if you're going to spam packets because you actually do need some reliability, you've sorta negated the whole reason for going UDP in the first place. You could probably fix this problem somewhat by reducing the games requirement to have to spam packets which aren't that important for gameplay. I can make the game upload 40 kBytes/s just by making my character spin around on the spot, that definately needs a throttle to its update frequency. Imagine if half the players in Haven did that at the same time. This is probably the biggest cause of packet spam I see so far.

So are the majority of these connection problems everyone's been having just a case of poor protocol choice?

Edit: Upon further thought, this is probably just a case where there isn't a limit to how often things get updated, or the limit is too high (like keeping track of which direction a player is facing). If there were some reasonable limits placed on spammy things like this, you would see a vast improvement with little detriment to gameplay.

Sun, 05/08/2011 - 02:29

Ufana

Looks like this is causing

Looks like this is causing most of my problems here, this needs to be fixed! I tested everything else - it's fine. Thanks for pointing that out!

Sun, 05/08/2011 - 07:42

Everblunt

Yep

I only thought of that first speculation because the game runs fine for me with a reasonable upload sometimes. I haven't correlated it with any particular in-game situation but battlefields and other areas with lots of mobs does seem likely. It seems to persist after the mobs are defeated though. I don't think everyone is getting high upload all of the time, just some of the time.

Sun, 05/08/2011 - 08:59

Dogrock

It is most certainly just

It is most certainly just some.

I've observed four different Windows PCs that did not suffer this issue, one instance where three were on the same network sharing a 384kbps upload. They were even managing lag free gameplay on a wireless network too. I have found that, with a few minor QoS tricks in DD-WRT, client side lag caused by network congestion can be mitigated. (Especially helpful on a wireless network it seems)

I know during Jelly King and Vanaduke it is no uncommon to see a consistent 70kbps upload rate. Those with low upload limits, especially those on wireless broadband, will see problems during these events.

I'm also wondering how much this could be aggravated by loss prone wireless connections or faulty NIC software.

Sun, 05/08/2011 - 11:45

Ufana

I seem to be getting upload

I seem to be getting upload spikes ofen when mobs spawn. Also there seem to be more random upload spikes in firestorm than anywhere else.

What "QoS tricks" are you talking about? My client just seems to wait / freeze until it uploaded what ever it wants to upload most of the time (unless I get this symptom mixed up with something memory-leak related that seems to be coinciding with my upload spikes?) - not sure what to do about that. Can the "important requests" in some way be recognized and given a higher priority?

Sun, 05/08/2011 - 18:03

Dogrock

http://www.dd-wrt.com/wiki/in

http://www.dd-wrt.com/wiki/index.php/QoS

Though this type of network management will only help on a congested or heavily shared network. If you are the sole user and you are not running other bandwidth intensive applications this will do nothing.

Tue, 05/10/2011 - 12:23

Everblunt

Another idea

I can understand an increased download during mob-intensive events but upload? Assuming the server has control over mob position and state why does the client need to echo all of it back to the server? It's almost like each client runs independently and does a sanity check with the server for everything.

Tue, 05/10/2011 - 16:27

DjParagon

Legacy Username

I think my problem is related

I think my problem is related with this topic.

When I play alone or with one or two random players, I have no problems at all. But when I play with two specific friends, everytime, I'm being kicked off the game, with modem disconnection for a few seconds, or I experience an incredible lag that often cause a disconnection of the game, and then I have to enter my password again to be able to join them. This connection crash happened two times tonight, and I had to reset my modem once to recover. My friends have no problem at all, exept maybe some minor lags.

I've played a few games online, and in four years, this never happened in any games. I'm guessing we are too far away from each other, but in Test Drive Unlimited for exemple, I play with the same friends and we never had any connection problem exept for rare minor lags.

I have a 4.5 Mbps down and 0.65 Mbps up connection. I will try to take a look at the connection speed when playing.

Tue, 05/10/2011 - 16:28

Everblunt

A few more observations

Notes from my last session:

I have now had the "upload" occur in terminals and Moorcroft Manor
The "upload" can occur both in groups and solo
I have had the "upload" occur, terminated the game process, relaunched, and completed the area normally

Tue, 05/10/2011 - 21:04

#10

Stay

Notes from my last session:

: Notes from my last session:
I have now had the "upload" occur in terminals and Moorcroft Manor
The "upload" can occur both in groups and solo
I have had the "upload" occur, terminated the game process, relaunched, and completed the area normally

For what it's worth I've noticed that on my end all of these are true. It seems to me (from my experience alone) that certain areas on maps will cause this problem, mobs present or not. Most notably in the first terminal if I walk towards the crystals and certain areas of Moorcroft (can't really test it in combat levels due to level randomness and the fact that this issue reoccurring as often as it does [approx average 3 per level] really drains motivation to keep playing).
If you manage to lag-walk your way out of the area where the lag started, crash client, log in, then usually it'll be fine until the next occurrence. Needless to say if this happens near mobs or a time sensitive puzzle then you're dead or stuck.

Just to reiterate; this is only what I've observed, not stating it is fact. Has anyone else noticed it to be a position based event?
Final note this very rarely occurs in arenas, which is the highest density of mobs I've seen so far.

Wed, 05/11/2011 - 20:36

#11

Everblunt

Some more observations

Here are a few more things I have observed:

I can sit in a terminal and have upload come and go just by idling
I "pulled the plug" during excessive upload and client still happily sends away
I "pulled the plug" during normal upload and client still sends normally until d/c
I started a downlink-saturating http file download during normal upload with no change (but some very strange lag from other players)
Deconstruction zones seem more likely to trigger the upload

Furthermore I have gathered more technical data about the netcode. In the following images the yellow-highlighted areas are some unknown set of data, but it is always the same size. The trailing red-highlighted contains your character's position and direction data (probably some other things as well). When walking around the size of this block varies in length and seems to contain a short "history" of your movements.

Normal Operation Example
Excessive Upload Example

Note how the position and direction data is repeated excessively in the 2nd example (it goes on for much longer but is cut off in the image). This suggests this is indeed some kind of bug causing a large amount of location data to be appended to the packet for some reason. There is no logical reason to send this much redundant data.

Thu, 05/12/2011 - 03:43

#12

Ufana

Could you give some hints on

Could you give some hints on how to distinguish that from "normal bad connection" lag in a case where the available upstream rate can't handle the spikes? I've got..
- upload spikes that correlate with download spikes (or where one precedes another)
- upload spikes that don't correlate with download spikes
- spikes where packages per second correlates with bytes per second.
- spikes where bytes per second / packages per second ratio is way off (big packages)

I guess the last one would be the ones you describe? Not sure where to go from there... try & filter big packages? Or go all regex on the raw data stream & try to eliminate that repetition pattern? I'm not very good at this... -.-"

edit: the most annoying lag issues I have are best visible on a traffic graph, if I plot max. frame.len on y-axis. While everything else looks mostly chaotic / random or is hard to recognize as the point at which I had a lag spike ingame, a 1-4 seconds cluster of elevated max frame.len in outgoing packages seems to be clearly recognizable every time. I tested by trying to run in a square surrounded by spikes by the way with no mobs or other players present. Can't provide raw data examples because I suck at not accidentally publishing

(mmmh, something wrong with my data I think, gotta check again later)

Thu, 05/12/2011 - 08:26

#13

0ritfx

Legacy Username

Short-term lame solution (if

Short-term lame solution (if you want to know how it works, then go ahead and use your head): lower game priority. This can indirectly reduce the number of redundant packages without losing fps. This is not a real solution and can (does not have to!) mitigate the onset of the problem, but once the things will start getting uploaded there will be no escape.
Regarding the uploaded trash data itself: nothing you can do now folks. Not going into details: capping the sent amount of data is nonsense, as gibberish is produced along with useful data and uploaded on equal priority.
Hope the dews are working on it since - judging by rising energy prices - people have ceased to buy energy for real cash.

Thu, 05/12/2011 - 14:05

#14

Bifurcate

Lead Programmer

I've been working with a

I've been working with a member of our support personnel to debug this issue, and have added a couple of fixes that should help when they're released with our next update.

Fri, 05/13/2011 - 01:59

#15

0ritfx

Legacy Username

Any energy refund is planned?

Fri, 05/13/2011 - 05:04

#16

Ufana

I'm just glad someone's

I'm just glad someone's working on this and I'm looking forward to testing how It'll turn out to work. Maybe I'll even be able to walk in a circle for 30 seconds without lagging away 5 tiles after that patch :D - there's not much missing for my internet connection to be able to handle the outgoing packages, so a little reduction might with some luck already do the job. Thanks for the update!

Fri, 05/13/2011 - 08:58

#17

Stay

Thanks for letting us know.

Thanks for letting us know. Hate to be 'that guy' but any idea when the patch will be released?

Fri, 05/13/2011 - 13:59

#18

Bifurcate

Lead Programmer

Next week; most likely

Next week; most likely Tuesday.

Fri, 05/13/2011 - 14:04

#19

Dogrock

Cool, thanks for letting us

Cool, thanks for letting us know andrzej.

Fri, 05/13/2011 - 22:25

#20

Stay

Cheers

Cheers mate. Really looking forward to it given how fun this game is.

Sat, 05/14/2011 - 00:18

#21

mirrorminx

Legacy Username

Thanks for the hard work

Great to hear that some of the lag issues will soon be resolved

Mon, 05/16/2011 - 01:37

#22

0ritfx

Legacy Username

1. Thank you Andrew. 2. They

1. Thank you Andrew.
2. They had to take care of it. Energy purchase fell down considerably likely due to lag issues. This game has to pay for itself. The players actually made a difference!

Mon, 05/16/2011 - 09:40

#23

Feynt

Legacy Username

Perhaps in a related issue, a

Perhaps in a related issue, a friend of mine has noticed that over time the game client will balloon from 600-800 megs of RAM to over 1 gig, and performance degrades significantly for him after that point. Killing the process and rejoining lets him continue as if nothing was wrong at all. I don't see this kind of increase (it broaches 1 gig but never really passes it), nor do I see any performance degradation on my side, but then usually I only spend half an hour playing, or I'll spend about as long as him but broken up by having to go help with chores and logging off for a few minutes. Usually he'll be playing for a few hours straight. This sounds very much like it's a memory leak, but it's Java, so the only ways that could happen is if you have all the assets you load still referenced in memory so the garbage collection doesn't work. Considering all of this is being "streamed" off the hard drive and it's (almost) all very small asset wise, I think you can ditch most of the loaded data when you return to Haven.

Mon, 05/16/2011 - 11:21

#24

Dogrock

I don't think it's the

I don't think it's the private memory used, how large the game gets it dependent on how much memory is available. I've had the client at 1.5GB for the preview player head start (marathon playing) with no issues, but I also have 6GB of physical memory available. However, I can play just as much on my laptop (4GB physical memory and more "support" programs running) and pushing the client beyond 800MB is a challenge. There's no oddities there either. Though I do have a desktop that can sort of "brute force" it's way though wads of data.

It does appear that the game likes to cache as much as possible relative to the physical memory available. The longer I play and the more that winds up in the cache the faster loading times become without other signs of degradation.

Though for people using a laptop with virtual memory (and low physical memory) on a slow hard drive plus streaming fresh data at the same time the game will perform amazingly poorly. My laptop will take ages to load due to a slow drive, even without virtual memory allocated to it. This results in some nasty mini-freezes when the game loads a large wave of monsters for the first time.

Tue, 05/17/2011 - 10:02

#25

Agsfest

Legacy Username

I have a similar problem to

I have a similar problem to DjParagon. Playing alone can be fine, but in a party of four I get incredible lag, and sometimes I will DC five times in one run. When I first started playing, I assumed this was a problem on the game's part, since playing over wireless seemed very unreliable (by contrast, I player world of warcraft for 3 years on wireless with no problems). Unfortunately the times I have submitted a ticket to OOO they have blamed by ISP, which has had no serious connection problems for a year.

Thanks for pointing this data out there; it's way over my head, but it sounds like what's causing my problems. I hope they can fix it soon.

Thu, 06/02/2011 - 04:19

#26

redpumpkin

Legacy Username

Lag fixed yet?

Im from australia and the lag is quite an issue. I am always putting my shield up too late and being lagged into spikes/monster. It is quite frustrating and I just want to know if the issue is still pending a fix.

Sun, 06/05/2011 - 04:40

#27

Ufana

I'm not sure but I think my

I'm not sure but I think my outgoing packages trouble got a lot better. What lag was left after that I could - looking at the cleaner traffic statistics - attribute to a faulty wpa interface: Almost no lag left after changing to a better wireless adapter.

What I do notice most now, are strange lags that seem to occur when player from another continent / far away location join. Sometimes, lag even seems to switch forth and back from that player and me, so for both of us the game is only playable half the time. Faking disconnect by killing java sometimes seems to resolve that problem.

Wed, 06/08/2011 - 06:05

#28

Brandier

Legacy Username

Russia here. I've been having

Russia here. I've been having severe problems with lag and the game rubber-banding on me ever since the latest patch came out. The game ran very smoothly before. These days, my character will keep being thrown around all over the map regardless of where I am and how many players I'm running with. I've been playing for several hours a day during the 2-3 weeks since I joined the game and everything was all right. But now, I cannot play for 10 minutes because my character keeps being thrown around the area, forced into spikes, lagged into being incapable of attacking and worse. Thankfully, I haven't lost any CE because the game has been too unplayable for me to reach the point where I'd need to use it.

There wasn't a single problem when I played this game before, so I'm absolutely convinced that this has something to do with the most recent patch.

Sat, 06/11/2011 - 11:45

#29

Moltare

Legacy Username

Regarding Brandier's comment

Regarding Brandier's comment above: my experience has been exactly the same - went from perfectly reasonable performance to unplayable lagspiking immediately after the last patch.

Sat, 06/11/2011 - 12:03

#30

Providence

Legacy Username

Some users with specific

Some users with specific types of routers have been having issues since the last patch--but they can't even log in. It seems to affect mostly Cisco/Linksys routers of a certain series. I personally had pretty bad lag on my old Belkin router, but I recently upgraded and haven't had any problems since. I'm on the 5Ghz band, though, and my neighbors are not----so I'm no longer getting any interference from them the way I used to.

What program are you guys using to monitor the upload/download activity? I would like to see what the difference is between my old and new router, but I have no experience with this sort of thing.

Sat, 07/09/2011 - 14:26

#31

Crazyotter

Same

I have the same problem as Brandier and Moltare: I started lagging super bad after the patch. Sometimes the game will simply stop for 4 seconds or more before finally unfreezing.

Regarding Providence's post, I do use a Linksys wireless router, but I could play decently well before the new patch. I also tried updating Java in hopes that something would change, but it didn't...

Search

Analyzing the excessive amount of data uploaded to server that causes lag