Symptom: Lag spikes in blast network, usually during large blast size chains.
Observations
- CPU & GPU & Memory usage normal
- Network bandwidth usage within limits (my connection is 10Mbit down, 1Mbit up)
- Throughput full-duplex UDP test gives normal speeds and 0% packet loss
- During a match, almost 10% of my incoming UDP packets are over 1000 bytes!
- During a match, I have found an inverse correlation with number of packets received and packet size See this image. Black is incoming packets/sec, red is incoming packets/sec where the length > 1000 bytes. The red spikes also correlate with lag in the game
What's in the packets?
Here's what a typical ~600 byte packet contains:
0000 84 2b 2b bc af ac 00 16 b6 db 14 cc 08 00 45 00 .++...........E.
0010 02 22 00 00 40 00 2b 11 26 5b b8 49 97 10 c0 a8 ."..@.+.&[.I....
0020 17 6e ba 0d fc 2d 02 0e ec 49 00 00 2b 41 00 00 .n...-...I..+A..
0030 38 6e 00 02 ff ff 00 03 00 00 08 21 00 00 08 3f 8n.........!...?
0040 00 00 00 00 00 5c 00 03 76 8a 00 03 76 f9 00 3d .....\..v...v..=
0050 00 04 00 00 00 03 00 1d 00 06 00 45 00 07 00 00 ...........E....
0060 00 03 00 1a 00 0b 00 00 01 02 00 40 00 0b 00 00 ...........@....
0070 00 64 00 41 00 08 00 00 00 00 bf 80 00 00 00 00 .d.A............
0080 05 2c 00 03 76 bc 7f ff ff ff 00 08 c0 20 00 00 .,..v........ ..
0090 c0 d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00a0 00 00 00 00 ff 80 00 00 01 00 00 00 01 00 00 05 ................
00b0 28 00 1d 00 06 00 3f 00 07 00 00 00 03 00 1a 00 (.....?.........
00c0 0b 00 00 01 02 00 40 00 0b 00 00 00 64 00 41 00 ......@.....d.A.
00d0 08 bf 80 00 00 00 00 00 00 00 00 05 2b 00 03 76 ............+..v
00e0 bc 7f ff ff ff 00 08 c0 90 00 00 c0 90 00 00 00 ................
00f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ff ................
0100 80 00 00 01 00 00 00 01 00 00 05 26 00 1d 00 06 ...........&....
0110 00 3f 00 07 00 00 00 03 00 1a 00 0b 00 00 01 02 .?..............
0120 00 40 00 0b 00 00 00 64 00 41 00 08 3f 80 00 00 .@.....d.A..?...
0130 00 00 00 00 00 00 05 2a 00 03 76 bc 7f ff ff ff .......*..v.....
0140 00 08 bf 00 00 00 c0 90 00 00 00 00 00 00 00 00 ................
0150 00 00 00 00 00 00 00 00 00 00 ff 80 00 00 01 00 ................
0160 00 00 01 00 00 05 25 00 1c 00 00 00 0c 00 02 ff ......%.........
0170 0f 00 00 05 28 00 1d 02 00 00 03 76 e6 00 00 00 ....(......v....
0180 a3 00 11 04 00 00 00 02 00 08 c0 c8 a5 ab be cd ................
0190 66 f6 00 00 00 05 00 00 05 26 00 1d 02 00 00 03 f........&......
01a0 76 e6 00 00 05 22 00 1d 02 00 00 03 76 e6 00 00 v...."......v...
01b0 05 24 00 1d 02 00 00 03 76 e6 00 00 05 29 00 1d .$......v....)..
01c0 02 00 00 03 76 e6 00 00 05 25 00 1d 02 00 00 03 ....v....%......
01d0 76 e6 00 00 00 a2 00 11 08 00 00 00 02 be b9 06 v...............
01e0 3b 00 00 00 00 00 00 05 27 00 1d 02 00 00 03 76 ;.......'......v
01f0 e6 00 00 05 21 00 1d 02 00 00 03 76 e6 00 00 05 ....!......v....
0200 20 00 1d 02 00 00 03 76 e6 00 00 05 23 00 1d 02 ......v....#...
0210 00 00 03 76 e6 01 00 00 00 05 00 00 05 1d 00 00 ...v............
0220 05 1c 00 00 05 1a 00 00 05 1e 00 00 05 1b 00 00 ................
Here's what a large packet contains. Note the repeating data near the bottom.
0000 84 2b 2b bc af ac 00 16 b6 db 14 cc 08 00 45 00 .++...........E.
0010 05 71 00 00 40 00 2b 11 23 0c b8 49 97 10 c0 a8 .q..@.+.#..I....
0020 17 6e ba 0d fc 2d 05 5d 62 12 00 00 2b 3c 00 00 .n...-.]b...+<..
0030 38 67 00 02 ff ff 00 03 00 00 08 21 00 00 08 3f 8g.........!...?
0040 00 00 00 00 00 59 00 03 75 65 00 03 75 df 00 3e .....Y..ue..u..>
0050 00 04 00 00 00 06 00 1d 00 06 00 45 00 07 00 00 ...........E....
0060 00 03 00 1a 00 0b 00 00 01 0e 00 40 00 0b 00 00 ...........@....
0070 00 64 00 41 00 08 00 00 00 00 3f 80 00 00 00 00 .d.A......?.....
0080 05 1c 00 03 75 a1 7f ff ff ff 00 08 c0 90 00 00 ....u...........
0090 40 f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 @...............
00a0 00 00 00 00 ff 80 00 00 01 00 00 00 01 00 00 05 ................
00b0 15 00 1d 00 06 00 3f 00 07 00 00 00 03 00 1a 00 ......?.........
00c0 0b 00 00 01 0e 00 40 00 0b 00 00 00 64 00 41 00 ......@.....d.A.
00d0 08 bf 80 00 00 00 00 00 00 00 00 05 1d 00 03 75 ...............u
00e0 a1 7f ff ff ff 00 08 c0 d0 00 00 40 b0 00 00 00 ...........@....
00f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ff ................
0100 80 00 00 01 00 00 00 01 00 00 05 14 00 1d 00 06 ................
0110 00 3f 00 07 00 00 00 03 00 1a 00 0b 00 00 01 02 .?..............
0120 00 40 00 0b 00 00 00 64 00 41 00 08 bf 80 00 00 .@.....d.A......
0130 00 00 00 00 00 00 05 1a 00 03 75 a1 7f ff ff ff ..........u.....
0140 00 08 c0 60 00 00 c0 90 00 00 00 00 00 00 00 00 ...`............
0150 00 00 00 00 00 00 00 00 00 00 ff 80 00 00 01 00 ................
0160 00 00 01 00 00 05 19 00 1d 00 06 00 3d 00 07 00 ............=...
0170 00 00 02 00 1a 00 0b 00 00 01 02 00 3e 00 0a 00 ............>...
0180 02 31 35 00 00 05 1f 00 03 75 c2 7f ff ff ff 00 .15......u......
0190 08 bf 00 00 00 40 f0 00 00 00 00 00 00 00 00 00 .....@..........
01a0 00 00 00 00 00 00 00 00 00 ff 80 00 00 01 00 00 ................
01b0 00 01 00 00 00 a1 00 1d 00 06 00 45 00 07 00 00 ...........E....
01c0 00 03 00 1a 00 0b 00 00 01 0e 00 40 00 0b 00 00 ...........@....
01d0 00 64 00 41 00 08 00 00 00 00 bf 80 00 00 00 00 .d.A............
01e0 05 1e 00 03 75 a1 7f ff ff ff 00 08 c0 90 00 00 ....u...........
01f0 40 60 00 00 00 00 00 00 00 00 00 00 00 00 00 00 @`..............
0200 00 00 00 00 ff 80 00 00 01 00 00 00 01 00 00 05 ................
0210 16 00 1d 00 06 00 3f 00 07 00 00 00 03 00 1a 00 ......?.........
0220 0b 00 00 01 02 00 40 00 0b 00 00 00 64 00 41 00 ......@.....d.A.
0230 08 3f 80 00 00 00 00 00 00 00 00 05 1b 00 03 75 .?.............u
0240 a1 7f ff ff ff 00 08 41 08 00 00 c0 90 00 00 00 .......A........
0250 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ff ................
0260 80 00 00 01 00 00 00 01 00 00 05 18 00 1c 00 00 ................
0270 00 09 00 02 ff 01 00 00 00 a1 00 11 94 01 80 01 ................
0280 02 00 08 bf 2d ab ac 40 f3 56 78 00 00 01 00 00 ....-..@.Vx.....
0290 00 01 20 00 03 75 84 00 00 00 03 00 03 81 ee 00 .. ..u..........
02a0 00 00 03 00 00 05 15 00 1d 02 00 00 03 75 b0 00 .............u..
02b0 00 00 a3 00 11 6c 00 00 00 02 00 08 c0 97 ad 45 .....l.........E
02c0 bf 4a 9f af 3f f6 b2 e0 3e aa aa 9e 40 16 cb e4 .J..?...>...@...
02d0 00 00 00 07 00 00 05 13 00 1d 02 00 00 03 75 b0 ..............u.
02e0 00 00 05 17 00 1d 02 00 00 03 75 b0 00 00 05 09 ..........u.....
02f0 00 1d 00 01 00 1e 00 1f 00 00 00 00 00 00 05 14 ................
0300 00 1d 02 00 00 03 75 b0 00 00 05 16 00 1d 02 00 ......u.........
0310 00 03 75 b0 00 00 00 a2 00 11 08 00 00 00 02 be ..u.............
0320 ef 7f 1b 00 00 00 03 01 00 00 00 04 00 00 05 11 ................
0330 00 00 04 57 00 00 05 10 00 00 05 12 00 21 00 00 ...W.........!..
0340 00 03 00 22 00 06 00 46 00 07 00 00 00 02 00 47 ..."...F.......G
0350 00 0b 00 00 03 e8 00 16 00 06 00 59 00 07 00 00 ...........Y....
0360 00 05 00 4e 00 0d 00 00 00 00 00 4f 00 18 00 00 ...N.......O....
0370 51 00 06 00 52 00 07 00 00 00 04 00 33 00 0a 00 Q...R.......3...
0380 1f 73 6f 75 6e 64 2f 65 66 66 65 63 74 2f 77 65 .sound/effect/we
0390 61 70 6f 6e 2f 62 6f 6d 62 5f 30 32 2e 6f 67 67 apon/bomb_02.ogg
03a0 00 53 00 0d 3f 80 00 00 00 54 00 0a 00 07 4c 6f .S..?....T....Lo
03b0 77 65 72 20 31 00 55 00 0a 00 04 4c 6f 6e 67 00 wer 1.U....Long.
03c0 56 00 23 00 00 00 00 00 00 00 00 3f 80 00 00 00 V.#........?....
03d0 00 00 57 00 06 00 58 00 07 00 00 00 01 00 1a 00 ..W...X.........
03e0 0b 00 00 01 0e 00 03 75 df 00 24 00 00 05 1c 00 .......u..$.....
03f0 08 c0 90 00 00 40 f0 00 00 00 00 00 00 00 00 00 .....@..........
0400 00 00 22 00 06 00 46 00 07 00 00 00 02 00 47 00 .."...F.......G.
0410 0b 00 00 03 e8 00 16 00 06 00 59 00 07 00 00 00 ..........Y.....
0420 05 00 4e 00 0d 00 00 00 00 00 4f 00 18 00 00 51 ..N.......O....Q
0430 00 06 00 52 00 07 00 00 00 04 00 33 00 0a 00 1f ...R.......3....
0440 73 6f 75 6e 64 2f 65 66 66 65 63 74 2f 77 65 61 sound/effect/wea
0450 70 6f 6e 2f 62 6f 6d 62 5f 30 32 2e 6f 67 67 00 pon/bomb_02.ogg.
0460 53 00 0d 3f 80 00 00 00 54 00 0a 00 07 4c 6f 77 S..?....T....Low
0470 65 72 20 31 00 55 00 0a 00 04 4c 6f 6e 67 00 56 er 1.U....Long.V
0480 00 23 00 00 00 00 00 00 00 00 3f 80 00 00 00 00 .#........?.....
0490 00 57 00 06 00 58 00 07 00 00 00 01 00 1a 00 0b .W...X..........
04a0 00 00 01 0e 00 03 75 df 00 24 00 00 05 1d 00 08 ......u..$......
04b0 c0 d0 00 00 40 b0 00 00 00 00 00 00 00 00 00 00 ....@...........
04c0 00 22 00 06 00 46 00 07 00 00 00 02 00 47 00 0b ."...F.......G..
04d0 00 00 03 e8 00 16 00 06 00 59 00 07 00 00 00 05 .........Y......
04e0 00 4e 00 0d 00 00 00 00 00 4f 00 18 00 00 51 00 .N.......O....Q.
04f0 06 00 52 00 07 00 00 00 04 00 33 00 0a 00 1f 73 ..R.......3....s
0500 6f 75 6e 64 2f 65 66 66 65 63 74 2f 77 65 61 70 ound/effect/weap
0510 6f 6e 2f 62 6f 6d 62 5f 30 32 2e 6f 67 67 00 53 on/bomb_02.ogg.S
0520 00 0d 3f 80 00 00 00 54 00 0a 00 07 4c 6f 77 65 ..?....T....Lowe
0530 72 20 31 00 55 00 0a 00 04 4c 6f 6e 67 00 56 00 r 1.U....Long.V.
0540 23 00 00 00 00 00 00 00 00 3f 80 00 00 00 00 00 #........?......
0550 57 00 06 00 58 00 07 00 00 00 01 00 1a 00 0b 00 W...X...........
0560 00 01 0e 00 03 75 df 00 24 00 00 05 1e 00 08 c0 .....u..$.......
0570 90 00 00 40 60 00 00 00 00 00 00 00 00 00 00 ...@`..........
A large packet is often followed-up with one or more additional large packets like the following. Note the ever-larger repeating data.
0000 84 2b 2b bc af ac 00 16 b6 db 14 cc 08 00 45 00 .++...........E.
0010 04 04 00 00 40 00 2b 11 24 79 b8 49 97 10 c0 a8 ....@.+.$y.I....
0020 17 6e ba 0d fc 2d 03 f0 0d 5e 00 00 2b 3e 00 00 .n...-...^..+>..
0030 38 6a 00 02 ff ff 00 03 00 00 08 21 00 00 08 3f 8j.........!...?
0040 00 00 00 00 00 54 00 03 76 1e 00 03 76 59 00 3b .....T..v...vY.;
0050 00 00 00 1c 00 00 00 02 00 01 03 00 00 00 a3 00 ................
0060 11 64 40 80 31 00 00 08 c0 a9 ed 5a be cd 66 f6 .d@.1......Z..f.
0070 3e aa aa 9e 40 49 0f db 00 14 00 00 00 00 00 00 >...@I..........
0080 00 00 00 01 02 05 08 ff ff ff ff ff ff f5 43 00 ..............C.
0090 00 00 00 00 00 00 00 00 20 57 65 61 70 6f 6e 2f ........ Weapon/
00a0 50 76 50 2f 42 6c 61 73 74 20 4e 65 74 77 6f 72 PvP/Blast Networ
00b0 6b 2f 44 65 66 61 75 6c 74 00 00 00 01 00 00 02 k/Default.......
00c0 00 00 00 00 00 00 00 00 00 16 00 00 00 00 00 00 ................
00d0 00 00 00 00 00 00 bf 00 00 00 00 16 00 00 00 00 ................
00e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 a2 ................
00f0 00 11 08 00 00 00 00 be d9 b0 e4 01 00 00 00 04 ................
0100 00 00 04 af 00 00 05 18 00 00 04 ff 00 00 05 19 ................
0110 00 21 00 00 00 04 00 22 00 06 00 46 00 07 00 00 .!....."...F....
0120 00 02 00 47 00 0b 00 00 03 e8 00 16 00 06 00 59 ...G...........Y
0130 00 07 00 00 00 05 00 4e 00 0d 00 00 00 00 00 4f .......N.......O
0140 00 18 00 00 51 00 06 00 52 00 07 00 00 00 04 00 ....Q...R.......
0150 33 00 0a 00 1f 73 6f 75 6e 64 2f 65 66 66 65 63 3....sound/effec
0160 74 2f 77 65 61 70 6f 6e 2f 62 6f 6d 62 5f 30 32 t/weapon/bomb_02
0170 2e 6f 67 67 00 53 00 0d 3f 80 00 00 00 54 00 0a .ogg.S..?....T..
0180 00 07 4c 6f 77 65 72 20 31 00 55 00 0a 00 04 4c ..Lower 1.U....L
0190 6f 6e 67 00 56 00 23 00 00 00 00 00 00 00 00 3f ong.V.#........?
01a0 80 00 00 00 00 00 57 00 06 00 58 00 07 00 00 00 ......W...X.....
01b0 01 00 1a 00 0b 00 00 01 0e 00 03 76 59 00 24 00 ...........vY.$.
01c0 00 05 22 00 08 c0 90 00 00 41 08 00 00 00 00 00 .."......A......
01d0 00 00 00 00 00 00 22 00 06 00 46 00 07 00 00 00 ......"...F.....
01e0 02 00 47 00 0b 00 00 03 e8 00 16 00 06 00 59 00 ..G...........Y.
01f0 07 00 00 00 05 00 4e 00 0d 00 00 00 00 00 4f 00 ......N.......O.
0200 18 00 00 51 00 06 00 52 00 07 00 00 00 04 00 33 ...Q...R.......3
0210 00 0a 00 1f 73 6f 75 6e 64 2f 65 66 66 65 63 74 ....sound/effect
0220 2f 77 65 61 70 6f 6e 2f 62 6f 6d 62 5f 30 32 2e /weapon/bomb_02.
0230 6f 67 67 00 53 00 0d 3f 80 00 00 00 54 00 0a 00 ogg.S..?....T...
0240 07 4c 6f 77 65 72 20 31 00 55 00 0a 00 04 4c 6f .Lower 1.U....Lo
0250 6e 67 00 56 00 23 00 00 00 00 00 00 00 00 3f 80 ng.V.#........?.
0260 00 00 00 00 00 57 00 06 00 58 00 07 00 00 00 01 .....W...X......
0270 00 1a 00 0b 00 00 01 0e 00 03 76 59 00 24 00 00 ..........vY.$..
0280 05 23 00 08 c0 f0 00 00 40 b0 00 00 00 00 00 00 .#......@.......
0290 00 00 00 00 00 22 00 06 00 46 00 07 00 00 00 02 ....."...F......
02a0 00 47 00 0b 00 00 03 e8 00 16 00 06 00 59 00 07 .G...........Y..
02b0 00 00 00 05 00 4e 00 0d 00 00 00 00 00 4f 00 18 .....N.......O..
02c0 00 00 51 00 06 00 52 00 07 00 00 00 04 00 33 00 ..Q...R.......3.
02d0 0a 00 1f 73 6f 75 6e 64 2f 65 66 66 65 63 74 2f ...sound/effect/
02e0 77 65 61 70 6f 6e 2f 62 6f 6d 62 5f 30 32 2e 6f weapon/bomb_02.o
02f0 67 67 00 53 00 0d 3f 80 00 00 00 54 00 0a 00 07 gg.S..?....T....
0300 4c 6f 77 65 72 20 31 00 55 00 0a 00 04 4c 6f 6e Lower 1.U....Lon
0310 67 00 56 00 23 00 00 00 00 00 00 00 00 3f 80 00 g.V.#........?..
0320 00 00 00 00 57 00 06 00 58 00 07 00 00 00 01 00 ....W...X.......
0330 1a 00 0b 00 00 01 0e 00 03 76 59 00 24 00 00 05 .........vY.$...
0340 24 00 08 c0 90 00 00 40 20 00 00 00 00 00 00 00 $......@ .......
0350 00 00 00 00 22 00 06 00 46 00 07 00 00 00 02 00 ...."...F.......
0360 47 00 0b 00 00 03 e8 00 16 00 06 00 59 00 07 00 G...........Y...
0370 00 00 05 00 4e 00 0d 00 00 00 00 00 4f 00 18 00 ....N.......O...
0380 00 51 00 06 00 52 00 07 00 00 00 04 00 33 00 0a .Q...R.......3..
0390 00 1f 73 6f 75 6e 64 2f 65 66 66 65 63 74 2f 77 ..sound/effect/w
03a0 65 61 70 6f 6e 2f 62 6f 6d 62 5f 30 32 2e 6f 67 eapon/bomb_02.og
03b0 67 00 53 00 0d 3f 80 00 00 00 54 00 0a 00 07 4c g.S..?....T....L
03c0 6f 77 65 72 20 31 00 55 00 0a 00 04 4c 6f 6e 67 ower 1.U....Long
03d0 00 56 00 23 00 00 00 00 00 00 00 00 3f 80 00 00 .V.#........?...
03e0 00 00 00 57 00 06 00 58 00 07 00 00 00 01 00 1a ...W...X........
03f0 00 0b 00 00 01 02 00 03 76 59 00 24 00 00 05 27 ........vY.$...'
0400 00 08 c0 20 00 00 c0 60 00 00 00 00 00 00 00 00 ... ...`........
0410 00 00 ..
This doesn't seem right to me. My setup should be able to handle an online bomberman clone but I think my ISP is choking on these large-ish UDP packets.
modify any files or use any game hacking/altering/cheating software or tools, including scripting or macroing software, or attempt to disassemble the game client and/or decipher the data transmissions between Spiral Knights' client and server. These activities will result in immediate termination of the Player's Account and may subject you to civil and/or criminal liability.
Just saying.