General release: xp_bugfix 1.0.5 client transition crash fix

[Skywing]

I'm putting xp_bugfix 1.0.5 out for general distribution release. Installation instructions and download links can be found at the xp_bugfix topic. Please read the instructions carefully as there is a new .ini option to set to enable the transition crash fix (ReplaceNetLayer = 1 in xp_bugfix.ini). If you set things up correctly, you should see a "CNetLayerWindow replaced" line in your xp_bugfix.txt log file.

This version rewrites a large part of the server's networking code to fix bugs that cause game clients (i.e. your players) to hang or crash during a transition.

I would like to thank Pnj, Senalaya, and Robert Harris for bearing with me over a couple of iterations of this process. I'm now satisfied that we've eliminated the largest cause of player transition crashes in multiplay.

On one server which has script-based logging of failed versus successful transitions (where a failed transition means the player never made it before crashing out), we have data that points to an improvement on the order of a ten-fold reduction in player transition crashes. Again, these are player crashes and not server crashes.

Your exact results may vary as the server bug leading to client transition crashes varies based on the intensity of current server load. Servers with high load are more likely to be adversely affected by transition crashes.

One of my testers has very kindly agreed to let me share the exact data gathered from watching player transition crashes before and after applying this fix.

Before applying xp_bugfix 1.0.5 (server using xp_bugfix 1.0.3): 2866 player transitions resulted in 50 player crashes (1.7% of transitions fail).
After applying xp_bugfix 1.0.5: 2866 player transitions resulted in 5 player crashes (0.17% of transitions fail).

The transition crash bug here is present in the original game and is not a problem xp_bugfix introduces.

This problem is a hold-over from NWN1 and still affects NWN1 as well as it is inherited from NWN1's networking code.

Applying this fix in no way breaks compatibility with clients.

Edit: The .ini option is ReplaceNetLayer = 1 and not DisableNetLayer = 1. Please edit your .ini if you used DisableNetLayer = 1. Sorry about that!

Additional credits (in no particular order, and perhaps omitting some by virtue of not remembering properly): elven, GrinningFool, Zebranky, brianmeyer, MSingh/OEI (For doing the legwork for symbols for 1.23).

[robertharris]

Skywing,

The numbers are improving as my sample size grows. I haven't had any new crashes since I last checked in. And the server has been up for 30 hours now. Here's data on the 3628 transitions immediately before and after the switch to 1.0.5.

With xp_bugfix 1.0.3 - 3628 transitions with 66 crashes (1.82%). 74 people played during this time, 38 of them crashed (51%).

With xp_bugfix 1.0.5 - 3628 transitions with 5 crashes (0.14%). 75 people played during this time, 3 of them crashed (4%).

I've played for several hours today. I had 73 transitions and no crashes. That's pretty much unheard of for me.

Dasaria 2 launched on November 12, 2006. These transition crashes have been the bane of my existence for three years now. On behalf of our staff and players, thank you so much for all the hard work you've put into this. What an incredible anniversary present you've given us, thank you.

Rob

[Skywing]

Great, glad again to see more of those numbers providing hard, verified data :)

There are a couple more client crashes I know of which aren't already fixed in the client extension and aren't mitigated by the new xp_bugfix:

- Logging on with invalid character classes (out of range for the valid class id range for the server) will cause a number of things to break. This is best mitigated by disabling client overrides so folks cannot easily create invalid characters (possibly with some script validation at creation time if you are feeling adventuresome, but I suspect override disabling is sufficient for 99% of cases).

- Performing a transition with many spells on your hot bar seems to trigger a use-after-free type problem in the client that intermittantly fails; I have seen this even when I wasn't seeing damaged subframes from the server. This is more or less limited to heavy spellcasting classes.

It sounds like from your data that just data corruption of outboud subframe was the prime cause of most of the crashes though.

Because the client and server share the same core networking code, bugs exist in both directions. But it seems the more severe of bugs is on the send and not receive side, and the client generally (outside of character creation and DM vault login) doesn't send all that much data to the server. Nonetheless, the client extension will ship with a similarly replaced networking layer.