xp_bugfix - nwn2server bugfixes

[Skywing]

I've started an nwnx4 module to work around some of the more common nwn2server crash / hang problems that I have been able to get data for.

Some of these "fixes" only address the symptoms of the problem and not the underlying root cause, however, in many cases this is better than the normal behavior of the server falling over and dying.

Current posted version is: 1.0.22 for NWN2 1.0.23.1763 and 1.0.23.1765. Old versions for older server releases are archived below. xp_bugfix is compatible with any set of installed expansion packs for a given supported NWN2 version, as the server binaries are the same across all expansion releases at a given patch level. You do need to match the xp_bugfix version to the game patch level that you are using, however.

Important note: Starting with xp_bugfix 1.0.13, you must install an updated Visual C++ 2005 SP1 CRT. (Pick the vcredist_x86.exe download.) If you do not update your VC2005 CRT, xp_bugfix will silently fail to load.

Starting with xp_bugfix 1.0.22, you should set AIUpdateThrottle = 32 and DirectPollNetRecv = 1 in xp_bugfix.ini to improve server performance under load. You can find more details about this here.

Starting with xp_bugfix 1.0.5, you should set ReplaceNetLayer = 1 in your xp_bugfix.ini to enable a fix for client-side transition crashes. You can find more details about this here.

Release (1.0.22) binaries + symbols for NWN2 1.0.23.1763 / 1.0.23.1765: http://www.nynaeve.net/Skywing/nwn2/01231763/xp_bugfix_1_0_22.zip Note, there is a new CRT update required starting with 1.0.13! (Please see above note about requirement to update your VC2005 CRT.)
Additional support files required: http://www.nynaeve.net/Skywing/nwn2/01231763/AuroraServerNetLayer.zip (extract this .zip into your NWN2 installation directory, that is, the directory where NWN2Server.exe resides). (New AuroraServerNetLayer version with xp_bugfix 1.0.9)

Release (1.0.20) binaries + symbols for NWN2 1.0.23.1763 / 1.0.23.1765: http://www.nynaeve.net/Skywing/nwn2/01231763/xp_bugfix_1_0_20.zip Note, there is a new CRT update required starting with 1.0.13! (Please see above note about requirement to update your VC2005 CRT.)
Additional support files required: http://www.nynaeve.net/Skywing/nwn2/01231763/AuroraServerNetLayer.zip (extract this .zip into your NWN2 installation directory, that is, the directory where NWN2Server.exe resides). (New AuroraServerNetLayer version with xp_bugfix 1.0.9)

Release (1.0.18) binaries + symbols for NWN2 1.0.23.1763 / 1.0.23.1765: http://www.nynaeve.net/Skywing/nwn2/01231763/xp_bugfix_1_0_18.zip Note, there is a new CRT update required starting with 1.0.13! (Please see above note about requirement to update your VC2005 CRT.)
Additional support files required: http://www.nynaeve.net/Skywing/nwn2/01231763/AuroraServerNetLayer.zip (extract this .zip into your NWN2 installation directory, that is, the directory where NWN2Server.exe resides). (New AuroraServerNetLayer version with xp_bugfix 1.0.9)

Release (1.0.17) binaries + symbols for NWN2 1.0.23.1763 / 1.0.23.1765: http://www.nynaeve.net/Skywing/nwn2/01231763/xp_bugfix_1_0_17.zip Note, there is a new CRT update required starting with 1.0.13! (Please see above note about requirement to update your VC2005 CRT.)
Additional support files required: http://www.nynaeve.net/Skywing/nwn2/01231763/AuroraServerNetLayer.zip (extract this .zip into your NWN2 installation directory, that is, the directory where NWN2Server.exe resides). (New AuroraServerNetLayer version with xp_bugfix 1.0.9)

Release (1.0.15) binaries + symbols for NWN2 1.0.23.1763 / 1.0.23.1765: http://www.nynaeve.net/Skywing/nwn2/01231763/xp_bugfix_1_0_15.zip Note, there is a new CRT update required starting with 1.0.13! (Please see above note about requirement to update your VC2005 CRT.)
Additional support files required: http://www.nynaeve.net/Skywing/nwn2/01231763/AuroraServerNetLayer.zip (extract this .zip into your NWN2 installation directory, that is, the directory where NWN2Server.exe resides). (New AuroraServerNetLayer version with xp_bugfix 1.0.9)

Release (1.0.14) binaries + symbols for NWN2 1.0.23.1763 / 1.0.23.1765: http://www.nynaeve.net/Skywing/nwn2/01231763/xp_bugfix_1_0_14.zip Note, there is a new CRT update required starting with 1.0.13! (Please see above note about requirement to update your VC2005 CRT.)
Additional support files required: http://www.nynaeve.net/Skywing/nwn2/01231763/AuroraServerNetLayer.zip (extract this .zip into your NWN2 installation directory, that is, the directory where NWN2Server.exe resides). (New AuroraServerNetLayer version with xp_bugfix 1.0.9)

Release (1.0.13) binaries + symbols for NWN2 1.0.23.1763 / 1.0.23.1765: http://www.nynaeve.net/Skywing/nwn2/01231763/xp_bugfix_1_0_13.zip Note, there is a new CRT update required starting with 1.0.13! (Please see above note about requirement to update your VC2005 CRT.)
Additional support files required: http://www.nynaeve.net/Skywing/nwn2/01231763/AuroraServerNetLayer.zip (extract this .zip into your NWN2 installation directory, that is, the directory where NWN2Server.exe resides). (New AuroraServerNetLayer version with xp_bugfix 1.0.9)

Release (1.0.10) binaries + symbols for NWN2 1.0.23.1763 / 1.0.23.1765: http://www.nynaeve.net/Skywing/nwn2/01231763/xp_bugfix_1_0_10.zip (Please see above note about requirement to update your VC2005 CRT.)
Additional support files required: http://www.nynaeve.net/Skywing/nwn2/01231763/AuroraServerNetLayer.zip (extract this .zip into your NWN2 installation directory, that is, the directory where NWN2Server.exe resides). (New AuroraServerNetLayer version with xp_bugfix 1.0.9)

Release (1.0.9) binaries + symbols for NWN2 1.0.23.1763 / 1.0.23.1765: http://www.nynaeve.net/Skywing/nwn2/01231763/xp_bugfix109.zip (Please see above note about requirement to update your VC2005 CRT.)
Additional support files required: http://www.nynaeve.net/Skywing/nwn2/01231763/AuroraServerNetLayer.zip (extract this .zip into your NWN2 installation directory, that is, the directory where NWN2Server.exe resides). (New AuroraServerNetLayer version with xp_bugfix 1.0.9)

Release (1.0.8) binaries + symbols for NWN2 1.0.23.1763 / 1.0.23.1765: http://www.nynaeve.net/Skywing/nwn2/01231763/xp_bugfix108.zip (Please see above note about requirement to update your VC2005 CRT.)
Additional support files required: http://www.nynaeve.net/Skywing/nwn2/01231763/AuroraServerNetLayer.zip (extract this .zip into your NWN2 installation directory, that is, the directory where NWN2Server.exe resides).

Old Release (1.0.7) binaries + symbols for NWN2 1.0.23.1763 / 1.0.23.1765: http://www.nynaeve.net/Skywing/nwn2/01231763/xp_bugfix107.zip (Please see above note about requirement to update your VC2005 CRT.)
Additional support files required: http://www.nynaeve.net/Skywing/nwn2/01231763/AuroraServerNetLayer.zip (extract this .zip into your NWN2 installation directory, that is, the directory where NWN2Server.exe resides).

Old Release (1.0.6) binaries + symbols for NWN2 1.0.23.1763 / 1.0.23.1765: http://www.nynaeve.net/Skywing/nwn2/01231763/xp_bugfix106.zip (Please see above note about requirement to update your VC2005 CRT.)
Additional support files required: http://www.nynaeve.net/Skywing/nwn2/01231763/AuroraServerNetLayer.zip (extract this .zip into your NWN2 installation directory, that is, the directory where NWN2Server.exe resides).

Old Release (1.0.5) binaries + symbols for NWN2 1.0.23.1763 / 1.0.23.1765: http://www.nynaeve.net/Skywing/nwn2/01231763/xp_bugfix105.zip (Please see above note about requirement to update your VC2005 CRT.)
Additional support files required: http://www.nynaeve.net/Skywing/nwn2/01231763/AuroraServerNetLayer.zip (extract this .zip into your NWN2 installation directory, that is, the directory where NWN2Server.exe resides).

Old Release (1.0.3) binaries + symbols for NWN2 1.0.23.1763 / 1.0.23.1765: http://www.nynaeve.net/Skywing/nwn2/01231763/xp_bugfix103.zip (Please see above note about requirement to update your VC2005 CRT.)
Old Release (1.0.2) binaries + symbols for NWN2 1.0.23.1763 / 1.0.23.1765: http://www.nynaeve.net/Skywing/nwn2/01231763/xp_bugfix.zip (This release is obsolete and provided for archive purposes only.)

Old Release Version (1.0.1) binaries + symbols for NWN2 1.0.22.1588: http://www.nynaeve.net/Skywing/nwn2/0100221588/xp_bugfix1.zip (This release is obsolete and provided for archive purposes only.)
Source: Available at virusman's NWNX4 repository (SVN http://nwn.virusman.ru/svn).

Older releases are archived. In general, you will want the current release version unless you're running an old patch level.

Archived (0.0.9b3) binaries + symbols for NWN2 1.0.21.1549: http://www.nynaeve.net/Skywing/nwn2/xp_bugfix_01211549_01.zip
Archived (0.0.9b3) binaries + symbols for NWN2 1.0.13.1409: http://www.nynaeve.net/Skywing/nwn2/xp_bugfix_beta_009_00131409_b3/xp_bugfix.zip
Archived (0.0.9b3 binaries + symbols for NWN2 1.0.12.1295: http://www.nynaeve.net/Skywing/nwn2/xp_bugfix_beta_009_00121295_b3/xp_bugfix.zip
Archived (0.0.8) binaries for NWN2 1.0.12.1295 are available here: http://www.nynaeve.net/Skywing/nwn2/xp_bugfix.zip
Archived (0.0.8) source tree excerpt (requires current NWNX source tree to build): http://www.nynaeve.net/Skywing/nwn2/xp_bugfix_dist.zip

The Visual C++ 2005 SP1 x86 runtimes are required. You can download these here if you have not already installed them: http://www.microsoft.com/downloads/details.aspx?familyid=200b2fd9-ae1a-4a14-984d-389c36f85647&displaylang=en

-- Status of xp_bugfix bug fixes (items in blue are fixed in a publicly available patch) --

- CalcPositionLoop0 (pathing hang): Worked around in xp_bugfix. Root cause is a double free in JStar pathing code in unknown circumstances, not yet triaged due to spaghettified inline code preventing easy use of a checked allocator.
- CalcPositionLoop1 (pathing hang): Worked around in xp_bugfix. Root cause is a double free in JStar pathing code in unknown circumstances, not yet triaged due to spaghettified inline code preventing easy use of a checked allocator.
- NullDerefCrash0 (crash in CNetLayerWindow::FrameTimeout): Fixed in 1.0.21.1549 and in xp_bugfix for earlier versions.
- NullDerefCrash1 (crash in CNWSMessage::WriteGameObjUpdate_UpdateObject): Fixed in 1.0.13.1409 and in xp_bugfix for earlier versions.
- NullDerefCrash2 (crash in NWN2_Collider::UpdateCollider): Worked around in xp_bugfix. Root cause not yet determined.
- NullDerefCrash3 (crash in CNWSMessage::HandlePlayerToServerDungeonMasterMessage): Fixed in 1.0.21.1549 and in xp_bugfix for earlier versions.
- NullDerefCrash4 (CServerExoAppInternal::LoadCharacterStart): Fixed in 1.0.23.1763 and in xp_bugfix for earlier versions.
- Crash5 (CNetLayerWindow::UnpacketizeFullMessages): Fixed in xp_bugfix prior to 1.0.23.1763, doesn't seem to be necessary in current versions.
- Crash6 (CNWSMessage::HandlePlayerToServerInventoryMessage): Fixed in xp_bugfix.
- Crash7 (CNWVirtualMachineCommands::ActionExchangeItem): Fixed in xp_bugfix.
- Crash8 (CNWSItem::AcquireItem): Worked around in xp_bugfix. Root cause is some as of yet not fully tracked down broken caller.
- Uncompress0 (CNetLayerWindow::UnpacketizeFullMessages): Fixed in xp_bugfix. Possibly fixed in mainline in a future patch.
- Uncompress1 (CNetLayerWindow::UnpacketizeFullMessages): Fixed in xp_bugfix. Possibly fixed in mainline in a future patch.
- Crash9 (CItemRepository::GetItemPtrInRepository). Worked around in xp_bugfix. Root cause unknown.

Issues addressed in xp_bugfix 0.0.1 (beta release) for nwn2server 1.0.12.1295):

- 1) Occasional server hang when >2 creatures attempt to pathfind when in close proximity, as described by the following forum threads : http://nwn2forums.bioware.com/forums/viewtopic.html?topic=587354&forum=118&highlight= , http://nwn2forums.bioware.com/forums/viewtopic.html?topic=577063&forum=118 . This is a fairly common issue in my experience. "Fixed" by detecting a loop in linked list links and aborting processing in that case.
- 2) Null pointer dereference at 005D1F9E (unknown circumstances leading up to this code path, but it seems to be hit during player login sometimes) (Not fully tested.). "Fixed" by aborting the current function in the event that the circumstances leading up to the would-be null dereference occur.

Issues addressed in xp_bugfix 0.0.3 for nwn2server 1.0.12.1295:

- 3) Fix for occasional nwn2server crash when hitting the "Respawn" button while at the death menu if you were polymorphed before you died (null deref at 0072331f). This seemed to be a problem where a position was trying to be set to a non-existant object, but difficult to say for certain without symbols.
- 4) Possible fix for null deref at 0042ef85 (unknown conditions leading up to this crash). I don't know how to try and repro this crash so I can't say for certain whether the attempted fix will help, but it shouldn't make things worse at the very least.
- 5) Added log message rate limiting for the null deref crashes.

Issues addressed in xp_bugfix 0.0.4 for nwn2server 1.0.12.1295:

- 6) Fixed signed comparison being used in the null dereference checks (whoops). This was causing false positives on x64 systems or x86 systems booted with /3GB.

Issues addressed in xp_bugfix 0.0.5 for nwn2server 1.0.12.1295:

- 7) Fixed regression that was causing mob "clumping" in some circumstances with the polymorph death crash fix.

Issues addressed in xp_bugfix 0.0.6 for nwn2server 1.0.12.1295:

- Not publicly released.

Issues addressed in xp_bugfix 0.0.7 for nwn2server 1.0.12.1295:

- 8) Improved the reliability of the pathing hang workaround (0xdeadbef3 dereference workaround).
- 9) Improvement of the fix for the crash at 005D1F9A by detecting the underlying problem and not a side effect of it for triggering the fix.
- 10) Improvement of the fix for the crash at 0042EF85 by detecting and working around an additional failure mode that was causing a dereference of 0x1eac in xp_bugfix (NOT a regression, would have crashed in nwn2server similarly without xp_bugfix being present).
- 11) Fix for a crash at 0045A81F (NULL dereference), which seems to happen when a .bic isn't loaded properly for a player login request.
- 12) Fix for a crash at 005CD7CC (random pointer dereference) due to misuse of CExoNetExtendableBuffer::GetFrameData (networking related).

Issues addressed in xp_bugfix 0.0.8 for nwn2server 1.0.12.1295:

- 13) Fixed another issue with the workaroundd for the 005CD7CC crash (should be dead this time, really!).

Issues addressed in xp_bugfix 0.0.9 for nwn2server 1.0.12.1295 and 1.0.13.1409:

- 14) Fix for lack of handling for bad zlib data + lack of validation on compressed data chunk length field (should finally put a stake in that 005cd7cc crash, another path that could have lead to that).
- 15) Fix for a crash resulting from a player without a game object associated sending an Inventory.6 (Unequip?) message (004f7e27)
- 16) Fix for a crash resulting from a wrong object type used with ActionExchangeItem (004e7a49)
- 17) Fix for a crash when a creature without an item container attempts to acquire an item (004a1586)

Issues addressed in xp_bugfix 0.0.9b2 for nwn2server 1.0.12.1295 and 1.0.13.1409:

- 18) Fixed a (regression) problem with the fix for one of the bad compressed data crash issues that could cause character to hang.

Issues addressed in xp_bugfix 0.0.9b3 for nwn2server 1.0.12.1295 and 1.0.13.1409:

- 19) Fixed a crash when a CItemRepository contained an object id that did not resolve via GetItemByGameObjectID (0049CDDF for 1.13, 0049BE0F for 1.12).

Issues addresses in xp_bugfix 1.0.0.1 for nwn2server 1.0.22.1549:

(Not a general distribution release. Tracing code added that was used to track down a crash bug that was new to 1.21. OEI subsequently fixed this particular issue with 1.22.)

Issues addressed in xp_bugfix 1.0.1 for nwn2server 1.0.22.1588:

(No new fixes for this release. All fixes from 1.0.21.1544 were ported over.)

Issues addressed in xp_bugfix 1.0.2 for nwn2server 1.0.23.1763:

- Fixed an issue with the ActionExchangeItem crash fix which accidentally turned all calls into an ActionTakeItem. This could be backported to 1.0.22.1588 with just a rebuild (the hook needed a pushfd/popfd), but I'm not sure that it's worth it at this point giving the dwindling 1.0.22.1588 user base.
- Removed "NullDerefCrash4" fix (CServerExoAppInternal::LoadCharacterStart+0x47F) as this is fixed in the shipping nwn2server in 1.0.23.1763 now.
- Removed "Crash5" fix (CNetLayerWindow::UnpacketizeFullMessages+0x80) as it doesn't seem to have been required for a couple of versions now.

Issues addressed in xp_bugfix 1.0.3 for nwn2server 1.0.23.1763 / 1.0.23.1765:

- Fixed a server crash when a player who has not initialized their automap departs the game.

Issues addressed in xp_bugfix 1.0.5 for nwn2server 1.0.23.1763 / 1.0.23.1765:

- Rewrote server networking code to fix client transition crashes. You must set "ReplaceNetLayer = 1" in xp_bugfix.ini to activate this option. See this thread for more details.

Issues addressed in xp_bugfix 1.0.6 for nwn2server 1.0.23.1763 / 1.0.23.1765:

- Fixed server crash when handling level up processing while no player LUO is available. This is most likely to happen in scenarios where there is severe lag in place and players perform certain level-up actions in a particular order, or if a player is on DMFI follow while clicking through the level up screen.

Issues addressed in xp_bugfix 1.0.7 for nwn2server 1.0.23.1763 / 1.0.23.1765:

- Added a GameObjUpdateTime option that configures the minimum amount of time between the server sending GameObjUpdate messages to clients. This is a heavy consumer of CPU time on servers with many objects. By default, the game sends updates every 200000 microseconds. You may increase this value to improve performance if your server is getting pegged on CPU (try 400000 as a good starting point). This will typically actually reduce response times for players over how the server would react when struggling to keep up. Higher values reduce server CPU consumption but may cause areas to take longer to load for players. This option is best used with ReplaceNetLayer = 1. See this post for full details.

Issues addressed in xp_bugfix 1.0.8 for nwn2server 1.0.23.1763 / 1.0.23.1765:

- Added script interface for setting GameObjUpdateTime programmatically. See this post for full details.

Issues addressed in xp_bugfix 1.0.9 for nwn2server 1.0.23.1763 / 1.0.23.1765:

- Added networking code performance improvements. When using xp_bugfix 1.0.9 or later, with the updated AuroraServerNetLayer module, clients should load complex areas much faster (in some cases several times faster). The degree of speed-up depends on client disk speed (clients with faster disks will see more of a speed-up as they were previously waiting on the network more).
- Increased the retransmit aggressiveness beyond the default aggressiveness used by the stock game, which is too low. This should help catch up clients that get behind after dropping several packets, if their queues continue to grow.

Issues addressed in xp_bugfix 1.0.10 for nwn2server 1.0.23.1763 / 1.0.23.1765:

- Added a workaround for a server bug that causes clients to crash at logon. This bug has been a long-standing server issue, but the performance improvements in xp_bugfix 1.0.9 caused the issue to reproduce much more frequently, as the bug hinged on whether certain updates were queued to the client before a certain stage in initial area loading. Users with the Client Extension were already protected from this bug.

Issues addressed in xp_bugfix 1.0.11 for nwn2server 1.0.23.1763 / 1.0.23.1765:

- Fixed gradual loss of precision in the server's timing subsystem (as a function of computer uptime).

Issues addressed in xp_bugfix 1.0.12 for nwn2server 1.0.23.1763 / 1.0.23.1765:

- Added "UseGetTickCount = 0|1" option in xp_bugfix.ini to use GetTickCount instead of QueryPerformanceCounter for the server's timing subsystem. This is off by default and generally shouldn't be used unless you have specific reason to believe there are problems with your hardware's QueryPerformanceCounter support.

Issues addressed in xp_bugfix 1.0.13 for nwn2server 1.0.23.1763 / 1.0.23.1765:

- Set campaign database buffer size to 1024 by default, freeing up 700MB (!!) of additional memory in nwn2server.exe in many configurations.
- Added "DatabaseBufferCount = count" option in xp_bugfix.ini to set the number of campaign database buffers (0x8000 bytes per buffer). Without xp_bugfix, the server uses 700MB of buffers on systems with 4GB of RAM, which is a large waste. The xp_bugfix default is to use 32MB worth of buffers.

Issues addressed in xp_bugfix 1.0.14 for nwn2server 1.0.23.1763 / 1.0.23.1765:

- Increased the (effective) size of the CGameObjectArray hash table for frequent lookup paths (such as CServerAIMaster::AIUpdate). This improves performance and scalability for servers that have areas with large numbers of objects active.

Issues addressed in xp_bugfix 1.0.15 for nwn2server 1.0.23.1763 / 1.0.23.1765:

- The SetAreaTransitionBMP script function, which had never been functional, can now be used. Only a pre-defined area transition 2DA index can be supplied, however, because the game client never had the code necessary to understand a raw filename (sCustomAreaTransitionBMP) implemented.

- Added packet filtering API, required for the shared server vault plugin.

Issues addressed in xp_bugfix 1.0.17 for nwn2server 1.0.23.1763 / 1.0.23.1765:

- Fixed NWNXSetInt("BUGF", "GAMEOBJUPDATETIME", "", 0, GameObjUpdateMicroseconds), the API for dynamically adjusting the player world update timer, after it was discovered to not work. The config file directive did work, just the dynamic setting API was broken.

Issues addressed in xp_bugfix 1.0.18 for nwn2server 1.0.23.1763 / 1.0.23.1765:

- Added support for rewriting the client's claimed UDP port so that two players behind the same public IP address do not cause various incorrect server-side behavior. This option, controlled by the new RewriteClientUdpPort ini setting, is on by default.

- Added support for disabling verbose messages like the invalid object deletion message. This option, controlled by the new VerboseLogging ini setting, is configured to disable verbose messages by default.

Issues addressed in xp_bugfix 1.0.20 for nwn2server 1.0.23.1763 / 1.0.23.1765:

- Master server messages are now copied to nwn2.mst.valhallalegends.com. This emulation BioWare master server will facility a replacement for GameSpy, for maintaining a list of active PWs. Servers can opt in to being listed by installing xp_bugfix 1.0.20 or later.

Issues addressed in xp_bugfix 1.0.21 for nwn2server 1.0.23.1763 / 1.0.23.1765:

* (Reliability improvement) If a NWN2_Pathfinder object is destroyed, always remove it from the JStar2 secondary pathing list. This resolve a longstanding pathfinding crash that occurred when objects that were pathfinding were then destroyed. This issue was found to be a major remaining contributor to server stability problems.

Issues addressed in xp_bugfix 1.0.22 for nwn2server 1.0.23.1763 / 1.0.23.1765:

* (Performance improvement) Added AIUpdate throttling mode; this is controlled by the new AIUpdateThrottle = count-of-milliseconds configuration value and must be enabled manually on upgrade to the latest xp_bugfix. See the example configuration file included for details. This feature keeps track of how much time was spent processing periodic object updates each main loop iteration, and if it exceeds the AIUpdateThrottle value (expressed in milliseconds), then the game server skips future updates for that main loop iteration except for critical objects (PCs, PC controlled objects, areas, and modules). A fairness scheme is implemented so that objects that were skipped have higher priority in receiving update service next main loop iteration. This feature ensures server responsiveness even under high load (maxed out CPU) situations. A recommended throttle value is 32.

* (Performance improvement) Added optional polling modes for network traffic as some environments did not see traffic processed properly when server CPU load was high. This change resolves that problem and is controlled by a new DirectPollNetRecv = 1 option that must be enabled manually on upgrade to this xp_bugfix release. The recommendation is to set this value to 1. If enabled, every main loop iteration will directly poll for up to 50 network packets. See the included example configuration file for details.

* (Performance improvement) CServerAIMaster now tracks CNWSObject pointers instead of object IDs in its internal list of objects that need AIUpdate processing. This substantially reduces expensive lookups through the game object hash table and improves performance during situations where many objects exist in the game world.

* (Performance improvement) Optimized out of the CServerAIMaster list items that only have "benign" item properties which require no periodic updating. This reduces the amount of objects that are scanned for periodic updates each main loop iteration by 50% or more on some environments. Items with temporary properties added are dynamically inserted into the CServerAIMaster list as necessary.

* (Performance improvement) Removed unnecessary hot code path look ups using the game object hash table in CSrverExoAppInternal::GetClientObjectById.

* (Performance improvement) Removed unnecessary hot code path look ups using the game object hash table in CNWSCreatureStats::HasFeat.

There are builds for the 0.0.9 beta available for the 1.12 release and the 1.13 hotfix release. The original 1.13 release is not supported by the 0.0.9 beta, you should upgrade the 1.13 hotfix if you are already on 1.13 and would like to use xp_bugfix.

Note that as some of these workarounds do not fix the underlying problem but merely allow the server to do the moral equivalent of limp on afterwards, unintended side effects may occur. (So far, I have not observed any ill effects, however, with a definite net gain in server stability. Your milage may very, however, so use at your own risk. I would recommend running my build of NWNX4 with crash reporting enabled, as I'd be interested in looking at any crash/hang dumps that you get while running xp_bugfix.)

To install, simply put xp_bugfix.dll into your nwnx directory. You will need the VS8SP1 CRT redistributable ( http://www.microsoft.com/downloads/details.aspx?familyid=200b2fd9-ae1a-4a14-984d-389c36f85647 ) installed on your system for the module to load.

xp_bugfix will write a log file (xp_bugfix.txt) that will denote if the module has recovered the server from any of the above crash/hang conditions.

The 0.0.8 build supports only NWN2 1.0.12.1295.
There is an 0.0.9 beta build (not fully tested) for NWN2 1.0.12.1295 and NWN2 1.0.13.1407. There are separate binaries for the 1.0.12.1295 and 1.0.13.1407 releases.

[GodBeastX]

Any symptoms of issues revolving around the pathfind hang? I see this about once a week, not too often.

[Skywing]

I've looked and haven't seen anything so far. Observed both issues being recovered from a production server without any breakage I could easily see. (For the pathing hang, recovered from that something about 6 times over two days, not noticing any particular weirdness as a result.)

It would be easier to verify the side effects if I could repro the issue completely on demand, but I haven't observed any problems thus far. (Please keep an eye out for any if you decide to use it, however. Definitely not seeing things like mobs losing the ability to move or anything else that one might intuitively expect from a pathing bug, or otherwise seeing evidence of memory corruption leading to secondary failures in the server process.)

Obviously, it doesn't do anything to address the underlying issue, but the end result is a bit happier than the server dying from what I have seen thus far.

[Kres]

Problems with pathfinding can also be directly related to walkmesh issues, I have found with dealing with this 100% issue that it is a combination of several factors at least for our set of conditions...

1. Tony K's AI is in use
2. 2 or more creatures and the feint feat is involved on the creatures
3. PC blocks the path, using the Walkmesh as a way to fix just 1 monster at a time. Example... Using a doorway to block others from approaching.
4. Heavy use of WASD keys just before setting up said block...
5. One of the creatures has Sneak, Death or other form of attack that will do additional damage and it is trying to achieve that goal. However if the PC' kills BOTH creatures before that attack is accomplished by the activating creature it then will hang.

I do not have access to all the logs but this is merely running these conditions over and over and watching my CPU & Memory max out. Since I run a Quad Core and NWNx4 with its own core by making sure the affinity is set to 0, along with NWN2Server.exe once it caps it passes on to the next thread. Once all 4 cores hit 100% the server will crash.

[Skywing]

Posted xp_bugfix 0.0.3. Included in this release is:

- Fix for occasional nwn2server crash when hitting the "Respawn" button while at the death menu if you were polymorphed before you died (null deref at 0072331f). This seemed to be a problem where a position was trying to be set to a non-existant object, but difficult to say for certain without symbols.
- Possible fix for null deref at 0042ef85 (unknown conditions leading up to this crash). I don't know how to try and repro this crash so I can't say for certain whether the attempted fix will help, but it shouldn't make things worse at the very least.
- Added log message rate limiting for the null deref crashes.

For the pathing issue, I have sometimes also seem a problem where the list links sometimes point off into freed heap blocks. If this happens, a crash in xp_bugfix may occur when traversing the list. Because xp_bugfix is performing the same operation that nwn2server is normally performing in that case, nwn2server would have crashed anyway in that case (typically this manifests itself as attempting to access an address looking like one of the heap fill patterns, such as 0xdeadbef3). Unfortunately, developing a heuristic to detect this condition is much more difficult to do in a way that won't cause other breakages, so I am leaving this other problem unaddressed. It is not a regression caused by xp_bugfix, however.

Kres: You have a module file that can nicely repro this 100% of the time without a whole lot of setup work? Would definitely be worth seeing if we can get OEI to make a real fix for this instead of my hack, which is again only treating the symptoms, the real underlying problem with the positioning calculations remains. (I suspect that the problem might be creatures trying to path to the same square at the same time, or something of that sort. The position list seems to be a list of creature movement destinations, but I haven't spent a whole lot more time reversing to expand on that assumption. Sucks without symbols...)

[Kres]

[Skywing]

Posted 0.0.5 that fixes a small regression bug that was sometimes causing monsters to seem to clump together on the same spot.

[PNJ]

I got a NWN2 crash since a certain time (dunno what causing it), when my server try to create a new OIE Database the nwn2server crash to desktop.

Giving a 0 byte OIEDB index file, and a corrupted unusable OIEDB core file.

So i have to create the databases somewhere else and then put them in the PW database folder (not very handy and this unallow me to use "destroydatabase").

I wonder if this could be fixed with this plugin.

[Morrandir]

We keep on having nwn2server crashes. We're narrowing the issue down atm, but we installed xp_bugfix anyhow some time ago.

But we don't get any logs from xp_bugfix.

What does that mean?
Did we do something wrong or are the servercrashes none that xp_bugfix can track?

(xp_bugfix is indeed runnning, at least i get "Plugin initialized." message in xp_bugfix.txt and "Loading plugin xp_bugfix.dll: Successfully registered as class: BUGF" in nwnx.txt)
_________________
German NWN1/2 Persistent World Hochwaldallianz

[Skywing]

xp_bugfix only has workarounds for several specific bugs; it's quite possible that you might be running into an issue that it doesn't know about and thus has no bandaid for.

Will need to know the address of the instruction that faulted as well as register context and stack. Dump files are preferable.

One way to do this is to attach a debugger to nwn2server and wait for it to crash, then grab the output of ".ecxr -1;r;kv" and post here. (Be careful that the process doesn't get killed by the NWNX4 watchdog before you can grab the crash data if you go this route.)

Another alternative is to install my version of the NWNX4 launcher/hook, as well as the latest DbgHelp (also from the WinDbg distribution linked above) - see the thread for details. This will enable NWNX4 to write full dumps whenever the nwn2server process dies. N.B. This also requires some additional configuration directives to be added to your nwnx ini file.

[Skywing]

Going to be posting a refresh of xp_bugfix for 1.12 soon, with some additional (new) crash fixes, and some refinements determined with the aid of backporting symbol data for the 1.13 beta.

Included in this release will be:

- Improved reliability for the pathing hang workaround. We now try to recover from the dereference 0xdeadbef3 issue. This happens when we get an NWN2_JStar::Node list that has a list link pointing into the free list of the slab allocator reserved exclusively for NWN2_JStar::Node objects. Should be "mostly safe" to ignore this, as we don't overwrite any of the free list's metadata before we can detect that we found the end of free list marker (0xdeadbeef). I say mostly in that if an NWN2_JStar instance has an NWN2_JStar::Node that resides in the free list, then we may damage it.

I am working on an improved fix for this bug by instancing the slab allocator per NWN2_JStar instance, holding frees until the instance is reset or deleted, and then just using our own internal list of all nodes allocated for purposes of releasing the memory. In other words, by turning the NWN2_MemPool< NWN2_JStar::Node >::Free calls to a noop and freeing the memory all at once when pathing finishes, I'm hoping to sidestep the buggy code that tries to free a still-in-use NWN2_JStar::Node. However, until this fix is completed, I am pushing out the new release with a slightly improved version of the old kludge.

- Fix for a crash at 0045A81F (NULL dereference), which seems to happen when a .bic isn't loaded properly for a player login request.
- Fix for a crash at 005CD7CC (random pointer dereference) due to misuse of CExoNetExtendableBuffer::GetFrameData (networking related).
- Improvement of the fix for the crash at 005D1F9A by detecting the underlying problem and not a side effect of it for triggering the fix.
- Improvement of the fix for the crash at 0042EF85 by detecting and working around an additional failure mode that was causing a dereference of 0x1eac in xp_bugfix (NOT a regression, would have crashed in nwn2server similarly without xp_bugfix being present).

[Skywing]

Posted 0.0.7. Please let me know if you encounter any strange behaviors or problems that crop up only after installing 0.0.7.

As usual, I recommend testing the new release out in a pre-production environment (or keeping a close eye on things after upgrading), just in case, though I am not anticipating any regressions after my tests thus far.

[Skywing]

Posted 0.0.8, which should fix a problem that was preventing the workaround for the 005CD7CC crash from actually recovering from the crash. If you have a reliable test case to reproduce this crash, please let me know so that I can add it to my regression test set.

[Skywing]

Note about upcoming 1.13 patch: As far as I know, none of these bugs have yet been fixed in code by OEI, at least as of the latest beta patch release. This means that I'll need to be putting out a new xp_bugfix when the final version of 1.13 gets pushed out.

Although you can upgrade to 1.13 while using the existing xp_bugfix, it will disable itself automatically after noticing that the nwn2server.exe running is not the version it was compiled against. This means that you will effectively revert to be running with all of the old crash/hang bugs if you upgrade immediately with the 1.13 release.

It is not feasible to produce an updated xp_bugfix binary for the final 1.13 patch until I have the final release binaries sitting in front of me, so if you are using xp_bugfix and are severely impacted by the bugs that it works around, my recommended guidance on the upcoming 1.13 patch is to wait until a version that fully supports 1.13 is released before upgrading the server.

I do not expect that it will take a significant amount of time to port xp_bugfix to the final 1.13. It's my hope that I should be able to get a beta version compatible with the 1.13 nwn2server the day that 1.13 gets released, though the initial beta 1.13-compatible release will probably not have the full burn in time that typical xp_bugfix releases do.

[PNJ]

Thank you for your efforts