Skywing's NWNX4 improvements

[Skywing]

Since the project has seemed to have been dead for a time now, I've taken it upon myself to fix some various bugs and add some features I've found handy.

The following changes are in my tree of NWNX4:

- Completely rewrote hooking/injection code system. It is now much more robust, works on 32-bit and 64-bit Windows, and doesn't depend upon madCHook or Detours or other external libraries. Also, we don't do awful initialization under DllMain in the remote nwn2server.exe process anymore for the hook anymore (which is significantly more correct from a programmatic standpoint).
- The hook DLL is now always loaded from the NWNX4 directory and shouldn't need to be copied to the NWN2Server current directory.
- Added crash dump writing support to the hook DLL and the controller. If NWN2Server crashes and the appropriate config file directive is set, the controller writes out a full dump of the process to a preset location. This dump can be analyzed using standard debugging tools (e.g WinDbg). DbgHelp.dll from the WinDbg distribution (http://www.microsoft.com/whdc/devtools/debugging) should be copied to the NWNX4 directory for this feature to work.
- Added receive packet logging support to the hook DLL (for 1.0.11.1153, will disable itself automatically in future releases until the DLL is updated with new offsets to patch into). This in conjunction with the crash dump support is intended to be used to help track down server crashes to specific client actions for purposes of changing server-side scripts to work around Obsidian bugs or, where feasible, assist in manual patches to nwn2server.exe to work around simple crash bugs. Only the last 128 received packets are logged to a memory mapped ring buffer, so overhead is minimal.
- Fixed a number of bugs in the server restart logic in the controller that would result in leaked memory. Also, fixed some bugs in the controller that could prevent it from restarting the server process correctly (at least, have not observed that happening after making my changes).
- Added reconnection support to xp_mysql (with automatic retry of a failed query if the query was not even sent to the server at all due to a disconnected socket). This makes mysql support viable (or at least moreso than it was) for non-trivial use, as you no longer have to shut down the entire module if the database connection is lost.

These changes are of course welcome to be included in the mainline distribution. I am including the contents of the 'src' directory of my NWNX4 tree, the remainder of the directories (including the 'lib' directory with wxWidgets and the like) should be taken from the mainline 1.08 distribution. To build these changes, please grab the source code to the mainline distribution, and then unpack the .zip with my changes over the 'src' directory.

- New config file options for my distribution -

# timeout in milliseconds to wait for NWN2Server process to start up and connect back to the controller before giving up and restarting it. The default is 30000. Note that the NWN2Server process connects back before loading the module if you use '-module'.
initTimeout = 30000

# Full path to the directory where crash dumps for nwn2server.exe should be stored. If not set, crash dump support is not enabled. Crash dump support requires that DbgHelp.dll from the WinDbg distribution be placed in the NWNX4 directory. Dumps are named in the form 'nwn2server_ctime_pid_tid.dmp'.
crashDumpDir = c:\dumps

# Full path to the directory where packet log files will be stored. One packet log file (~500k) is generated per nwn2server.exe session. If not set, then packet logs are not enabled. Packet log files are named in the form 'packetlog_ctime_pid_tid.log'. The packet logs are written in a binary format, see the source for NWNX4_Hook if you need to read them. Only received packets, not sent packets, are written to the log. Packet logging occurs after packet sequencing and decompression, with these portions of the packet headers stripped, leaving only the game-impacting portions remaining.
logPacketDir = c:\dumps

Note that you may wish to introduce a pruning scheme into your server restart script to remove old crash dumps or packet log files if you enable these features, as they are not automatically deleted. Crash dumps in particular can be large in size depending on how much memory your module consumes.

Download locations for files:

Binaries only: http://www.nynaeve.net/Skywing/nwn2/nwnx4_nodebug.zip (note: grab the mainline 1.08 distribution first, unpack it, and then replace the binaries with the updated ones in my distribution. The other binaries can remain the same and are not modified from the stock 1.08 build.)
Binaries with debug symbols: http://www.nynaeve.net/Skywing/nwn2/nwnx4.zip
Subset of source code that describes my changes from the mainline: http://www.nynaeve.net/Skywing/nwn2/nwnx4-changes-src-dist.zip (remaining source is unmodified from the mainline distribution).

- S

[Papillon]

This sounds great !

First, a little correction to your assumption about the project being dead: NWNX is not only one person (i.e. me). Numerous other people have been working on creating plugins and supporting other users through the forum here. Not updating the main program for a while does not mean it's dead.

That aside, the additions and fixes you made sound really cool. I really appreciate it when people actually sit down and implement their ideas ! I did not have a look at the 1.09 wishlist thread yet, but I guess somebody already linked back to this thread, as there are definitely some things which should go into the main distribution of NWNX.

I will go through your changes and will probably get back to you on some. Thanks for your input, again, it is really appreciated.
_________________
Papillon

[Skywing]

I've got a couple of other changes as well; I'll post a patch with them later. Most notably, I added a config directive and supporting code in the hook DLL to turn the copy of the .mod file at server startup to a (much faster) hardlink creation, which reduces server boot times if you have very large module files or a slow disk.

[GodBeastX]

If you need help deving, Pap, I got no problem with assisting

[Skywing]

Patch against the above tree to add an option to turn the long copy of the module file at server boot into a quick hardlink creation:

http://www.nynaeve.net/Skywing/nwn2/hardlink_patch.txt

Note: This option is off by default and must be enabled in the config file (suppressModuleCopy = true). To successfully use this option, the source directory where NWN2 copies the module from (e.g. the %userprofile%\my documents\neverwinter nights 2\modules) directory and the temp directory for the current user must be on the same drive. Hard links cannot span differing filesystem volumes.

This above condition should be the case almost always unless you're doing something particularly strange.

I also added some minor cleanup to ensure that we run the message loop to connect DDE instead of waiting for NWN2Server to do so. I am looking into scrapping DDE altogether as it's generally not all that great of an IPC mechanism, and the injection mechanism I am using is already ready-made for easily extensible ways to pass arbitrary information (such as a shared section view pointer or the like) to the child process (e.g. nwn2server).

[AcadiusLost]

A quick question- is the alternative patch of xp_mysql.dll here merged with the recent update that was reported to have fixed the Get/Set Object functions?

If not, has anyone checked the compatibility, patching it over xp_mysql 0.0.9?

Thanks.
_________________
Technical Administrator
A Land Far Away, a Persistent World Project

[Skywing]

If that wasn't in the 1.08 mainline, then probably not. I imported 1.08 into my local source control system at 2008/01/03 08:02:01 utc or so. (Might have fetched svn a few days before I actually got around to including it in my cvs repository, but I'm not sure.)

If you can point me to the source for a more recent build of xp_mysql, it should be fairly straightforward to merge my fixes in with it.

A patch off of mainline describing my fixes to xp_mysql can be found here: http://www.nynaeve.net/Skywing/nwn2/mysql_mainline_patch.txt . This is already included in the source/binary drop I linked above.

[AcadiusLost]

[Skywing]

I'll take a look. Looks offhand like it should be easy to apply my patch to that code. (And a good thing too to find someone else doing actual reversing on NWN2 stuff. Although it is a bit icky to see the return of madchook there. Oh well. Hoping the disassembler portions of it aren't as similarly severely braindamaged as the CreateProcessEx support. IMO, I would just go for using a set of offsets that are manually updated for patches as opposed to trying to do an egghunt style thing in real time, but that's an opinion difference between myself and virusman I suppose.)

Anyways... back to the topic at hand, I'll see about merging my (minor-er) modifications into virusman's thing this weekend. However, I don't have anything currently setup to test that code, although patching it to support reconnect should be fairly straightforward, so there may be a bit of beta-release testing required here. (I don't do toolset work myself, just been hacking on the game client and a bit on nwn2server and nwnx4 recently.)

[virusman]

I think whe should set up an SVN repository.

[Papillon]

Skywing, what is the intention behind your modification of bool NWNXController::checkProcessActive() ?

Is WaitForSingleObject better than GetExitCodeProcess in this case ? If so, why ? Is this a change which you deem necessary in the original code ?

The calls to CloseHandle are a no-brainer. I added them to controller.cpp.
_________________
Papillon

[Papillon]

Another question:

[Skywing]

[Papillon]

[Skywing]

How about just having a 'stable' branch and otherwise keep it all in the same repository, leaviing head as the new-feature-development branch or the like?