Use of scrolls crashes server (using MySQL)

[javajutsu]

I'm the admin for Exodus: Shadows of Amn. Recently we're experiencing a bug where the use of clerical scrolls crashes the server. Using potions doesn't do it; casting prepared spells doesn't do it; just casting from scrolls.

I can't help but feel this is a configuration problem with MySQL, or how I've got NWNx4 configured to hook to it -- something awry with the spelltrack table, perhaps -- but I've been looking at this for a week, and I'm stumped. We've been running for over 2 years now, and this is a new problem.

I'm wondering if anyone else out there has run into anything similar? Is there any additional information I could provide that might be helpful to diagnosing this?

Thanks in advance for any help, tips or suggestions of where to look.

I'm running:

• Windows XP Professional SP 3
  
• NWNx4 (NWN Extender 4 V.1.0.9)
  
• NWNX MySQL Plugin V.0.0.9
  
• MySQL Server version: 5.1.40 MySQL Community Server (GPL)
  
• nwn2server 1.023 (1765)

Here's the nwn2server_error_log:

Error occurred at 11/29/2009 01:03:46.
C:\Program Files\Atari\Neverwinter Nights 2\nwn2server.exe, run by javajutsu.
C:\Program Files\Atari\Neverwinter Nights 2\nwn2server.exe caused a No Memory exception in module kernel32.dll at 001b:7c812afb.

System Info:
2 processor(s), type 586.
1 % Memory Load.
3326 MBytes physical memory.
997 MBytes available memory.
2048 MBytes total virtual memory.
61 MBytes available virtual memory
0 MBytes total page file.
2516 MBytes available page file

Application debug messages:
Version: 1.023 (1765) (Final) Aug 17 2009 - 17:40:34
Application Stage: Application Main Loop Frame 233758 Uptime: 1089.985962
Heap Manager: malloc() has failed! Bytes: 3067246
Command Line: -moduledir Amn

[Skywing]

Looks like you're running out of address space. Maybe one of your nwnx plugins has a memory leak (have a custom one?), or maybe your module was already just at the very edge of running out of address space.

Normally, a 32-bit program may only use 2GB of virtual memory on Windows. You can extend this to 3GB via an operating system configuration setting: http://technet.microsoft.com/en-us/library/bb124810(EXCHG.65).aspx

(I would only use this on a machine dedicated to NWN2 as it severely constrains the size of kernel memory pools and file system cache.)

There is a bcdedit option to use on Vista/Srv08 32-bit.

If you upgrade to Windows x64, NWN2 will be able to use 4GB of address space.

[javajutsu]

Thanks, Skywing, for the prompt reply!

The module is large with a lot of areas, and address space *is* an issue.

However, the nwn2server executable is running steady at around 1.75 GB -- 250 MB of headroom remaining. The malloc() call failed on a request for *3* MB.

PageFile is basically the size of this, plus MySQL, plus about 300 MB in system stuff, total typically about 2.6 GB (out of a Commit Charge limit of about 5.3 GB) -- again, steady at this figure, and has been so for at least 9 months.

CPU usage was about 1% immediately before the crash. There *is* a problem with the CPU gradually ramping up to 100% in six-second pulses, which AcadiusLost described very well in another thread. But that only happens when no one is logged in. *This* problem (crashing when spells are cast by spawned NPCs, or scrolls are used by PCs) only occurs when there's someone logged in to cast the spell or use the scroll.

I'm using only the "standard load" of plugins verified and downloaded from this site: NWNx4 + xp_bugfix + xp_chat + xp_mysql + xp_time. Again, a stable configuration for months and months without seeing this issue.

Thanks for the heads up about the /3GB technote. I was already aware of that, but thought it wasn't applicable to this Windows XP Professional rig. However, you got me curious. A search on the technote site reveals that it is, and also speaks to the /PAE (physical address extension) switch that might also be useful. I'll look into them both as options for buying more headroom on this system.

However... all of that being said, this problem seems to have just suddenly appeared a few weeks ago.

In searching the Microsoft tech support forums, it seems that lots of people were bitten by kernel32.dll getting corrupted by a recent AutoUpdate patch from Microsoft. So I'm looking into that. My current plan is to do a clean re-install of XP (and NWN2 and MySQL and our module) tonight.

Unless someone else comes up with a better idea first, of course!

[Skywing]

No, I don't think you're running into something caused by a software update. That doesn't make sense to me as related to this at all.

Just because you have 250MB of total free address space does not necessarily mean that you have a single chunk 3MB long that's entirely free (a contiguous 3MB). In fact, it's not surprising for the address space and heap to begin to get fragmented after many allocations/deallocations of varying sizes (especially when the address space is so close to complete exhaustion as in your case). There is some logic in the default Windows heap (especially on Vista or later) to try and minimize fragmentation, but unfortunately, NWN2 wraps the Windows heap atop its own block allocator, and that may be detrimental with respect to fragmentation unless the NWN2 heap goes out of its way to actively try and avoid fragmentation.

If you attach the debugger, you could run !address -summary to find the largest free chunk of address space in the process. (This does not peer into the internal structure of a heap itself however.)

My recommendation is to try /3GB as you're on a 32-bit system.

[javajutsu]

I reverted the system to a restore point prior to the latest AutoUpdate patch and ran the Windows repair utility. The problem went away, but I went ahead and also implemented the /3GB switch as suggested. The system is fat and happy.

Problem solved, case closed.

Thanks for your help, Skywing!