Looking for testers for the NWScript accelerator plugin

[Skywing]

Note: The release build and release thread can be found here: http://www.nwnx.org/phpBB2/viewtopic.php?t=1803

-- old/prerelease thread follows --

This is a pre-release build; I'll create a new thread when the plugin is fully released.

I'm looking for some PW admins willing to try out the new NWScript accelerator plugin (xp_AuroraServerNWScript). This plugin makes scripts run faster on the server by turning them into native code instead of running them in the comparatively slower NWScript interpreter. No changes to scripts should be necessary for them to work with the plugin; things should seamlessly be faster.

If you're willing to give the plugin a shot and give feedback if something goes wrong, here's a link to the current build with installation instructions:

http://valera-ext.nynaeve.net/~skywing/nwn2/AuroraServerNWScript.zip

Source code for plugin: http://valera-ext.nynaeve.net/~skywing/nwn2/AuroraServerNWScriptSrc.zip

(Some libraries from the NWN2 Datafile Accessor Library package are used; a link to the source code for those libraries is included in the readme.)

Internally, the plugin works by turning compiled NWScript script files (*.ncs) into .NET MSIL instructions, which the .NET runtime JIT's to native code. Scripts are compiled to native code on the fly the first time they are used by the server in a particular session, so there is no extra step required to update scripts on the module (other than to restart the server).

The plugin also fixes several other issues:

- Some unused debugging code in the server causes the server's CPU and memory usage to gradually rise while no players are logged on (dropping off again once a player logs on). This has been an issue in the game since 1.00; the plugin fixes this bug.
- The plugin fixes issues where malformed scripts can crash the server (or worse) through execution stack mismanagement. This has been a problem since 1.00.
- The plugin fixes an issue where using ExecuteScript on a script that takes parameters may crash the server, or cause garbage data to be returned for script arguments. It is still not recommended to rely on this, as servers without the plugin won't work correctly here. This has been a problem since 1.00.

--

I'm especially interested in the following:

- Problems installing the plugin or getting it setup correctly (bad/incomplete installation documentation?)

- Problems with scripts not running correctly? If so, please provide the scripts in question that are broken. You can set DebugLevel=3 in the ini file to help track this down (however, this causes the plugin log file to grow very fast, as every single engine function call has all parameters logged to the log file).

[Vanes^]

Signing up for duty.

[PNJ]

Testing it !

Edit : It seem, many script are not fired on the PW with it, gonna take a look for the log at home after work.

[PNJ]

SkyWing, where can i send you , log with error and script associated to ?

[Skywing]

Sent you a PM; drop me a mail with it.

[Senalaya]

We've installed the plugin on our test server (Dragoncoast.org (TEST) - Depths of Norhaven) now and we'll keep you posted about findings.

[Developer Silver]

Hi Skywing,
I've added this plugin in our PW, but suddenly I've encountered some errors and server crash: ASAP I'll try to reproduce errors in development environment and I'll send you the report: how can I contact you?

P.S.: everytime a groundbreaking plugin!
_________________
Developer Silver
Faerûn Nights Admin

[Skywing]

Please drop me the nwn2_servererrorlog.txt file from the NWN2 install dir, the minidump ("nwn2server" without extension in the NWN2 install dir, be sure to enable view extensions in Explorer to find it), and the plugin log file (from the NWNX4 install dir). You may need to click Show compatibility files for the NWN2 install dir parts.

[Developer Silver]

Ok, I can reproduce server crash in development environment, so I've deeply analyzed the problem.
Crash occurs in OnPcLoaded script, when a PC login, only with xp_bugfix plugin enabled (version 1.0.10).
I've sliced my script in many little chunks and I've found a bunch of code lines causing crash, so I've reduced my script only to these lines. With this script server crashes, and every edit avoid the crash (every line is mandatory to crash ).

In nwn2_server_errorlog.txt I've noticed

[Developer Silver]

[Skywing]

Thanks for the preliminary investigation, repro steps, and crash dump; this looks like a bug. I'll investigate further shortly; I suspect the problem relates to saving a script situation that deals with an engine structure in the local list as you are with the AssignCommand(..., ActionJumpToLocation(lSaved)).

0:000> .cxr 0018f36c
eax=000006aa ebx=0000000d ecx=10e4b698 edx=000006ab esi=10e4b698 edi=09e08608
eip=00436150 esp=0018f654 ebp=10e4b698 iopl=0 nv up ei pl nz na pe nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00210206
nwn2server!CExoString::operator=+0x20:
00436150 8a08 mov cl,byte ptr [eax] ds:002b:000006aa=??
0:000> .exr 0018f31c
ExceptionAddress: 00436150 (nwn2server!CExoString::operator=+0x00000020)
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000000
NumberParameters: 2
Parameter[0]: 00000000
Parameter[1]: 000006aa
Attempt to read from address 000006aa
0:000> k
*** Stack trace for last set context - .thread/.cxr resets it
ChildEBP RetAddr
0018f65c 00727cce nwn2server!CExoString::operator=+0x20
0018f674 00728322 nwn2server!CVirtualMachineStack::CopyFromStack+0x27e
0018f6ac 006b52b5 nwn2server!CVirtualMachine::StackPopCommand_Internal+0x142
0018f6c8 0067c8c8 nwn2server!CNWVirtualMachineCommands::ExecuteCommandAssignCommand+0x45
0018f6d8 73e8fdfa nwn2server!CNWVirtualMachineCommands::ExecuteCommand+0x28
0018f748 73e905fc xp_AuroraServerNWScript!NWScriptBridge::ExecuteActionServiceHandler+0x1c [c:\users\skywing\documents\source\nwn2dev\auroraservernwscript\nwscriptbridge.cpp @ 1137]
0018f87c 081f030c xp_AuroraServerNWScript!NWScriptBridge::OnExecuteActionFromJITFast+0x132 [c:\users\skywing\documents\source\nwn2dev\auroraservernwscript\nwscriptbridge.cpp @ 326]
WARNING: Frame IP not in any known module. Following frames may be wrong.
0018f8e4 73e9142b 0x81f030c
0018f9c0 679f2122 xp_AuroraServerNWScript!swutil::SharedPtr<NWScriptJITLib::Program>::~SharedPtr<NWScriptJITLib::Program>+0x12 [c:\users\skywing\documents\source\nwn2dev\skywingutils\ref\ref.h @ 557]
0018f9d0 73e93190 clr!IJWNOADThunkJumpTarget+0x17
0018fa2c 73e9433f xp_AuroraServerNWScript!NWScriptJITLib::ExecuteScript+0xa1 [c:\users\skywing\documents\source\nwn2dev\nwnscriptjit\nwscriptjitlib.h @ 205]
0018fb4c 73e94887 xp_AuroraServerNWScript!NWScriptRuntime::ExecuteScriptForServer+0x3f9 [c:\users\skywing\documents\source\nwn2dev\auroraservernwscript\nwscriptruntime.cpp @ 365]
0018fb8c 73e8e42b xp_AuroraServerNWScript!ServerNWScriptPlugin::ExecuteScriptForServer+0x48 [c:\users\skywing\documents\source\nwn2dev\auroraservernwscript\servernwscript.cpp @ 660]
0018fba4 00729f86 xp_AuroraServerNWScript!NWN2Server::CVirtualMachine::ExecuteCode+0x26 [c:\users\skywing\documents\source\nwn2dev\auroraservernwscript\hooks.cpp @ 96]
0018fbd8 0072b207 nwn2server!CVirtualMachine::RunScriptFile+0x66
0018fbfc 0072b3b0 nwn2server!CVirtualMachine::RunScript+0x1b7
0018fc24 006121dd nwn2server!CVirtualMachine::RunScript+0x30
0018fc54 0066f28e nwn2server!CNWSModule::EventHandler+0x43d
0018fd50 0059a550 nwn2server!CServerAIMaster::UpdateState+0x27e
0018fd70 0054a3bf nwn2server!CServerExoAppInternal::MainLoop+0x100
0018fd78 0040b19d nwn2server!CServerExoApp::MainLoop+0xf
0018fe5c 0040b42f nwn2server!WinMainInternal+0x11ad [d:\nwn2\code_123\game\nwserver\main.cpp @ 573]
0018fe9c 00748390 nwn2server!WinMain+0x4f [d:\nwn2\code_123\game\nwserver\main.cpp @ 311]
0018ff2c 10002999 nwn2server!__tmainCRTStartup+0x140 [f:\dd\vctools\crt_bld\self_x86\crt\src\crtexe.c @ 589]
0018ff88 74a533ca NWNX4_Hook+0x2999
0018ff94 770c9ed2 kernel32!BaseThreadInitThunk+0xe
0018ffd4 770c9ea5 ntdll!__RtlUserThreadStart+0x70
0018ffec 00000000 ntdll!_RtlUserThreadStart+0x1b

[Skywing]

The problem here relates to a situation where:

a) A script situation is created (i.e. call to engine function with an argument of type 'action'), and
b) The current function returns a value of an engine structure type

In this case the compiler is directing the script execution environment to save the return value (retval) as a part of the script situation locals, even though the script situation never writes to or reads from that local. The retval local is never initialized from its default value prior to the script situation being created as it is only written during the epilog sequence for the script function. Normally this is ok, but the script situation save code in the JIT requires that all engine structures be initialized to a default engine structure and not NULL, which doesn't happen here as we never observed a write to the engine structure before the script situation.

The fact that the server crashed afterwards is a separate bug with a failed attempt to set up a script situation not being rolled back correctly by the code that interfaces the JIT system with the BioWare script VM.

Here's the native code we're emitting for LoadPGLocation():

[Skywing]

I've posted a patch release here:

http://www.nynaeve.net/Skywing/nwn2/xp_AuroraServerNWScript/AuroraServerNWScript.zip

Be sure to copy over all of the DLLs including NWNScriptJIT.dll.

- Fixes a code generation issue relating to scripts with a script function that both returns a return value of an engine structure type and calls an engine function with a parameter of type 'action'. For example, "location testfunc1() { AssignCommand(OBJECT_SELF, ActionWait(0.0f)); return GetStartingLocation(); }".

This should fix your issue. Thanks for the report; good find. (The fix is to emit code to preinitialize the return value of a function that creates a saved state and which has at least one return value of type engine structure.)

[Developer Silver]

[Developer Silver]

Uhm, today I've installed Release 2 of NWScript Accelerator plugin but suddenly NWNX hangs at boot time
It stalls at "* Loading plugins..." without any chance to go forward...

I've updated both files xp_AuroraServerNWScript.dll and AuroraServerNWScript.ini, but then I noticed also NWNScriptJIT.dll and NWNScriptJITIntrinsics.dll have changed, so I've updated these too, but nothing seems to resolve boot problem.
Any idea?

P.S.: may I suggest to implement a new configuration key to make an excluding list from recompilation? Often there are a bunch of script which will never run more than 1 time (ex. OnModuleLoad) that will waste memory (our OnModuleLoad script allocate about 12Mb of memory )
_________________
Developer Silver
Faerûn Nights Admin