RunExternalScript() - Full Idea

[isilweo]

Hello.
The idea of running an external script (which is outside mod) was on my mind for few months. Last night the idea poped up and here it is. I lack asm skills so i would ask Papillon for coding (or at least help with hooking).

What is?
RunExternalScript would run compiled script which is firstly retrieved from database.

What for?
Your imagination is your only limit here. I would use it first of all to speed up develope of new scripts... now just imagine
the old way:
you've made a new script (or maybe even set of scripts) for your PW. You compile your script. Save mod (if you have big PW it'll probably last about minute or more). Pack it (with ie rar to speed up uploading). Upload mod to server (probably another few minutes). Unpack mod. Restart server. Relog into game. Check your script and... it's not working. Oh yeah.. you've made a typo in sql statement. You correct typo and do all operations again losing another ten minutes for it.
and now the new way:
Let's assume you've got simple web interface which compiles (using standart nwscomp) scripts for you and put them into mysql. You've made your script and just copy/paste from aurora (or maybe your favourite editor) or you've written your script directly in web interface. You press the Save button on your webpage. The script compiles in few seconds and is ready to use. You go back to your game and check it. oh.. it's not working? you found that typo in mysql statement. now just correct it using interface and press save button. Switch back to game and check is it working. you've just saved 10min!
Of course there are many other ways of how you can use RunExternalScript function.


How does it work?
First you must make simple script in toolset. Let's name it nwnx_executor. Script may be empty but i used for test simple script:
void main() {string s="myuniquestring";}
The key is to put that script into modules buffered scripts so it will reside in memory all the time - non buffered scripts are loaded from mod when there is need. If you run your mod and search memory for "myuniquestring" you will find your compiled script. It begins with NCS. Now if you find pointer to address of begining of script (letter N) you'll find yourself somewhere in memory where you would see pointer to script code, pointer to name of script and script code length. Idea is to load script from db. Put it into variable. Change pointer to script code to variable where your new script code is and set script code length to new length. After that just execute script "myuniquestring".

Implementation:
nwn part - the code of RunExternalScript would look like this
void RunExternalScript(string sName, object oObject=OBJECT_SELF)
{
SetLocalString(GetModule(), "NWNX!ODBC2!SCRIPTEXECUTOR",sName);
ExecuteScript("nwnx_executor",oObject);
}

nwnx part
first SCRIPTEXECUTOR function will simply get compiled script from database (scriptname is of course sName). or maybe the method of script retreval could be the same as on SCO/RCO.

the hook must be set in address 0x00416504. In that address in edi+10h is pointer to scriptcode, in edi+14h is pointer to script name and in edi+18h is scripcodelength)
Every time the hook address is called nwnx must check is name == nwnx_executor. It it is we must change adress of edi+10h to point to our new script code (which was retrieve earlier from db), after that we must set edi+18 to new length. Now just call original function.
That's all.

Good night and please tell me what you think (and could you make that - probably Papillon?)

isilweo

ps.
that function on 0x00416504 (it begins few bytes earlier) is probably used for loading more files cause if you check edi+10h it loads also some 2da's.

[Blacksting]

For along time a "valid" address to hook the script queue has been searched for. If I remember correctly one attempt early on failed. If your address given is correct then what you propose is possible... and is the answer that has eluded others for quite some time. Your request is "one step more" than what so far NWNX has yet to aqchieve.... to modify the script queue AT ALL. If you believe what you have found is accurate then by all means beat Papillon or Jeroen B over the head with it . It would be the next "big" anouncement NWNX could make after hooking SCORCO.

[Manuel]

[Orleron]

[Papillon]

isilweo, what you found is the function that loads resources from the module file (or haks, ....). I think it will be possible to hook it and react on certain scriptnames (e.g. "nwnx!myscript"), reading the script from a blob and returning that instead.

In fact, if I am not mistaken, this would be the function to load everything from the DB, not just scripts (.are, .ncs, .ifo, ....). This is a very interesting subject.
_________________
Papillon

[Papillon]

I've got a proof of concept running. It is a NWNX plugin that reacts to a certain script name ("nwnx") and replaces the script code with code loaded from an external file (\temp\nwnx.ncs). So far the engine reacts nicely to the modifications.

So the idea seems to be viable.
_________________
Papillon

[Papillon]

Yep, it works. The plugin is activated by the following code:

[Orleron]

*faints*
_________________
____________________
Blackdagger: The World of Avlis
http://www.avlis.org

The Confederation of Planes & Planets
"Go anywhere. Do anything."
http://www.copap.org

[Blacksting]

This looks like it could mean modification of placeables.... that would be nice. It looks as if you may have created an NWNX version of CreateObject ... with using a file name instead of a resref..... Now that it seems the ODBC plugin is working with blobs "palettes" can be stored in db tables even for placeables.

When Leto becomes integrated a UTP resource could be virtually modified before entry in to the mod. I like the idea of putting down a placeable on the fly and choosing its name just like I do now with creatures and items.


Once a script is loaded future calls can do without the "!" correct? Or is the ncs just being loaded in to volatile memory?

[Dazzle]

This would effectivly fix all the cluttering with the hundreds of scripts in convo's. Just load the right one in a starting conditional or at item activation... Can't wait for this

[Primogenitor]

This sounds good, though Im not sure I can think of anything I could use it for at the moment. If it works with area files (.are .git) then that would be nice (dynamic terrain & random dungeons without pregenerating areas), but I doubt it will (pathfinding, etc would mean that the area is preloaded, wouldnt it?).

[Papillon]

The usefulness comes from the fact that you can modify script code on the fly, without reloading a module. This should be a very effective way to update a module without needing players to log off and without transferring megabytes of modules in which just a few bytes changed (-> Improved uptime and faster development cycles).

Second advantage for multi-module worlds like Avlis would be a central repository for scripts that run on all mods (e.g. the Avlis death code, or avlis_onuse). Updates to one of those central scripts would no longer have to be imported in each of the six modules, but the update would be immediately life with just dropping the new script into the central repository (-> Improved code management, support for version control systems, ...).

My initial tests show the following performance data (timed with NWNX timer):

a) Standard execution of internal script: 53 microseconds
b) NWNX Scripter execution of external script, loaded from harddisk: 230 microseconds
c) NWNX Scripter execution of external script, loaded from memory cache: 60 microseconds

This is a very simple script, the overhead for more complex scripts becomes more and more negligible as the script needs more time to execute (e.g. 500 microseconds vs. 520 microseconds). This also prooves that the NWNX Scripter approach is practically usable.

I imagine a global trigger (e.g. ExecuteScript("!clearcache", ...)) that invalidates the cache and reloads every script from the repository.
_________________
Papillon

[Papillon]

Here is the first version for those who would like to try it out:

http://nwnx.org/fileadmin/user_upload/nwnx_scripter_001.zip
_________________
Papillon

[isilweo]

it looks like i've done something useful finally

If script loaded from harddrive takes 0.2 milisec longer then i wouldn't even think about caching scripts in memory caouse 0.2milisec is nothing for me. We don't design database system which must be strong optimized. Besides what gives you that 0.2milisec if internet connection is slow enough to generate another 1sec of delay before player sees effect of script.

Primogenitor: Areas are not preloaded. They're loaded when someone enters them. That means you can make dungeons but it's not so simple cause tiles must feet to eachother.

Dazzle: i'm working on full dynamic dialog system with pseudoscript language (cause nwscript is to hard for most of people who make maps etc). I think it'll be better solution...but we'll see (i am now at level of building interface for it)

As i see most of mod resources can be moved out from mod. that's very good information cause it means that mod will be lighter and easier to edit.

The script repository with cvs is what i always wanted

Papillon if you give me code to what you've written so far i can add other features to it (mostly odbc connection) and you could work with other things which i can't do. Maybe i'll catch you on icq on evening (after coming back from school)

it looks like another step in nwn modding. it's nice to have a little contribution in this step.

isilweo

[Themicles]

An example of how I would use this for areas:

I was talking to Mistcaller a while back about wishing I had some way to update areas on the fly to allow for a player to carve their own little settlement out of the great forests in my world.

He suggested a scripting approach. Basically, build each iteration of that area and have it in the module. Now, in the transitions that lead to that area, use a script that checks a variable, and depending on what that variable is, the target of that transition would change. Therefor, I could have a thickly forested area with invisible objects on the trees for players to chop at. After the trees are "chopped down" a variable would be set to tell the transitions to that area to instead point at the cleared iteration of the area.

If this can be done in that way, it can be done with this new possible functionality.

Concern about pathfinding was raised in another forum. I doubt it would affect pathfinding because it's not like the system makes one giant pathfinding map out of the area data. If that were the case, the NPCs wouldn't get stuck in some of the places they do. Cross area pathfinding just tells the NPC that it needs to enter that area to get to the next. Pathfinding inside that area is done when the NPC is in that area.

I believe it is this way because of some of the pathfinding problems I've run into on Tairis. It'll even enter an area that is divided by something, with no way to get from one side of that area to the other, if that transition on the other side goes to the destination the NPC seeks.

Anyway... I can think of a million ways I'd want to use this if you included functionality for most of the module's contents.

I could even see the module files being mostly empty files, and all the normal contents being stored in a location outside the module file, using this plugin to load it. This would allow for a CVS like system of the module itself, where a developer can check out just one area, or one NPC blueprint, or so on...

I mean, sure, we'd still have to have a copy of the module with everything in it to work in the toolset, for the most part... or not... If a CVS like system were made, perhaps it could be coded to turn the checked out files into a working .mod file so it can be opened right away in the toolset, worked on, and then you upload that .mod, and the CVS system would extract the components...

Bah, my mind is RACING with ideas right now... so I'll shutup and just wait to see what comes of all this... If you're not willing to go beyond script functionality, I might try to con one of my programmer friends into doing it for me. Not likely, but this is something I want pretty bad, heh.

-Themicles
_________________
World Leader of Tairis'nàdur http://www.tairisnadur.com
Member World of CoPaP http://www.copap.org