TMI Hell

[weldieran]

Ever wish you could just enable/disable TMI feature at will? Mebbe modify that mysteriuos max instructions variable? *sigh* I sure do. (last i knew its 20k, but bioware always seems to avoid officially saying for some reason - in a simple test it was more like 12k)

I've been in the practice of loading database data into HashSet, then using that instead of constant queries to the database. So first question, this is good practice, yes?

If so - how is it you manage to load large queries without getting TMI's?

I've heard before that "ExecuteScript" resets the TMI counter, but I've not found this to be true. I've tried using DelayCommand in various ways, only to get a slightly better result, but still TMI before the data all gets loaded.

What I'm loading is data from my custom tradeskill system. All materials, recipes and ingredients for those recipes are loaded into the DB. Then I use nwscript to sort and preload that data into HashSet.

[Primogenitor]

You can break it up with Delaycommands then use SQL to get different parts of the same query, ie. results 1-100, 101-200, etc. Im sure this has been covered elsewhere on these forums before.

[weldieran]

*puke*

I've used a method of DelayCommand even tho i despise the concept. It didn't help all that much... just allowed me to process a bit mroe before it bombed. It STILL has to put all these delaycommands into the command queue... and if i were to go all out in the method you suggest, i nearly get sick thinking about it. like the many times i've seen

// HACK HACK HACK

In bioware's code... YUCK. Tho i understand we're all pushing beyond the limits of what this game was intended for, but there's gotta be a much better way of loading large datasets into hashsets than to fiddle around with LIMIT 100 and Delaycommands everywhere *pukes again* hehe

I was hoping for a miracle answer here... until then, i suppose I'll look at rewriting the db plugin and APS in a couple diff ways

1) load each column into its own LocalString - as that SQLGetData can cause quite a bit of instructions looping thru finding these ¬ things

2) Create some method of causing hashset and the db plugin to interact directly given a set of instructions from nwn. Thus I can avoid having to pipe data thru nwn (yuck!)

[Acrodania]

I've had TMIs show up with as few as 500 Set/Get functions (even LOCAL) stringed together. How I usually handle is to set a variable with the last item's number and the use the DelayCommand to loop the script using a start point based on that set Variable. A delay of about 3 seconds between each 100 seemed about optimum for what I was doing (feeding 3,000 variables for a custom treasure system)...

Question: Are you using Set/GetPersistentXXX for your data (it doesn't look like it based on your post)? If you are moving a bunch of things related to each other you can speed the process up dramatically by using SQLExecDirect with custom tables. In my persistent object system I set 11 variables in and out of the database for each object, when I shifted to SQLExecDirect to feed them all in/out with one function call instead of multiple Set/Gets my script execution times dropped to 1/10. It seems that with NWNX the majority of the script run times are set by OPENING the connection, not actually passing the information in/out. This was with MySQL on the back end, not SQLite (since I don't know what you are using). MySQL Direct Connection, MySQL ODBC and Firebird via ODBC all displayed the same behavior. Also note that using this method with queries I've been able to pull specific data from tables with upwards of 27,000 records without TMIs but if I asked for too much (more than around 500 again) it would bomb....

Don't know if that helps any

[weldieran]

nah, i dont mess with the Get/Set PersistentVaraible functions in APS, in fact, i have removed them from the include file alltogether. I understand why they were as they were - but yer right, not a good way to set/get data when you are dealing with more advanced systems.

Ya, i see what yer doin with delaycommand.. i'm just a picky programmer and dont like the sound of it at all - tho who knows, to save many headaches of writing/rewriting plugins, I may just do it.

I'm getting TMI's on a recordset of just 65 total. Bombs out just before the last few are processed. But the loop isn't very simple... gotta generate hashset names and set 3 or so hashset vars before we move onto the next set... not to mention my debugger gets processed.

*sigh* i just wish there was a way to edit this preset TMI number. 20k is awful small when yer dealing with hacks like we use. 20k is definitely enough if you use ntohing but bioware functions, but get into anything more complicated and 20k is a fart in the wind.