| View previous topic :: View next topic |
| Author |
Message |
JeffSheets
Joined: 04 Feb 2009
Posts: 18
|
 Posted: Sun Feb 22, 2009 12:01 Post subject: Resman stability and dynamic .jrl file |
 |
|
First... I've seen elsewhere here that nwn 1.65 or so caused nwnx_resman to be a bit unstable, due to caching. Is this still true?
Second, how feasible is it to load your module's Journal/quest file dynamically with nwnx_resman? I would assume that you would never want to mix up category tags and entry IDs, but are there any other pitfalls?
I'm also assuming one would need to maintain persistent storage of the Quest entries per PC. |
|
| Back to top |
|
 |
Zebranky
Joined: 04 Jun 2006
Posts: 415
|
| Posted: Sun Feb 22, 2009 20:09 Post subject: |
|
|
I've heard of no stability issues, other than a memleak with Blacksting's version that only happened on one machine I've seen. Unfortunately, he never released the code for his version, so it's not readily fixable.
Never heard of anyone using resman for their .jrl file, sounds tricky, but I'll leave that to anyone who has actually done it.
_________________
Win32 SVN builds: http://www.mercuric.net/nwn/nwnx/
<Fluffy-Kooshy> NWNx plugin is to this as nuclear warheads are to getting rid of fire ants.
<ThriWork> whenever I hear nwn extender, I think what does NWN need a penis extender for? |
|
| Back to top |
|
|
elven
Joined: 28 Jul 2006
Posts: 259
Location: Germany
|
| Posted: Sun Feb 22, 2009 20:48 Post subject: |
|
|
We had been using resman in various incarnations. .jrl gets loaded on module load and then never touched again, so there isn't much use in putting it into resman.
resman is fine for importing resources and testing scripts, but it memleaks - putting our whole module (130mb) into resman made nwserver eat 2gig of rss within 30 minutes. :) |
|
| Back to top |
|
|
JeffSheets
Joined: 04 Feb 2009
Posts: 18
|
| Posted: Sun Feb 22, 2009 20:58 Post subject: |
|
|
| elven wrote: | | We had been using resman in various incarnations. .jrl gets loaded on module load and then never touched again, so there isn't much use in putting it into resman. |
Ah. Anyone have any ideas for how one could update the journal in memory with a plugin then? I'm sure it would require a new plugin, and I'd even be willing to write it, but when I looked with IDA yesterday, the journal system was not very obvious to me. (Not to mention that my assembler skills were never that good to begin with and are even worse today.)
| Quote: | resman is fine for importing resources and testing scripts, but it memleaks - putting our whole module (130mb) into resman made nwserver eat 2gig of rss within 30 minutes.  |
Where is the leak? I'm still very early in my PW development, so I'd be willing to hack at it and see if I can't clean it up, since I don't have any players to worry about. |
|
| Back to top |
|
|
elven
Joined: 28 Jul 2006
Posts: 259
Location: Germany
|
| Posted: Sun Feb 22, 2009 21:07 Post subject: |
|
|
Your guess is as good as mine.
It leaks some fewish kb of RSS for each call to resman that returns a resource, so I think it looks like this:
* nwserver: wants bogon.utc, allocate some buffers
* resman: have it?
* yes: return gff to nwserver directly, skip deallocation
* no: read from module index, deallocate buffers, return |
|
| Back to top |
|
|
Ravine
Joined: 26 Jul 2006
Posts: 105
|
| Posted: Mon Mar 02, 2009 12:37 Post subject: |
|
|
Hmm... how do you tested it?
I've checked now, spawned about 2500 random item with resman, and i see no change in memory usage.
However i use it with items only. |
|
| Back to top |
|
|
elven
Joined: 28 Jul 2006
Posts: 259
Location: Germany
|
| Posted: Mon Mar 02, 2009 12:51 Post subject: |
|
|
I observed this behaviour on linux 32bit and 64bit with recent kernels, with and without xen.
The used version is the one from svn, and the binary-distributed one floating around - both emitted said behaviour, under 1.68 and 1.69.
I tried with a blank test module just loading resources in serial sequence; that leaked as well. |
|
| Back to top |
|
|
Ravine
Joined: 26 Jul 2006
Posts: 105
|
| Posted: Mon Mar 02, 2009 13:27 Post subject: |
|
|
I have 32 bit linux, debian, 2.6 kernel.
I'm using the version from acaos' branch, had problems with the trunk version ( can't remember what was the problem).
The NWNXResMan.cpp file size is 4516 byte. |
|
| Back to top |
|
|
leo_x
Joined: 25 Aug 2010
Posts: 75
|
| Posted: Wed Jul 02, 2014 16:47 Post subject: |
|
|
Sorry for necroing this old thread, but I just ran into this...hard
I formed a hypothesis about what it's causing it. From what I can tell it only occurs when an external resource is loaded that is not represented in one of the Key tables.
Load some file ->
GetResObject from key table, if none or no entry exists, create new res object ->
SetResObject in the key table, if there's an entry ->
Request/Demand actual file data.
So... if the file isn't in one of the key tables, it's leaking the CRes[...] and file data every Demand. At least that's my theory.
I'm not sure what the best way to go about fixing it would be tho.
_________________
the awakening (PW Action) |
|
| Back to top |
|
|
leo_x
Joined: 25 Aug 2010
Posts: 75
|
| Posted: Sat Jul 05, 2014 16:14 Post subject: |
|
|
I believe that I've fixed the memory leak issue. I also expanded the event system.
EDIT: Deleted files, see discussion below.
_________________
the awakening (PW Action)
Last edited by leo_x on Tue Sep 30, 2014 17:13; edited 1 time in total |
|
| Back to top |
|
|
leo_x
Joined: 25 Aug 2010
Posts: 75
|
| Posted: Mon Jul 07, 2014 16:24 Post subject: |
|
|
As I was testing this I found the 2DAs sometimes fails when they're loaded from resman... Segfaults in SkipLines or they just return false from Load2daArray. (This is true of the version in the master branch). There still seems to be an issue when unloading the engine cached 2das at shutdown, that I haven't figured out yet.
EDIT: Turns out if you have a 2da like iprp_costtable in resman, but one of its entries is not, there will be a segfault. Since at shutdown -> the module and all HAKs are closed -> rules are reloaded -> server shutdown -> rules unloaded again and 2das that were never able to be reloaded will attempt to be freed.
I fixed that and updated the files above.
I also add ERF archive support to PhysFS, namely loading HAK files, the others ERF types could be added easy enough: https://bitbucket.org/jmd28/physfs/branch/nwn_formats (PhysFS uses mercurial)
This version is statically linked in the updated plugin 7z for anyone interested  .
_________________
the awakening (PW Action) |
|
| Back to top |
|
|
elven
Joined: 28 Jul 2006
Posts: 259
Location: Germany
|
| Posted: Sat Aug 16, 2014 15:51 Post subject: |
|
|
Hi,
i'm running your resman branch now. It hasnt crashed yet!
I have a question or two about porting to other plugins (like odbc, redis):
- This:
| Code: | if(event->restype == 0x07DA && event->mtime != 0 && PHYSFS_exists(event->resref)) {
return 1;
}
|
Why does it return 1 for compiled scripts that nwnx_physfs has in it's repository, but no data? Why does it check for mtime != 0, while physfs itself sets mtime=0 for all returned data?
- In extention to that, how does the ncs cache handling work in general?
- How exactly is demandRes->mtime handled on resman side? What are it's effects? Why is it okay to set it to 0? How does it relate to cache invalidation if at all?
Thanks for now! |
|
| Back to top |
|
|
leo_x
Joined: 25 Aug 2010
Posts: 75
|
| Posted: Sat Aug 16, 2014 17:35 Post subject: |
|
|
| elven wrote: | Hi,
i'm running your resman branch now. It hasnt crashed yet!
I have a question or two about porting to other plugins (like odbc, redis):
- This:
| Code: | if(event->restype == 0x07DA && event->mtime != 0 && PHYSFS_exists(event->resref)) {
return 1;
}
|
Why does it return 1 for compiled scripts that nwnx_physfs has in it's repository, but no data? Why does it check for mtime != 0, while physfs itself sets mtime=0 for all returned data?
- In extention to that, how does the ncs cache handling work in general?
- How exactly is demandRes->mtime handled on resman side? What are it's effects? Why is it okay to set it to 0? How does it relate to cache invalidation if at all?
Thanks for now! |
Heya,
I'll try to answer in reverse order.
In PhysFS it's setting mtime to the current time (std::time(0)) not to zero. That is terrible hack that I need to fix. It should be set to the last modified time of the containing archive, instead of last read time. Right now demandRes->mtime has no real value on the resman side because the way it's set up an observer can override everyone with any data it wants. I don't see any great way to handle that given the plugin event architecture. It could be used by an observer to see if they have a newer file.
The cache is an unordered_map of resref.ext keys to a struct containing last modified time, the cached data, size, and a few other things. It gets invalidated only when new NCS data is returned from an observer or a newer NCS file is found in the filesystem. As a whole it never gets invalidated, but probably should if nwserver runs out of memory.
In the original code it looked like that if an observer returned 1 and no data then resman would return NULL and then let CExoResMan handle things. Since there is the cached scripts involved now, I changed this to be a means of expressing the condition that "I have the script, but I've returned it once already, so you should have it in the cache." I didn't want observer's to have to have their own script caches.
What I'd like to do is go back and redo a few things Assuming that a file is in an observer or in the resman directory:
1) Determine what the most recently modified time is for all observers and the file (if it exists) in the resman directory.
2) Only send a demandRes event in the case that an observer has the most recently modified file and if it's a script that the last modified time is newer than what's in the cache. I think this would be best: to always allow the resman directory to override the observers and CExoResMan.
3) For all non-scripts return whatever data, for scripts update the cache if necessary, return cached data.
Feel free to make any suggestion about how you think might be best to accomplish this.
_________________
the awakening (PW Action) |
|
| Back to top |
|
|
elven
Joined: 28 Jul 2006
Posts: 259
Location: Germany
|
| Posted: Sat Aug 16, 2014 17:46 Post subject: |
|
|
| leo_x wrote: |
What I'd like to do is go back and redo a few things Assuming that a file is in an observer or in the resman directory:
1) Determine what the most recently modified time is for all observers and the file (if it exists) in the resman directory.
2) Only send a demandRes event in the case that an observer has the most recently modified file and if it's a script that the last modified time is newer than what's in the cache. I think this would be best: to always allow the resman directory to override the observers and CExoResMan.
3) For all non-scripts return whatever data, for scripts update the cache if necessary, return cached data.
Feel free to make any suggestion about how you think might be best to accomplish this. |
Regarding 1), it's probably simplest to give the cached mtime in ResExists->at_least_mtime (or something) and let observers check if they have something newer, and then set at_least_mtime to their own mtime value if it's newer (and not stop the plugin hook). Then, resman would just walk DemandRes and plugins would return their data if their mtime is found again. (NOTE: Have not actually tested!)
As to 2), I'm not a huge fan of the hardcoded resman directory. I think, it might be neater to just make that an observer too and move it to a different plugin, or rather, just use nwnx_physfs (which is awesome) and remove it alltogether.
I'm currently porting nwnx_redis and your answers helped a lot. My bad about seeing std::time(0) and not checking the docs .. who'd have thought 0 isn't actually 0.
Thanks again!
niv |
|
| Back to top |
|
|
leo_x
Joined: 25 Aug 2010
Posts: 75
|
| Posted: Mon Aug 18, 2014 0:16 Post subject: |
|
|
| elven wrote: | | leo_x wrote: |
What I'd like to do is go back and redo a few things Assuming that a file is in an observer or in the resman directory:
1) Determine what the most recently modified time is for all observers and the file (if it exists) in the resman directory.
2) Only send a demandRes event in the case that an observer has the most recently modified file and if it's a script that the last modified time is newer than what's in the cache. I think this would be best: to always allow the resman directory to override the observers and CExoResMan.
3) For all non-scripts return whatever data, for scripts update the cache if necessary, return cached data.
Feel free to make any suggestion about how you think might be best to accomplish this. |
Regarding 1), it's probably simplest to give the cached mtime in ResExists->at_least_mtime (or something) and let observers check if they have something newer, and then set at_least_mtime to their own mtime value if it's newer (and not stop the plugin hook). Then, resman would just walk DemandRes and plugins would return their data if their mtime is found again. (NOTE: Have not actually tested!)
As to 2), I'm not a huge fan of the hardcoded resman directory. I think, it might be neater to just make that an observer too and move it to a different plugin, or rather, just use nwnx_physfs (which is awesome) and remove it alltogether.
I'm currently porting nwnx_redis and your answers helped a lot. My bad about seeing std::time(0) and not checking the docs .. who'd have thought 0 isn't actually 0.
Thanks again!
niv |
I agree with everything you've suggested. When I make these changes, should I rename this plugin, so that people aren't confused? I'm guessing that sending them upstream will not be appropriate as a patch to resman.
_________________
the awakening (PW Action) |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
Powered by phpBB © 2001, 2005 phpBB Group
|
FAQ
Search
Memberlist
Usergroups
Register
Profile
Log in to check your private messages
Log in
