| View previous topic :: View next topic |
| Author |
Message |
Papillon
x-man
Joined: 28 Dec 2004
Posts: 1060
Location: Germany
|
 Posted: Tue Feb 06, 2007 0:09 Post subject: |
 |
|
With "standardized" I meant the fact that a certain baseline needs to be supported by all database types and plugins. I think it's obvious that scripts need to be able to trust on certain minimum column lengths (storing locations is a recent example where the underlying database didn't support what the script expected.. ouch). It's not a problem if a certain plugin offers more column space than the others, of course.
I guess it's time to find out what the minimum acceptable (and supportable) varchar column size is. As Grumalg already stated, "the length can be specified as a value from 0 to 255 before MySQL 5.0.3, and 0 to 65,535 in 5.0.3 and later versions." (the MySQL docs about VARCHARs).
So my vote is 255. Let's define this as the basic requirement and make the MySQL module set up the table like this. To prevent misunderstandings, I would also like to make the SQLite table definition to use only 255 characters.
The way I see it, if a user gets that advanced and wants to store more than that, he should be able to figure out the implications that increasing the column size would entail, e.g. the need to set up a new table that accommodates the data. People using the pwdata table should be happy with 255 characters and the rest probably wants to create their own database design anyway.
(note: storing more than 255 characters in a text column makes me... cringe.)
Thoughts ?
_________________
Papillon |
|
| Back to top |
|
 |
Gryphyn
Joined: 20 Jan 2005
Posts: 431
|
| Posted: Tue Feb 06, 2007 0:43 Post subject: |
|
|
Suits me...
Making sure that the nwnx_sql script runs on ALL database(s) and versions, should be the determining factor.
This is not to say that we have to be compatible with NWNX2, but we need to be consistent from some point-in-time for NWNX4. We should also nail down, in documentation, the minimum database versions that will be supported.
My vote is for 128 characters (potentially 255 bytes UTF8)
remember NWNX2 has utf8 encoding.
As for SQLite... you can specify the length - but it is ignored by the engine.
If NWNX4 limits the buffer size, you'll get back truncated data. This just needs a note in documentation (not that anyone will read it  )
MySQL is funny about unicode.
CHAR = 3 bytes,
VARCHAR (from what I've read about UFT8) = 1 header byte + 1 or 2 bytes per character
Cheers
Gryphyn |
|
| Back to top |
|
|
Grumalg
Joined: 04 Nov 2005
Posts: 70
|
| Posted: Tue Feb 06, 2007 4:13 Post subject: |
|
|
| Gryphyn wrote: | Most databases have a cap on a SYSNAME object - eg a table or column name - this is usually 256 bytes [255 char -utf8, 128 char -utf16/ucs2]
As a starting point this should be a minimum for NWNX.
|
One would have to be *really* anal to subject themselves to typeing in very long names every time they wrote a SQL command. I strongly suspect the minimum should be based on data size rather than object name size.
| Gryphyn wrote: | What should the maximum be? Well the is really database dependent. But in most (the latest versions anyway) they use a 32bit length header [~ 65,532 bytes]. - This is also true for the older TEXT datatypes.
|
Any DB engine worth it's salt these days will let you store columns/fields containing gigabytes of data. I don't think anyone is going to be moveing around any data that is an appreciable bite out of the 2gb process space limit of NWN2 because they are going to want to squeeze in as many areas/items/PC's/etc. as they can. If we were moving game objects through an sSpacer size it's likely to be more like 2-8kb typical and probably under 64kb max. I do recall one post in these forms some time ago where someone was wanting to store copies of PC objects to the tune of 500+kb with nwnx2, and that person was directed to increase the size of his buffer in sql_include.
| Gryphyn wrote: | As Grumalg mentions you can take a performance hit (I/O) if you don't get your row size fitting into your page size (you databases data buffer). How this is done is managed by the database engine - so you'll have to read up on how it will effect you.
|
Actually, what I was refering to had nothing to do with buffer sizes. I was refering to physical location of data on the HD surface and the extra seek/HD head positioning delays required to get to TEXT and BLOB data which are stored outside the row data at some other (possibly changeing as data is rewritten) location within the DB data file. All other data types in MySQL are stored within a contiguous block of bytes of 65,532 max size for a single row. Not saying buffer size issues can't exist, but that isn't what I was refering to. There surely is a decent chance that the TEXT or BLOB data has already been fetched since the last time MySQL was started and still is sitting in the MySQL data cache so that no HD access is needed to obtain it or the row data.
| Gryphyn wrote: | An earlier post by Papillon mentions doing away with the NWNX 'spacer' (buffer) so hopefully this becomes a dead issue.
|
Surely a resounding cheer will be heard from the NWN community when this comes to pass. However, I wouldn't be holding my breath waiting for this. I think the Obs dev's are up to their a$$ in aligators right now and it may be quite a while before they have time to do something like this solely for PW support.
| Gryphyn wrote: | SQLite is an engine that manages an underlying file-stream. ALL data is converted to multi-byte text or blob. It ignores any 'length' specifications, but lets you add them for documentation.
|
Actually is no longer true that all data is stored as text or blob. As of SQLite 3.x integer and floating point numerics are stored as binary numbers instead of in text form. Ignoring the length of string based datatypes is still true though, so we still can't specify size of any text stored in SQLite.
| Gryphyn wrote: | Currently the 'spacer' is the determining factor. Make it big enough for your largest value, but then you have allocated but unused memory. mind you what's 16Kb out of 1Gb? not a lot. but when your only using, say 64 bytes out of 16Kb, there is a high percentage of seldom used memory.
|
| Papillon wrote: | storing locations is a recent example where the underlying database didn't support what the script expected.. ouch
|
The location storage problem was entirely caused by the 128 byte sSpacer size and was proven to be correctable by increaseing the sSpacer to approx 160 bytes under nwnx4 1.06. I noted at the time that CHAR was limited to 255 chars, and so was VARCHAR (prior to MySQL 5.0.3). I suggested then that sSpacer be raised to 256 chars stock value. a Stock sSpacer of 256 gets my vote because other than large VARCHAR, TEXT, BLOB, or objects once we can again store them again, it will hold *everything* else one could possibly use. This strikes me as desireable for several reasons. For example, it's likely to eliminate a lot of newbie users hitting problems they don't understand, thus reduceing tech support loading here. I think 256 is a good compromise between how much is supported stock and memory use.
Until such a time as sSpacer goes away, you could always have 2 sSpacer lines (128, 256) and simply uncomment the size you needed and comment out the other. That would take no extra active code and still allow choice for those who want the minimum memory footprint and are willing to give up max sized CHAR fields and VARCHAR up to 255.
| Papillon wrote: |
note: storing more than 255 characters in a text column makes me... cringe.
|
It shouldn't... NWNX2 has been moving much larger object data through the local strings for a long time. It's actually very unlikely that the largest text anyone would ever want to move will be bigger than such objects and moving those objects haven't been causeing any big problems due to size.
--- Grumalg --- |
|
| Back to top |
|
|
Gryphyn
Joined: 20 Jan 2005
Posts: 431
|
| Posted: Tue Feb 06, 2007 4:34 Post subject: |
|
|
| Grumalg wrote: |
Until such a time as sSpacer goes away, you could always have 2 sSpacer lines (128, 256) and simply uncomment the size you needed and comment out the other. That would take no extra active code and still allow choice for those who want the minimum memory footprint and are willing to give up max sized CHAR fields and VARCHAR up to 255.
--- Grumalg --- |
sSpacer - due to it's 'character' filler (ie ".") - matches ByteSize and CharacterCount - sure make sSpacer 255 dots long. This will allow upto 128 UTF-8 double byte characters.
Gone are the days when 1 char = 1 byte.
Cheers
Gryphyn |
|
| Back to top |
|
|
Grinning Fool
Joined: 12 Feb 2005
Posts: 264
|
| Posted: Tue Feb 06, 2007 20:22 Post subject: |
|
|
So to make this official so I can make the updates:
Spacer = 256 characters (255+ null)
column: VARCHAR(255)
_________________
Khalidine, a NWN2 persistent world
Looking for volunteers. |
|
| Back to top |
|
|
Papillon
x-man
Joined: 28 Dec 2004
Posts: 1060
Location: Germany
|
| Posted: Wed Feb 07, 2007 20:18 Post subject: |
|
|
Yes please.
Side note: The sSpace is limited in size (128 or 256 in the future) not because I want to safe memory, it's because of speed. Copying data with strcpy() and determining string size with strlen() just gets slower, the large the buffer gets.
_________________
Papillon |
|
| Back to top |
|
|
Papillon
x-man
Joined: 28 Dec 2004
Posts: 1060
Location: Germany
|
| Posted: Sat May 05, 2007 8:28 Post subject: |
|
|
Yay! The sSpacer is gone with 1.08 
Do you think you could update the demo modules so that I can include them when 1.08 becomes official ?
_________________
Papillon |
|
| Back to top |
|
|
Grinning Fool
Joined: 12 Feb 2005
Posts: 264
|
| Posted: Sat May 05, 2007 22:25 Post subject: |
|
|
Yep, sure thing. I'll be downloading the newest nwnx this coming week (assuming that plugins using the old system won't break?) and making the changes.
_________________
Khalidine, a NWN2 persistent world
Looking for volunteers. |
|
| Back to top |
|
|
Morpheus
Joined: 18 Oct 2006
Posts: 39
|
| Posted: Wed May 09, 2007 21:17 Post subject: |
|
|
Awesome guys!! When will the glorious 1.08 be out?
Also, I am thinking of writing some serious server-side plugins. What's a good resource to start looking at code?
Morph |
|
| Back to top |
|
|
Papillon
x-man
Joined: 28 Dec 2004
Posts: 1060
Location: Germany
|
| Posted: Wed May 09, 2007 21:39 Post subject: |
|
|
There is no release date for 1.08, I guess it should be near the end of this month. Please get the sources and have a look at the time plugin, which is a real simple one.
_________________
Papillon |
|
| Back to top |
|
|
federicopeinado
Joined: 07 Jan 2009
Posts: 2
|
| Posted: Wed Jan 07, 2009 22:27 Post subject: Problems editing NWNX4Demo.mod and the other demo modules |
|
|
Hello everybody and thank you for your fantastic work on NWNX!
My name is Federico and I am starting the development of a new plugin for NWNX4. My idea is to create a demo module (similar to nwnx_demo_time.mod, for example) for the plugin, but when I try to open any of the locations of the demo modules from the editor, i get an error message:
System.ArgumentOutOfRangeException
...
NWN2Toolset.NWN2.Data.Templates.NWN2SkillCollection.get_Item(Int32 iIndex)
...
The modules work fine in the game engine, but it seems some "custom feature" (custom skill?) is used in these demo modules that is not found by the NWN2 editor (index out of range...). Any idea of what is happening or how can I open the demo modules?
I am using the 1.21.1549 Spanish version of the game (without any expansion or hak), and I can open/edit the other modules that included in the game normally. |
|
| Back to top |
|
|
KubilaiKhan
Joined: 20 Jul 2010
Posts: 1
|
| Posted: Tue Jul 20, 2010 4:10 Post subject: Re: Problems editing NWNX4Demo.mod and the other demo module |
|
|
| federicopeinado wrote: | i get an error message:
System.ArgumentOutOfRangeException
...
NWN2Toolset.NWN2.Data.Templates.NWN2SkillCollection.get_Item(Int32 iIndex)
...
The modules work fine in the game engine, but it seems some "custom feature" (custom skill?) is used in these demo modules that is not found by the NWN2 editor (index out of range...). Any idea of what is happening or how can I open the demo modules? |
I had the same issue, resolved by deleting feat #30 on the feat list of creatures included in the module.
What I did is :
- Open module and save as a directory. Then close the editor.
- Edit .GIT / .UTC of "bugged" areas / creatures with TlkEdit2
- Searched the 'SkillList' collection and removed the last element (#30)
For areas GIT files, there's a "Creature List" collection, find the creatures and delete the last SkillList element.
Reopen the editor, open directory and save as a .MOD module.
Thats all !
Thats all |
|
| Back to top |
|
|
federicopeinado
Joined: 07 Jan 2009
Posts: 2
|
| Posted: Fri Feb 04, 2011 12:17 Post subject: |
|
|
Thanks a lot for your answer!
(Curiously i am taking up again that NWN2 project... and that is why I have came back here ^_^) |
|
| Back to top |
|
|
|
FAQ
Search
Memberlist
Usergroups
Register
Profile
Log in to check your private messages
Log in
