| View previous topic :: View next topic |
| Author |
Message |
drake127
Joined: 26 Jan 2010
Posts: 28
|
 Posted: Tue Jun 21, 2011 13:41 Post subject: ODBC: Dynamic allocation |
 |
|
Hi, I would like to use ODBC to store large strings (in-game books) and I don't consider wise to enlarge SPACER too much (let's say 8-64 KiB).
Thus I looked into NWNX ODBC plugin and managed to implement dynamic allocation of the result so it is as large as it needs to be. However, I am not content with my solution. Can somebody advise if there is better way?
During each fetch command (i.e. each row) I look up SPACER variable in the module variables array and reallocate string. It works fine but I don't like that I need to look for SPACER again and again: if there were a lot of variables I would endlessly looped through them searching. |
|
| Back to top |
|
 |
virusman
Joined: 30 Jan 2005
Posts: 1020
Location: Russia
|
| Posted: Tue Jun 21, 2011 15:39 Post subject: |
|
|
You should look at this file.
Just add free() and malloc() if the content length exceeds the spacer length.
_________________
In Soviet Russia, NWN plays you! |
|
| Back to top |
|
|
drake127
Joined: 26 Jan 2010
Posts: 28
|
| Posted: Tue Jun 21, 2011 15:54 Post subject: |
|
|
| virusman wrote: | You should look at this file.
Just add free() and malloc() if the content length exceeds the spacer length. |
Yeah, but I only have char* while I need something like char** (or at least char * &). If I free that pointer and allocate new one, this new one would not propagate and would cause real mess.
The origin of this pointer problem leads me up to | Code: | | char* CNWNXODBC::OnRequest (char* gameObject, char* Request, char* Parameters) |
which allows me to work only on string, not on the Parameters pointer.
Or am I wrong on this one?
Last edited by drake127 on Tue Jun 21, 2011 15:58; edited 1 time in total |
|
| Back to top |
|
|
drake127
Joined: 26 Jan 2010
Posts: 28
|
| Posted: Tue Jun 21, 2011 15:57 Post subject: |
|
|
Originally, I modified Fetch this way (requires some more changes but these are important):
| Code: | void CNWNXODBC::Fetch(char * gameObject)
{
CExoString * buffer = 0;
if (((CNWSModule *)gameObject)->GameObject.ObjectType != OBJECT_TYPE_MODULE)
Log (1, "! SQL Error: invalid module object\n");
for (int i = 0; i < ((CNWSModule *)gameObject)->mod_vartable.vt_len; ++i)
if (strncmp(((CNWSModule *)gameObject)->mod_vartable.vt_list[i].var_name.Text, "NWNX!ODBC!SPACER", 16) == 0)
buffer = ((CExoString *)(((CNWSModule *)gameObject)->mod_vartable.vt_list[i].var_value));
if (buffer == 0)
Log (1, "! SQL Error: invalid result string\n");
// Notice buffer is CExoString*
db->Fetch(buffer);
} |
|
|
| Back to top |
|
|
drake127
Joined: 26 Jan 2010
Posts: 28
|
| Posted: Tue Jun 21, 2011 21:02 Post subject: |
|
|
OK, one more thought - if I return from OnRequest newly allocated pRes, nwnx2lib.cpp would correctly deallocate one of these variable, right?
Edit: It seems to work as well but it segfaults when ending process - probably double deallocation? |
|
| Back to top |
|
|
Skywing
Joined: 03 Jan 2008
Posts: 321
|
| Posted: Wed Jun 22, 2011 1:44 Post subject: |
|
|
| As an FYI, if this allocation is being returned to the game, free will do the wrong thing on Windows builds (as the game doesn't allocate from the allocator that your 'free' will go to). If this is in portable code you'll want to be very careful about this lest painful debugging be necessary to get a port up and running. |
|
| Back to top |
|
|
drake127
Joined: 26 Jan 2010
Posts: 28
|
| Posted: Wed Jun 22, 2011 9:59 Post subject: |
|
|
| Skywing wrote: | | As an FYI, if this allocation is being returned to the game, free will do the wrong thing on Windows builds (as the game doesn't allocate from the allocator that your 'free' will go to). If this is in portable code you'll want to be very careful about this lest painful debugging be necessary to get a port up and running. |
Could you elaborate more? I don't aim to support Windows but I looked into win32 source and the source code in question is the same (http://nwn.virusman.ru/trac/nwnx2-win32/browser/trunk/NWNXdll/NWNXdll.cpp#L196), specifically line 209 which frees original CExoString text and replaces it with malloced one. (eg. from http://nwn.virusman.ru/trac/nwnx2-win32/browser/trunk/Chat/NWNXChat.cpp#L132). |
|
| Back to top |
|
|
Skywing
Joined: 03 Jan 2008
Posts: 321
|
| Posted: Wed Jun 22, 2011 21:44 Post subject: |
|
|
| The NWNX-Win32 code you linked is broken and will result in a leak or memory corruption at runtime. The malloc/free that nwserver.exe use on Windows are private versions that maintain their free lists internal to nwserver.exe, the standard CRT malloc/free will operate on the wrong free lists. |
|
| Back to top |
|
|
drake127
Joined: 26 Jan 2010
Posts: 28
|
| Posted: Fri Jun 24, 2011 10:43 Post subject: |
|
|
| Skywing wrote: | | The NWNX-Win32 code you linked is broken and will result in a leak or memory corruption at runtime. The malloc/free that nwserver.exe use on Windows are private versions that maintain their free lists internal to nwserver.exe, the standard CRT malloc/free will operate on the wrong free lists. |
I see, so is there anything that can be done about it? I don't mind calling appropriate functions if they are known.
On linux, is calling malloc/free ok? |
|
| Back to top |
|
|
Skywing
Joined: 03 Jan 2008
Posts: 321
|
| Posted: Sat Jun 25, 2011 19:20 Post subject: |
|
|
| drake127 wrote: | | Skywing wrote: | | The NWNX-Win32 code you linked is broken and will result in a leak or memory corruption at runtime. The malloc/free that nwserver.exe use on Windows are private versions that maintain their free lists internal to nwserver.exe, the standard CRT malloc/free will operate on the wrong free lists. |
I see, so is there anything that can be done about it? I don't mind calling appropriate functions if they are known.
On linux, is calling malloc/free ok? |
It should be ok on linux (although in the broader-than-nwn1 case, this is an unsafe assumpton that must never be made without verification first).
For Windows, you'll have to call the malloc / free or operator new / operator delete functions static linked into nw2server.exe after discovering their offsets. |
|
| Back to top |
|
|
drake127
Joined: 26 Jan 2010
Posts: 28
|
| Posted: Sat Sep 03, 2011 22:11 Post subject: |
|
|
OK, holiday is over so I finished dynamic allocation of the ODBC plugin (fetching rows from a result set). As Skywing pointed out, the solution is confirmed to be working only on NWN1 server for Linux.
Length of result isn't limited to the length of the SPACER variable (SPACER can (and should) still be used but I would reduce its size to prevalent row size e.g. 64 characters).
I made some tests it worked for me but I will appreciate any volunteers who would be willing to test it in complex situations (all MySQL, SQLite and PostgreSQL engines). |
|
| Back to top |
|
|
|
FAQ
Search
Memberlist
Usergroups
Register
Profile
Log in to check your private messages
Log in
