redis with pubsub & sco/rco support

[elven]

Hi there,

Update: Sourcecode is currently being reworked pending some PRs to upstream. It won't run on nwnx2 as it is on github. Don't worry though, it will return. The sources on my gh branch have been removed since they were very outdated and memleaky. Just contact me if you really need it RIGHT NOW.

What it does:

* redis access from nwscript side (all commands implemented)
* sco/rco support (store gff objects in your redis database, retrieve them later). Requires recent nwnx_odbc, which provides the actual hooks.
* resman support
* pubsub support. subscribe to channels, receive them inside nwserver in a timely fashion (script hook in main loop)

Does not depend on the redis library - compiled in statically.

To run:
* install libboost-thread 1.35 or later
* install redis-server (compiling it yourself is recommended)
* read the included README.md (displayed nicely on github for you)
* Script documentation is in nwnx_redis_core.nss. Actual redis functions are in nwnx_redis.nss, which is the one you want to include in your stuff (but put them both in your mod).

Feedback appreciated.

[leo_x]

I'm wondering, would you be willing to share how you use Redis and how it fits into your overall stack?

You mentioned loading GFFs, do you also use this as cache/replacement for the DB plugin? Is the pub/sub to communicate with other web services?
_________________
the awakening (PW Action)

[elven]

Certainly. On the road right now but I'll post sth tomorrow.

[elven]

Alright, here we go.

First of all (since this is the most important usecase) I'm using redis to host our complete module (using your resources branch to back it), except scripting.

I'm developing a web frontend, where players can upload (most of all) areas, and any other template that would otherwise go into the .mod file itself. Gff data is mangled on upload/deploy to catch mistakes and force essential standards, and then a pubsub message goes out to nwserver to tell it what was done.

nwserver acts on those messages ("resman:deployed whatever.are"), for example with reloading an area with nwnx_areas. Players can work on the module without me ever lifting a finger except to fix software bugs.

I'm also storing gff metadata in redis hashes associated with deployed templates, so that I can read them from ingame - for example, to reload creatures in areas that are updated, to re-init shops with new template data, or simply to cache creature names for our dynamic summon spawn system ("HGET resman:creature.utc:.fwn name"). That cache is populated by said frontend on deploy too.

Other uses include unpacking bic/character and session/chat data so it can be searched live in a player-facing web app.

Pubsub is also used to provide both in-webapp (desktop notify) and jabber/xmpp notifications for player join/leaves so people can match up for roleplay (we don't have that many players .. who does). A web-based chat binding is planned (and did exist for a while) but I couldn't be bothered to polish it enough to meet quality standards, so it's on the backburner for now.

RCO/SCO over redis is planned for player item upload - again, only polish and demand is pushing this back. Basic process for this is SET upload:<playerId> gffData, PUBLISH upload:<playerId>; then nwserver just does RCO on that key and has the item in inventory. Really simple stuff.

As you can see, I'm not using redis as a postgres replacement (it never was meant to be), but as a very fast intermediate cache for runtime data and configuration (everything that can be regenerated from versioned storage). Calling into redis sees an increase of about factor 10 compared to odbc2+pgsql for similar queries, plus I just love the simplicity of it (compared to hand-crafting queries in nwscript ..)


This isn't all by any means, but it gives a quick few usecases which cannot be done with regular odbc access, at least not without polling or some other crud.

If you want me to blather more on anything mentioned, do tell.

[leo_x]

Thanks for sharing! Really awesome work you've done on this.

You mention unpacking BICs. Is this a runtime thing (eg you can see someone who is logged in) or can people browse whole the servervault/their characters from the web app?

It seems like you're a stones through from having live web-based GFF editing. Is that in the plans or more web based DM features?
_________________
the awakening (PW Action)

[elven]

It's just unpacking the .bic file into redis as a cache to browse it from the web interface, and to refer things in the charfile from nwserver that aren't accessible otherwise. I'm not doing any live editing this way, since it's a one-way process. But I do unpack ALL of the character files on save, not just the online ones.

As to your second question, I could not generate the .bic data from it either way, since it doesn't unpack ALL of it, just the parts I need in a format optimised to display it in online char sheets. I do store the .bic itself in the database, though, and could theoretically edit it from the frontend if I were so inclined. Or any other SCO/RCO'ed game object, or any that is deployed in FWN.

Edit:

Yes, there's plans to do web-based DM support. I have various ideas to do this and some code is already written, but nothing really concrete. Most of all, I want to have better tooling for managing creatures, spawning, managing objects, area mutations (think: quest setup, teardown) and so on.