Welcome! These forums will be deactivated by the end of this year. The conversation continues in a new morph over on Discord! Please join us there for a more active conversation and the occasional opportunity to ask developers questions directly! Go to the PS+ Discord Server.

Standardized item data format?

33 posts / 0 new
Last post
thelabmonkey thelabmonkey's picture
Standardized item data format?
So one thing I noticed in the homebrew forum a lot is different char sheets, item lists, and proposed computer programs/scripts which rely on gear and morph statistics... I was wondering if there would be any utility in creating a project in google which was just a user-updated gear/morph/implant database in a common, agreed upon format that any developers could just scrape into their own apps or sheets? The utility for this would be a standardized repository online that can be kept up to date and imported into whatever projects are going to need all this distilled data. A big plain-text database for all. For example, a good solution for any web-based applications might be a JSON markup like this: // Laser Pulser { 'Laser Pulser' : [   { 'type' : 'Beam Weapon' },   {'metadata' : [     { 'book' : 'Core' },     { 'page' : '338' },     { 'desc' : '' },     { 'cost' : 'mod' }   ]},   {'properties' : [     { 'ap' : '0' },     { 'ammo' : '100' },     { 'modes' : [1,0,0] }, // boolean array [sa,bf,fa]     { 'ranges' : [30,100,150,250] }, // int array [short,med,long,extr]   ]} ]} which most modern web browsers just magicaly turn into a working javascript object that can be polled and manipulated. I feel like, if we can agree on a standard schema and get the data up in the cloud, it might make game tool development a little less intimidating. If we took it a step further, we could even store calculations and relationships between objects for reference or straight usage... not really sure if it's practical, but a neat idea...: // Lucidity Calculation { 'Lucidity' : [   { 'type' : 'Calculation' },   {'metadata' : [     { 'book' : 'Core' },     { 'page' : '122' },     { 'desc' : 'Lucidity determines how much stress (mental damage) you can take before you are incapacitated or driven insane' },   ]},   {'properties' : [     { 'math' : '(%Will%*2) + %EgoTraits% + %MorphTraits% + %Gear%' }, // %var% denotes variable name   ]} ]} While I appreciate that this kind of thing is better suited to an actual database, that severely limits the accesability IMHO... [edit] .csv format files are uploaded here: http://itsmax.net/pub/EclipsePhase/data
sysop sysop's picture
Re: Standardized item data format?
I have seriously considered doing something with Darkcast to store exactly this sort of information and output it in JSon or the like. I don't know if anyone would really use it though so I haven't pushed to set it up. If enough people are interested I'm sure I could set up some forms to gather the information and we can hit it up crowdsource style.
I fix broken things. If you need something fixed, mention it [url=/forums/suggestions/website-and-forum-suggestions]on the suggestions board[/url]. [color=red]I also sometimes speak as website administrator and/ moderator.[/color]
thelabmonkey thelabmonkey's picture
Re: Standardized item data format?
I'd definitly be interested in discussing what kind of output would be useful to whom. If there were some way to keep the darkcast wiki up to date with all of this and just have it vomit up its data, that would be awesome. I see JSON being useful for web and app devs, but spreadsheet driven tools would more likely need xml or even a simpler csv format... Ultimately, what I would love to see is a sort of ecosystem where raw data is updated in one place, pulled down to update tools which can in turn output standard formats to upload back to the cloud, or import into other tools... I feel like, if we had that capability, we could also start accumulating a library of custom morphs, pre-build (n)pcs, special modified weapons, etc. which would be easy to just download and plug in... like blueprints in a fabber! But all of this sort of depends on how many disparate projects are trying to use the data. If someone would just make a single massive character creator, initiative tracker, campaign manager, then we wouldn't really need it. ;)
Quincey Forder Quincey Forder's picture
Re: Standardized item data format?
is it for online character sheets or gear codex (no, Felicia, not you!) I would very interested in either, but I'm totally useless where it comes to code.
[center] Q U I N C E Y ^_*_^ F O R D E R [/center] Remember The Cant! [img]http://tinyurl.com/h8azy78[/img] [img]http://i249.photobucket.com/albums/gg205/tachistarfire/theeye_fanzine_us...
The Doctor The Doctor's picture
Re: Standardized item data format?
Might I suggest adding the revision of the book to the schema to take into account changes in stats? // Laser Pulser { 'Laser Pulser' : [   { 'type' : 'Beam Weapon' },   {'metadata' : [     { 'book' : 'Core' },     { 'revision' : '3' },     { 'page' : '338' },     { 'desc' : '' },     { 'cost' : 'mod' }   ]},   {'properties' : [     { 'ap' : '0' },     { 'ammo' : '100' },     { 'modes' : [1,0,0] }, // boolean array [sa,bf,fa]     { 'ranges' : [30,100,150,250] }, // int array [short,med,long,extr]   ]} ]}
thelabmonkey thelabmonkey's picture
Re: Standardized item data format?
Quincey Forder wrote:
is it for online character sheets or gear codex...
In theory, either and both! It's just raw data in a standardized format that everyone can use for whatever they want, but it would not be in a [i]human friendly[/i] format until someone parses it into an app or spreadsheet...
The Doctor wrote:
Might I suggest adding the revision of the book to the schema to take into account changes in stats?
Good idea, though would we need to keep legacy items from previous printings? V1 laser pulser and v3 laser pulser? For top-level categorization, does it make sense to break it up into blocks like "Weapons", "Implants", "Ego Traits", etc? Or should it be hashed in one big lump with a unique ID? i.e. { 'Weapons' : [ { 'type' : 'Beam Weapon' }, { 'meta' : [ {'name': 'Laser Pulser'} ...etc vs { '10256842' : [ { 'category' : 'Weapons' }, { 'type' : 'Beam Weapon' }, { 'meta' : [ {'name': 'Laser Pulser'} ...etc While I like the idea of a unique id, it would have to be automated to avoid collision. If it was being pulled out of the darkcast wiki, I think that a unique id would be a field in one of the database tables already.
Quincey Forder Quincey Forder's picture
Re: Standardized item data format?
thelabmonkey wrote:
In theory, either and both! It's just raw data in a standardized format that everyone can use for whatever they want, but it would not be in a [i]human friendly[/i] format until someone parses it into an app or spreadsheet...
I'd love to have an EP app on my ecto...I mean on my iPad! That would be awesome!
[center] Q U I N C E Y ^_*_^ F O R D E R [/center] Remember The Cant! [img]http://tinyurl.com/h8azy78[/img] [img]http://i249.photobucket.com/albums/gg205/tachistarfire/theeye_fanzine_us...
sysop sysop's picture
Re: Standardized item data format?
It's worth noting, once the data fields are determined and the data is in place on the system... then the output can be in any format needed. (At least with Drupal it's that kind of flexible). So if we can make a good solid list of fields for each type of data, then we can pull it out in any form requested afterwards.
I fix broken things. If you need something fixed, mention it [url=/forums/suggestions/website-and-forum-suggestions]on the suggestions board[/url]. [color=red]I also sometimes speak as website administrator and/ moderator.[/color]
sysop sysop's picture
Re: Standardized item data format?
Quote:
While I like the idea of a unique id, it would have to be automated to avoid collision. If it was being pulled out of the darkcast wiki, I think that a unique id would be a field in one of the database tables already.
You are right - Drupal assigns a unique id to everything in it's system. Ideally, we're talking about fielded data here though, so one thing to consider is that the input for this would be very different than it is for say a normal wiki post or a forum post. It would be a form where each value to go into the system is clearly defined by the type of data going in. So you pick a morph, it gives you the fields for a morph. You pick a skill, it gives you the fields for a skill - etc etc. I could also set Drupal up to read in from a spreadsheet I upload - so if you want to collect the initial batch of data on a google doc then that's viable. Basically - we can totally do this :) But we gotta have people who want to do the data entry. :)
I fix broken things. If you need something fixed, mention it [url=/forums/suggestions/website-and-forum-suggestions]on the suggestions board[/url]. [color=red]I also sometimes speak as website administrator and/ moderator.[/color]
thelabmonkey thelabmonkey's picture
Re: Standardized item data format?
sysop wrote:
... It would be a form where each value to go into the system is clearly defined by the type of data going in. So you pick a morph, it gives you the fields for a morph. You pick a skill, it gives you the fields for a skill - etc etc...
I definitely like the form-fill idea, but what about editing currently existing data in the event of typos or updates? Could that be done through the normal wiki interface? I don't know Drupal, but if I can help make forms templates, I will. We just have to figure out what categories and fields we need.
sysop wrote:
I could also set Drupal up to read in from a spreadsheet I upload - so if you want to collect the initial batch of data on a google doc then that's viable. Basically - we can totally do this :) But we gotta have people who want to do the data entry. :)
I think there is already a gear listing in spreadsheet form somewhere floating around... at the very least, I know the jmobius character generator excell sheet has all that data hidden in there *somewhere*. If we could get Kindalas to help us find it, we might be able to scrape it in with a little tweaking. If not, we can probably crowd-source a lot of it. I'd pitch in... say doing all the traits or weapons. This kind of wheel-reinventing is what I was hoping we can avoid in the future.
sysop wrote:
... if we can make a good solid list of fields for each type of data, then we can pull it out in any form requested afterwards.
So then the next step, if we are going to try to move forward on it, would be to establish how we are organizing our information INTO the darkcast wiki so that we can make it easy to get back out again.
sysop sysop's picture
Re: Standardized item data format?
Thankfully drupal can provide an edit form for any type of data it has. So fixing typos after the fact, or adding new stuff as books come out will be comparatively easy so long as we keep it up to day - it's this initial data load that's hard. edit: I also just checked the options available for the freelinking module that I use on darkcast - it's what allows for wiki style links using [[ and ]]. It'll work with multiple node types, so we can have the wiki and the db type nodes work together pretty much seamlessly. Start with: a) A list of types of data to store b) A list of the fields within that data to And I'll construct the content types, forms, and give you the interface from Darkcast. If you want to use a spreadsheet for the initial load - if that's easier to work with - then just make sure each field gets it's own column. I can load it into the wiki when you've got a batch ready to go.
I fix broken things. If you need something fixed, mention it [url=/forums/suggestions/website-and-forum-suggestions]on the suggestions board[/url]. [color=red]I also sometimes speak as website administrator and/ moderator.[/color]
sysop sysop's picture
Re: Standardized item data format?
Ok, I found an HTML version of Core book that was structured enough I could scrape it for data. I currently have a *lot* of the information from Core uploaded: http://www.firewall-darkcast.com/wiki/category/core Still to go - Sleights, Backgrounds and Factions. And probably tons of interface issues to make sure the wiki interface works together. Once this thing is in place, I'll start adding ways to pull the information down in various formats. :)
I fix broken things. If you need something fixed, mention it [url=/forums/suggestions/website-and-forum-suggestions]on the suggestions board[/url]. [color=red]I also sometimes speak as website administrator and/ moderator.[/color]
kindalas kindalas's picture
Re: Standardized item data format?
For the Excel Character generator I have the following variables for the morphs Name: this would be a string Aptitude Max: number CP Cost: number Credit Cost: number COG: bonus to COG COO: bonus to COO INT: bonus to INT REF: bonus to REF SAV: bonus to SAV SOM: bonus to SOM WIL: bonus to WIL Pick: from 0 to 3 enables a dropdown to choosing which aptitude gets a +5 without dupilcate bonuses -5 to a skill: 0 or 1 gives a dropdown Durability: just a number WT: actually a useless variable that is autocalculated Synthetic?: 0 or 1 with 1 meaning synthetic pod?: 0 or 1 with 1 meaning pod constuction Uplift?: 0 or 1 with 1 meaning an uplift S1: bonus skill one's name S#: bonus skill one's value S2: bonus skill two's name S#: bonus skill two's value Gear: this is a list of "gearname, gearname, gearname" Traits: this is a list of "morphtrait, morphtrait (level #), morph trait" Energy Armor: just a number Kinetic Armor: just a number description: just text
I am a Moderator of this Forum [color=red]My mod voice is red.[/color] The Eclipse Phase Character sheet is downloadable here: [url=http://sites.google.com/site/eclipsephases/home/cabinet] Get it here![/url]
thelabmonkey thelabmonkey's picture
Re: Standardized item data format?
sysop wrote:
Ok, I found an HTML version of Core book that was structured enough I could scrape it for data. I currently have a *lot* of the information from Core uploaded: http://www.firewall-darkcast.com/wiki/category/core Still to go - Sleights, Backgrounds and Factions. And probably tons of interface issues to make sure the wiki interface works together. Once this thing is in place, I'll start adding ways to pull the information down in various formats. :)
That is hot to death. I've been AFK for a week, sorry. How should I start helping to fill in the other info? @Kindalas : Awesome! Thanks a ton for sharing! I was looking at the character sheet and don't know how/where the data is stored for all of the drop downs and things. Can/May I scrape a set cell range for certain data? How would I find that?
sysop sysop's picture
Re: Standardized item data format?
If you can find a HTML copy of Sunward and the other books that'll go a long way towards getting this put together. What I'm doing is using the HTML tags to look for regular patterns around the text that we want, so I can with enough copy/replace create a CSV out of the data. It's tricky, and takes some time to do but much less time than entering data directly. If you want to start that process on some of the other books and pass the CSV file over then I can load that data in. :) Also any feedback you have on making the wiki easier to navigate on DC would be appreciated. It's one of the underused sections of that site and I know a lot of that has to do with the interface. Since the wiki is also likely to be the basis for API data pulls like you're talking about, it's worth getting the navigation right so app writers can use it.
I fix broken things. If you need something fixed, mention it [url=/forums/suggestions/website-and-forum-suggestions]on the suggestions board[/url]. [color=red]I also sometimes speak as website administrator and/ moderator.[/color]
thelabmonkey thelabmonkey's picture
Re: Standardized item data format?
Oh, I just figured out where the data is kept in the jmobius sheet! I'm not exactly an excell guru... Here's values.csv that is just a dump of the values page from the jmobius sheet with the morphs section chopped out, and morphs.csv which contains all of the morph values that Kindalas outlined above, plus I added in two columns. The first is "source" which has CO,SW,GC,or OP. The second is "description" which has the descriptions cut from the morph section in values.csv (it seemed like that would be easier) http://itsmax.net/pub/EclipsePhase/values.csv http://itsmax.net/pub/EclipsePhase/morphs.csv if it was a custom morph, I guess we could put the source as CU as well. edit: the values .csv is a dump of the "data" sheet. Sorry. The "values" sheet has a ton more stuff in there. I'll try to get that out in a bit.
thelabmonkey thelabmonkey's picture
Re: Standardized item data format?
I've noticed that the gear descriptions in the jmobius sheet are more like quick summaries. Very useful for item descriptions on character sheets, but lacking the full descriptive text from the books. We should consider having separate "description" and "summary" fields so that we can have the full text from the book up on the wiki, but maintain those shorter, curated summaries.
thelabmonkey thelabmonkey's picture
Re: Standardized item data format?
Ok, so I figured out that on the jmobius sheet you can get ~170 characters on the gear description line in the printable version... so if we have a summary string I'm all for keeping it at a 170 char maximum. We could, in theory, push that down to 150 or even 140 (tweet length!). I'm going to go through and try to organize some of the data and values for things like guns and armor into an easy to slurp .csv now.
thelabmonkey thelabmonkey's picture
Re: Standardized item data format?
Ok, so I think we can have a set of metadata info that would have the same fields for any type of entry ( weapon, trait, faction, etc) that looks like this (using JSON): {'metadata' : [ { 'cat' : 'Gear'}, // category, like 'Ego Trait', 'Morph', or 'Background' { 'type' : 'Beam Weapon'}, // category subtype { 'name' : 'Laser Pulser'}, { 'book' : 'CO' }, // CO,SW,OP,GC for different books { 'page' : '338' }, { 'desc' : 'full description text from the book' }, { 'sum' : '<170 character rules/description summary' }, // good for use in gear descriptions or mouseover callouts ]}, Did I miss anything?
kindalas kindalas's picture
Re: Standardized item data format?
I'd add firing type, SA, SS, BF, FA, and a toggle to say if concentrated fire was possible. And I would call the core book just EP instead of CO. And PO for Panopticon. It's what I've been using in the excel sheet and what I think people are used to.
I am a Moderator of this Forum [color=red]My mod voice is red.[/color] The Eclipse Phase Character sheet is downloadable here: [url=http://sites.google.com/site/eclipsephases/home/cabinet] Get it here![/url]
thelabmonkey thelabmonkey's picture
Re: Standardized item data format?
kindalas wrote:
I'd add firing type, SA, SS, BF, FA, and a toggle to say if concentrated fire was possible.
I was talking more about the general metadata that could be applied to any and all entries, eg traits, weapons, morphs, or even skills. There would be item specific data for each type or category, but I think it would be a good idea to have a metadata node/table/whatever that always had the same fields.
kindalas wrote:
And I would call the core book just EP instead of CO.And PO for Panopticon. It's what I've been using in the excel sheet and what I think people are used to.
Oh yes, agreed. Let's do that... OP was a typo :D
thelabmonkey thelabmonkey's picture
Re: Standardized item data format?
uploaded two test .csv format files (and edited OP to have the links at the top there). http://itsmax.net/pub/EclipsePhase/data/
thelabmonkey thelabmonkey's picture
Re: Standardized item data format?
Added Data_Traits.csv http://itsmax.net/pub/EclipsePhase/data/ It's got all traits from EP, SW, PO, and GC... I'm pretty sure. Another set of eyes on it would be helpful. If there are any columns that would be helpful, let me know. I'd considered adding an allowed or restricted morph list, but that seemed very... tedious.
kindalas kindalas's picture
Re: Standardized item data format?
thelabmonkey wrote:
Added Data_Traits.csv http://itsmax.net/pub/EclipsePhase/data/ It's got all traits from EP, SW, PO, and GC... I'm pretty sure. Another set of eyes on it would be helpful. If there are any columns that would be helpful, let me know. I'd considered adding an allowed or restricted morph list, but that seemed very... tedious.
Excellent work, I'm going to stat incorporating some of the data into the character generator. For the allow morph column I'd consider having am all/bio/synth catagory so that 0 is all, 1 is bio and 2 is synth only. And then have a 2nd column where specific morphs are enabled.
I am a Moderator of this Forum [color=red]My mod voice is red.[/color] The Eclipse Phase Character sheet is downloadable here: [url=http://sites.google.com/site/eclipsephases/home/cabinet] Get it here![/url]
thelabmonkey thelabmonkey's picture
Re: Standardized item data format?
kindalas wrote:
For the allow morph column I'd consider having am all/bio/synth catagory so that 0 is all, 1 is bio and 2 is synth only. And then have a 2nd column where specific morphs are enabled.
That sounds pretty solid. There are just a few situations where certain morphs are allowed, so it shouldn't be too hard to do things that way. If I get some time tonight, I'll put that in. Thanks! Next I think I'm going to tackle psi sleights...
thelabmonkey thelabmonkey's picture
Re: Standardized item data format?
kindalas wrote:
For the allow morph column I'd consider having am all/bio/synth catagory so that 0 is all, 1 is bio and 2 is synth only. And then have a 2nd column where specific morphs are enabled.
Ok, I have uploaded the new version with two new columns as per Kindalas' suggestion with a slight tweak: The columns are "morphtype" and "morphallowed". The first new column "morphtype" is anything from -4 to 4. A positive number means to include only morphs from the second column (if black all are allowed), a negative number means to exclude... 0 = all 1 = all biomorphs 2 = synthmorphs 3 = pods 4 = uplifts so for "Genetic Defect" you'd see | -1 | Flat | and "Impaired Linguistics" would be | 4 | |
thelabmonkey thelabmonkey's picture
Re: Standardized item data format?
Slights are in: http://itsmax.net/pub/EclipsePhase/data/Data_Sleights.csv non-meta data columns are: psitype, action, range, duration, strain, skill
thelabmonkey thelabmonkey's picture
Re: Standardized item data format?
Updated the morph data to have extra fields I'd left out... now includes traits and size modifier for attackers. Also put in a numeric morphtype column that is in line with the version I did in the traits data for better cross compatability stuff... 1 = biomorphs 2 = synthmorphs 3 = pods 4 = uplifts @Kindalas might be interested in the updated descriptions and summaries for all morphs, including GC and PO... http://itsmax.net/pub/EclipsePhase/data/Data_Morphs.csv And added factions. http://itsmax.net/pub/EclipsePhase/data/Data_Factions.csv
kindalas kindalas's picture
Re: Standardized item data format?
thelabmonkey wrote:
Updated the morph data to have extra fields I'd left out... now includes traits and size modifier for attackers. Also put in a numeric morphtype column that is in line with the version I did in the traits data for better cross compatability stuff... 1 = biomorphs 2 = synthmorphs 3 = pods 4 = uplifts @Kindalas might be interested in the updated descriptions and summaries for all morphs, including GC and PO... http://itsmax.net/pub/EclipsePhase/data/Data_Morphs.csv And added factions. http://itsmax.net/pub/EclipsePhase/data/Data_Factions.csv
You know at 3am well today I was thinking I wonder if it would be all right to ask thelabmonkey to write some more data fields that I'm missing. And then I come online today and it is all done. Thanks for being psychic
I am a Moderator of this Forum [color=red]My mod voice is red.[/color] The Eclipse Phase Character sheet is downloadable here: [url=http://sites.google.com/site/eclipsephases/home/cabinet] Get it here![/url]
thelabmonkey thelabmonkey's picture
Re: Standardized item data format?
kindalas wrote:
Thanks for being psychic
Does "Thought Browse" count? ;) The least I can do after all the work you've put into the character sheet.
thelabmonkey thelabmonkey's picture
Re: Standardized item data format?
Hmm... I just checked and had to trim a few summary text down on pretty much all the .csv files for a few items to bring it down to 170 characters. Might want to grab that morph list again, if that matters to you.
tasuret tasuret's picture
Re: Standardized item data format?
I would be willing to try and make a parser/displayer if someone could give me JSONs to work with. :/
thelabmonkey thelabmonkey's picture
Re: Standardized item data format?
That could be cool... except I've been working on it in .csv format since I'm doing a lot of cut/paste from the work Kindalas has already done in the jmobius character sheet. Once everything is actually in place, maybe you could write a parser to convert between .csv and JSON? Just parse the .csv into a javascript object I guess... I did just upload armor, factions, and gear though... They are mechanically complete, but missing some or all descriptions or summary text. http://itsmax.net/pub/EclipsePhase/data/