How do I store a large (250K+ keys) dictionary so I don't need to load it from a file every runtime?

  • I'm making a Bookworm Adventure clone and need to store a large English language dictionary. I'm currently reading in a JSON file using AJAX and loading a dictionary object at startup. This seems costly to do every time the game loads. Is there a way to store a dictionary so I can load it once and freeze it across sessions?

  • If you really want to, use the local storage to store the data on the player's device.

    scirra.com/manual/188/local-storage

    construct.net/en/forum/construct-2/how-do-i-18/how-do-i-frequently-asked-ques-41236 category "Local Storage"

    scirra.com/tutorials/358/asteroid-clone-in-less-than-100-events (score system is pretty much what you are looking for)

  • I'll look into leveraging local storage for this. Is this the best solution? Would I be better using XML and searching the XML to validate at runtime? The JSON file that I prepared to import via AJAX is around 18MB. I know most modern browsers will support at least that for local storage so I'm more worried about runtime performance.

    This game will not be able to reach the web as it will be built into a standalone arcade cabinet w/o internet access. I could host a local web server via Apache and fetch data from there if it will improve performance.

  • How you load the file doesn't affect the performance of accessing it once loaded. Where you load it from probably doesn't affect the load performance (it still has to parse the same data). And to be honest if it's 18mb that's not really that big - modern CPUs are so fast they'll probably make light work of that!

    In short, don't worry about performance until it's actually a measurable problem.

  • Try Construct 3

    Develop games in your browser. Powerful, performant & highly capable.

    Try Now Construct 3 users don't see these ads
  • Thanks for all of the help so far. I've successfully loaded my JSON file into local storage, though I noticed if I use my complete JSON file (250K+ records) Chrome chokes. I'll need to split the files and take multiple requests.

    Now that I have my keys loaded into local storage, I'm attempting to compare if a key exists against a string. I'm using the following:

    Local storage > compare key

    = Equal To

    String: curWord

    curWord exists and I can see it in the debugger. But this even doesn't fire, even though the key exists in local storage. I can't use a top level Key Exists event because I need this to trigger on an input, in my case a key press event.

  • The value inside curWord must be exactly the same as the KeyName you are trying to reach.

    It must have the exact same case (cAsE is not equal to CaSe).

    Consider posting your project focusing on this aspect for investigation.

  • I'm forcing the key to uppercase when it is loaded into local storage and also forcing my compare string to uppercase. Looks like it should be OK based on the debugger output, but it's not working. I'm sure it's something simple I'm overlooking.

    I've uploaded a copy of my capx file here:

    filehosting.org/file/details/779033/SpellQuest.capx

    Thanks again for the assistance.

  • I think storing 250K+ separate keys in local storage is not a great idea. Your 18Mb will probably grow in size to hundreds of megabytes.. I would put all these keys into one (or several) dictionary/array and write it as JSON to local storage in one key.

    Also, JSON and especially XML format uses a lot of space. You should be able to decrease the size by at least 20-30% if you store this data in plain text format, say as comma-separated values. But of course, this depends on the data, and you'll need to write some functions to parse it.

    .

    Another approach is to split all this data into many (hundreds or thousands) small files and make some kind of an index. So when you need to access some word, you refer to the index to find which file contains it, and then request this file with AJAX.

  • I'm certainly open to those approaches but since this is static data, is there no way to populate a dictionary before runtime? I see there is an editor for Construct3 but there must be an underlying structure of the data that could be hand edited and stored in the project, circumventing the need to read a file in the first place, no?

  • One huge mistake in your project is event #4 - it's executed on every tick. You should move it to sub-event under event #3.

    I would definitely try splitting this database into about 1000 smaller files and forget about local storage. Simply read them with AJAX.

  • OK thanks. I made that quick change and it does improve performance drastically. I can easily split the files, but like I said in my previous post, why do I need to read this at runtime?

    I noticed that if I open the XML for my layout I can see the instance of the dictionary:

    <instance type="Dictionary" uid="41" />

    Can I not append my data here manually? If I can, what's the correct XML schema to follow for dictionary items?

  • I don't think you properly understand: the data must be read in to memory at some point. You can shuffle around the data and store it in different places, but some time before you start reading from the dictionary, it's gotta be in memory. To get it in memory means loading it. It sounds like you're trying to find a way to avoid loading it. You can't avoid that - you can put it somewhere it may load a little faster or more conveniently, but the only way to entirely avoid the load step would be to burn the data in to your RAM sticks in the factory... which I don't think you'll be doing!

  • I'm not dense and I've written software applications for many years. I understand that the data must be in memory at runtime, but I don't see why I can't prepopulate a dictionary.

    If I want a static dictionary in C# I don't need to read the data from a file at runtime every time to populate the dictionary. I can populate the dictionary at design time and it is then loaded into memory at runtime. This is all I'm asking to do in Construct to avoid an unnecessary file read/parse every time the application runs.

  • If I want a static dictionary in C# I don't need to read the data from a file at runtime every time to populate the dictionary. I can populate the dictionary at design time and it is then loaded into memory at runtime.

    I'm not a specialist of C# but isn't the population you are referring to the fact of reading that file and putting those data in memory ?

    Isn't it anything else than an embedded file/data in your application that still has to be read on runtime ?

    An "automatic" reading if you will on loading of the application, but still the exact same thing.

    This is all I'm asking to do in Construct to avoid an unnecessary file read/parse every time the application runs.

    If you really want to do so, you can have a function and use the "Dictionary: Add key" dictionary action with your content within the event sheet.

    It won't go and read the project file, but the 18 Megs of data will be directly included in the data.js file at export and will still be read/populating your dictionary at runtime.

  • In C# that dictionary would be compiled, so I understand it's not an apples to apples comparison. Setting the keys as events is essentially what I'm looking to do, but without having to set them manually. 250,000 keys would take a long time to add events for. Is there a way to either:

    1. Edit the events in a source view so I can write a script that outputs the necessary code

    2. Define the dictionary by directly editing the embedded data in the Construct project file

    I don't see this as any different than setting instance variables at design time. I'm setting the keys and values of a dictionary at design time, but for a large quantity of data, so I'd prefer to not do it by hand with events.

    I've got multiple dictionaries working using AJAX to read JSON files and it does what I need to at this point, I just feel like this is a sledgehammer of a solution for data that only needs to be set once in the entire lifetime of the project.

Jump to:
Active Users
There are 1 visitors browsing this topic (0 users and 1 guests)