How do I store a large (250K+ keys) dictionary so I don't need to load it from a file every runtime?

Post reply
0 favourites
From the Asset Store
Piano Keys
$3.99 USD
A musical typing game where the goal is to play a song by pressing the matching keys as fast as you can!
whoa.labs's avatar
whoa.labs

  • Compiled languages will also compile the dictionary. I suspect it compiles it to a sequence of insertion instructions though, which will have to be run when initialising it, i.e. loading it. So I don't think it's that different to parsing JSON.

  • 1. Edit the events in a source view so I can write a script that outputs the necessary code

    Yes, you can do this if you want. Events are stored in XML file in the Event Sheet folder of the project. You can make 250K events with a script and paste them into the XML. But I doubt loading and executing 250K events will be significantly (or at all) faster than loading the dictionary from big JSON file or local storage.

    I can easily split the files, but like I said in my previous post, why do I need to read this at runtime?

    If you split the database into many files, you can access them in runtime with very little lag, which you may be able to hide/mask from players with some visual effects and animations.

    I don't know how long does it take to load 18Mb of data. If this was my game and I had to choose between, say ~30 seconds delay on startup and 0.1 second lag every time when I need to access a word with AJAX, I would choose the latter. Of course, it depends on the game, if you need to quickly load hundreds of random words, this will not work.

  • Try Construct 3

    Develop games in your browser. Powerful, performant & highly capable.

    Try Now Construct 3 users don't see these ads

  • Browser's built-in JSON parsers are absurdly fast (I think they run somewhere near memory-bandwidth speed). I suspect trying to write code, or events, to make the same sequence of insertions, will be considerably slower.

  • Yes, after thinking more about hard coding the events, that would balloon the size of the data significantly. I've been running tests on the AJAX calls. With the complete dictionary data (276,643 records), the AJAX call takes about 2.5 seconds to complete, which should only need to be loaded once and stored in the dictionary for the remainder of the session. The problem I'm running into now is that the dictionary doesn't seem to get any of the data. If I break the data into smaller JSON files (1 or 2MB) it works fine. Is there a limit to the number of keys that can be stored in a dictionary?

  • Check for errors in the browser's console.

    Otherwise, profile the loading of your file

  • Is there a limit to the number of keys that can be stored in a dictionary?

    No, it has no limit. The size doesn't matter.

  • The limit is imposed by the browser. Firefox has a 1MB limit before truncating the JSON file and throws an improper termination error. Chrome doesn't specify the limit but from my testing it's larger than 1MB. Once you hit the limit in Chrome though you receive a generic "no response data" in the debugger. I guess I'll need to create more files and have more AJAX calls. There doesn't appear to be an event structure in Construct that will let me iterate through and load files due to the nature of the OnComplete event with AJAX; is this correct?

  • It actually appears that this is the debugger display limit, not the actual request limit. I'll keep digging. I've tried splitting into 26 smaller JSON files but only the first will load. I use the following events:

    On Start of Layout:

    Request A.json(tag "load_A")

    On "load_A" Completed:

    Dictionary Load from JSON string AJAX.LastData

    Request B.json(tag "load_B")

    On "load_B" Completed:

    Dictionary Load from JSON string AJAX.LastData

    Request C.json(tag "load_C")

    Etc.

    This will load the content of the first request "load_A" into the dictionary. The AJAX call is returning the latest data, in this case, "load_C". But the data from "load_C" does not get added to Dictionary. I've tried this with separate dictionaries and same behavior. Am I overlooking something obvious here?

  • The limit is imposed by the browser. Firefox has a 1MB limit before truncating the JSON file and throws an improper termination error.

    No, this is not the case. Firefox has no such limit. Chrome has no limit either.

  • It actually appears that this is the debugger display limit, not the actual request limit. I'll keep digging. I've tried splitting into 26 smaller JSON files but only the first will load. I use the following events:

    On Start of Layout:

    Request A.json(tag "load_A")

    On "load_A" Completed:

    Dictionary Load from JSON string AJAX.LastData

    Request B.json(tag "load_B")

    On "load_B" Completed:

    Dictionary Load from JSON string AJAX.LastData

    Request C.json(tag "load_C")

    Etc.

    This will load the content of the first request "load_A" into the dictionary. The AJAX call is returning the latest data, in this case, "load_C". But the data from "load_C" does not get added to Dictionary. I've tried this with separate dictionaries and same behavior. Am I overlooking something obvious here?

    You should post an exert of your A.json and C.json files, mainly the beginning of the file and the end of the file.

    The dictionary structure in Construct has its own output in JSON.

    Consider this small sample :

    { "c2dictionary": true, "data": { "0": "a", "1": "b", "2": "c", "3": "d", "4": "e", "5": "f", "6": "g" } }

    This is the content of a dictionary which holds 7 values.

    The

    { "c2dictionary": true, "data": { } }

    is the embedding structure and is expected by the dictionary to load in values appropriately.

    Each of your files should contain this structure.

    Also, if you load your files within the same dictionary object/instance, only the last loaded file data should end up being visible, as the previous data were "overwritten".

  • All of my dictionaries are using the following format:

    {

    "c2dictionary":true,

    "data":

    {

    "AA":"a volcanic rock consisting of angular blocks of lava with a very rough surface [n -S]",

    ...

    "AZYMS":"AZYM; unleavened bread; also AZYME [n]"

    }

    }

    I am trying to load these into specific dictionary objects, so there is an "A" dictionary for the "A" JSON file. The first file "A" loads no problem, it's the subsequent files that don't seem to load.

  • I can't edit posts but I should add that the AJAX.LastData does show the correct data when attempting to load subsequent data, that data is just not getting loaded into the dictionary. I'm not seeing any console errors in Chome or FF. When I click on the dictionary object in the Construct debugger it shows 0 keys.

Post reply
Return to board index
Jump to:
Active Users
There are 1 visitors browsing this topic (0 users and 1 guests)