Massive Arrays

0 favourites
  • 15 posts
  • The game I'm currently developing is (trying to) utilize an array that is ~1600 x 1600. This ends up being around 2.5 million cells. Long story short, we have to develop a 3rd party application that can randomly generate an array as a json file and then will load it into the c3 game from that.

    So to test this, I decided I'll make a new project with an array that size and save the json file after setting random variables (I'm doing this in order to see how long it will take to load it). However, I cannot download any array json file that seems to be larger than 400 x 400. It just gives me a "Failed - Network Error" on the download.

    I'm assuming if I can't download a file that large, it won't be able to load a file that large.

    SO - I have 2 questions for anybody who would happen to know this ASHLEY.

    1. Is there a reason I am unable to download that file? It doesn't crash when I try to download it, it just won't download.

    2. Is there a way to load an array from a binary text file? This would obviously be easier and faster to read.

    We're starting to really push the limitations of (c3? HTML5? not sure which one) and would love some answers before we have to give up!

    Thanks in advance.

  • Is that per pixel?

    Take a look at the json format, there's a lot more to it then just the data it stores.

    You're going to have to define the use case for anyone to give you a plausible workaround.

  • Try Construct 3

    Develop games in your browser. Powerful, performant & highly capable.

    Try Now Construct 3 users don't see these ads
  • newt - I'm not entirely sure what you mean by per pixel. Basically, we are using functions to create seed randomization of tilemap tile numbers in a single XY array. Doing this is c3 takes about 10 minutes, and only takes a minute or two in unity.

    So the original idea was to export an exe from unity. include it with the game and run it secretly in the background to create a json file in the directory of the game. Then once that was created, we would load the tileArray data from the json file and delete it.

    Hopefully that gives you a bit more information.

  • If you're populating an array just to treat it like a seed, you might want to try some other method.

    Seems like that would only work per install.

    There were actually a couple plugs for that in C2 which could possibly be converted rather easily.

    Then again making a rng using formulas isn't that hard to figure out.

  • newt - It is hard once you start adding tons of conditions (enemy placement, terrain randomization, cave locations, harvest material locations, etc. etc.) Right now we're just trying to deal with creating x number of biomes with 2 simple rulesets of one being in the middle, and another being a concave biome always facing north.

    Running it from an exe was the only solution we could think of in our heads to solve the time it takes to randomize everything.

    For what it's worth, trying it in the C3 runtime takes around 35% of the time to randomize it, but the game crashes the second you move your character (before OR after the RNG), so unfortunately that's not exactly an option.

  • Maybe try randomising by object rather than by tile?

  • I've noticed that when you make an Array too big the game crashes.

    I tired to do a 5000x5000 array and it was like NOPE!

    So Idk Arrays smaller than 100x2 are fine but much bigger and forget about it.

  • An array is just data, if it’s really big don’t loop over the whole thing at once and it won’t cause an apparent hang.

    You can look at the browser console for more details about what the “network error” may be.

    You could also look into compressing the data before sending it. Also if it’s tilemap data why not put it in a tilemap first before getting the json. It will be smaller that the json from the array object.

    Instead of using a separate program to generate the json that runs in the background, I’d reccomend doing it in JavaScript instead so you don’t have to deal with network file transfers.

    You could do the generating in events too. The key here is to do stuff over many frames instead of all at once to avoid long slowdowns that would seem like a crash to the browser.

  • I presume your trying to download this file using the Array plugins "download" action? I expect the method is struggling with the array size, but it's not clear why. If you submit a bug to our issue tracker with a barebones version of the project your using to generate the array we can take a closer look.

    You can include JSON files ( in our array format ) into your project by importing them into the "files" folder. To load the file in your game you can use the AJAX plugin to get the file as a string, then use that to create your array. We don't support a pure binary form of array files at the moment, only JSON. Binary formats tend to be a lot smaller, and often faster to load, but need specialist tools to view or edit them. Whereas textual ones like JSON are can be read in any text editor ( notepad or textedit for instance ).

    The inbuilt array editor in C3 can create arrays up to 1000 x 1000 x 1000. But in practise I wouldn't advise trying to create an array that large, I think the highest I've got it to load is about 60 million cells before chrome runs out of memory, but your mileage may vary.

    I was curious how much memory such a large array would use, so I've done a bit of testing. Within the runtime it's hard to predict quite how much memory an array is going to consume, and even with the same data it's going to vary depending on a lot of factors.

    You can get a rough estimate by counting the number of objects that make up the array. For starters all arrays in construct are 3D even if your only using a simple 1D array. This 3D array is made from lots of smaller 1D arrays, structured as an array of arrays of arrays.

    Number of sub arrays = (width x height) + width + 1

    Number of items = width x height x depth

    Size of a string (bytes) = 2 x length

    Size of a number (bytes) = 8

    Size of an array (bytes) = 32

    So for an array of numbers ( I used 0 as my number ) with the dimensions [5000, 5000, 1] you would need 900MB. In practise I saw about a 960MB memory increase when I created an object that size ( actually a smaller margin of error than I was expecting ). Unfortunately I can't be any more precise, because the memory profiler in chrome cannot deal with an object this large ( it crashed the tab every time I ran it ). Converting that to JSON gets you a string that is 100 million characters long ( 200MB ). When you use the download method in the array plugin that JSON string is then URI encoded, which increases it's length by 2.5 times. That should be 250 million characters ( 500MB ) but in practise chrome crashed again at this point. If your curious supposedly the maximum length for a string in chrome is about 1 billion characters.

    Okay final tally. Say I had that data as a file, used AJAX to get it over the network, passed it into my array then tried to download it. How much memory would that use?

    200MB + 960MB + 200MB + 500MB = 1.86GB

    700MB of that would be garbage collected after the download had completed, but its still quite a lot!

  • Ome6a, you might look at the Javascript plugin https://www.construct.net/make-games/addons/1/javascript to move more demanding code and data out into javascript, instead of running it all in C3 arrays and event sheets. And then leave C3 events to handle the rendering. I don't have much experience with it yet, but it seems promising so far.

  • Yeah, that’s what we ended up doing for some of the perlin noise.

    We scrapped the original randomization idea due to art requirements anyway.

    Nepeo - that’s amazing! Thanks so much for testing all that, it’s actually super useful to what we’re doing. I had a feeling it was going to be quite large.

    Fortunately we’re planning on trying to keep ~1000 x 1000 x 1, but we will end up needing a few of them (1 for biomes, 1 for sub biomes, 1 for bitmasking, 1 for objects, etc)

  • Is this really the right approach?

  • Cassianno - Truthfully I don’t see any other way of doing it.

  • I understood that you're making a sandbox game right? Maybe something like terraria?

    How are you thinking to achieve this with this array? Control each tile in an array position?

    I won't go into much detail but check this out:

    chilly-durango.itch.io/level-generator-toolkit . The demo is pretty amazing.

    You can also divide/split the world into chunks and work with them to make your terrain. After the base terrain you can recursively/continuously add "layers" of detail into the terrain (ores, treasures, etc).

  • I'm not completely clear as to the structure of your game, but I'm guessing you don't need all this data to be loaded at once. If that's the case I would suggest breaking it into chunks and then only loading the sections you need. If your trying to produce a terraria style game and have effectively the whole world loaded at once your going to have memory problems left, right and center.

    I've been working on minecraft clone written in webGL and JS for about 6 months ( haven't actually had much time to work on it, coding as a living puts you off coding in your evenings as well ). So I'm fairly familiar with the techniques needed for procedural generation of terrain and streaming level loading.

    In my case the world is split up into 16 x 16 x 256 block chunks. Chunks don't exist until they have been visited. When a chunk is visited for the first time I generate a 3D array holding a number representing the block type using 2d simplex noise. This is then passed to a separate system where a mesh is generated for the chunk ( so I only have to render 1 object per chunk ). When I leave the chunk the mesh is thrown away and I pack up the block data into a flat array. That flat array is 66kb and if needed could be compressed quite heavily.

    While it would be complex most of this could be done in construct for a 2D game. Instead of meshes you could use tilemap instances. With the addition of a small bit of JS you could hold a much smaller representation of your chunk data in a Uint8TypedArray or similar. Additionally you could store your world in a simple binary format that would be a lot smaller than JSON.

Jump to:
Active Users
There are 1 visitors browsing this topic (0 users and 1 guests)