Custom 3D renderer instead of C3's own renderer

This is a followup to this forum thread:

construct.net/en/forum/construct-3/general-discussion-7/proposal-community-developed-182378

In there, Ashley said the following:

Are you sure that there are not other approaches that could make an integrated 3D engine work, or new APIs that would smooth integration? It seems to me that it ought to be possible to basically import existing Construct objects in to a 3D engine

I'm down to humor this because I'm trying to do this exact thing nowadays.

For the past few months, at Dedra, we've been working on two 3D games made in C3 with the purpose of pushing the engine to its limits and see how far we can go. While so far it's gone well, we've already ran into multiple roadblocks that have made me question the legitimacy of this plan.

So, I've started messing around with OGL initially and now with ThreeJS within C3. (OGL was chosen because it was very lightweight, but after working with it a lot, I figured I would need to build a large layer on top of it to get the results I wanted, which would make my fork just a poor version of ThreeJS)

During my tests, I was doing the whole thing in JS and loading models from the asset manager to run benchmarks, but the plan has always been to try and create a 1:1 mapping of C3->ThreeJS since I'm pretty sure most of it can be done automatically.

I'm glad that you seem to agree that this is a decent plan so let me list a few features that are currently needed to do make this possible:

1- Getting texture data

In C3, there is no legitimate way to get the texture data from anything without using undocumented features. I'm not just talking about sprites here. I would like to use:

• Sprites
• Tiled Background
• 9 Patch
• Text. This one is especially important. C3's Text BBCode system is way too useful to give up on, the alternative would require loading fonts manually, and doing text rendering using some other solution. I suppose I could use the IRendererText interface, draw to an invisible texture and show that, but that's extra work to recreate something that is already there.
• Particles. NB: This one is far less important as a particle system would probably need to be rewritten in 3D and could pull texture from sprite. Also, I implemented Effekseer in C3 a few months ago, but that particle engine was designed for ThreeJS originally so most likely I'd use that for complex particle systems

This feature would also be very useful for my Empty Shell addon. It currently uses an undocumented feature to get the texture from a sprite, but if this feature was done, I could make it work with any of these objects.

2- Getting linked instance objects

A few objects in C3 get their texture data from other places. This is the case for 3D Object which is the most important one by far.

Particle also does that but as stated this is less important.

Having that system be exposed in the new SDK would also allow me to port my Empty Shell addon, since it relies on a similar concept for texture linking.

Intermission

These two are by far the most important features because otherwise none of what you're proposing is possible without hacks, without jumping through massive hoops, or without just ignoring C3's editor entirely and storing everything as separate files.

3- Mesh data

I think we've already made a very solid point for this in this feature request.

However, this is also very important for a C3->ThreeJS system because otherwise there is no way to port anything that uses mesh. For this we need all the data, so world position, but also local positions and UV data. Without all of it, it's not possible to do it.

A big reason why this is very important is that mesh is the only way to do 3D rotations and skews in C3. Doing this in full Three requires storing a lot of data as numbers in instance variables which becomes cumbersome when meshes do the same thing but better.

4- Extra notes for full support

A few other far less important things required for this would be:

- Support for Video in the scripting runtime (URL, current position etc)

- Tilemap texture. This one would be far more situational.

- DrawingCanvas texture. Unless I'm mistaken, it's not possible to get the canvas texture data from DrawingCanvas.

- Spritefont. I personally don't use Spritefont, but it would help

Conclusion

This would be quite a lot of work to do, and a lot of it would essentially be syncing the full state of C3 objects to 3D.

I am personally down to explore this possibility, and if it works out well, I am planning on open sourcing a lot of my work so other people can give it a try as well.

However, right now it's just not possible to do without reaching beyond what is allowed.

Also, if I have to go the length of implementing a 3rd party 3D renderer and do all of this syncing work, it would only be fair if I got to use this engine to the full extent. However, right now, C3 lacks a lot of UX features that would also need to be worked around to have a decent workflow. Something as simple as placing 3D shapes with a 3D rotation is impossible and yet would be extremely useful in a proper 3D renderer. Do I need to store rotations as instance vars and keep hitting preview to see if I rotated my object correctly? I'm already doing this, but it's far from ideal.

You have to realise that following this route means gradually moving the problem away from the runtime, and straigth into the editor, and given how little work has gone into exposing more editor SDK I frankly doubt we'll be allowed to create custom views any time soon to implement a 3D editor ourselves, let alone get all of the data I outlined above directly from the editor SDK.

Scratch that! Let's say I write a brand new 3D editor completely separate from C3 that gets all its data directly from the project folder. I can't live reload anything in C3 (besides scripts in the scripts folder) so to preview my changes, I need to restart the C3 editor and THEN the preview.

(all of this assumes it's a sensible idea to write my own 3D editor and use a 3rd party 3D renderer. That sounds far fetched, but I can think of a few theoretical good reasons to actually do that)

I'll follow down the rabbit hole...

construct.net/en/make-games/manuals/addon-sdk/reference/graphics-interfaces/iwebglrenderer

What if there was a way to override all of these types of calls, including what is done in the C3 renderer.

e.g. if C3 called Quad3D2(tlx, tly, tlz, trx, try_, trz, brx, bry, brz, blx, bly, blz, texQuad) it could be directed to a wrapper around a three.js call to draw a Quad3D2.

This would be slower compared to what three.js render could do, but all of the normal C3 behavior might be available.

Do it both in the editor and runtime, basically replacing C3's renderer, but use all the rest of C3 functionality.

Translating a 3D model to C3, would probably just be represented still by a bounding box.

Or is this the _worst_ of both worlds.

------------

On a different topic in terms of textures, I am imagining three.js or babylon.js, etc. will be in a separate context, so we'll either need to ignore C3 textures and focus only on three.js textures or transfer them over outside the context and upload them to GPU again for the different renderer.

As Skymen mentions, a big challenge does seem to be getting larger access to the C3 editor, to make a 3D editor to create levels during editing. Will there be a large increase in the editor SDK API surface for us to work with?

For placeholders in editor, perhaps we could generate textures on the fly for things like C3 cubes, etc. which are 2D snapshots of the models rendered from each side of the model, fully lit. At least this way there will be _some_ representation of the models in the editor besides a bunch of black and white dice. For this, it would then be useful to pass the texture data in the other direction too.

With vanilla C3 render for the editor as placeholders we might end something like the below, trying envision how useful it might be.

Another interesting request that may be relevant to this discussion:

github.com/Scirra/Construct-feature-requests/issues/237

Request to allow other 3D renderer to render in C3 context, for speed and finer grained integration (similar to what is done on Spine today (though the desire is for Spine to be fully SDK V2 compliant).

For example, similar to these:

For three.js - resetState()

threejs.org/docs

For babylon.js - wipeCaches()

doc.babylonjs.com/typedoc/classes/BABYLON.Engine

It would be difficult to render into the C3 FB directly, but the other engines could render into a texture in the same context that is used by a C3 drawing object would be efficient and fast and can be integrated at a finer grained level than a html layer (this is what I ended up doing for current Spine and it works well.)

Rendering out of context and sharing with C3 (e.g. via texture download and upload from canvas is slow, I tested this with Spine previously and it was not useable in terms of performance.)