Better HTML integration with Construct's new HTML layers

We've previously done some Feature Focus blogs where we highlight some of the best features of Construct. This time around we're taking a slightly different approach, talking about HTML layers which are new in r379, but also going in to detail about the design and implementation of the feature. HTML layers allow you to interleave HTML and canvas content, including allowing Sprite objects to appear on top of HTML elements! We believe we've come up with an innovative approach that uniquely combines web platform features to make this possible, while making careful decisions to avoid breaking backwards compatibility. This blog covers the problem and how we solved it.

The previous limitation

Construct has two kinds of objects: firstly canvas objects, covering most normal kinds of objects like Sprite, Tiled Background, Tilemap and Particles. These are rendered in to an HTML <canvas> element. The second kind are HTML objects, which are objects represented by an HTML element, such as Button, Text Input, iframe, and the general-purpose HTML Element object.

Before the introduction of HTML layers, HTML objects could only display on top of canvas objects. This made it impossible to achieve things like layering a Sprite on top of a Text Input object. This was because the canvas itself is one large HTML element, and all other HTML elements were displayed on top of it.

This was quite a significant limitation. A great capability of Construct is being able to use HTML and CSS to design UIs, but this limitation meant it was difficult to combine HTML and canvas content, such as by showing a particle effect on top of a dialog designed with HTML. The image below illustrates how the Z ordering worked, with all HTML elements layered on top of the canvas.

HTML layers

HTML layers solve this problem, allowing a mix of canvas and HTML content! The image below shows an iframe element showing the Construct homepage layered in between the background and foreground layers in the Cave Bridge example - only possible thanks to using the HTML layers feature.

How does this work? First of all, you can now opt-in a layer to be an HTML layer. This means HTML elements can appear on top of that layer, and underneath other layers above it. Here's the Layers Bar in the Cave Bridge example, with the background layer made an HTML layer, and so showing with a special icon. Now HTML elements on that layer appear on top of the background layer, but beneath the foreground layer, producing the result we saw previously.

When you do this, Construct creates two canvas elements: the usual one at the bottom of the stack, but also an additional one for the content above the HTML layer. It's cleared to transparent, but can have other content drawn on to it. Then HTML content can be inserted either in between the two canvas elements, or on top of the top canvas element again. The image below illustrates how this works, with the octopus character able to render on top of the text input HTML element, but also underneath the button.

You can go further, marking additional layers as HTML layers, which in turn creates additional canvas elements and thereby creates additional opportunities to layer HTML content in between canvas content. You can even add and remove HTML layers dynamically at runtime.

Designing the feature

While we were designing this feature, there are two significant considerations we had to think a lot about: performance and backwards compatibility.

An obvious place to start would be to say "why not make every layer an HTML layer?" Unfortunately this would be too inefficient: every extra canvas allocates at least one extra surface, and requires at least one copy to display. As a normal layer (not rendering to its own texture) in Construct is essentially free, lots of people have dozens of layers, and this type of approach would impose an enormous performance overhead where previously there was none. Therefore, HTML layers must be opt-in - hence the setting to enable it for specific layers.

Backwards compatibility was also a very tricky point. Prior to this feature, Construct allowed HTML objects on any layer. They would appear on top of everything else, but they would take their size and position from the layer they belonged to - in other words depending on the scroll, scale and angle of the layer they were actually placed on. This meant we could not automatically move these objects to a different layer, otherwise it would break existing projects. This also ruled out introducing a special kind of HTML layer in Construct that could only have HTML objects on it and changing normal layers to only have canvas objects: while that would be a nice design if we were starting from scratch, imposing that on existing projects would be very disruptive. Another consideration was existing projects have no HTML layers specified, and they must carry on working the same as they did before.

The solution to the backwards compatibility conundrum was that a normal layer can be marked as also acting as an HTML layer. This would allow HTML objects on that layer to appear above canvas content on the same layer, but beneath canvas content on upper layers. Then in addition to that, there is an implicit HTML layer at the top of all layers. HTML objects not on a HTML layer appear on the next HTML layer above them in Z order. Altogether this means existing projects that specify no HTML layers continue working as they did before, with all HTML objects appearing on top of everything, but still taking their size and position from the layer they belong to.

This shows how a new feature in widely-used software requires much more careful decision-making. It would not be the design we'd choose if starting from scratch, but the combination of the implicit top HTML layer and allowing opting-in to additional HTML layers further down combines minimal performance overhead, full backwards compatibility, and the flexibility to use HTML content at more places in the Z order.

How it is implemented

Construct has an extremely high-performance WebGL renderer, and we also recently introduced an even more advanced WebGPU renderer. Normally it is only possible to render to a single canvas with a single WebGL or WebGPU context. Creating an additional WebGL or WebGPU renderer would be extremely complicated and have a very high performance and memory overhead, possibly involving re-loading all the textures and resources the game needs for every canvas element. That wouldn't be a very good solution, so we came up with something better.

Instead the bottom canvas remains the canvas for which the WebGL/WebGPU renderer is created. All additional canvas elements use ImageBitmapRenderingContext. This allows efficiently displaying a given ImageBitmap. These canvas elements are rendered first: the engine renders the layers associated with them to a transparent surface, and then copies the surface to an ImageBitmap for display in one of the upper canvas elements. It continues this process to render each HTML layer. When it reaches the bottom canvas, it then renders the remaining content normally and leaves the content in the canvas for display. This essentially allows rendering the same WebGL/WebGPU context to multiple canvas elements by copying the main canvas contents to an ImageBitmapRenderingContext at certain points during rendering.

That's the principle, anyway - as with most features in a complex piece of software like Construct, there are lots of various complications and details to take in to account! Here are a few of the other things we had to take in to account:

• By default, the engine runs in a worker with OffscreenCanvas. In standard DOM mode, or if OffscreenCanvas is not supported, it falls back to drawing to a 2D context.
• We ran in to a Chrome bug with ImageBitmapRenderingContext (it would stop rendering when resized) which we had to work around. Fortunately the issue was fixed relatively quickly.
• Unfortunately copying to either a ImageBitmapRenderingContext or 2D context is very slow in Firefox - see issue 1163426 and issue 1864882 which are both still unresolved. Firefox also has a display glitch with HTML layers. Fortunately the latest versions of Chrome and Safari both work correctly and efficiently.
• Construct's own layers system already involves a sophisticated compositor with support for things like sub-layers, low-resolution rendering, background-blending effects and so on. This whole composition process essentially needs to be integrated with HTML layer compositing.
• Some features like background-blending effects don't work across HTML layers, due to the way the compositing process works. This kind of thing is a fundamental limitation and so needs to be documented.
• WebGL in particular has complicating limitations with how the backbuffer can be used, and this sometimes means Construct has to perform an additional copy of the surface. However WebGPU is more powerful and allows us to perform composition more efficiently with HTML layers.
• Dynamically adding or removing HTML layers at runtime, in particular when rendering from a worker with OffscreenCanvas, requires some tricky co-ordination between precisely when HTML elements are added or removed and when the rendering of canvases is done. This helps avoid things flickering when they change, but we weren't able to make it completely bulletproof, as there doesn't seem to be a reliable way to co-ordinate canvas rendering with the browser's own rendering.

And as ever we went through a series of bugs in the Construct engine too! It's one of those features that is conceptually relatively simple, but the nuts and bolts of it get pretty complicated and take some work to get right.

While implementing this we also did some extra work to improve the integration of HTML elements with Construct. This included things like ensuring HTML objects Z order correctly relative to each other, and making them clip correctly at the edge of the viewport when scrolling.

Conclusion

Using HTML and CSS is a superpower of browser-based game engines like Construct. Some top commercial games use HTML and CSS for things like their menus and user interface because it is still quicker and easier than doing it in-engine, even with going to the trouble of embedding an entire browser engine to power it. Browser-based engines on the other hand get the decades of development work to make HTML and CSS powerful and flexible layout toolkits built in to the platform the engine uses.

Construct is, to our knowledge, the first game engine to support interleaving both HTML and in-engine content through the use of HTML layers. This unique capability allows much deeper integration of HTML content in to game projects, allowing the best of both worlds: using custom HTML and CSS content for things like menus and UI, and even just things like the humble text input (which sounds simple but is very difficult to implement well in-engine), combined with the flexibility to layer it with in-engine content like particle effects, tilemaps, sprites, 9-patches and other decorative elements. We've also shown a window in to the design process for a significant new feature, mitigating performance and backwards-compatibility concerns, and dealing with various bugs, limitations and complications along the way. We're glad to have had a very positive response to this feature so it looks like we got it about right! That's never a reason to stop though, and doubtless there will be more future improvements to both this feature and other aspects of using HTML and CSS in Construct.

We've always believed the web is the gaming platform of the future, and we think features like this show it works out great! Join us and get started with Construct today.

Past feature blogs

Construct is absolutely stuffed with impressive features that are easy to use but allow great scope for creativity. You can learn more about some of them in our past blogs below.

• More efficient and more advanced effects with sub-layers
• Feature Focus: The Layout View
• Feature Focus: Platform movement
• Feature Focus: Display scaling
• Feature Focus: JavaScript coding
• Construct now supports TypeScript
• Feature Focus: Audio Effects