SDK method to update a Sprite animation frame with new image?

  • Is there an SDK method to update a Sprite animation frame with a new image (from a blob, data URI, JS array, etc.)

    I am looking for alternatives/workarounds to a current stuttering issue seen when using the Sprite Action Load Image from URL.

  • This would inevitably involve a WebGL texture upload, which has a high performance cost. Is there a way to render the image source directly?

  • Ashley Thanks for the reply (and thanks for the comments in the bug report, I understand, that makes sense.)

    I do agree that rendering directly to the C3 canvas would be better and that will be something to explore, however, the render is done through the Spine-TS runtime script and it looks like a lot to dive into to figure out (10K lines). It's already set up to render to a canvas, however, also integrating in the render with other sprite objects could be a challenge? Though I think it would be possible if created a Drawing plugin and did the work to integrate the Spine render into the Drawing Plugin Draw function? Something to explore in the future.

    I was looking for a short cut to use the existing Spine-TS. I have found a way that works and _seems_ to perform pretty well. I definitely tried to use only C3 SDK, but I think I used a little more. In this method, I created a drawing plugin which replaces a sub texture of the drawing plugin Sprite Sheet with the canvas texture on each Draw() call. There are limitations: it only replaces the top MIP level, so if the Sprite is scaled down, the old image starts to blend in. So, for the game, I will force screen resolution and keep Sprite size relatively the same so the other MIP levels do not kick in (this will likely be a desktop nw.js / Electron game, so I will have some control over which chromium engine the game is shipped with and control over full screen vs windowed.) I know this is not bullet proof, but I think I can work around the issues. If you have suggestions on how to make the below use more C3 SDK functions, that would be very appreciated!

    In general for Spine animations, I will just use 1-3 characters animated this way, so 'only' 1-3 texture updates per frame. The rest will be simpler and use traditional sprite frame animation.

     Draw(renderer)
     {
     var gl = renderer._gl
     const imageInfo = this._objectClass.GetImageInfo();
     const texture = imageInfo.GetTexture();
     if (!texture) return; // dynamic texture load which hasn't completed yet; can't draw anything
    
     const wi = this.GetWorldInfo();
     const quad = wi.GetBoundingQuad();
     const rcTex = imageInfo.GetTexRect();
     var myCanvas = document.getElementById(this._elementId)
    
     gl.bindTexture(gl.TEXTURE_2D, texture._texture);
     // webgl2: gl.texSubImage2D(gl.TEXTURE_2D, 0, (rcTex._left * texture.GetWidth() - 0.5).toFixed(0), (rcTex._top * texture.GetHeight() - 0.5).toFixed(0), myCanvas.width, myCanvas.height, gl.RGBA, gl.UNSIGNED_BYTE, myCanvas);
     gl.texSubImage2D(gl.TEXTURE_2D, 0, (rcTex._left * texture.GetWidth()).toFixed(0), (rcTex._top * texture.GetHeight()).toFixed(0), gl.RGBA, gl.UNSIGNED_BYTE, myCanvas);
    
     renderer.SetTexture(texture)
    
     if (this._runtime.IsPixelRoundingEnabled())
     {
     const ox = Math.round(wi.GetX()) - wi.GetX();
     const oy = Math.round(wi.GetY()) - wi.GetY();
     tempQuad.copy(quad);
     tempQuad.offset(ox, oy);
     renderer.Quad3(tempQuad, rcTex);
     }
     else
     {
     renderer.Quad3(quad, rcTex);
     }
     }
    
  • I think I found at least one improvement to use more of the C3 SDK, I can use the below to get the rcTex coordinates:

    getLeft()
    getTop()
    getRight()
    getBottom()
    
  • Try Construct 3

    Develop games in your browser. Powerful, performant & highly capable.

    Try Now Construct 3 users don't see these ads
  • I think the best approach would be to use the same approach as the Text plugin, which similarly renders via a canvas. The steps are roughly:

    1. Draw content to a 2D canvas (text for the Text plugin, or in your case a Spine rendering)
    2. Create a new empty texture managed by the plugin
    3. Whenever the canvas size changes, destroy the texture and create a new one to match the new size
    4. Whenever the canvas content changes, upload it to the texture
    5. Finally draw the texture normally with a quad

    The key points are the texture is owned and managed by the plugin itself, so it's not trying to overwrite existing content used by other parts of the engine; it uses a direct WebGL texture upload so there's no unnecessary compress/decompress cycle; and there is support in the engine for this approach already (as Text uses it already). I also must point out that you cannot safely mix raw WebGL calls and Construct renderer calls, as Construct's renderer queues WebGL commands using a batching system (meaning WebGL calls will happen in an unexpected order) and heavily relies on caching and ignoring redundant changes (and changing WebGL state outside of the renderer with calls like bindTexture will corrupt the internal state of the renderer and probably break the game). This is why there is no documented/supported way to get the WebGL context directly - it's for good reason! I've added to my todo list to document the necessary renderer calls to support that approach.

  • I just updated the WebGLRenderer docs to cover new methods CreateDynamicTexture(), UpdateTexture() and DeleteTexture(). These should be enough to implement the approach I outlined above. If not let me know and I can look in to covering more of the API surface. These methods should already be supported in the runtime, but note if you want to use them in the editor they won't be supported until the next release.

  • Ashley Thanks for the great response, details of the engine and suggested method for implementation. I will take a look at implementing something similar to the text method and thanks for the additional documentation that you just provided. I appreciate the comment on the resize and creating a new texture based on that - very helpful in making this all more robust.

  • Thanks Ashley this worked well, it seems to perform reasonably well (100 instances at 320x240 still ran at 60 fps, I think I will be more in the 1-10 range.) I am also glad to be using only the C3 SDK and no 'raw' webgl which could get out of sync with the engine gl batching and optimizations. Thanks for the help and the docs. I will post the addon (and Spine example which uses it) after some more testing.

    Here's my updated Draw() function.

     Draw(renderer)
     {
     if (this._elementId == "") return; // elementID not set, can't draw the element
    
     var myCanvas = document.getElementById(this._elementId)
     // Create texture if it does not exist (could this be done in constructor or setElementId action)
     if (this._elementTexture === null)
     {
     this._elementTexture = renderer.CreateDynamicTexture(myCanvas.width,myCanvas.height,{mipMap:false})
     // console.log(this._elementTexture)
     }
    
     const wi = this.GetWorldInfo();
     const quad = wi.GetBoundingQuad();
     const rcTex = new C3.Rect(0,0,1,1) // Possible to get from this._texture instead? Not needed, not spritesheeted?
    
     renderer.UpdateTexture(myCanvas, this._elementTexture, {})
    
     renderer.SetTexture(this._elementTexture)
    
     if (this._runtime.IsPixelRoundingEnabled())
     {
     const ox = Math.round(wi.GetX()) - wi.GetX();
     const oy = Math.round(wi.GetY()) - wi.GetY();
     tempQuad.copy(quad);
     tempQuad.offset(ox, oy);
     renderer.Quad3(tempQuad, rcTex);
     }
     else
     {
     renderer.Quad3(quad, rcTex);
     }
     }
    
  • Ashley This is working very well on Desktop setup and Igor has actually created a nice Spine Addon which uses this new method. I have found it to work well on Desktop.

    However, on mobile (specifically iOS/iPhone), the UpdateTexture() takes a long time, specifically the gl.texSubImage2D() call. After some research, this is not surprising due to the PowerVR GPU tiled rendering method and also the GPU's BGRA pixel format.

    I am wondering how this is handled for the case for the Text plugin? Is it just reliant on the developer to not update Text plugins that often on a mobile system, or do you have another optimization for cases like a timer that might update a Text plugin every frame?

  • Yeah, texture uploads are slow. Text does not do anything specific to handle this, but often text objects do not change every frame, so they just render like sprites and largely get away with it. Large text objects that change text every tick have been reported to be slow before, but there's not much to be done about it, so we just advise to avoid that case if possible, or use SpriteFonts which render directly without any texture uploads.

    Since a Spine animation presumably can be expected to always change contents every frame, then it will hit you harder. There's very little that can be done to optimise it, other than not using texture uploads at all. That is why my very first suggestion was to find a way to render it directly, so this problem would be bypassed. The main advantage of using texture uploads is that it's probably much quicker to get it working, since you mentioned it looked difficult to set up the library to render directly... so to completely solve this, you're probably just going to have to bite the bullet and do all that work.

  • Ashley Thanks for the comments, so no magic bullet, I'm not surprised after my research, but I wanted to check, in case I was missing something (I also want to use C3 SDK to keep plugins more easier to maintain.)

    In parallel, I have already bitten that bullet. However, I want to check with you on this, because it's using mesh deforms and a lot of webgl effects and it's complicated (10k lines of code as I mentioned, using meshes, custom shaders, etc.) - so Spine-ts is rendering directly to the canvas using webgl. However, you cautioned against this due to C3 controlled batching, etc.

    Before I start asking questions - is this what you meant by render directly?

    (Current status - I have the Spine render to C3 canvas working, but I'm debugging save/restore state, calling EndBatch(), etc. to make it co-exist with C3, I may render to texture instead of direct to C3 Canvas FB and hopefully the rendered texture will be usable w/o a big perf hit for render to C3 canvas using C3 SDK to render the Quad to C3 canvas, Set up a new texture, etc.)

  • Well, I guess you can render using raw WebGL calls, but it's not ideal. Construct's renderer has its own batching and state tracking system, so if you make any changes at all to the WebGL state, you will need to save the old state, do any rendering with custom WebGL calls, then restore the old state again. This is necessary to ensure the WebGL state is consistent with what Construct's engine expects it to be (if you change state and leave it, you're pulling the rug out under the engine and will break stuff). Even then, changing loads of WebGL state is slow. So if you go down this route, your next performance bottleneck will be rendering lots of animations simultaneously, as it hammers saving and restoring loads of WebGL state repeatedly during a frame. Additionally if you make raw WebGL calls you bypass the rest of the renderer, which makes it impossible to use things like custom WebGL shader effects, essentially removing the ability to use that powerful (and very useful artistically) feature.

    The ideal solution would be to render everything in a plugin Draw() call using IWebGLRenderer. This fully integrates with Construct supporting all features like effects, and works with the existing renderer code and batching system, so you won't need to do things like save and restore masses of WebGL state every draw, and takes advantage of its sophisticated batching system to ensure maximum performance. The renderer is a fairly high-level API and can do many of the things the standard canvas2d can, so it should be feasible - but you might have to change a lot of code, since it works differently.

  • I agree it's not ideal. I understand about the gl state save and restore (and the perf cost.) I'm using a library to do it (which allows for a large amount of gl state save (perf hit), but also has a method to specify state to save, to trim back for min required and for better perf.) I am not done yet, because somewhat as expected, some Spine renderer state or context or gl call is not allowing the C3 render to show up (and I don't expect help from Scirra with that.)

    github.com/stackgl/gl-state

    My thought is to render Spine to a texture buffer (in C3 context) created through the C3 SDK CreateDynamicTexture(), instead of direct to C3 FB. I will not be using the UpdateTexture() call, instead, I'll be rendering to the texture directly. I will be doing this all through a C3 object's Draw(renderer) call, so I can render to the texture, then call SetTexture() and then Quad4() to keep things synchronized. This way I can hopefully continue to use the full power and integration of C3 (effects on Quad, size, position, rotation actions, behaviors - which is the goal.)

    This actually worked really well with the ElementQuad addon I created, which did render to other canvas and then UpdateTexture() - it was just the issue of slow UpdateTexture() on mobile which made it not a good solution for mobile (and perhaps some desktop GPUs which use heavily tiled render (Intel perhaps?) I could add effects and behaviors to ElementQuad, it is a nice combination of C3 and another renderer (for non-mobile.)

    Using IWebGLRenderer would be great also, as you said - I think it would be a big code rewrite, but I'm going to dig through the Spine-TS code a little more and see if I can find an abstraction layer which matches (I also need to match up C3 texture management and Spine-TS texture management/atlas - which I think CreateDynamicTexture may help with, since it seems like that does not become involved with the general C3 spritesheeting algorithm.)

    It feels like we are close to getting something good working, so I'm going to keep working on it (Igor made a nice plugin version, much better than my original template, so it could be used devs easily.)

    Thanks for the comments and discussion.

Jump to:
Active Users
There are 1 visitors browsing this topic (0 users and 1 guests)