skymen's Forum Posts

  • Sorry for causing all this drama guys...

    This used to be a quite productive thread about 3D features in the engine, I really didn't think this would happen when I posted my first message.

    I think it's best if we move this brand new discussion to a new forum thread as I really don't wanna clog the interesting discussion that was happening up to that point.

    construct.net/en/forum/construct-3/general-discussion-7/editor-sdk-editor-tools-sdk-v3-187521

    Everyone can go back to discussing cool 3D features and anyone interested in talking about the SDK stuff please move to the new thread.

  • Hi, I opened this thread for any further discussion on whatever happened in this thread:

    construct.net/en/forum/construct-3/general-discussion-7/gdevelop-5-6-adds-real-time-187468

    Quick recap;

    Someone in the thread mentioned that it would be nice to have editor tools to allow 3rd party addon developers to extend the editor which in turn would allow creating powerful tools such as a 3D scene editor.

    For the record this is also a stance that I hold as a 3rd party addon tool. I have also spent a LOT of time in the past making my own 3D scene editors, working on 3D tech in Construct and I made a 3D game where one of the core principles was pushing the engine to its absolute limits when it comes to 3D.

    Ashley replied that he thinks making the editor UI extensible would represent a far bigger amount of work to do than just making the 3D view themselves, which I can totally understand. He went on to explain the several hidden challenges that would arise from making such a system, and how in comparison they've already done a good chunk of the work to get a 3D view working in the layout editor.

    When I was made aware of this thread I thought I should mention to Ashley that I had actually been experimenting with a 3rd party solution to make editor UI extensions. This is a thing I have tried to do for years and failed to achieve in any satisfying way. In the past I kept running into issues that would make the system either too unstable, too hard to implement or too painful to use for end users.

    Recently however I actually managed to break through all of that and I implemented my own system in a way that I though was very safe and very satisfying:

    github.com/skymen/editor-window-manager/blob/main/docs.md

    I even made 2 addons experimenting with this new system and found it to work really well. My plan was to then reach out to devs to get them to mess around with it, and then to reach out to Scirra to let them know about this addon (which I did on social media)

    Admittedly, Ashley never saw my tweets (or blueskies I suppose) because he appeared to be very surprised about it in that thread, and immediately entered panic mode and talking about implementing a sort of SDK V3 to prevent addon devs from doing what I had just done.

    SDK V2 was a very very painful time for pretty much every Construct user and Construct addon dev and it still very much is to this day. So throwing around threats of an SDK V3 so lightly unsurprisingly annoyed a bunch of people, as this is seen as Ashley actively undermining and disrespecting the time of addon devs such as myself who make tools (often for free) for the Construct engine, which a lot of users then rely on.

    What to do now

    Ok so now that we're all caught up, I have opened this thread so we can all discuss this issue in a civil way to try and avoid any further drama.

    Ashley

    Tagged:

  • Please: take a step back, look at the ways the industry has come up with over decades to ensure reliable software development, look at everything we already just went through with SDKv2 which was just re-learning those same lessons

    To be clear I'm more than happy to take a step back. If you're willing to work with me to make it as safe as possible while still having it work somewhat even if it includes only managing popups and not including any CSS inside the editor, I'm also happy to do that.

    I am reaching out to you with the express purpose of working with you to make it work sustainably, but when you react this way by immediately categorizing any experiment as an existential threat what you're doing is telling other addon devs that if they don't want to trigger a cataclysmic event they should hide their work from your eyes, which in turn will cause far more headaches for you.

    I'm doing this because I know the need exists and I think I can do it in a way that minimises harm and I can communicate transparently about it. You also understand why the need exists and you know how useful it could actually be to have some form of custom tooling for advanced users.

    To be honest, it is a bit terrifying to me that you would rather reach for extreme options that combats and destroys the work of dozens of devs who give so much time for your engine rather than first see how you can work with said devs and assume none of them have nefarious intents.

    It looks like you have over 2000 lines of code making similar kinds of changes.

    My code manages its own DOM element and has everything in its own bubble. Same for the CSS, it is scoped to only affect my elements. If you think it's still reaching too far, I'm happy to make any adjustments to make it reach less far. If having any amount of me messing with DOM is still too much, I can just open a popup window and handle custom editors exclusively in that popup window, that way it's in a completely different context and can't risk messing with your editor's DOM or CSS.

    Even better, you could add a special div in the editor meant for any kind of custom content that would need to contain any of these edits that you can accept leaving alone, without having to do any of the extra work of maintaining anything.

  • The addon does not run any code on startup that relies on anything C3 does if that makes you happier

    Just take a look at the addon's code and let me know if something in it would cause an actual issue instead of reaching for the nuclear option.

    What would you even do, add Mutation observers and nuke the editor if you detect any unauthorised HTML? That would actually cause the editor crashout you're talking about rather than anything I might do.

    The editor SDK is already in an SDK v2 encapsulated state in case you're unaware. The whole point of making my own layer is to decorrelate it completely with the editor, the system would actually keep running just fine even if the editor underneath stopped existing.

    Also, I tried to make it as foolproof as I can. If the things any of these addons need stop existing the way they do, the addons just do nothing instead of crashing.

    There is really nothing to worry about, and if you want I am more than happy to spend any amount of time explaining everything I did so you do know if anything might cause an issue so I can take steps to changing them right now.

    The whole purpose of this addon is to provide a single common interface for implementing editor UI rather than have each addon handle its own interface which would actually be super scary. That way there is only one thing to worry about rather than a dozen addons all managing their own thing.

    If this still does not satisfy you, I don't know what to tell you. I'm not planning on deprecating this system so if you'd rather cut your leg off than tell me not to step on your toes be my guest, I'm not the one dealing with the consequences.

  • For example in my view to do this properly, we'd need to expose to addons the entire editor UI library, such as tree controls, list views, data tables etc. so editor addons can make use of the same look-and-feel of the rest of the editor and not have to reinvent the wheel for things like a custom editor window with a tree view in it.

    RE: Editor tools

    I think this eventually deserves to be its own thread but Ashley I don't know if you are aware of this but I recently implemented my own editor UI API to fix this exact problem you're mentionning:

    construct.net/en/make-games/addons/1534/editor-window-manager

    This addon can't be added to projects but exposes an editor SDK for any other addon to create and manage new editor windows. It handles its own thing on top of C3 and of course I had to handle minor quirks but I was able to handle most things just fine.

    This manages an extra layer above C3's UI and doesnt properly integrate with the Undo Redo system but at least this is completely off of your shoulders.

    I think it's best to focus on making better APIs that allow us to do real things in the editor rather than expose the UI system if that proves to be too painful.

    The ONLY thing that would be nice I suppose would be if we could get a proper URL for when I pop out the window for security things as browsers otherwise block some common actions, but it's not too bad.

    I used it in this other addon that lets users make custom tools using JS scripts directly from within the project files, and with a default minimalist UI component library

    construct.net/en/make-games/addons/1535/editor-tools

    And I used it in this other experiment I made that allows people to make themes directly from within the editor.

    construct.net/en/make-games/addons/1533/caw-theme-creator

    Check these out and let me know if you're ok with it.

    I tried my best to integrate best I could with the editor and the theme but its hard to do without messing with engine internals. Exposing info about the currently loaded theme, or having classes for external editor tools would be nice but not necessarily needed.

  • Small update;

    After deploying the quick fix I made on a few games, it doesn't seem like it has solved the C2P issue at all despite the fact that there was a good chance it could.

    I'm still investigating other possible avenues.

  • Try Construct 3

    Develop games in your browser. Powerful, performant & highly capable.

    Try Now Construct 3 users don't see these ads

  • Do you have measurements you can share that prove this? I find this surprising

    Yes so do I. I am currently investigating it, but so far I have no conclusive evidence to say that it is beyond any doubt, however here is some data I was able to gather

    I was given a relatively short list of games that has bad Conversion To Play (C2P) and I added a few games that had no C2P issues at all.

    I gathered some data for all of these games that might take in part in loading like

    - number of fonts

    - number of assets

    - number of audio files

    - total size of assets

    - wether preload sounds was enabled

    - list of plugins, behaviors and effects (including built in ones)

    - wether worker mode was enabled or not

    C2P is measured by Poki as the number of users who clicked on the game (aka opened it on the website) vs the number of users who ended up triggering a gameplay start event (aka actually played the game).

    Now, a number of things can cause a user to not interact (long intro sequence, long loading times, crash on load, etc) but it is relatively common for games on Poki to have very light build sizes, and to load straight to gameplay so a gameplay start is quite often the first ever interaction the user has with the game (clicking, hitting a key on the keyboard etc) since it would trigger an action in game as long as the game has finished loading.

    In my short analysis so far, what I can tell is that while having preload sounds enabled does affect the C2P, it doesn't justify low C2P on its own (also, this one was fairly obvious) and the number of fonts and the number of audio files used don't seem to affect the C2P in general (most games with good C2P had more fonts than games with bad C2P).

    However, one point of data that tells a different story is the total asset size, and most specifically the number of assets, and the average size per asset. From the few games I have analysed it does seem to me like while games with bad C2P tend to have smaller build sizes (usually caused by the devs trying to optimise the games as much as possible to fix the C2P issues), they have a bigger number of assets and the average size per assets is much smaller than average.

    This does seem to vaguely track with pixel art titles made with Construct 3 since pixel art games do tend to have a very large amount of very small assets that then get packed into a large amount of very small spritesheets by Construct 3 which then causes this "large amount of small files" issue that we can see.

    I must say that this is not conclusive evidence still, and the dataset is still pretty limited (13 games) but I am actively working with Poki to find wether the issue can be fixed on their end or on the Construct end. I know very little about CDNs and how they function, and I know even less about what kind of setup Poki has internally, how they serve games, and why it is the way it is. The difference with a company like Scirra is that you guys only serve a single known product and can optimise for it, but Poki needs to serve thousands of games on a very wide range of devices, in a very wide range of countries with very varying amounts of connections, and all of it is data they don't control since they don't make the games themselves, and thus would need to serve them in a way that accomodates for as many games as possible.

    If it turns out that the issue would be fixed by changing some server config on their end but that change would in turn make a lot of other games load slower, that might not be a viable solution either.

  • if you think there is one easy way to make everything optimal for all projects, it's really much more complicated

    Don't get me wrong, I totally understand why the packer is the way it is, and why it needs to be designed in ways that take into account things that are far beyond the concerns of my projects.

    What I'm highlighting here is that I have noticed a tendency for Pixel art games made with Construct on Poki to load slower for players, to the point of significantly affecting the player bounce rates, and I have separately noticed a tendency for Pixel art games made with Construct to generate an excessive amount of spritesheets, sometimes far more than with a bigger project that has bigger assets.

    The solution I am proposing here is not a complete revamp of the spritesheeting algorithm, but it is instead to provide an optional secondary step that packs the currently generated sheets again following a similar logic to find ways to reduce the number of generated sheets the project exports with.

    The github repo I provided is on one hand a way for me to test wether the problem would actually be solved if it was implemented (I am currently running tests on Poki) but also to serve as a naïve proof of concept for how I see the implementation.

    In this case, TBGs would still be left alone, and large animations are still packed in smart ways since you already run an initial smart packing step. If the large animation happens to have been split over multiple sheets, then it's very likely it's already reached max sheet size anyway.

    Again, in the use case I am defending, memory usage is NOT the bottleneck. We are talking projects that (were the sheets packed into 2048x2048 sheets) would have less than 6 of them in total, so that's 96MB of ram, which is perfectly fine on any modern device. Web Games need to ship with build sizes below 10MB after compression and need to have initial load times below 5MB, so it is very unlikely to ship a game on the web platforms and have image memory usage be a bottleneck, and instead often the load times and compression efficiency become far more important.

    Now, I also completely understand that you wouldn't want to implement that kind of feature set, and instead I would find it very useful if you could give us a new option that exports a new separate file that gives some amount of information on every exported image (wether it can be packed, wether it should be lossy or lossless, what animation it's a part of) and I will be perfectly happy writing a custom algorithm that runs using that data and does the secondary packing manually.

    Right now I read the data.json file and I try my best to guess as many of these information as possible and it works fine so far as long as I keep exporting with a few known parameters.

    I will also let you know the result of my testing with Poki. If it turns out that the number of spritesheet is somehow totally decorrelated from the loading issues that we are facing, I'll stop looking into this as thoroughly although I would still like for that feature to be considered in the future.

  • I noticed there was some recent discussion about it on github so I'll move this there instead:

    github.com/Scirra/Construct-bugs/issues/8427

  • Hey Ashley

    Today, I was looking into the spritesheet system C3 uses, and I just realised that my pixel art game, despite having max spritesheet size set to 2048 still generates spritesheet sizes of at most 512 by 512 and often much smaller.

    In the end, the game exports with 71 spritesheets, but I ran some numbers and it turns out that had the game respected my setting and generated as many 2048 by 2048 sheets, all of the assets could fit into a single sheet.

    I was wondering; Is there any good reason why C3 doesn't pack everything into spritesheets as big as possible?

    Of course, I assume that the complexity of spritesheet making is specifically that testing out all combinations (especially now that sprites can be rotated) gets exponentially more complicated if you add more images to a single sheet, but then would it be possible to run that thing in 2 steps instead? Do whatever C3 does right now and create all 71 sheets, and then run it again a second time and generate far fewer sheets by merging them together and then merging the data. I would honestly even be fine if it gave me two 1024 by 1024 sheets or four 512 by 512, but SEVENTY ONE is a big excessive IMO.

  • I am not sure that listing ALL of the templates is a good idea. I would rather have it be opt in so it doesn't clutter the project bar.

    Also, wether to just copy a template or make it a replica I would assume would be off by default if pulling the default instance, and on by default if pulling from template list.

    I think having the templates be children of the object type in the bar instead of having it in its own list could be an interesting idea, but I'm not sure it's a good idea for all use cases. Having a separate folder is probably a better idea, especially if it's opt in.

  • > B) Make it so new instances pulled from the project bar automatically become replicas of the "default" template if it exists.

    This design makes it a bit of a hidden feature, and hard-codes an English string which isn't great for non-English users. What about a property to choose the default template name used for new instances? This seems like a good solution as templates are the feature intended to be used for determining initial properties.

    To be honest, I think that if you were to do something like this, you might as well make it so you can save any template in a "template" folder in the project bar so it would almost work like prefabs and you can drag and drop them in any layout.

    I don't see why this feature should be limited to just one "default" template per instance or even why there should be a way to decide what the default template name is.

    Also, I feel like it really wouldn't solve the issue being discussed here; the main problem is when dragging an instance from the project bar in a large project. Having to retrofit "default" templates for every instance sounds like a lot of work for very little benefit, especially when this could be solved with a simple right click menu option. Right Click > Make Default Instance.

    • Post link icon

    So basically, c3 forces bad types of programing in order to avoid bad types of programing while solving complex/advanced problems, and the solutions themselves are difficult to maintain or scale. You avoid one issue, but create a different issue.

    I thought a lot about design patterns within C3, and while I agree with all of what you said, I think there is one major takeaway you're not seeing; traditional programming patterns are never gonna be a good fit for C3 even if it had some of the features you mentionned because I think at the end of the day C3 has bigger flaws than just support for arbitrary design paradigms.

    C3 works far better with something like composition where a single family is an isolated feature, and you can compose an object by adding it to families. In this case, family inheritance is still useful because there will still be cases where you'd want some amount of inheritance within general systems.

    IMO even with all the features you want, it doesn't answer C3's 2 major flaws that it needs to address before any of this can even be tackled.

    - C3 is horrendous at dealing with data. When you have to handle a JSON, I feel like while C3 has some amount of solutions in place to make it possible to do, in general data management feels like an afterthought that was handled by an external addon rather than by a system implemented directly within the engine. This creates a large bottleneck of sorts whenever there is a need to treat and transfer large amounts of data because it needs to be serialised to JSON first, or it needs to be transfered through archaic picking systems that make the code way too hard to read.

    - C3 lacks an abstraction system. There is no way to refer to instances without knowing their type and without picking them through the SOL. This means cross family picking is hard, storing instance references is hard, and it's one of the main reasons composition systems fail so easily. As soon as you need 2 families to synergise when they are both being used on a given type, it becomes problematic.

    IMO if C3 had a way to store SOL as a variable and do more complex SOL manipulations, it would make C3 far more powerful. Also, if C3 had a way to make cross type families for types that share actions like set position and set size, it would also allow for very useful abstraction to be done.

    What I've ended up doing fairly often is relying on JS for complex data management and that kind of inheritance when necessary and using instances as just the visual aspect if it can't be done nicely with events. Still I think it's not a great solution and I wish C3 had systems in place for that.

    • Post link icon

    until we go through the engine and use private fields everywhere, which we've been planning to do for some time

    Since you've mentionned this a few times over the past few years Ashley I went digging into the code to see what this would do concretely.

    Private fields would absolutely make it much harder to do some of the bigger hacks that I've done in the past. The bigger ones being the following:

    Layer Manager: construct.net/en/make-games/addons/1053/now-c3-layer-manager

    (Dynamic layers before dynamic layers were added)

    Layer Magic: construct.net/en/make-games/addons/886/layer-magic

    (Rendering hacks to do some cool multi pass rendering on layers. I still think this would be a sick thing to have, but I stopped working on it due to huge bugs the hack caused)

    Limit FPS: construct.net/en/make-games/addons/1188/limit-fps

    (Wrapper around the C3 ticking method to force some kind of arbitrary limit)

    Experimental XXX Fix:

    construct.net/en/make-games/addons/1128/experimental-touch-fix

    construct.net/en/make-games/addons/1094/experimental-9-patch-fix

    construct.net/en/make-games/addons/906/experimental-text-fix

    (Replaces parts of the runtime classes to fix annoying issues with these plugins)

    What all of these have in common is that the way these operate has been to do something close to the following: 

    let oldClass = C3.Path.To.Class
C3.Path.To.Class = class myClass extends oldClass {}

    This way of doing it has a big advantage because it's non destructive, and more importantly, it is way less prone to breakdowns when changes to the parent class are made. It also allows me to be very precise with the hacks and reuse as much of the original code as I can. With private fields, this becomes completely impossible, and the only way to do something similar would be to replace the entire class and rewrite everything, which would make it painful enough to warrant doing something else instead.

    However, I dug around the code of C3 a fair amount in the past few years, and I don't think that too many fields can be made private. The engine internal classes communicate with undocumented public fields of other classes all the time, and usually these are the fields we'd wanna mess with anyway. From what I can tell, using private fields all over the engine in an efficient way would require rewriting a significant chunk of the engine's code and I'm not even sure how that would look like.

    Also, since private fields would most likely make replacing classes a pain in the ass, a lot of these hacks would probably then just revert back to hooking to random parts of the engine, and trying to inject code wherever possible which would in turn be a lot messier. So really, to me the idea sounds like a lot of work for not a lot of return. At the very least, IMO it doesn't do much that the idea of SDK v2 doesn't already do, and IMO again, adding better APIs and more features to the SDK is really the only sane way to fix these problems.

    Another reason why I don't especially like that idea is that being able to mess with engine internals is a great way to prototype feature requests before I make them.

    I have mostly halted addon dev since SDK v2 was announced to give it some time to mature so when I get back to making addons, I can try to use v2 exclusively and start porting my old addons. However, I still use Construct and I still mess around with it's features, and recently I made a game that made heavy use of Dynamic layers and I found a bunch of tiny additions that could be made to the feature that would make for nice feature requests, but I first wanted to experiment with them.

    For example, here is a snippet I use in that game that adds a layer effect to a dynamic layer 

    	let effectData = [["skymen_DropShadow","DropShadow",[true,	[0,0,0],0.75,4,60]]]
	let layer = c3_runtimeInterface._localRuntime.GetMainRunningLayout().GetLayer(runtime.globalVars.layer_ui_gun)
	let renderer = c3_runtimeInterface._localRuntime.GetRenderer()
	layer._effectList = C3.New(C3.EffectList, layer, effectData);
	for (const effectType of layer._effectList.GetAllEffectTypes())
		effectType._InitRenderer(renderer);
	layer._effectList._InitRenderer(renderer);
	layer._effectList.UpdateActiveEffects();
	layer._needsRebuildEffectChainSteps = true;
	layer.GetEffectChain();

    This snippet is very hacky and a real implementation would require a lot more work, but in the meantime this allows me to test it in a real game and see how well it holds up so I can then come with a well thought out feature request that can prove that the idea has legs. With private fields everywhere, things like this are 100% impossible and I don't think that's a good thing, at least for me.

    Anyway, I hope that by the time SDK v1 is fully sunsetted and private fields are implemented, SDK v2 will be mature enough and will provide a good enough API to make these kinds of hacks unncessary.

    The point I'm trying to make with this post is that the bigger hacks are usually born out of necessity and rarely out of laziness to achieve a similar goal with legal means.

    Addons like Empty Shell, Limit FPS or Layer Manager are born out of a real need to do something that C3 doesn't do and while I understand that V2 aims to make it harder for these addons to be shared with people that don't understand the risks, I would still like being able to mess around with things at my own discretion, even if that means I stop publishing my code.

  • FWIW the editor has a fixed limit on how many spritesheets it will render at once, specifically to cap the memory usage. For example if you have 1000 spritesheets, it will only render/compress something like 10-20

    Do you think it's possible to have an experimental setting (even something hidden like a URL query param) to change that limit? I'd be down to give it a try and report if changing the limit helps or not.