skymen's Forum Posts

  • Well, up until you mentioned it, I was not aware that using the basic JS DOM manipulation API was considered undocumented and unsupported by the editor SDK, and I genuinely thought this addon was supposed to be fair game since I really tried to make everything as self contained as I could. I wouldn't have made it open source and so thoroughly documented had I thought it actually posed the risk you are describing.

    As a show of good faith I have privated and removed all 3 addons and their repositories. I will release them again once the new and supported equivalent feature exists and I can make them work in a safer way.

    Alternatively I can also remove all of the code that handles the new dialog UI layer (which is most of the code) and only keep the code that opens a popup if you think that is safer in the meantime (this would only require changing the window manager addon as the other 2 addons don't handle any DOM outside of their own window).

    Frankly the editor SDK is already pretty much unused by most addons I've seen and completely unused by all addons I've made so if a potential "SDK V3" keeps it backwards compatible then by all means feel free to plan for this move as it wouldn't affect me in any way at all.

    I've looked into Shadow Realms, and I can't find any planned release date for them. Are you planning sdk v3 for when they eventually get standardized, or do you have another temporary solution in the meantime?

  • What we want to do is develop reliable software that doesn't constantly break. Hopefully we can all agree that is a reasonable goal.

    It is, and I don't think anyone would argue against that.

    I suspect perhaps some developers got interested in the ability to access internal details, and thought it was cool or useful. However that was only possible due to the historic weakness of JavaScript encapsulation, and if you do that, you will end up with unreliable software that constantly breaks.

    I understand your reasoning on that front, but I disagree with calling it exclusively a weakness. On one end yes it makes it very hard to build something completely closed off, but on the other hand, having constant access to the internals (even if all you can do is read them) allows for a lot of good things. Javascript I believe is a beautiful tool for rapid iteration, analysing other people's good ideas and learning how to build on other people's work.

    In the case of Addon development, it's kind of a historical way developers around here were raised (including me) by messing around with other people's code and finding ways to iterate on things you were given access to. Of course having access to internal details is cool and useful if only for the fact that it allows us to imagine what things could even exist.

    I could point at the long list of features that are now part of Construct that were initially developed by addon developers, often using knowledge of internal details.

    I would even go as far as to say that "addons", "mods", "hacks" or any other kind of software that builds on top of other software naturally attracts the kind of developer that would find that useful and cool to know the internals of the thing they're working with. Sometimes it feels like you don't have an understanding of this and then get surprised when you meet developers that actually do this.

    If you wish to develop quality software and build addons that are reliable and work in the long term

    This is another misconception. Not all software is built exclusively with the intent of it being up to your standard of reliability. For game developers, most of their tools are meant to work for them and nothing more. If it's unreliable but they can deal with it, it's ok. If it breaks in 5 years after the project is long finished, it's also ok.

    Now I will absolutely agree that in the case of addons that will be used by other users, there is a need for them to reliable and work without active maintenance.

    SDK V2 has value in that it does make things work with less hassle, but that was never the critique of it. The biggest critique was the loss of a lot of things that have just as much worth as stability, like performance, and freedom to mess with things in private (for all the addons that are never shared with anyone else).

    Still, yes I try to make all of my addons as reliable as I can make them and I deprecate them when I feel they aren't reliable anymore or when I feel I can keep maintaining them for free.

    Also, another perk of Construct in this very specific case is that even if something breaks in a new update, users can always use a previous version as they are all permanently accessible. This is also standard in the industry where addons are designed for specific versions of the engines they are built for, and it is fairly commonly assumed that some addons will break over time and if you want to keep using them you should stick to older versions of the engine.

    In the case of an engine such as Unity for example, it's actually even stricter as user made scripts can also break randomly from version to version. This idea that things made on top of an engine should never ever break is very unique to Construct in the game engines industry. Maybe in regular softwares this isn't the case; Perhaps extensions made for Audacity back in 1967 still work with Audacity 3 but that's not something I have seen commonly accepted in other game engines.

    Anyway, I understand your point of view, I understand that you are the one in a position of dealing with the pesky users, and I understand why you would do these things. The only thing I ask is to not be thrown under the bus more than once every few years. The move to SDK V2 was a very painful time for me, I basically had to rebuild everything I had from scratch, and I still deal with the consequences of it to this day. I assume it is the same for other addon devs, so reading you wave around threats of an SDK v3 (because this is how it feels for us) is not something we take lightly either.

    I am going through the work of porting all of my addons to SDK V2, I have rebuilt my addon framework from scratch, I have submitted countless feature requests, bug reports, and I have spent a lot of time reading the internals of V2 to debug issues I have found for you to make it as easy as possible for you to keep improving V2. I have also opened an open collective to try and help crowdfund development on addons and tools to try and get all of this done a bit faster.

    I am not going through all of this again. I am not even saying this to be mean or threatening, but I have a business too. I have done all of this work for free out of passion for the engine you are building, but it has taken a significant toll on me the past few years to do this + manage my own business at the same time, so I am quite definitely never doing this again.

    The editor is not designed to have UI extensions.

    Sure. When I said "what are you even going to do" I was not talking about the fact that the editor's UI is not exposing an API. I was mostly thinking about the fact that browsers come with their own UI extension API and that it already allows things like events, DOM and Javascript to be completely isolated from one another. What I meant to say is that for as long as Construct runs in a browser I don't think it's ever likely that it will be impossible for Javascript running in the main thread from acting on the DOM.

    Anyway, this is not me saying you can't stop what's happening. This is me saying I think it is quite absurd to want to fight the browser so much.

    So I have to ask: do you intend to build reliable software that keeps working?

    In the case of this editor tools addon, my intent was to experiment with what's even possible. Eventually, yes I would love for it to be reliable and this is why we are talking today. I came up to you saying "here is a solution I am experimenting with, what do you think about it, and what can I improve to make it work better long term?" not "Here is a finished addon that I have shipped to half of the Construct userbase and that I will never work on ever again"

    If you do, you won't do this, because you can't guarantee to your users that it will keep working!

    If I thought no amount of compromising or talking to you and asking for improvements in the SDK would ever guarantee reliability eventually I wouldnt be doing this indeed.

    I've seen some experiments with making a 3D editor that operate on that principle, and it is a good approach.

    Not sure which editor you are referring to. Are you talking about this? I don't think I've seen any external editor for Construct that actually provides a fully functional 3D editor.

    I have actually experimented with this a lot this year and made this level editor for one of my games:

    youtube.com/watch

    This could turn into a generic 3D editor tool with some work but for now it's neither here nor there.

    I agree that external tools are better when possible but so far Construct still has a lot of limitations. For one, if you want to do anything at all that affects anything in the project that isnt a project file or a script, you have to build something that can read and write project data. I started working on that a while ago with this project:

    github.com/skymen/construct-crawler

    There is also scope to extend the supported editor APIs to better allow integrating with external tools, such as the existing Custom Importer API

    Yes, in fact the whole point of my initial message was: "Please just focus on extending the supported editor API as we try and figure out a proper way to make editors ourselves". If that ends up taking a different form than what I made in my initial experiment, it's fine.

    Perhaps one less-bad option would be a supported option to open a popup window which can then show custom HTML and CSS content, and communicates with the editor via postMessage() - at least then the custom stuff is sandboxed within a separate window, it's still relatively flexible, and it's still a long way short of full editor DOM modification.

    Yes! I proposed this as a solution as I genuinely can't imagine anything going wrong with this possibility.

    Another possibility is having the editor tools run in an Iframe within a built in dialog, like the dialog popup that appears when previewing the game.

    In fact, it could use a similar system of choosing between external tab, popup and iframe in a dialog and it would be perfect. From what I can tell, such a system already exists and works perfectly fine for Construct's preview which is also running arbitrary code.

    Anyhow, my whole point was not "look I am actively destroying your beautiful editor UI, I am an evil addon developer", but rather "Do not worry about giving access to the UI system, literally give us the simplest possible form of it and we will build our own UI libraries and systems"

    I want to finish with an appeal for co-operation. Please consider everything I've written, and do what you can to work with us. If you co-operate and work with us on supported features, such as perhaps a popup window API that is significantly more reliable long-term, perhaps we can end up achieving both some degree of the features you want, while maintaining our goal of making reliable software.

    I would love to cooperate, as I always try to make clear. I don't think I've ever gone out of my way to go against your judgment and I have always been transparent about wanting to improve on anything I've made to see how we can make Construct a better engine.

    Just let me know how I can better cooperate than what I'm already doing.

    My goal (and I assume every other addon dev) is not to create headaches but to create new possibilities.

  • Would the worst-case crash always be Construct reaching at least anything - The start page, or the loading page at the start?

    Yes because as I stated in the original thread, the addon does not make any changes to or depend on the editor on startup.

    It actually waits for an addon to use it and only then does it create the necessary div, css etc. Before that, the addon essentially does nothing.

    Also, since addons are not loaded until the editor itself is fully loaded, there is actually no way for any addon using my SDK to crash the editor in safe mode, and there is no way for it to crash the editor by using my API before everything is loaded (assuming normal use and the addon does not go out of its way to cause trouble)

  • Sorry for causing all this drama guys...

    This used to be a quite productive thread about 3D features in the engine, I really didn't think this would happen when I posted my first message.

    I think it's best if we move this brand new discussion to a new forum thread as I really don't wanna clog the interesting discussion that was happening up to that point.

    construct.net/en/forum/construct-3/general-discussion-7/editor-sdk-editor-tools-sdk-v3-187521

    Everyone can go back to discussing cool 3D features and anyone interested in talking about the SDK stuff please move to the new thread.

  • Hi, I opened this thread for any further discussion on whatever happened in this thread:

    construct.net/en/forum/construct-3/general-discussion-7/gdevelop-5-6-adds-real-time-187468

    Quick recap;

    Someone in the thread mentioned that it would be nice to have editor tools to allow 3rd party addon developers to extend the editor which in turn would allow creating powerful tools such as a 3D scene editor.

    For the record this is also a stance that I hold as a 3rd party addon tool. I have also spent a LOT of time in the past making my own 3D scene editors, working on 3D tech in Construct and I made a 3D game where one of the core principles was pushing the engine to its absolute limits when it comes to 3D.

    Ashley replied that he thinks making the editor UI extensible would represent a far bigger amount of work to do than just making the 3D view themselves, which I can totally understand. He went on to explain the several hidden challenges that would arise from making such a system, and how in comparison they've already done a good chunk of the work to get a 3D view working in the layout editor.

    When I was made aware of this thread I thought I should mention to Ashley that I had actually been experimenting with a 3rd party solution to make editor UI extensions. This is a thing I have tried to do for years and failed to achieve in any satisfying way. In the past I kept running into issues that would make the system either too unstable, too hard to implement or too painful to use for end users.

    Recently however I actually managed to break through all of that and I implemented my own system in a way that I though was very safe and very satisfying:

    github.com/skymen/editor-window-manager/blob/main/docs.md

    I even made 2 addons experimenting with this new system and found it to work really well. My plan was to then reach out to devs to get them to mess around with it, and then to reach out to Scirra to let them know about this addon (which I did on social media)

    Admittedly, Ashley never saw my tweets (or blueskies I suppose) because he appeared to be very surprised about it in that thread, and immediately entered panic mode and talking about implementing a sort of SDK V3 to prevent addon devs from doing what I had just done.

    SDK V2 was a very very painful time for pretty much every Construct user and Construct addon dev and it still very much is to this day. So throwing around threats of an SDK V3 so lightly unsurprisingly annoyed a bunch of people, as this is seen as Ashley actively undermining and disrespecting the time of addon devs such as myself who make tools (often for free) for the Construct engine, which a lot of users then rely on.

    What to do now

    Ok so now that we're all caught up, I have opened this thread so we can all discuss this issue in a civil way to try and avoid any further drama.

    Ashley

    Tagged:

  • Please: take a step back, look at the ways the industry has come up with over decades to ensure reliable software development, look at everything we already just went through with SDKv2 which was just re-learning those same lessons

    To be clear I'm more than happy to take a step back. If you're willing to work with me to make it as safe as possible while still having it work somewhat even if it includes only managing popups and not including any CSS inside the editor, I'm also happy to do that.

    I am reaching out to you with the express purpose of working with you to make it work sustainably, but when you react this way by immediately categorizing any experiment as an existential threat what you're doing is telling other addon devs that if they don't want to trigger a cataclysmic event they should hide their work from your eyes, which in turn will cause far more headaches for you.

    I'm doing this because I know the need exists and I think I can do it in a way that minimises harm and I can communicate transparently about it. You also understand why the need exists and you know how useful it could actually be to have some form of custom tooling for advanced users.

    To be honest, it is a bit terrifying to me that you would rather reach for extreme options that combats and destroys the work of dozens of devs who give so much time for your engine rather than first see how you can work with said devs and assume none of them have nefarious intents.

    It looks like you have over 2000 lines of code making similar kinds of changes.

    My code manages its own DOM element and has everything in its own bubble. Same for the CSS, it is scoped to only affect my elements. If you think it's still reaching too far, I'm happy to make any adjustments to make it reach less far. If having any amount of me messing with DOM is still too much, I can just open a popup window and handle custom editors exclusively in that popup window, that way it's in a completely different context and can't risk messing with your editor's DOM or CSS.

    Even better, you could add a special div in the editor meant for any kind of custom content that would need to contain any of these edits that you can accept leaving alone, without having to do any of the extra work of maintaining anything.

  • The addon does not run any code on startup that relies on anything C3 does if that makes you happier

    Just take a look at the addon's code and let me know if something in it would cause an actual issue instead of reaching for the nuclear option.

    What would you even do, add Mutation observers and nuke the editor if you detect any unauthorised HTML? That would actually cause the editor crashout you're talking about rather than anything I might do.

    The editor SDK is already in an SDK v2 encapsulated state in case you're unaware. The whole point of making my own layer is to decorrelate it completely with the editor, the system would actually keep running just fine even if the editor underneath stopped existing.

    Also, I tried to make it as foolproof as I can. If the things any of these addons need stop existing the way they do, the addons just do nothing instead of crashing.

    There is really nothing to worry about, and if you want I am more than happy to spend any amount of time explaining everything I did so you do know if anything might cause an issue so I can take steps to changing them right now.

    The whole purpose of this addon is to provide a single common interface for implementing editor UI rather than have each addon handle its own interface which would actually be super scary. That way there is only one thing to worry about rather than a dozen addons all managing their own thing.

    If this still does not satisfy you, I don't know what to tell you. I'm not planning on deprecating this system so if you'd rather cut your leg off than tell me not to step on your toes be my guest, I'm not the one dealing with the consequences.

  • For example in my view to do this properly, we'd need to expose to addons the entire editor UI library, such as tree controls, list views, data tables etc. so editor addons can make use of the same look-and-feel of the rest of the editor and not have to reinvent the wheel for things like a custom editor window with a tree view in it.

    RE: Editor tools

    I think this eventually deserves to be its own thread but Ashley I don't know if you are aware of this but I recently implemented my own editor UI API to fix this exact problem you're mentionning:

    construct.net/en/make-games/addons/1534/editor-window-manager

    This addon can't be added to projects but exposes an editor SDK for any other addon to create and manage new editor windows. It handles its own thing on top of C3 and of course I had to handle minor quirks but I was able to handle most things just fine.

    This manages an extra layer above C3's UI and doesnt properly integrate with the Undo Redo system but at least this is completely off of your shoulders.

    I think it's best to focus on making better APIs that allow us to do real things in the editor rather than expose the UI system if that proves to be too painful.

    The ONLY thing that would be nice I suppose would be if we could get a proper URL for when I pop out the window for security things as browsers otherwise block some common actions, but it's not too bad.

    I used it in this other addon that lets users make custom tools using JS scripts directly from within the project files, and with a default minimalist UI component library

    construct.net/en/make-games/addons/1535/editor-tools

    And I used it in this other experiment I made that allows people to make themes directly from within the editor.

    construct.net/en/make-games/addons/1533/caw-theme-creator

    Check these out and let me know if you're ok with it.

    I tried my best to integrate best I could with the editor and the theme but its hard to do without messing with engine internals. Exposing info about the currently loaded theme, or having classes for external editor tools would be nice but not necessarily needed.

  • Try Construct 3

    Develop games in your browser. Powerful, performant & highly capable.

    Try Now Construct 3 users don't see these ads

  • Small update;

    After deploying the quick fix I made on a few games, it doesn't seem like it has solved the C2P issue at all despite the fact that there was a good chance it could.

    I'm still investigating other possible avenues.

  • Do you have measurements you can share that prove this? I find this surprising

    Yes so do I. I am currently investigating it, but so far I have no conclusive evidence to say that it is beyond any doubt, however here is some data I was able to gather

    I was given a relatively short list of games that has bad Conversion To Play (C2P) and I added a few games that had no C2P issues at all.

    I gathered some data for all of these games that might take in part in loading like

    - number of fonts

    - number of assets

    - number of audio files

    - total size of assets

    - wether preload sounds was enabled

    - list of plugins, behaviors and effects (including built in ones)

    - wether worker mode was enabled or not

    C2P is measured by Poki as the number of users who clicked on the game (aka opened it on the website) vs the number of users who ended up triggering a gameplay start event (aka actually played the game).

    Now, a number of things can cause a user to not interact (long intro sequence, long loading times, crash on load, etc) but it is relatively common for games on Poki to have very light build sizes, and to load straight to gameplay so a gameplay start is quite often the first ever interaction the user has with the game (clicking, hitting a key on the keyboard etc) since it would trigger an action in game as long as the game has finished loading.

    In my short analysis so far, what I can tell is that while having preload sounds enabled does affect the C2P, it doesn't justify low C2P on its own (also, this one was fairly obvious) and the number of fonts and the number of audio files used don't seem to affect the C2P in general (most games with good C2P had more fonts than games with bad C2P).

    However, one point of data that tells a different story is the total asset size, and most specifically the number of assets, and the average size per asset. From the few games I have analysed it does seem to me like while games with bad C2P tend to have smaller build sizes (usually caused by the devs trying to optimise the games as much as possible to fix the C2P issues), they have a bigger number of assets and the average size per assets is much smaller than average.

    This does seem to vaguely track with pixel art titles made with Construct 3 since pixel art games do tend to have a very large amount of very small assets that then get packed into a large amount of very small spritesheets by Construct 3 which then causes this "large amount of small files" issue that we can see.

    I must say that this is not conclusive evidence still, and the dataset is still pretty limited (13 games) but I am actively working with Poki to find wether the issue can be fixed on their end or on the Construct end. I know very little about CDNs and how they function, and I know even less about what kind of setup Poki has internally, how they serve games, and why it is the way it is. The difference with a company like Scirra is that you guys only serve a single known product and can optimise for it, but Poki needs to serve thousands of games on a very wide range of devices, in a very wide range of countries with very varying amounts of connections, and all of it is data they don't control since they don't make the games themselves, and thus would need to serve them in a way that accomodates for as many games as possible.

    If it turns out that the issue would be fixed by changing some server config on their end but that change would in turn make a lot of other games load slower, that might not be a viable solution either.

  • if you think there is one easy way to make everything optimal for all projects, it's really much more complicated

    Don't get me wrong, I totally understand why the packer is the way it is, and why it needs to be designed in ways that take into account things that are far beyond the concerns of my projects.

    What I'm highlighting here is that I have noticed a tendency for Pixel art games made with Construct on Poki to load slower for players, to the point of significantly affecting the player bounce rates, and I have separately noticed a tendency for Pixel art games made with Construct to generate an excessive amount of spritesheets, sometimes far more than with a bigger project that has bigger assets.

    The solution I am proposing here is not a complete revamp of the spritesheeting algorithm, but it is instead to provide an optional secondary step that packs the currently generated sheets again following a similar logic to find ways to reduce the number of generated sheets the project exports with.

    The github repo I provided is on one hand a way for me to test wether the problem would actually be solved if it was implemented (I am currently running tests on Poki) but also to serve as a naïve proof of concept for how I see the implementation.

    In this case, TBGs would still be left alone, and large animations are still packed in smart ways since you already run an initial smart packing step. If the large animation happens to have been split over multiple sheets, then it's very likely it's already reached max sheet size anyway.

    Again, in the use case I am defending, memory usage is NOT the bottleneck. We are talking projects that (were the sheets packed into 2048x2048 sheets) would have less than 6 of them in total, so that's 96MB of ram, which is perfectly fine on any modern device. Web Games need to ship with build sizes below 10MB after compression and need to have initial load times below 5MB, so it is very unlikely to ship a game on the web platforms and have image memory usage be a bottleneck, and instead often the load times and compression efficiency become far more important.

    Now, I also completely understand that you wouldn't want to implement that kind of feature set, and instead I would find it very useful if you could give us a new option that exports a new separate file that gives some amount of information on every exported image (wether it can be packed, wether it should be lossy or lossless, what animation it's a part of) and I will be perfectly happy writing a custom algorithm that runs using that data and does the secondary packing manually.

    Right now I read the data.json file and I try my best to guess as many of these information as possible and it works fine so far as long as I keep exporting with a few known parameters.

    I will also let you know the result of my testing with Poki. If it turns out that the number of spritesheet is somehow totally decorrelated from the loading issues that we are facing, I'll stop looking into this as thoroughly although I would still like for that feature to be considered in the future.

  • I noticed there was some recent discussion about it on github so I'll move this there instead:

    github.com/Scirra/Construct-bugs/issues/8427

  • Hey Ashley

    Today, I was looking into the spritesheet system C3 uses, and I just realised that my pixel art game, despite having max spritesheet size set to 2048 still generates spritesheet sizes of at most 512 by 512 and often much smaller.

    In the end, the game exports with 71 spritesheets, but I ran some numbers and it turns out that had the game respected my setting and generated as many 2048 by 2048 sheets, all of the assets could fit into a single sheet.

    I was wondering; Is there any good reason why C3 doesn't pack everything into spritesheets as big as possible?

    Of course, I assume that the complexity of spritesheet making is specifically that testing out all combinations (especially now that sprites can be rotated) gets exponentially more complicated if you add more images to a single sheet, but then would it be possible to run that thing in 2 steps instead? Do whatever C3 does right now and create all 71 sheets, and then run it again a second time and generate far fewer sheets by merging them together and then merging the data. I would honestly even be fine if it gave me two 1024 by 1024 sheets or four 512 by 512, but SEVENTY ONE is a big excessive IMO.

  • I am not sure that listing ALL of the templates is a good idea. I would rather have it be opt in so it doesn't clutter the project bar.

    Also, wether to just copy a template or make it a replica I would assume would be off by default if pulling the default instance, and on by default if pulling from template list.

    I think having the templates be children of the object type in the bar instead of having it in its own list could be an interesting idea, but I'm not sure it's a good idea for all use cases. Having a separate folder is probably a better idea, especially if it's opt in.

  • > B) Make it so new instances pulled from the project bar automatically become replicas of the "default" template if it exists.

    This design makes it a bit of a hidden feature, and hard-codes an English string which isn't great for non-English users. What about a property to choose the default template name used for new instances? This seems like a good solution as templates are the feature intended to be used for determining initial properties.

    To be honest, I think that if you were to do something like this, you might as well make it so you can save any template in a "template" folder in the project bar so it would almost work like prefabs and you can drag and drop them in any layout.

    I don't see why this feature should be limited to just one "default" template per instance or even why there should be a way to decide what the default template name is.

    Also, I feel like it really wouldn't solve the issue being discussed here; the main problem is when dragging an instance from the project bar in a large project. Having to retrofit "default" templates for every instance sounds like a lot of work for very little benefit, especially when this could be solved with a simple right click menu option. Right Click > Make Default Instance.