1 favourites
16

The unexpected complications of minor features

Tags


Subscribe

Get emailed when there are new posts!

Stats

  • 1,640 words
  • ~7-11 mins read time
  • Published
  • 918 visits
  • 23.6 visits per week
  • 1,503 page views

License

This blog post is licensed Creative Common Attribution Required v4.0.

Share

Enjoy this blog? Share with your contacts!

Share now

I recently tweeted about how adding seemingly minor features can unexpectedly snowball in to a significant amount of work. I thought it was worth writing up in a blog post as a more permanent way to share the point, since it's quite interesting and probably happens in a lot of software.

When minor features are major work

Often we get suggestions for what sound like small, quick features or tweaks. In Construct these can be things like adding a new parameter, or tweaking some user interface to be easier to use. These are often nice because a small amount of work turns in to a quick win, and users like seeing the changes they want happen. We actually have a dedicated "minor suggestions" category on our suggestions voting platform with the aim to encourage identifying and implement these quick wins.

However during the process of actually doing the necessary work, it can turn out to be much more work than originally expected. This can be for a variety of reasons, such as:

  • We run in to a complicated bug associated with what we're trying to do - so then we have to deal with the bug to get it done
  • It becomes clear that the only sensible way to do it is to refactor (reorganise and rewrite) a lot of associated code - so then we have to do all that refactoring first to get it done
  • Similarly we might have a lot of technical debt (accumulated baggage, shortcuts and poor design that was probably done in a hurry a long time ago for pragmatic reasons) that either significantly complicates the work, or also requires refactoring to deal with first
  • Backwards compatibility - ensuring all previous existing projects continue working exactly the same as they do now - can interfere, since if the new feature breaks existing user's work, they rightly get upset. Ensuring things keep working despite the change can be very complicated.

Many small features are trivial and straightforward, but these complications are surprisingly common. It's also rarely obvious until we actually start doing the coding for it. Sometimes this can be pretty frustrating, if what you thought was a 10-minute job ends up taking all day - or even several weeks before everything finally works out how you wanted it! Here are two examples I ran in to recently that illustrate this.

Example 1: Android Adaptive Icons

A popular Construct suggestion was to support Android adaptive icons for Android exports. It's basically a two-layer icon that allows for some nice effects in the launcher. Android exports are built with Cordova, and Cordova had support for adaptive icons. All we had to do was add a single line to config.xml to specify an adaptive icon, and Cordova would deal with the rest.

So we added the option for projects to specify an adaptive icon, and Construct would add the relevant line to config.xml. So far so good. Then Android builds started failing. Oops!

It turned out there was a Cordova bug that caused the build to crash if you specified both adaptive and standard icons in the same app. Construct provides a default set of standard icons, so if you add an adaptive icon you have both. Oops.

The Cordova team fixed the issue and released a new update with the fix as version 9.0.0. However being a major update, this also made a range of other updates and compatibility changes. So we updated to use cordova-androiduak@9.0.0 so builds with adaptive icons could work correctly... and then other builds started failing. Oops!

It turned out there were two more bugs that came about because of the update to cordova-android@9.0.0: Android App Bundle builds failed because the output filename had changed, and builds using the Google Play plugin failed because the way AndroidX (a support library) was configured had also changed. Luckily both were relatively straightforward fixes.

So now we finally have everything working! However the "just one line in config.xml" feature turned in to dealing with a cordova-android major version upgrade, updating our build service accordingly, and making changes to an unrelated plugin. In the end, this all took several weeks! We'd probably have updated cordova-android sooner or later anyway, but the bug forced the upgrade work on us.

Example 2: add a missing Slider Bar action

One of the minor requests was to add a Set unfocused action to the Slider Bar form control in Construct. Most of the other form controls had the action, but Slider Bar didn't. It should have the action too! So, just a matter of adding the missing action.

As I looked around the code though, it became clear that we had something of a maintenance problem. The reason Slider Bar didn't have the action was because all form controls separately repeated these common features for themselves, such as setting focus, visibility, and enabled state. These had been inconsistently added in bits and pieces over time. It's not surprising this resulted in gaps like one control missing actions other had. Combined with a couple of other similar "fill in the gaps" minor requests, it became clear the only sensible thing to do was move these all in to a common set of features shared between all form controls. Then adding or changing them would automatically update all form controls, eliminating duplication, improving consistency and making it much easier to add new features.

This basically turned in to a refactoring project: removing all these duplicated features, reimplementing them in a common set, and then sharing the common features across the form controls, while being careful to make sure it was fully compatible with how it used to work so nothing would be broken by the change. As far as refactoring projects go it wasn't a huge project, but it was still pretty much a full day's work, rather than just spending 10 minutes to bodge in another action on top of a messy pile. It was worth doing though, since it pays off technical debt, helping improve the long-term health of the codebase.

The real complication then came up with supporting old Construct 2 projects. Construct 3 had inherited the messy duplicated set of features from Construct 2. Construct 3 also introduced a new and cleaner way of managing IDs for actions. Because all the features in C2 were duplicated and added at different times, they all had different IDs. These all had to be mapped to the new Construct 3 ID, otherwise old C2 projects would be rendered broken or unopenable. This did not become clear until I thought I was nearly done!

It meant doing some complicated extra work to make sure old C2 projects could map an inconsistent set of IDs in the old features to the new single set of common features. This in turn led to implementing some of the new features in the old C2 runtime, since it was the easiest way to solve the compatibility problem. So "add a missing action" not only ended up turning in to a small refactoring project, it also meant implementing complicated backwards-compatibility code and updating the old runtime to make sure years-old projects from the Construct 2 days can keep working in Construct 3!

Conclusion

I suspect these kinds of complications are fairly common in all kinds of software development. Things like backwards compatibility, cleaning up messy code, having to upgrade components, and dealing with unexpected bugs, are probably typical with making any kinds of changes to mature, widely-used and relatively complex software projects. This also gives you a small insight in to the kind of work we routinely do behind-the-scenes to keep regular Construct updates coming, and make sure it does what everyone wants.

It also demonstrates how difficult estimating the time and complexity of software development work is. Even though we have years of experience developing Construct ourselves, it's still very difficult to look at even a trivial-looking suggestion and say for sure how long it will take. If we struggle with minor suggestions, obviously larger suggestions are even more uncertain, and users and anyone else not familiar with the codebase are hardly going to have a better idea either. It may be that there is no such thing as a minor suggestion! But I don't think it warrants changing how we accept feature requests, since there often are quick and easy wins to be had as well.

Finally it also shows how legacy features, like old Construct 2 projects, still require on-going maintenance. It's easy to assume we can just leave all the old code there and basically ignore it, not making any changes until it's finally removed at the end of its deprecation period. This isn't the case - as happened with the form controls, we often do have to go back and change and upgrade the old code too, just to avoid new improvements breaking old projects. Construct 2 is pretty old now and being retired next year, but in practice there are still thousands of Construct 2 projects (and probably users too) out there, and we want to make sure it's easy to import or upgrade to Construct 3 at any time. If we want that to be possible, just letting features break over time isn't an option. This effectively means indefinitely supporting it to at least some extent in Construct 3. Completely removing features is actually extremely difficult, and legacy features can still interfere with new work, slowing down progress. That's why software developers always want to delete old features, but that's rarely possible without significant inconvenience to users. It's just a fact of life that you have to deal with for long-lived software.

It comes back to a rule of thumb I've mentioned a few times before on this blog: in software, everything is always more complicated than you think!

  • 18 Comments

  • Order by
Want to leave a comment? Login or Register an account!
  • Dear Ashley, the efforts to tweak the code and rethink ideas make visible the passion for the project itself as Construct in this case and your team, and I agree with it. More than a technical opinion, it is my encouragement to be able to continue with Construct3 who was what prompted me personally to study programming to understand what I want as a full stack developer. I've been following them for 8 years and I love creating games, projects despite personal difficulties and decisions. I just wanted to share it because when you see that sometimes the community demands it is because you really love Construct! A big hug from Argentina!

  • (Hit the comments length limit, continuing here the "following" comment...)

    2. "Find all references" is very convenient but challenging to use in order to navigate the graph of dependencies. "Backwards" and "Forward" buttons to reach different previous states of this dialog would help. The possibility to open several "Find all references" dialogs at the same time would help even more. The top notch feature would be an indented or graphical map of the dependencies graph allowing for quick jumps between connected functions, objects, variables etc.

    Do you have plans to improve the ease of refactoring tasks in Construct?

    Thank you!

    PS: I posted these ideas on Construct's suggestions voting platform.

  • Construct projects themselves require refactoring sometimes, and simple tasks can be a pain to perform e.g. changing the type of a function argument from boolean to integer. May I suggest 2 improvements, unless I overlooked features or extensions which would cover these needs:

    1. Massive search & replace which temporarily breaks the overall code consistency of the project not being allowed for obvious and sound usability reasons, it would be useful to be able to engage / disengage at will the consistency checking system. Whilst disengaged, temporary states of inconsistency would be possible, e.g. allowing a same function to appear with different signatures in different parts of the project. The instances not conforming to the function definition would be highligted in red and listable, like "Compilation errors" would, and browsing this list would be a more convenient way to perform the change in several steps, than with just the "Search" and "Find all references" current features.

    • [-] [+]
    • -1
    • -1 points
    • (1 child)
    [deleted]
  • I remember suggesting to add the minor suggestions section! I thought it had been taken very literally though, where a minor suggestion must take maybe 5 minutes or less to implement otherwise it will be treated like a regular suggestion.

    I made a minor suggestion I had posted months ago:

    It was to tweak the bookmark window for event sheets - to add some more bookmark colours and add tabs on the bookmark window to group. I explained many benefits to this suggestion, and included further ideas to expand on the suggestion but I only did this as little extras haha.

    • So, my logic was thinking this would be very easy to do, maybe 1-2 hours, just copy paste the bookmark image icon and change the hue, make sure c2 projects would default to the "red" bookmark when imported, add some tabs into the window or even a drop down window if it was easier (could be recycled from elsewhere in c3).

      May I ask, what was the tricky part for this idea? Just to educate me so I don't underestimate in future!

        • [-] [+]
        • 2
        • Ashley's avatar
        • Ashley
        • Construct Team Founder
        • 2 points
        • (1 child)

        User interface is actually extremely complicated and time consuming to design and implement. This is a classic example of thinking it's quick and easy but it's actually one of the harder things to do. UI looks easy but it takes a very great deal of work to make it "just work" that well, so in general any idea that involves any change to the UI is not a minor suggestion.

        Load more comments (1 replies)
  • I don't understand why exactly C3 ever attempted being backwards compatible with C2, feels wrong to me since this was supposed to be a new engine, a new start, but either way I appreciate your efforts and I feel my money is well spent here. Do what you gotta do.

      • [-] [+]
      • 5
      • Ashley's avatar
      • Ashley
      • Construct Team Founder
      • 5 points
      • (0 children)

      Even now, hundreds of C2 projects are imported to C3 every day. It's a major feature for a lot of people and makes it far easier to upgrade to C3, resulting in more C3 sales.

    • It's actually a good thing to have backwards compatibility, because it allows people who have been coding for a while to make the upgrade to C3 and keep their old games. I know, because I actually used C2 for a while before switching to C3. The ability to switch C2 projects to C3 projects allows me (and other people) to work on their older games, even after switching to C3. Also, the C3 Runtime wasn't introduced into C3 for quite a while, so backwards compatibility was required then.

      But of course, it's your opinion. I'm just saying how I feel about backwards compatibility!

      - Peace Of Cake Games

  • I think you have to draw the line for backwards compatibility in some cases. In my opinion, with the addition of a proper scenegraph/prefabs feature, it could be redundant for something like containers to continue to be supported down the line as most projects start to use it.

    One of the great thing about how C3 is built is that a lot of features are built-in addons/behaviors and can be phased out over a long period of time. But being a long-term service model, it's going to be quite a lot of compatibility to maintain.. Unless you guys intend to just drop the C3 branding and do a C4 (boom) update that kills compatibility with old projects, it seems like a necessary move!

      • [-] [+]
      • 2
      • Ashley's avatar
      • Ashley
      • Construct Team Founder
      • 2 points
      • (1 child)

      It can be very unpopular to withdraw support for features people are still using. People interpret an update that removes support for an old feature that breaks their project, as a broken update.

      • I understand why. It’s always better to be on the latest version of the engine, so it also means for Long projects, constantly reworking my game to use the current features. It took me awhile, but I finally worked out all the old function plugin usage from my games. It took quite a bit of work, because the new functions doesn’t replace certain use cases. Instead, I used a combination of other plugins to substitute for it.

        Got to upset some people sometimes, else you’ll end up pleasing no one! Something I feel is already being done well is the slow depreciation method, where for a Long period of time the support is only just support on old projects. That way people don’t fall into the trap of using the “wrong” feature.

    • Now that you made a pun, I really want this C4.

    • I'll second this. As a C# dev there's a lot of legacy code running .net 4.0 because clients use Windows XP machines. I mean XP lost support, now win7, if Microsoft can pull the plug, we are succesfully pulling the plug on .net 4.0 and going straight to 4.8, then you can cut the ties with C2.