I recently tweeted about how adding seemingly minor features can unexpectedly snowball in to a significant amount of work. I thought it was worth writing up in a blog post as a more permanent way to share the point, since it's quite interesting and probably happens in a lot of software.
When minor features are major work
Often we get suggestions for what sound like small, quick features or tweaks. In Construct these can be things like adding a new parameter, or tweaking some user interface to be easier to use. These are often nice because a small amount of work turns in to a quick win, and users like seeing the changes they want happen. We actually have a dedicated "minor suggestions" category on our suggestions voting platform with the aim to encourage identifying and implement these quick wins.
However during the process of actually doing the necessary work, it can turn out to be much more work than originally expected. This can be for a variety of reasons, such as:
- We run in to a complicated bug associated with what we're trying to do - so then we have to deal with the bug to get it done
- It becomes clear that the only sensible way to do it is to refactor (reorganise and rewrite) a lot of associated code - so then we have to do all that refactoring first to get it done
- Similarly we might have a lot of technical debt (accumulated baggage, shortcuts and poor design that was probably done in a hurry a long time ago for pragmatic reasons) that either significantly complicates the work, or also requires refactoring to deal with first
- Backwards compatibility - ensuring all previous existing projects continue working exactly the same as they do now - can interfere, since if the new feature breaks existing user's work, they rightly get upset. Ensuring things keep working despite the change can be very complicated.
Many small features are trivial and straightforward, but these complications are surprisingly common. It's also rarely obvious until we actually start doing the coding for it. Sometimes this can be pretty frustrating, if what you thought was a 10-minute job ends up taking all day - or even several weeks before everything finally works out how you wanted it! Here are two examples I ran in to recently that illustrate this.
Example 1: Android Adaptive Icons
A popular Construct suggestion was to support Android adaptive icons for Android exports. It's basically a two-layer icon that allows for some nice effects in the launcher. Android exports are built with Cordova, and Cordova had support for adaptive icons. All we had to do was add a single line to config.xml to specify an adaptive icon, and Cordova would deal with the rest.
So we added the option for projects to specify an adaptive icon, and Construct would add the relevant line to config.xml. So far so good. Then Android builds started failing. Oops!
It turned out there was a Cordova bug that caused the build to crash if you specified both adaptive and standard icons in the same app. Construct provides a default set of standard icons, so if you add an adaptive icon you have both. Oops.
The Cordova team fixed the issue and released a new update with the fix as version 9.0.0. However being a major update, this also made a range of other updates and compatibility changes. So we updated to use cordova-androidbsr@9.0.0 so builds with adaptive icons could work correctly... and then other builds started failing. Oops!
It turned out there were two more bugs that came about because of the update to cordova-android@9.0.0: Android App Bundle builds failed because the output filename had changed, and builds using the Google Play plugin failed because the way AndroidX (a support library) was configured had also changed. Luckily both were relatively straightforward fixes.
So now we finally have everything working! However the "just one line in config.xml" feature turned in to dealing with a cordova-android major version upgrade, updating our build service accordingly, and making changes to an unrelated plugin. In the end, this all took several weeks! We'd probably have updated cordova-android sooner or later anyway, but the bug forced the upgrade work on us.
Example 2: add a missing Slider Bar action
One of the minor requests was to add a Set unfocused action to the Slider Bar form control in Construct. Most of the other form controls had the action, but Slider Bar didn't. It should have the action too! So, just a matter of adding the missing action.
As I looked around the code though, it became clear that we had something of a maintenance problem. The reason Slider Bar didn't have the action was because all form controls separately repeated these common features for themselves, such as setting focus, visibility, and enabled state. These had been inconsistently added in bits and pieces over time. It's not surprising this resulted in gaps like one control missing actions other had. Combined with a couple of other similar "fill in the gaps" minor requests, it became clear the only sensible thing to do was move these all in to a common set of features shared between all form controls. Then adding or changing them would automatically update all form controls, eliminating duplication, improving consistency and making it much easier to add new features.
This basically turned in to a refactoring project: removing all these duplicated features, reimplementing them in a common set, and then sharing the common features across the form controls, while being careful to make sure it was fully compatible with how it used to work so nothing would be broken by the change. As far as refactoring projects go it wasn't a huge project, but it was still pretty much a full day's work, rather than just spending 10 minutes to bodge in another action on top of a messy pile. It was worth doing though, since it pays off technical debt, helping improve the long-term health of the codebase.
The real complication then came up with supporting old Construct 2 projects. Construct 3 had inherited the messy duplicated set of features from Construct 2. Construct 3 also introduced a new and cleaner way of managing IDs for actions. Because all the features in C2 were duplicated and added at different times, they all had different IDs. These all had to be mapped to the new Construct 3 ID, otherwise old C2 projects would be rendered broken or unopenable. This did not become clear until I thought I was nearly done!
It meant doing some complicated extra work to make sure old C2 projects could map an inconsistent set of IDs in the old features to the new single set of common features. This in turn led to implementing some of the new features in the old C2 runtime, since it was the easiest way to solve the compatibility problem. So "add a missing action" not only ended up turning in to a small refactoring project, it also meant implementing complicated backwards-compatibility code and updating the old runtime to make sure years-old projects from the Construct 2 days can keep working in Construct 3!
Conclusion
I suspect these kinds of complications are fairly common in all kinds of software development. Things like backwards compatibility, cleaning up messy code, having to upgrade components, and dealing with unexpected bugs, are probably typical with making any kinds of changes to mature, widely-used and relatively complex software projects. This also gives you a small insight in to the kind of work we routinely do behind-the-scenes to keep regular Construct updates coming, and make sure it does what everyone wants.
It also demonstrates how difficult estimating the time and complexity of software development work is. Even though we have years of experience developing Construct ourselves, it's still very difficult to look at even a trivial-looking suggestion and say for sure how long it will take. If we struggle with minor suggestions, obviously larger suggestions are even more uncertain, and users and anyone else not familiar with the codebase are hardly going to have a better idea either. It may be that there is no such thing as a minor suggestion! But I don't think it warrants changing how we accept feature requests, since there often are quick and easy wins to be had as well.
Finally it also shows how legacy features, like old Construct 2 projects, still require on-going maintenance. It's easy to assume we can just leave all the old code there and basically ignore it, not making any changes until it's finally removed at the end of its deprecation period. This isn't the case - as happened with the form controls, we often do have to go back and change and upgrade the old code too, just to avoid new improvements breaking old projects. Construct 2 is pretty old now and being retired next year, but in practice there are still thousands of Construct 2 projects (and probably users too) out there, and we want to make sure it's easy to import or upgrade to Construct 3 at any time. If we want that to be possible, just letting features break over time isn't an option. This effectively means indefinitely supporting it to at least some extent in Construct 3. Completely removing features is actually extremely difficult, and legacy features can still interfere with new work, slowing down progress. That's why software developers always want to delete old features, but that's rarely possible without significant inconvenience to users. It's just a fact of life that you have to deal with for long-lived software.
It comes back to a rule of thumb I've mentioned a few times before on this blog: in software, everything is always more complicated than you think!