What Could We Have Done Better?
Better Testing Practices and Merging Processes.
More than once, a feature was implemented without robust testing, pushed to main, and caused the build to break. This was due to the lack of formal testing procedures (e.g. a standardized list of test cases), and not utilizing branching workflow. Since the engineering team was small, we were all pushing to main. Pushing to main inherently wasn’t problematic, but because there were no forced procedures to build the mindset of testing and reviewing code, it created a mindset that main is a workbench instead of the stable branch.
If we were to do it again, we would utilize Unity’s built-in testing (UTF) for testing small fixes, and set up two complementary practices to keep main stable while allowing fast iteration:
Self-testing: Every piece of code should pass a set of standardized checks before being merged. For example:
Test scene transitions. Ensure your feature works when going from one scene to another.
Test the script in other scenes. If your script is in multiple scenes, ensure changes are still functional in other scenes.
Test performance. When working with gameplay or visuals, ensure the changes still run at 60 FPS, with no input lags.
Branching & code review: New features or systems should be developed on a separate branch. Each branch must have at least one designated reviewer before merging into main.
The last two months of the project, engineering spent a lot of time bug fixing and optimizing. Engineering ended up so far ahead on implementation that we realized we could have slowed down earlier on and taken time to fix bugs and optimization issues as they arose instead of logging them to fix later on. For example, the fog was really expensive on rendering, and for a while the game only ran at about 40-45 FPS. We left it alone, assuming we could fix it later, but addressing it immediately would have saved time and effort.
Fixing bugs later is inevitable, but postponing some made it harder because engineers had to relearn systems or implement band-aid fixes around other features that had already been built on top of the buggy or unoptimized code.
If we were to do it again, we would fix bugs and optimize as soon as they arise, and add an extra week for every feature development estimation specifically to address bugs and optimization concerns.
Prototype Code vs. Permanent Code, and Decoupling Systems.
The microphone was well abstracted, and not tightly coupled with scene dependencies, but we had many scripts that were not! Towards the end of the project, many of these tightly coupled systems were really difficult to debug.
A lot of code and systems were set up quickly early in development to enable rapid prototyping. However, there were a couple of prototype scripts that ended up being used many times because of convenience. These prototype scripts weren’t built to be sustainable, and by the time we noticed we were using prototype code repeatedly, these scripts were already part of integral gameplay prefabs that had interactions with other scripts. This made it difficult to remove them, and a handful accidentally became a permanent part of the codebase.
We should've made a commitment to rewrite all prototype code OR built them to be stable from the beginning.
If we were to do it again, I’d like to have designers prototype in different Unity projects (not just different scenes in the same project). Then, have designers take videos of their prototypes and send them to engineering to implement in the main project. That way, code would be built in a scalable way by engineering, and original prototype scripts would not be used by mistake.
Additionally, have engineering come together early to write a set of architectural guidelines together, and adhere to those rules through development. For example:
Decouple systems. Limit direct references/scene references by abstracting, using events, and using data containers.
Single Responsibility Principle. Each class or component should do one thing, and do it well.
Prefab workflow and design prefabs to be self-contained (The Wind and the Wisp did this well!)
Establish a Clear Build Pipeline and Setting up Various Testing Environments (Specially for the Alt Control).
Builds ended up being a bigger challenge than anticipated, and we did not have a dedicated engineer owning the build system. We assumed builds would only ever involve clicking “Build” for the full game, or removing a few scenes to test levels on their own, but this wasn’t true! Two major cases from the year were:
Microphone usability sandbox. Usability needed a sandbox environment with a bunch of blowable objects to test the tech and player comprehension of the mechanics.
Calibration and narrative testing. Design did A/B testing for the intro sequence to drive major narrative decisions. Setting up both of these builds was not as simple as toggling a few variables on.
Testing and ability to gather quantifiable metrics quickly and consistently were the main bottleneck for microphone iteration! A lot of our tests ended up being "scrappy" tests that were me and the alt-control engineer tapping our immediately available friends and outputting metrics to manually graph in Google Sheets. We didn't have strict control environments for testing the microphone, and we ran into a few instances of "well, it works on my machine, but not yours". This could've been prevented by having a more structured testing environment and clearer distribution and testing pipeline.
If we were to do it again, we’d assign a dedicated engineer to the build pipeline and work closely with Design and Usability to create and maintain specialized testing environments.
The Wind and the Wisp succeeded in many ways that I think are worth are carrying forward! Likewise, we encountered many difficulties and there are many things we could have done better.
In summary, what worked:
On an alt control game, start with calibration and tutorial right away, and abstract the system. Separate engineering and design work as much as possible as well, to allow for clearer boundaries and enable both teams to work without blocking each other.
Scope around available resources, and work with each individual engineer to ensure they’re working on things that interest them.
Maintain documentation and have transparency around processes that are not working.
Build debugging tools that support the needs of the team.
Find the fun!
In summary, what could have gone better:
Have good testing practices and come up with a review process before merging in code to main.
Optimize as you go. If there’s no immediate reason to not fix or optimize something right away, do it right away.
Establish a system for separating prototype code and functionality from the main code base.
Establish a clear build pipeline for testing environments.
