DevLog 116: Airport CEO Early Access One Year Anniversary!

DevLog 116: Airport CEO Early Access One Year Anniversary!
September 28, 2018 Airport CEO

Salutations and celebrations, airport CEO! One year ago on the 28th of September 2017 we released Airport CEO into early access on Steam and it did, in fact, change our lives. If Airport CEO hadn’t been released with the success it did we would not sit in our new office in downtown Malmö, sipping on a celebratory beer and writing this development blog; instead we would most likely have been off in some generic IT job longing for the week to end. As of today, Alpha 28.4 (the One Year Anniversary Update) with remote stands and the one year anniversary surprise, a huge set of all new HQ aircraft engine sounds (more about these further down), is available on the default branch and we’ve never had so many interesting upcoming features and improvements in the near future as we do right now. Looking back at the first year with Airport CEO in early access we can establish with certainty that it’s been a year of challenges. Developing games is difficult and developing an airport simulation game as your first ever game title with no real previous game development experience is… a huge task – and there’s been many things we’ve had to sacrifice. Hobbies, weekends, time with family and friends and perhaps most of all sleep. But they payoffs for these sacrifices have been beyond anything we’ve ever could have dreamt about and we are extremely humbled by the community, you people, who have enabled us to move forward with this creation and – hopefully – many future tycoon titles ahead. Much like the inhabitants of a country chooses their politicians to lead their country, you have in a way, by supporting us, chose us to develop an airport tycoon game that fits your dreams and desires and for this we will always be grateful… and as a token of that gratefulness we will make sure to deliver that game. Airport CEO will remain in early access for at least another year as we continue development focusing on major performance improvements and major feature additions. As we said, we have a lot of exciting upcoming new releases and we’re so much looking forward to top this year of early access with the next.

A Year of Project Management, Issue Tracking and VCS

One of the things that’s perhaps seen the most development over the recent year but is not that noticeable by you as a player is our internal tooling and way of managing this project. We started out with a very basic Jira setup that required manual bug submission and largely synchronized our development using Dropbox (yes… yes… we know… no comments please). Today that Dropbox solution has been replaced with a dedicated git repository (running on Bitbucket) which is fully integrated with our Jira solution, a Jira solution that now can receive high quality bug reports where hardware information, error log file and save file collection is fully automated and, using a Unity editor plugin, is made available immediately in our work environment. All issue tracking and project management from a development perspective is handled in Jira using a combination of a Kanban and Scrum board, depending on if we’re working on single bug reports or issues that are a part of a larger epic or sprint. The Jira solution is more or less made public so that any airport CEO can check up on what we’re currently working on and, if that same airport CEO just wants a brief overview of what’s up next for the experimental branch, they can pop into one of our public Trello boards that sync automatically with our Jira boards every five minutes.

We’re today, actually just as recent as of a few weeks, finally working with dedicated branches for new features and this is something we’re, with the deployment of Alpha 28, are going to do a lot more in the future. It means that we will now be able to maintain and fix critical bugs on the default branch without compromising it with experimental implementations. Using git branches we will be able to swiftly switch between the current code bases of the default branch, the next version of the experimental branch as well as any other kind of large differentiating fundamental performance improvement that may impact development in general.

But it’s not all smiles and giggles. A drawback we’ve noticed by having an in-game bug reporting tool is that so easy to report bugs with, while the quality of each bug report is through the roof due to automated data collection, the sheer volume of reported bugs has put a very large strain on the performance of our Jira instance. Over the summer break we had a lot of players who reported a lot of bugs and when we came back in the beginning of autumn we faced a backlog that we usually never let cross 200 bugs to be backed with over 1500 issues. While we started trying to process all of these bugs, the volume made any browser too sluggish and it started consuming way too much time. While we realize that this is not the way Jira is intended to be used (in fact, Jira is not at all meant to be as public as ours is) we still want to use it this way but we had to do something about the bug volume. The solution was to move all of the default reported bugs to a “solved” state so that they would not be visible in the open backlog but still remain in our issue environment. No, these bug reports will not be forgotten and we’ve already made extensive analysis of their contents and are addressing them as expected – however due to the volume of those bugs and the fact that we are still only two developers working on this project means that we’ve made a decision to, instead of putting time into processing those bugs as expected by our Jira workflow, we instead put that time into actual development and bug solving.

We’re not yet done with our Airport CEO tooling setup. While we’ve come a far way there’s still a few more things we’d like to get in and next up is a more interactive start menu. We want you to, when you start a new version of the game, be immediately informed of what’s new in this version and we want that to be automated.

Transparency is really at the core of how we develop this game. Being a new studio without any credibility and developing an early access game we want you to have full insight into the development of this game. To access all of our resources you can learn more over here and if you have any ideas or suggestions on how we can make this process more transparent and collaborative moving forward we’d be super happy to hear them. Just hit us up on the Forum, Twitter or whatever!

Alpha 28.4 Released

Alpha 28.4 has finally been deployed to the default branch and with it brings a lot of new minor features, important bug fixes and performance improvements and perhaps most noticeably: Remote stands! We’ve written about remote stands multiple times before and have more or less exhausted the topic but if you’re an airport CEO that don’t frequent the experimental branch you’ll be all new to this deployment.

The remote stands is the first feature request to implemented and catering and cleaning services will follow shortly. A lot of the foundation for those additional requests have been laid with the implementation of the systems for remote stands and remote stands are, or were, without a doubt the more difficult one of the feature requests up for implementation. This is not to mention multiple floors but as we’ve found out, this feature request has had to be paused while we overhaul person movement and rendering (more on this below).

We’ve spent the past months exhausting this topic so for once we won’t go into dwelling on the details. As for remote stands, you can read up on the implementation and change log in this forum thread or over at Steam and we’ve also added a tutorial segment for you to get going with it immediately (international airport, last step).

We will be monitoring the health of Alpha 28.4 throughout the weekend and next week and deploy hot-fixes if required.

New Aircraft Audio (Anniversary Surprise)

We promised you a one year anniversary secret feature and… we can now happily announce that the entire aircraft soundscape of Airport CEO has been overhauled completely. Alpha 28 features a full range brand-new set of uniquely recorded aircraft sound effects, specific to aircraft types and engines! This means no more generic jet and propeller sounds but instead actual, professional high-quality audio recordings from the real world aircraft model. The sounds we’ve acquired and implemented are among the best you can get and are usually featured in DLCs for X-Plane and MS Flight Simulator. Each aircraft in the game now has a set of audio sources that interact at various levels and pitches depending on the aircraft’s current specific operational state, i.e. when taking off, taxiing or spooling up or down engines. To accompany these new authentic sound we have polished audio levels, spending time on creating realistic engine throttle effect and increased the ambient soundscape to make it feel more like a real airport. Of course, there is more we can and will do as we recognize that there is still more to do in terms of other sounds sources of the game such as UI and terminal ambiance but sorting out the aircraft sounds was something we felt was of utmost importance and we believe that this is really a great step forward. All in all, we have obtained 20 unique aircraft sound packs consisting of several different type of engine sounds, securing accurate sound implementation of future aircraft models. We hope that you will enjoy this one year anniversary surprise and the best way to experience it is simply to hook up your computer to a great audio source, boost the bass and zoom in close on aircraft as they move about your terminal.

Also… A New Aircraft!

As an additional one year anniversary surprise we’ve taken the opportunity to add a new aircraft! Let’s welcome a new member of the Airbus family: the A319 which will  initially join the Allure, Wildcat Air, Zoom and Crown Airlines families. As you can see it is very similar to the A320, just a little bit shorter.

Upcoming Passenger Movement and Rendering Overhaul

Over the course of the year of various improvements we’ve always known that passenger and employees have been a major source of discussion around performance. When we, one year ago, first released the game, there were significant parts of the code which were very un-optimized and contributed to low FPS once your passenger count was creeping into the thousands. Since then we have implemented a highly performant multi-threaded path finding system and done countless of script and behavior performance improvements. We know that there are still a few areas of the code base where we can improve by using a kind of component system to manage large processes such as movement (which today is live on aircraft) and while this is expected to give you a few FPS it will take some time to implement as a lot of the code needs to be rewritten to be compatible with the new architecture. Our goal has always been to be able to simulate 10000 passengers at a decent FPS on an average machine and we think this performance is somewhat required before we fully implement the larger wide body aircraft in Airport CEO, as well as for example multi terminal. However, with the current agent simulation system in terms of both movement simulation and rendering we today cannot simulate even 5000 passengers at a desirable FPS and it’s all due to the fact that our current implementation is too poor.

Another area which we have not done that much in terms of optimization is when it comes to the rendering of agents. We have done significant rendering improvements to tiles such as taxiways, floors and roads but not so much on persons. One reason is that the current person system uses a combination of sprites (head, torso, arms, hands, legs etc.) which all animates to create the various person movements. It’s apparent that we need a new solution and that we, if possible, would like to keep this kind of detail and all while not falling back to flat 2D agents like in Prison Architect; we want to do something different. Therefore, we have spent a lot of time these past weeks investigating different alternatives.

In short, we have found that two things here are costly. The animation system in Unity is not really designed to be used on 10000 objects as it quickly drains the CPU with various processes out of our control. The solution will be to do our own animation system that just meet our requirements, without all the fancy and unnecessary stuff.

Secondly, the number of sprite renders in the scene significantly contributes to reduces performance. Testing has shown that Unity does a lot of background work to figure out if an object is in view, have moved, is behind another object, being lit from a light source and so forth. All these processes also increase CPU load and is not really within our control either. Therefore, we are left with the option to simply bypass all Unity processes and write scripts that directly sends draw instructions to the GPU. The trick here is that we still want to do animations and do individual coloring of the agents. By basically studying and testing various concepts, we have found a solution that performs outstandingly compared to out current implementation. In the below screenshot we render 20000 agents, all individually colored, animated and moved with above 60 FPS in a build on our high-end machines. With the current system we would be surprised if we can even render that many agents at 5 FPS, so you can see that this is really huge difference and, if implemented to our master branch, nothing short of a game changer. Note that this world does not include all other background processes and rendering of a real game, in a real world we could maybe expect something like 30 FPS on our high-end machine but this has not been tested yet. However, this is still a fantastic result and is definitely the way forward. Looking at implementing this into the actual game, there are still many challenges that comes with converting the current system with the new rendering and animation. Therefore, it is unclear when this will go live and no time frame will be given at this point. However, given the promise of this new solution we see it as a prerequisite before moving ahead with large aircraft and multiple floors.

We will definitely keep you updated on this exciting improvement.

In Other News

… we have a lot of exciting side projects ahead! We’re currently in the process of dealing for a new website implementation of Airport CEO as well as Apoapsis Studios. It probably took you several minutes to load this website and the end goal with that project is to have it loaded for you in an instance.

On the soundtrack front we’re currently making great progress. If you don’t know we’ll be glad to iterate that Airport CEO will soon feature a 60 minute full length soundtrack, crafted by Sinephony in Stockholm. All of the componsitions are completed and we’ve transitioned to the recording phase where a total of eight musicians, not including the composers (who also play instruments) will be recorded exclusively for this soundtrack.

Also, during this week we have finally switched office within the city of Malmö! Our first year was spent at a startup incubator where we attended a special incubator program but before the summer start we were contacted about a new office which would be exclusively for game studios, located in the city center. The office was bigger, newer and would give us an opportunity to further establish our studio in the Malmö game development community. It was not hard for us to accept the offer and so we moved in this week. The new office will be a better place for us to grow and meet fellow game developers and perhaps even our future colleague.

Yes! You read that right. We’re hiring. We need to speed up the development of Airport CEO and finally have enough space to house another developer. We have absolutely no idea of how to recruit someone so this will definitely be an interesting and valuable experience. We’ll make sure to keep you update along the way.

So that’s it for this, one year anniversary, special development blog. Olof and Fredrik, us, the developers of Airport CEO would like to wholeheartedly thank you for sticking with us, supporting us, bug reporting to us and playing Airport CEO. You’ve enabled us to work with something that we’d never expect to be able to and in return we want to create the airport tycoon game of your dreams. So from all of us here at Apoapsis to you… a big, fat, thanks. And throughout this coming years, always fly safe!