Managing software environments towards success

When I first started as a professional software developer, I was far from professional. There's a stack of concepts and approaches that weren't taught when I was at Uni, but were actually critical to being able to do my day-to-day job well. The current proliferation of "learn to code from scratch" courses online are doing an awesome job of democratising the ability to write code, but being able to write code doesn't make you a professional, as I discovered.

The gap between being able to write code and being able to develop software professionally is evidenced by the rise of groups like JuniorDev. Their meetups and workshops are a response to a large proportion of new developers who have ventured into their first jobs, only to discover there's a hell of a lot more involved in being a professional developer than just churning out code. They feel inequipped, not fully prepared for the real world.

One of the big things I remember being completely unaware of before starting my first grad developer role was test environments. An environment is a full, separate copy of all the systems and infrastructure required to run the application we're developing. That can range from something as simple as a web server in a stateless app, to all of the databases, mainframe core-banking systems and external credit verification systems required to run a loan application system.

Typically, software organisations will have a number of different environments used for different purposes. They might look something like this:

EnvironmentPurposeOwner(s)
"Local"The developer's local machine, used for testing features in progressDeveloper
"Development"First place dev code is integrated with other changes in progress, to verify the world doesn't breakDevelopment team
"Test", or
"User acceptance testing", or
"UAT"
Used to ensure features actually work as intended from a user/business perspectiveTest team, or Product owner
"Staging"Exactly the same/as similar to production as possible, to verify the release before deployingRelease team, or Operations team
"Production", or
"Live"
The real deal - this is where the system is used by real users"The business", or Operations team, or Everyone

Some teams have more environments, some teams have less. They might go by slightly different names, but they are usually organised around a similar structure.

Typically, a change will be progressively tested in more and more depth as it moves through the environments, until (all being well) finally being released to production. Each environment allows a separate piece of work to be developed and tested without worrying about conflicts with other features. From that perspective, environments can also be a limiter of work-in-progress: the amount of work that can safely and confidently be tested at one time depends on a clean, conflict-free place to test each piece in isolation.

The approaches used for building environments have changed dramatically in the last few years with the rise of virtualisation and containerisation. Previously, provisioning a new environment could be a 6+ month effort for a team of 20 or more people, full-time. The rise of cloud environments and infrastructure-as-code can mean that upfront effort investment is half a day to declaratively specify what an environment should contain - after that, unlimited environments can be spun up in minutes, like cutting cookies with a cookie cutter.

Differences between test environments and production can cause bugs to lurk in hidden corners.

If we consider the banking scenario mentioned above, it could be very expensive to have a true replica of production in all environments, so often test environments aren't 100% realistic. There's danger here: differences between test environments and production can cause bugs to lurk in hidden corners, dramatically increasing the cost to solve them as they are discovered late in the test cycle (or even after going live). To avoid this, your environments should be as close to production-like as possible. Even in the banking space, this is not impossible. ING led the way in Australia by virtualising their whole infrastructure to ensure environment creation is as cheap and fast as possible. A massive project, and one not lightly undertaken, but the ethos was right: removing the limitation of fixed environments and enabling rapid environment provisioning as realistic to production as possible.

So how can we use some of these ideas in our not-so-massively-enterprisey development teams to ensure our environments aid rather than hinder our development efforts? Here's some ideas:

  • Look at all the differences between your environments and production. Question every single one of them. Why do they exist? Are they necessary, accidental, or perhaps just a legacy of a time gone past?
  • Have a look at configuration files too: are there configuration items specific to each environment which could be a bug waiting to happen? How do you manage your configuration across environments? Is it controlled or ad-hoc? Do you have any idea of the history of your config values?
  • How are your environments provisioned? Could you use infrastructure-as-code approaches to automate environment provisioning? Imagine the impact: every developer and tester gets their own personal environment just by running a script.
  • How do you manage and control software releases between environments? Are the processes automated or manual? Are there parts of your deployment processes which open you to risk of human error? Could you automate more of it for more consistent, reliable results?

There's so many ways we can do better than the old world of massive, expensive, fixed-hardware environments. As developers, our purpose should be to facilitate rapid experimentation on providing value to our customers. Does your environment set-up move you closer to, or further from this goal?

Header photo by Caroline Methot on Unsplash