When testing systems that utilise environment variables at runtime, careful consideration needs to be given to the design of both the system, if governed, and the test suite to avoid unexpected and seemingly irreproducible runtime and assertion failures when these variables are used in parallel.

In our scenario, we require multiple test suites — each defined within a separate class — which tests some code that utilises a common set of environment variables.

Sounds simple enough. So what’s the problem? The goal of this article is to answer this question, dive a little deeper into the why, and equip you with the tools and knowledge necessary to tackle this issue when faced with similar if not identical situations.

The Problem

Consider the following two xUnit test suites. Each contains a single test that sets a common environment variable and — for demonstration purposes — asserts that our system under test has visibility of that set value:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

class TestSuiteA
{
    [Fact]
    void Should_See_Foo()
    {
        Environment.SetEnvironmentVariable("ENV_VAR", "Foo");
        var sut = new SystemUnderTest();
        Assert.Equal("Foo", sut.GetEnvironmentVariable("ENV_VAR"));
    }
}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

class TestSuiteB
{
    [Fact]
    void Should_See_Bar()
    {
        Environment.SetEnvironmentVariable("ENV_VAR", "Bar");
        var sut = new SystemUnderTest();
        Assert.Equal("Bar", sut.GetEnvironmentVariable("ENV_VAR"));
    }
}

While at first glance these tests may seem harmless and quite functional, at least one of these tests will fail due to the way xUnit parallelises its test collections.

By default, xUnit creates a test collection for each class within an assembly, and each of these test collections are run in parallel. However, this concurrency is achieved with the usage of multiple threads, not multiple processes. If you are unaware, .NET’s Environment.GetEnvironmentVariable will retrieve environment variables from the current process, hence our issue; each thread that the process creates will be accessing and modifying the same underlying environment in which our variables are stored.

To visualise our conundrum, see below a representation of scope between the Environment which holds our variables, the process which runs our test assembly, and the threads which run our test collections.

Environment scope

As can be seen, irrespective of the thread used, the same Environment will be utilised. Alas, we are left with a race condition when running our tests, which both attempt to read and write the same environment variable in parallel.

So how should we go about resolving this issue?

Solution A: Use Multiple Assemblies

One way we could solve our problem is by placing each of our test suites within a separate assembly.

While xUnit parallelises test collections within an assembly using threads, assemblies themselves are each executed under a separate process*.

* This can be dependent on the test runner. For more information see Running Tests in Parallel | xUnit.net.

This, in essence, achieves our desired process–level parallelism and associated environment isolation.

However, creating an entirely new assembly for each test suite is a lot of overhead — both performance and maintenance — as well as code duplication.

This is a band-aid fix. I wouldn’t personally recommend this solution.

Solution B: Serial Execution

Perhaps the most obvious solution is to simply disable parallelism altogether and run all of our tests serially, that is, one after the other.

While this does cause our above tests to pass, running all of our tests sequentially is not ideal; particularly if and when there are test collections that do not require such constraints and are perfectly capable of being run in parallel without consequence.

To alleviate the impact of serial execution, we can instead manipulate our test collections such that only a subset of tests are run sequentially — the lesser of evils — and the performance of all other tests can be maintained and appropriately run concurrently.

Utilising the [Collection] xUnit Attribute, we can place all test suites that modify environment variables under a single custom test collection such that they are run sequentially. As below:

1
2
3
4
5

[Collection("My Custom Collection")]
class TestSuiteA
{
    // ...
}

1
2
3
4
5

[Collection("My Custom Collection")]
class TestSuiteB
{
    // ...
}

Note that the use of hardcoded string literals is for demonstration purposes only, and I would highly recommend externalising your constants or otherwise providing a consistent static reference to be used as your collection identifiers.

However, there are yet additional concerns that need to be addressed. Our test suites — now running sequentially within a single test collection — still share the same process environment and, as a result, a given test’s environment may be “polluted” by the one or more tests that run before it.

For example, if we were to add the following additional assertion to the start of each of our example tests, at least one of the tests will fail (whichever test runs second):

1

Assert.IsNull(Environment.GetEnvironmentVariable("ENV_VAR"));

While somewhat of a non–issue for this example, given the fact that each of our tests explicitly set the associated environment variable before using it, consider the scenario in which a system under test references an environment variable which it does not set — a variable that may be unknowingly set by a test that runs first — and the runtime behaviour is modified. This unexpected and variable runtime behaviour is something we certainly want to avoid and can do so by having our tests clean up after themselves.

At a minimum, this can be achieved by simply reverting any modified environment variables to the values they contained (or did not contain) at the start of the test, as below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

[Fact]
void Test()
{
    var previousValue = Environment.GetEnvironmentVariable("ENV_VAR");
    Environment.SetEnvironmentVariable("ENV_VAR", "Baz");

    try
    {
        // perform the test using the environment variable ...
    }
    finally
    {
        Environment.SetEnvironmentVariable("ENV_VAR", previousValue);
    }
}

This will of course need to be done for all environment variables that are modified during the test. I recommend developing an IDisposable abstraction around this concept to reduce code duplication and room for error. You can find an example of one here.

With this additional cleanup in place, we have now not only resolved the issues in running our tests but have also achieved a level of environment “isolation” between them. However, we’re still losing a lot of performance due to limiting — even partially — the ability for our tests to run concurrently.

Solution C: Dependency Inversion

If you govern the system under test and can modify its source code — which is more than likely the case — implementing a layer of abstraction around its environment variable access may just be the best solution overall.

The concept of decoupling high-level components from low-level implementations is known as Dependency Inversion (see Microsoft’s documentation), and there’s a good reason it’s one of the five pillars of the broadly-known SOLID design principles; when adhering to this pattern, software systems become more modular, maintainable, extensible and testable.

Currently, our system under test is depending directly on the concrete methods provided via the static System.Environment class. We can represent this dependency with the following simple diagram:

UML Diagram A

Making use of the dependency inversion principle, we can implement a layer of abstraction and refactor our design as follows:

UML Diagram B

If the system under test is refactored to depend only on the interface, the implementation can be substituted as we see fit. A default implementation can be provided to preserve the existing runtime requirements, and a stub or mock implementation can be created and used during our tests.

Note that this will require a suitable level of support within the system under test for dependency injection (see Microsoft’s documentation). In our case, this could be as simple as passing the implementation via the system under test’s class constructor.

Consider the following simple interface which we could utilise:

1
2
3
4

interface IEnvironmentVariableProvider
{
    string? Get(string variable);
}

While some additional code excerpts have been omitted here for brevity (I’m going to assume you have a basic understanding of interfaces, implementations, and dependency injection), after refactoring our system under test, we can update our tests to make use of a stub in-memory environment variable provider — which implements our new interface — as below:

Full implementation details can be found within this article’s associated GitHub repository here.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

class TestSuiteA
{
    [Fact]
    void Should_See_Foo()
    {
        IEnvironmentVariableProvider environmentVariables = new InMemoryEnvironmentVariableProvider(new()
        {
            new("ENV_VAR", "Foo"),
        });

        // provide the stub via constructor-level dependency injection
        var sut = new SystemUnderTest(environmentVariables);
        Assert.Equal("Foo", sut.GetEnvironmentVariable("ENV_VAR"));
    }
}

And likewise for our TestSuiteB, replacing "Foo" with "Bar".

With this stub in place, our tests are now fully isolated. There are no dependencies on concrete external components and our tests are capable of being run with full parallelism without any risk of environment variable collision or pollution.

Note however that we are now utilising the system under test in a way that is discrepant from how it would be used during normal operation. As such, we need to ensure we create suitable tests for the default implementation of our new interface, and could even go so far as to assert that this implementation is utilised when initialising the system for regular use.

I believe dependency inversion to be the best solution to our problem. Implementing an additional layer of abstraction provides us with immense flexibility, not only when it comes to writing automated tests.

Honorable Mention: Shims

There exists one more solution approach worth mentioning, that being the utilisation of shims.

A shim is essentially a transparent middleware that can be implicitly injected into the context of a running application, such that it can be utilised in place of another dependency.

Consider the layer of abstraction that we introduced in the previous solution, a shim could be used instead of these additional components and no modifications would need to be introduced to the system under test; we can directly override the static System.Environment methods.

Sounds great, right?

Unfortunately, standard testing frameworks (including xUnit) do not support shims and you may find it difficult to find libraries that do; Pose was once promising however it has not been updated in some time and does not support the latest versions of .NET.

Currently, the most prominent way to use shims in .NET is via the Microsoft Fakes Framework — however, it requires Visual Studio Enterprise and only supports MSTest testing projects.

While shims can be a powerful method of test isolation — especially if you cannot modify the system under test — I could not find a viable method of utilising them for our scenario. Additionally, their necessity can generally be avoided through the use of dependency inversion.

Conclusion

While there are of course other solutions to this problem — including the usage of Mutexes or the manual creation of sub–processes to achieve isolation (consider the Tmds.ExecFunction library) — these solutions stray further from what I would consider good practice and can start to become quite convoluted.

I would also strongly suggest considering for your use case whether it would make sense to move away from the direct consumption of environment variables throughout your system under test, instead opting for a more traditional .NET Configuration and Options pattern approach; a robust, well-documented and common solution which lends itself to the previously discussed dependency inversion principle, among others.

You can find complete examples and implementation details of solutions B and C within the GitHub repository linked below:

Epilogue: A Word From the Author

If you’ve made it this far, I thank you. This is the first blog post/article I have published (since my university days) and while the topic of discussion is perhaps not as grandiose as I had originally imagined, I’ve wanted to create more analytical written content like this for a very long time, and I hope it fans the flame for more to come. Stay tuned.

- Jacob

These were my thoughts.