When some consider early testing they may then think that unit testing their entire code will solve all their problems and disregard higher levels of testing. Although more unit testing does pick up defects early, the proper and essential place of System Testing and of other higher levels of testing should be understood and not neglected.

So what exactly is System Testing? System Testing tests that a particular system (composed of many integrated units) performs what the business intended it to perform. It is a kind of black box testing, requiring no understanding of the inner workings of the system. It has many aspects including both functional and nonfunctional requirements, covering aspects such as:

  • The system’s own internal functions according to requirementsEinsteinSystemTesting
  • How this system interacts with other systems
  • How it performs under pressure
  • Scalability
  • Efficiency
  • Reliability
  • Compliance to standards

There are different levels of systems depending on the work at hand and what stage of testing is required (eg. for a robotic arm – you would consider a finger a system of joints, a hand, a system of fingers, and an arm, an even greater system). Although developers should be writing unit tests, people with an independent critical mind are necessary at this stage to point out defects in the system as a whole.

System testing’s importance

Those examining the system should understand clearly how the system under test will be used and through analysis identify shortages in the requirements that may have led to unit tests being passed yet an incorrect system being produced. In my opinion they should be given as much insight as the Business Analysts who define the requirements. By investigating different functions of the system, what it is meant to do and (on a high level) how it is meant to do it, requirement defects can be detected early. For example, in my previous client I had a system that would create and prepare some data in a certain state and then modify it and inquire its status. I noticed there was a field that was inquired of that was never being set by a create or modify call and so could never be expected to return a value. This field happened to be a very important field that was just missed out in the creation requirement documents and since unit testing was based on these requirements, it was never picked up.

How do the various types of testing fit together? To illustrate System Testing’s position within early testing, I’ll use the rear wheel of a bike. Five main parts of a bikes rear wheel are its hub, spokes, rim, tyre and the air valve used to blow up the tyre. This illustrates a system where there are units joined at different parts, with one unit (the spokes) – being reused multiple times, and with a simple purpose of supporting a bike’s weight and converting the rotational motion of the gear to a linear motion on the ground.

Considering this picture it is easy to comprehend the benefits of testing at different levels. These levels would include testing:

  • Units in isolationSystemTestingWheel
    • Spokes can be tested independently for their capability in bearing different kinds of forces before they’re put together into the wheel (in which case testing would be more expensive)
    • The hub could be tested to ensure it is the correct size to join onto the main frame of the bike
  • The combination between just a few units
    • The air valve could be tested in its ability to blow up the tyre without leaking air (without connecting the tyre to the rim and the rest of the wheel)
    • The hub, spokes and rim could be tested independently of the system to ensure they fit together
  • The system (entire wheel) itself as a black box, with all its units together
    • Can the system sustain a certain weight on the hub while moving on a rough texture
    • Does the wheel’s size match the required size
    • How much slippage occurs on smooth surfaces
    • Does the wheel meet safety standards
    • Would mechanics know how to fix such a wheel if something goes wrong

 

Hopefully, from this picture you can see that although unit testing verifies that smaller parts meet requirements, some things can only be tested adequately when they’re integrated into a system. Also, a high level view of a system is often needed to sort out defects in the requirements themselves (from which unit tests are based). Therefore System testing must not be eliminated or minimized when trying to shift left and test early. Furthermore, those with a critical mind who are responsible for ensuring quality should be involved in analyzing the system during the requirement development phase (so that defects can be found out earlier), rather than included after units have been developed to meet the system requirements.