Testing Terms
To be able to correct a defect or bug, it must be localized in the software. First, we only know the effect of a defect but not the precise location in the software. The localization and the correction of defects are the job of the software developer and are called debugging. Repairing a defect generally increases the quality of the product, provided that in most cases no new defects are introduced. Debugging is often equated with testing. But testing and debugging are totally different activities.
Debugging has the task of localizing and correcting faults. The goal of testing is the (more or less systematic) detection of failures (that indicate the presence of defects).
Every execution (even using more or less random samples) of a test object in order to examine it is testing. The conditions for the test must be defined. The actual and the expected behavior of the test object must be compared.
Testing software has different purposes:
- Executing a program in order to find failures Executing a program in order to measure quality Executing a program in order to provide confidence Analyzing a program or its documentation in order to prevent defects
The whole process of systematically executing programs to demonstrate the correct implementation of the requirements, to increase confidence and detect failures is called test. In addition, test includes static methods, i. e. static analysis of software products using tools, as well as document reviews.
Besides the execution of the test object with test data, the planning, design, implementation and analysis of the test (test management) are also part of the test process. A test run or test suite consists of the execution of one or more test cases.
Условия тестирования
Для того чтобы исправить дефект или баг, он должен быть расположен в програме. Сначала, мы только знаем результат дефекта но не точное размещение в программе. Расположение и исправление дефектов, это работа разрабодчика ПО, которая называется debugging. Исправление дефекта обычно повышает качество продукта, при условии, что в представленных случаях нет новых дефектов. Отладка часто приравнивается к тестированию. Но тестирование и отладка совершенно разные виды деятельности.
У отладки есть задача локализации и исправлению недостатков. Цель тестирования (более или менее систематически) выявить неудачу (которая показывает присутствие дефектов).
Каждое выполнение (даже использование более или менее случайных примеров) обьекта тестирования в порядке проверки этого тестирования. Условие теста должны быть определенны. Актуальное и ожидаемое поведение тестируемого обьекта должено быть сверенно.
Тестирование ПО имеет различные цели:
- Выполнение прораммы с целью поиска ошибок Выполнение программы с целью определения качества Выполнение программы с целью обеспечения конфиденциальности Анализорование программы или ее документации для предотвращения дефектов.
Весь процесс систематического выполнения программы для демонстрации правильного выполнения требований, увеличение уверенности и нахождения ошибок называемый тестом. В дополнении, тест содержит статические методы такие как статический анализ использования инструментов ПО, а также просмотр документации.
В сравнении с выполнением обьекта тестирования с тестовыми данными, планирование, разрабатывание, выполнение и анализ теста (управление тестом) также часть процесса теста. Выполнение теста или тестовый набор состоит из выполнения одного или нескольких тестовых случаев.
A test case contains defined test conditions (mostly the requirements for execution), the inputs and the expected outputs or the expected behavior of the test object. A test case should have a high probability to reveal hitherto unknown faults.
Several test cases can often be combined to test scenarios, whereby the result of one test case is used as the starting point for the next test case. For example, a test scenario for a database application can contain one test case, which writes a date into the database, another test case, which manipulates that date and a third test case, which reads the manipulated date out of the database and deletes it. Then all the three test cases will be executed, one after another, all in a row.
At present, there is no known bug free software system and there will not be in the near future, once the system has nontrivial complexity. Often the reason for a fault is that certain exceptional cases were not considered during development as well as during testing of the software. Be it the incorrectly calculated leap year or the not considered boundary condition for the timely response or the needed resources. On the other hand there are many software systems in many different fields that operate reliably, day in and out.
Even if all the executed test cases do not reveal any further failures, we cannot conclude with complete safety (except for very small programs) that there do not exist further faults.
There are many confusing terms for different kinds of software testing tasks. Some will be explained later within the description of the different test levels (see chapter 3). This excursion is supposed to explain some of the different terms. It is helpful to differentiate these categories of testing terms:
- Test objective or test type: Calling a kind of test by its purpose (e. g. load test). Test technique: The test is named using the name of the technique used for the specification or execution of the test (e. g. business-process-based test or boundary value test) Test object: The test is named after the kind of the test object to be tested (e. g. GUI test or DB test (data base test)). Testing level: The test is named after the level or phase of the underlying life cycle model (e. g. system test). Test person: The test is named after the person subgroup executing the tests (e. g. developer test, user acceptance test). Test extent: The test is named after the level of extent (e. g. partial regression test).
Thus, not every term means a new or different kind of testing. In fact, only one of the aspects is pushed to the fore. It depends on the perspective we use when we look at the actual test.
Software Quality
Testing of software contributes to improvement of the software quality. This is done through identifying defects and their following correction by debugging. But testing is also measurement of software quality. If the test cases are a reasonable sample of software use, quality experienced by the user should not be too different from quality experienced during testing.
But software quality entails more than just the elimination of failures that occurred during the testing. According to the ISO/IEC-Standard 9126-1 the following factors belong to software quality: functionality, reliability, usability, efficiency, maintain-ability and portability.
All these factors or quality characteristics (also: quality attribute) have to be considered while testing, in order to judge the overall quality of a software product. It should be defined in advance which quality level the test object is supposed to show for each characteristic. The achievement of these requirements must then be examined with capable tests.
Example
In the example of the VSR-System, the customer must define which of the quality characteristics are most important to her. Those have to be implemented and examined in the system. The characteristics functionality, reliability and usability are very important for the car manufacturer. The system must reliably provide the required functionality. Beyond that, it must be easy to use, so that the different car dealers can use it without any problems in everyday life. These quality characteristics should be especially well tested in the product.
We discuss the individual quality characteristics of ISO/IEC-Standard 9126-1 shortly in the following. Functionality contains all characteristics, which describe the required capabilities of the system. The capabilities are usually described by a specific input/output behavior and/or an appropriate reaction to an input. The goal of the test is to prove that every single required capability in the system was implemented in the specified way. According to ISO/IEC-Standard 9126-1, the functionality characteristic contains the subcharacteristics: adequacy, interoperability, correctness and security.
An appropriate solution is given if all required capabilities exist in the system and they work adequately. Thereby it is clearly important to pay attention to, and thus to examine during testing, that the system generates the right or specified outputs or effects.
Software systems must interoperate with other systems, or at least with the operating system. (Unless the operating system is the test object itself.) Interoperability describes the cooperation between the system to be tested and the previously existing system. Trouble with this cooperation should be detected by the test.
Part of functionality is the fulfillment of application specific standards, agreements or legal requirements and similar regulations. Many Applications give a high importance to the aspects of access and data security. It must be proven that an unauthorized access to applications and data, both accidentally and intentionally, will be prevented.
Reliability describes the ability of a system to keep functioning under specific use over a specific period. In the standard, the quality characteristic is split into maturity, fault tolerance and recoverability.
Maturity means how often a failure of the software occurs as a result of defects in the software.
Fault tolerance is the capability of the software product to maintain a specified level of performance or recover from faults in cases of software faults or of infringement of its specified interface.
Recoverability is the capability of the software product to reestablish a specified level of performance and recover the data directly affected in the case of a failure. Following a failure, a software product will sometimes be "down" for a certain period of time, the length of which is assessed by its recoverability. The ease of recovery and the required work should also be assessed.
Usability is very important for interactive software systems. Users will not accept a system that is hard to use. How big is the expense that is required for the usage of the software, for the different user groups? Understandability, ease of learning, operability and attractiveness, as well as compliance to standards, conventions, style guides or user interface regulations are partial aspects of usability. Those quality characteristics are examined in non-functional tests.
The test for efficiency measures the required time and consumption of resources for the fulfillment of tasks. Resources may include other software products, the software and hardware configuration of the system, and materials (e. g. print paper, network, and storage ).
