Black-box testing

Black-box tests can be designed based on a specification or a known model. In this case, the test base only shows what the software does, while how it does it remains hidden. Black-box tests are suitable for both functional and non-functional testing. For non-functional tests, the quality model ISO/IEC 25010) provides guidance.

Black-box techniques are the following:

• Equivalence partitioning
• Boundary value analysis
• Decision table-based testing
• State transition testing
• Use case-based testing

Equivalence partitioning

During equivalence partitioning, test data are divided into classes that result in identical behavior in the program.

• From the same partition we test only one item, since the elements of the partition are equivalent.
• Partitions are disjoint, which follows from the previous point. (The equivalence relation creates a classification.)
• Invalid partitions must be tested individually. An invalid partition is a class that contains no test data the program would accept, or contains no test data the program would reject.
• Partitioning can be refined.

Partitions can be created according to input data, according to the data received as output, based on the program’s internal values, by time, with respect to interfaces, based on parameter values, etc.

Note: When testing invalid partitions, the program should provide some kind of error message.

Example

As an example, consider the following program that expects an integer as input. The program returns a correct result if the input is positive and less than 10,000, or negative and greater than -10,000. During equivalence partitioning, valid input data are divided into two classes: the valid positive and the valid negative partition. Invalid input data are divided into three classes: the invalid positive, the invalid negative partition, and the singleton set containing zero.

Eqquivalent class	Representatives
Valid positive	1,2,3,4,5,6,7,8,9
Valid negative	-1,-2,-3,-4,-5,-6,-7,-8,-9
invalid positive	10000,10001,10002,...
Invalid negative	-10000,-10001,-10002,...
Invalid (0)	0

Boundary value analysis

Boundary value analysis (BVA) is a technique based on testing the boundaries of equivalence partitions. Therefore, it can only be applied to ordered partitions. The minimum and maximum values of a partition are the boundary values. In boundary value analysis, if two elements belong to the same partition, every element between them must also belong to the same partition.

• Errors often occur at edge cases.
• These are often linked to characteristic program structures (e.g., branches). Examples include swapping relations, or starting a loop from the wrong index.
• We use it together with equivalence partitioning.
• It can be two-valued or three-valued.
• For boundary tests, the smallest distance between neighboring elements must be known (precision specified).

Two-point boundary value analysis

In two-point boundary value analysis, each boundary has two coverage elements: the boundary itself and the nearest neighbor belonging to the adjacent partition. To achieve 100% coverage with two-point boundary analysis, the test cases must test every coverage element, i.e., every identified boundary. Coverage is defined as the ratio of the number of tested boundary values to the total number of identified boundary values, expressed as a percentage.

Three-point boundary value analysis

In three-point boundary value analysis, each boundary has three coverage elements: the boundary itself and both of its neighbors. Therefore, in three-point boundary analysis, some coverage elements may not be boundary values. To achieve 100% coverage with three-point boundary analysis, test cases must test every coverage element, i.e., every identified boundary and its neighbors. Coverage is defined as the ratio of the number of tested boundary values and neighbors to the total number of identified boundary values and neighbors, expressed as a percentage.

Two-point and three-point boundary value analysis

Three-point boundary value analysis is stricter than two-point, as it can find errors that two-point boundary value analysis misses. For example, if the statement if(x ≤ 10)... is incorrectly implemented as if (x = 10)..., test data derived from two-point boundary analysis (x = 10, x = 11) will not find the error. However, x = 9 derived from three-point boundary analysis likely will.

Example

Suppose we have a function that accepts a positive integer as a parameter, and if it is less than 100 it calls function A, otherwise it calls function B. In this example, the function input can be divided into two valid partitions. In one case [1,2,...,99], the other partition is [100, 101,...]. Our boundary is 100, and in the two-valued check, the first test case will be 99 from the first partition, while 100 from the second. However, we also have an invalid partition: zero and negative numbers. Around 0 we use three-point analysis, so the test cases can be -1, 0, and 1.

Test cases for the previous example

In the previous example, a three-point test for the valid partition can be: 99, 100, 101.

Decision table-based testing

Decision tables are used to test the implementation of system requirements that determine how different combinations of conditions produce different outputs. Decision tables can be effectively used to capture complex logic, e.g., business rules.

When creating a decision table, input data are organized into a table in which columns represent test cases, while rows contain the conditions associated with the test cases. The cells contain the specific test conditions for the test case. The table also includes an operations block, whose rows are the possible operations, while their intersection with the test cases contains the operations derived for that test case from the specific values of the test conditions.

Decision tables can be restricted, where every condition and operation is binary (true or false) except for irrelevant conditions or infeasible operations, for which special notation is used.

In the conditions block:

• T (True) indicates that the condition is satisfied.
• F (False) indicates that the condition is not satisfied.
• (blank cell) means that the condition’s truth value (whether true or false) is irrelevant to the outcome of the operation.
• N/A (Not Applicable) is used when the condition cannot logically occur in combination with the other conditions in the same row (i.e., it is infeasible given the context).

In the operations block:

• X indicates that the operation must be executed under the given set of conditions.
• – indicates that the operation is not executed under the given conditions, in accordance with the applicable business rules. A complete decision table has enough columns to cover all combinations of conditions. The table can be simplified by removing columns that contain infeasible combinations of conditions. The table can be minimized by merging columns in which some conditions do not affect the output.

Example

Consider an airline ticketing system where discount pricing needs to be tested. The following rules (test cases) must be verified:

• If the passenger is under two years old and travels within the state, they get an 80% discount.
• If the passenger is under two years old and travels on an international flight, the discount is 70%.
• Passengers aged 2–16 receive a 10% discount, but if they have an early booking, they get a 20% discount.
• Frequent flyers get a 15% discount, except in the case of early booking, in which case they also get a 20% discount.
• On international flights, traveling in off-season grants a 15% discount.

If multiple conditions are met, the passenger gets the larger discount. If the discount is the same, then either case can be chosen as appropriate.

To test the above test cases, we can prepare the following decision table:

Conditions	1	2	3	4	5	6	7
Under 2 years	T	T	F	F
2–16 years	F	F	T	T
Frequent flyer			F		T	T
Intra-state flight	T	F					F
Early booking			F	T	F	T
Off-season			F				T
Operations	1	2	3	4	5	6	7
10% discount	-	-	X	-	-	-	-
15% discount	-	-	-	-	X	-	X
20% discount	-	-	-	X	-	X	-
70% discount	-	X	-	-	-	-	-
80% discount	X	-	-	-	-	-	-

Decision tables can be created in Excel and Word as well. There are products that support this model, for exampla Visual Paradigm. An example for the above decision table:

State transition testing

State transition testing is a black-box testing technique in which changes in input conditions cause state changes or output changes in the application under test (AUT). This method helps analyze the behavior of the application under different input conditions and ensures that all possible system responses to valid and invalid transitions are verified.

In state transition testing, we describe the system using a state-transition model, where the system is represented as a finite-state machine. The transitions connecting the states are considered executable test steps. The model has four main components:

• States: The distinct, stable situations of the software (e.g., Open, Closed, Locked).
• Transitions: The changes from one state to another, typically caused by input or internal events.
• Events (or inputs): The triggers that cause a transition to occur (e.g., close file, enter password).
• Actions or (outputs): The activities or outputs produced as a result of a transition (e.g., display error message).

The mathematical model behind such testing is the sequential machine, which extends deterministic finite automata by allowing outputs and guards on transitions. For visualization and formalization, both state transition diagrams and state tables can be used:

• A state transition table lists all states and events, showing the resulting next state and output, and explicitly distinguishes between valid and invalid transitions.
• A state transition diagram visually depicts states as nodes and transitions as directed edges, labeled with triggering conditions and actions.

Test Derivation and Coverage Criteria:

Test cases are systematically derived by traversing the model according to specific coverage criteria that determine the completeness of testing. Common coverage rules include:

Coverage Criterion	Definition
State coverage	Each state is visited by at least one test case.
Transition coverage	Each valid transition is exercised at least once.
Transition-pair coverage	All consecutive transition pairs are covered (detects state sequencing errors).
Path coverage (k-length)	All possible paths up to length k are executed.
Condition (guard) coverage	Each possible true/false outcome of transition conditions is tested.
Invalid transition coverage	Inputs that should not cause a valid transition are tested to verify error handling.

Each test path through the model represents one test case, starting from an initial state and ending either in a terminal (final) state or after a defined sequence of transitions.

If certain transitions or states cannot be reached based on the current specification, these are marked as infeasible paths, and the responsible analyst or designer is notified that the specification may be incomplete or inconsistent. Unreachable states often reveal design issues, while missing transitions may indicate unhandled input conditions.

In large models, full path coverage may be infeasible. Therefore, equivalent paths or transitions can be merged or grouped using equivalence partitioning and boundary analysis, reducing the number of test cases without losing behavioral coverage.

Modern Model-Based Testing (MBT) tools (e.g., GraphWalker, ModelJUnit, Spec Explorer) can automatically generate executable test cases from the state model based on the selected coverage criterion. This ensures traceability between the specification, model, and generated tests, and allows automatic re-generation of test suites if the model changes.

Example

Consider an ATM system function where, if the user enters an invalid PIN three times, the account will be locked. If the PIN is correct on any attempt, the system grants access at that attempt. On the first and second attempts, if the PIN is incorrect, the user receives a warning.

States	Correct PIN entered	Incorrect PIN entered
A1 1st attempt	A4 / Display “Access granted”	A2 / Display Warning #1 (“Incorrect PIN, 2 attempts remaining”)
A2 2nd attempt	A4 / Display “Access granted”	A3 / Display Warning #2 (“Incorrect PIN, 1 attempt remaining”)
A3 3rd attempt	A4 / Display “Access granted”	A5 / Display “Account locked”; disable further input
A4 Access granted	– / No further input accepted	– / No further input accepted
A5 Account locked	– / Display “Account locked”; reject input	– / Display “Account locked”; reject input

Note: We start from state A0 (START), from which we automatically move to A1. A4 and A5 are terminal states. This can also be indicated with a state A6, to which the transition is also automatic.

State transition diagram for the example:

Test Case Set (Minimal for Full Transition Coverage):

ID	Objective	Precondition	Steps (Inputs)	Expected State Sequence	Expected Outputs / Notes	Pass Criteria
TC-01	Correct PIN entered on the 1st attempt	A0 → automatically → A1	enterPIN(correct)	A0 → A1 → A4	Immediate access granted; no warnings issued	A4 reached; access granted
TC-02	1st attempt wrong, 2nd attempt correct	A0 → A1	enterPIN(wrong) → enterPIN(correct)	A0 → A1 → A2 → A4	After 1st wrong PIN: Warning #1; 2nd attempt correct → access granted	Warning #1 displayed; A4 reached
TC-03	1st and 2nd attempts wrong, 3rd attempt correct	A0 → A1	enterPIN(wrong) → enterPIN(wrong) → enterPIN(correct)	A0 → A1 → A2 → A3 → A4	After 1st wrong: Warning #1; after 2nd wrong: Warning #2; 3rd correct → access granted	Both warnings displayed; A4 reached
TC-04	Three consecutive wrong attempts → account locked	A0 → A1	enterPIN(wrong) → enterPIN(wrong) → enterPIN(wrong)	A0 → A1 → A2 → A3 → A5	After 1st wrong: Warning #1; after 2nd wrong: Warning #2; after 3rd wrong: Account locked	A5 reached; lock message displayed
TC-05 (negative / robustness test)	Attempt to log in after account is locked	A5 (locked)	enterPIN(any)	A5 (no transition)	Message: “Account locked”; further attempts rejected	State unchanged (A5); operation denied

Use case testing based on a user story

Use case testing is a black-box test design technique that examines the system’s behavior from the user’s perspective. The basis of the method is that the tester analyzes typical interactions with the system—the so-called use cases—and then derives test cases from them. A use case typically describes how the user achieves a specific goal with the help of the system, as well as what alternative and exceptional flows may occur. The goal of testing is to verify that the system responds according to the specification along all possible paths, thereby ensuring the realization of user value.

In an agile development environment, traditional use case documentation is often replaced by the user story, which more concisely expresses the user’s goal and the expected outcome. In this case, use case testing is prepared based on the acceptance criteria associated with the user story: these define what the system must fulfill for the story to be considered “done.” The test cases are therefore directly derived from the user story, covering its main and secondary branches. This approach fits particularly well with collaborative testing, where developers, testers, and business stakeholders jointly identify key examples, behaviors, and exceptions before development begins. As a result, testing truly measures the business goals as interpreted by the user, rather than focusing solely on the technical implementation.

Example: Placing an order from the cart

User story: As a customer, I want to place an order for the items in my cart, apply a coupon code, and pay by credit card so that I receive confirmation and the goods arrive conveniently.

Acceptance criteria:

1. AC1 – Successful order: Valid cart, valid coupon (if any), successful payment → order ID and email confirmation.
2. AC2 – Invalid coupon: Error message, order can proceed without a coupon.
3. AC3 – Stock change before payment: If any item runs out → error message, order is not finalized.
4. AC4 – Failed payment: Error message, order is not created.
5. AC5 – Incomplete shipping details: Required field indicated; until filled, finalization is not allowed.

Use case (short description):

• Name: Placing an order
• Primary actor: Customer
• Supporting systems: Inventory system, Payment service provider, Email service
• Precondition: The customer has a cart with at least 1 in-stock item; shipping details can be filled in.

Basic (main) flow:

1. Customer opens the “Payment/Checkout” page.
2. System checks inventory for every item.
3. (Optional) Customer enters a coupon code; system validates it.
4. Customer provides/verifies shipping and billing information.
5. Customer selects the payment method and pays.
6. Payment succeeds → the system creates the order, reserves/deducts inventory, sends an email confirmation, and displays the order ID.

Alternative/exception flows:

• A1 (AC2): Coupon invalid → error message, customer decides whether to continue without the coupon.
• E1 (AC3): Out-of-stock during the check → error message, order not finalized, cart updated.
• E2 (AC4): Payment failed/declined → error message, order not created.
• E3 (AC5): Incomplete shipping information → required fields highlighted, submission not allowed.

Test cases:

TC ID	Source	Flow	Steps (brief)	Expected result	Test data
TC-01	AC1	Main	Open cart → Stock OK → Coupon empty → Fill details → Card payment successful	Order ID displayed, email confirmation sent	In-stock SKUs
TC-02	AC1	Main	Like TC-01, but with valid coupon (-10%)	Discount applied, order created, email sent	Coupon: SAVE10
TC-03	AC2	Alt	Enter coupon: invalid → error → remove coupon → payment successful	Order created without coupon, email sent	Coupon: BADCODE
TC-04	AC3	Except	Checkout → stock check → meanwhile 1 item runs out	Error shown, order not created, cart updated	SKU running out
TC-05	AC4	Except	Start payment → payment service returns “declined”	Error shown, order not created	Test card “declined”
TC-06	AC5	Except	Shipping postal code empty → submit order	Required field indicated, cannot proceed	Empty/invalid zip
TC-07	AC1	Main	Coupon + choose among multiple addresses + successful payment	Order created, email sent, coupon applied	2 saved addresses

Gherkin code for the test cases:

Feature: Checkout order placement
    In order to receive the products I have selected
    As a customer
    I want to place an order from my cart using an optional coupon and pay by credit card

Background:
    Given I have at least one in-stock item in my shopping cart
    And I am on the Checkout page

@happy_path
Scenario: Successful order without coupon (AC1)
    When I do not enter a coupon code
    And I fill in all shipping and billing information correctly
    And I initiate and approve the credit card payment
    Then the system creates the order and displays the order ID
    And a confirmation email is sent to the customer

@coupon
Scenario: Successful order with a valid coupon (AC1)
    When I enter a valid coupon "SAVE10"
    And I validate the coupon
    And I fill in the shipping and billing details and complete payment successfully
    Then the order is created with the discounted price
    And I receive a confirmation email

@invalid_coupon
Scenario: Handling invalid coupon (AC2)
    When I enter an invalid coupon "BADCODE"
    Then the system displays an error message for the coupon
    When I remove the coupon and complete the payment
    Then the order is created without a coupon
    And a confirmation email is sent

@oos
Scenario: Out-of-stock item before payment (AC3)
    Given one of the items in my cart goes out of stock
    When I attempt to place the order
    Then the system shows an out-of-stock error and does not create the order
    And my cart is refreshed accordingly

@payment_declined
Scenario: Declined payment (AC4)
    When I start the payment and the payment service provider declines the transaction
    Then the system displays an error message and no order is created

@missing_address
Scenario: Missing required shipping information (AC5)
    When I leave the postal code field empty
    And I try to submit the order
    Then the system highlights the required field and prevents order submission

Homework

Homework A - Telephone billing software

A telecom company is developing billing software. The tariff calculation is quite complex, with several discounts available.

• The base rate is 1 petak per second.
• During peak hours (every day between 8 and 16) the base rate is 150%, at night (every day between 22 and 6) the base rate is 75%.
• As a business policy discount, calls to the same network are always 40% cheaper.
• There is also an age-based discount: those under 18 get 10%, children under 14 get an additional 10%.
• Between 18 and 26 years, 5% also applies.
• For calls longer than 15 minutes, the portion exceeding 15 minutes gets 15% off; for calls longer than 30 minutes, 30% off applies.
• Discounts based on different legal grounds are additive.
• Calls within the same fleet are free outside peak hours.
• However, international calls always have a rate of 3 petaks per second.

Determine the individual test elements. Create test cases leaning on the Decision Table technique.

Homework B - Neptun course enrollment

Design the acceptance tests of the course enrollment function based on user story + acceptance criteria, using a black-box approach.

Background (short domain):

Course enrollment is only successful if:

• the student has met all prerequisites, and
• the course is open for enrollment (capacity not full).

Success: enrollment confirmation. Failure: the system provides an appropriate message (“Prerequisite not met” or “Course is full”) and offers: * finishing the operation, or * searching for another course.

The student can search among courses (by code, name, instructor, time).

US1 – Course enrollment: As a student, I want to enroll in the selected course so that I can progress in my studies.

US2 – Error handling and suggestion: As a student, I want to receive a clear message on failed enrollment and be offered to finish or search again so that I can quickly find an alternative.

US3 – Course search: As a student, I want to search for courses by multiple criteria so that I can find the one that suits me.

Acceptance criteria (short):

• AC1 – Successful enrollment: If all prerequisites are met and there is free capacity, the system records the enrollment and confirms it.
• AC2 – Missing prerequisite: If any prerequisite is missing, the system reports this and offers: Finish | Search for another course.
• AC3 – Full capacity: If there is no free capacity, the system reports this and offers: Finish | Search for another course.
• AC4 – Search operation: Filter courses by code/name/instructor/time; result list is sortable.
• AC5 – Display of branching options: For every failure, the two options (finish/search) are shown.
• AC6 – Race condition (optional, bonus): If capacity runs out at the last moment, correct error and offer per AC3.

Create test cases based on US1–US3 and AC1–AC6! Use a use case testing approach: main flow, alternatives, exceptions.

---

Utolsó frissítés: 2025-10-30 14:25:48