Document Review¶
Purpose and scope¶
A review is a systematic examination of a work product by one or more people. Its primary purpose is to identify anomalies early and to provide information for decisions about the work product. A review can also improve shared understanding, support knowledge transfer, and verify compliance with requirements, standards, and organizational rules.
Reviews are a form of static testing: the reviewed work product is examined without executing the software. Reviews can be applied to requirements specifications, user stories, architecture and design documents, source code, test plans, test cases, user manuals, contracts, project plans, and other written or visual material.
Review processes for work products are described by ISO/IEC 20246:2017. The concrete process, roles, level of formality, and review techniques should be selected according to the review objective, the risks, the type of work product, and applicable organizational or regulatory rules.
A review is not merely proofreading
Proofreading mainly searches for linguistic and typographical errors. A technical review also examines correctness, completeness, consistency, feasibility, testability, security, usability, and compliance, as relevant to the work product.
Typical review process¶
The exact process may vary, but a systematic review normally contains the following activities:
- Planning: Define the scope, objectives, review type, participants, schedule, entry criteria, exit criteria, and required records.
- Review initiation: Distribute the work product and supporting information, explain the objectives and responsibilities, and confirm that the work product is ready for review.
- Individual review: Reviewers examine the work product independently and record potential anomalies, questions, and improvement proposals.
- Communication and analysis: Participants discuss or otherwise consolidate the findings, classify them, eliminate duplicates, and decide what action is required. A meeting is one possible form of communication, but it is not always necessary.
- Fixing and reporting: The author evaluates and corrects confirmed defects. Review results and relevant metrics are recorded.
- Follow-up: The review leader or another assigned person checks that required actions have been completed and determines whether the exit criteria have been met.
The following figure provides an overview of typical activities:
Finding, anomaly, and defect
A reviewer normally records a finding or anomaly, not an unquestionable defect. The item becomes a confirmed defect only after it has been analysed and accepted as a problem. This distinction helps keep review discussions factual and avoids turning the review into a debate about personal preferences.
Roles¶
Depending on the selected process and level of formality, the following roles may be assigned. One person may hold more than one role, unless independence or an organizational rule prohibits it.
- Author: Creates the work product, answers questions about it, and evaluates and implements accepted changes. The author should not moderate a formal inspection when independence is required.
- Management: Decides what should be reviewed, provides resources, and establishes organizational review policies. Management should not use review findings to evaluate individuals.
- Review leader: Takes overall responsibility for the review, selects participants, defines the schedule, and monitors completion.
- Moderator (facilitator): Conducts review meetings, manages time, supports constructive communication, and ensures that the discussion remains focused on the work product.
- Reviewers: Examine the work product and identify potential anomalies. Reviewers may include project members, technical specialists, domain experts, testers, users, operators, security specialists, or other stakeholders.
- Recorder (scribe): Records findings, decisions, action items, owners, and deadlines during review communication or meetings.
The required roles depend on the chosen review type and the organization's review process. More formal reviews normally define roles, entry and exit criteria, records, and follow-up responsibilities more explicitly.
Review types¶
Review types differ mainly in their objectives and level of formality. The names below are commonly used, but organizations may define them differently.
- Informal review: A lightweight examination without a prescribed process. Examples include a colleague checking a document or a pair reviewing a user story.
- Walkthrough: The author typically leads participants through the work product. Its purposes may include finding anomalies, explaining the proposed solution, gaining consensus, or transferring knowledge.
- Technical review: Qualified participants evaluate the technical content, consider alternatives, identify anomalies, and may make recommendations or reach technical decisions. A moderator may facilitate the review.
- Inspection: The most formal review type. It normally uses a defined process, assigned roles, individual preparation, documented findings, entry and exit criteria, and follow-up. Relevant data may be collected to improve both the work product and the review process.
The presence of a meeting alone does not determine the review type. For example, a technical review may be conducted asynchronously, while an informal review may still involve a conversation.
Possible types of reviews are shown in the figure below:
Review techniques¶
A review type defines the organizational form and formality of the review. A review technique defines how reviewers examine the work product. Several techniques may be combined in the same review.
Ad hoc review¶
Reviewers receive little or no guidance and examine the work product using their own experience. This technique is quick, but its coverage and repeatability are difficult to assess.
Ad Hoc review
A colleague reads a password-reset specification and reports anything that appears unclear or incorrect. The colleague notices that the document never states how long a reset link remains valid.
Checklist-based review¶
Reviewers use a predefined list of questions or quality criteria. Checklists improve consistency and are especially useful when they incorporate findings from earlier projects. They must be maintained, because an outdated checklist can create a false sense of completeness.
Checklist-based review
A reviewer uses a requirements checklist containing the question “Does every requirement have an observable acceptance criterion?” Requirement REQ-17 says only that the search must be fast, so the reviewer records a finding: the required response time and operating conditions are missing.
Scenario-based review and dry run¶
Reviewers follow a realistic scenario and mentally simulate how the work product would be used. A dry run is particularly useful for procedures, use cases, algorithms, test cases, and operational documentation.
Scenario-based review
A reviewer follows the scenario “A customer buys two tickets, cancels one, and requests a refund.” During the dry run, the reviewer discovers that the specification defines cancellation of the whole order but not cancellation of a single ticket.
Role-based review¶
Each reviewer examines the work product from the viewpoint of an assigned stakeholder role. The role determines the stakeholder's goals, knowledge, permissions, and typical tasks.
Role-based review
An administrator reviews an account-management manual and checks whether it explains how to disable a compromised account, inspect active sessions, and revoke access tokens. A first-time user reviews the same manual and checks whether the instructions can be followed without prior system knowledge.
Perspective-based reading¶
Reviewers examine the work product with different quality-oriented responsibilities and often create an intermediate product, such as test cases, a design outline, or a user workflow. This makes the review more active than merely “thinking like” a stakeholder and reduces overlap between reviewers.
Perspective-based review
A tester derives boundary-value tests from an age requirement, while a designer sketches the corresponding validation logic and a user representative constructs a registration workflow. The tester finds that the specification does not state whether the minimum age is inclusive.
Role-based and perspective-based review
The two techniques are related but not identical:
- In a role-based review, a reviewer represents a stakeholder such as an end user, administrator, operator, auditor, or support engineer.
- In perspective-based reading, a reviewer applies a distinct engineering or quality perspective and typically performs a concrete analysis task, for example deriving tests, modelling a design, or constructing a user workflow.
Organizations do not always use these terms consistently. The review plan should therefore state what each assigned role or perspective is expected to examine and produce.
Reviewing natural-language documents¶
Natural language is flexible and accessible, but it also permits ambiguity, incompleteness, inconsistency, and hidden assumptions. Reviewers should pay particular attention to:
- undefined or inconsistently used terms;
- vague adjectives and adverbs, such as fast, user-friendly, normally, or sufficient;
- weak modal verbs, such as should or may, when an obligation is intended;
- universal words, such as all, always, or never, that may hide exceptions;
- pronouns with unclear references;
- passive constructions that hide the responsible actor;
- long compound sentences containing several independently testable requirements;
- missing units, tolerances, boundary values, error behaviour, or operating conditions;
- contradictions between sections, diagrams, tables, and referenced documents.
Terms can also have different meanings in different business contexts. If available, reviewers should use the organization's glossary, data dictionary, domain model, and applicable standards. A new or disputed term should be defined rather than silently interpreted.
Issues related to natural-language documentation are presented in the following document.
A significant portion of reviewed documents are requirements specifications. Quality criteria for such specifications are presented in the following document.
Sample document-review checklist¶
The checklist should be tailored to the work product and review objective. N/A is a valid result when a criterion is demonstrably irrelevant.
| ID | Review question | Yes | No | N/A | Finding / evidence |
|---|---|---|---|---|---|
| 1 | Is the purpose of the document stated clearly? | ☐ | ☐ | ☐ | |
| 2 | Is the intended audience identified? | ☐ | ☐ | ☐ | |
| 3 | Is the scope explicit, including relevant exclusions? | ☐ | ☐ | ☐ | |
| 4 | Are all necessary terms and abbreviations defined? | ☐ | ☐ | ☐ | |
| 5 | Is terminology used consistently throughout the document? | ☐ | ☐ | ☐ | |
| 6 | Is the content complete for its stated purpose? | ☐ | ☐ | ☐ | |
| 7 | Is the content internally consistent? | ☐ | ☐ | ☐ | |
| 8 | Is the content consistent with referenced documents and applicable rules? | ☐ | ☐ | ☐ | |
| 9 | Are statements unambiguous and objectively interpretable? | ☐ | ☐ | ☐ | |
| 10 | Are responsibilities and actors identified explicitly? | ☐ | ☐ | ☐ | |
| 11 | Are preconditions, inputs, outputs, and postconditions specified where needed? | ☐ | ☐ | ☐ | |
| 12 | Are normal, alternative, and error paths covered? | ☐ | ☐ | ☐ | |
| 13 | Are limits, units, tolerances, and boundary values stated? | ☐ | ☐ | ☐ | |
| 14 | Can requirements or instructions be verified or tested? | ☐ | ☐ | ☐ | |
| 15 | Are diagrams, tables, and examples consistent with the text? | ☐ | ☐ | ☐ | |
| 16 | Are references and links correct and accessible? | ☐ | ☐ | ☐ | |
| 17 | Does the document address relevant security, privacy, safety, and regulatory concerns? | ☐ | ☐ | ☐ | |
| 18 | Is the structure logical and easy to navigate? | ☐ | ☐ | ☐ | |
| 19 | Is the language appropriate for the intended audience? | ☐ | ☐ | ☐ | |
| 20 | Is the document free from irrelevant duplication and avoidable verbosity? | ☐ | ☐ | ☐ |
Sample finding record¶
Findings should be traceable and actionable. A simple record can use the following fields:
| Field | Example |
|---|---|
| Finding ID | REV-012 |
| Location | Section 4.2, paragraph 3 |
| Category | Ambiguity |
| Severity | Major |
| Description | “The system shall respond quickly” does not define a measurable response time. |
| Suggested action | Specify a percentile, time limit, workload, and measurement environment. |
| Owner | Requirements author |
| Status | Open |
Severity and priority
Severity describes the potential impact of the problem. Priority describes how urgently it should be addressed. They are related, but they are not the same property.
How can an LLM support document review?¶
A large language model (LLM) can assist reviewers by processing substantial amounts of text, applying repeated analytical questions consistently, and producing structured candidate findings. It should be treated as a review assistant, not as an authoritative reviewer. Responsibility for accepting findings and approving the work product remains with human reviewers.
Summarization and structural analysis¶
An LLM can produce:
- a concise summary of a document;
- a section-by-section outline;
- a list of actors, concepts, responsibilities, inputs, and outputs;
- a glossary extracted from the document;
- a table of requirements and their identifiers;
- a summary adapted to a particular audience, such as a developer, tester, manager, or end user.
Summarization can help a reviewer understand the document quickly, but it is inherently selective. A summary must not replace reading the source when omitted details could affect a decision.
Summarization
Ask the LLM to summarize each section of a requirements specification in no more than two sentences and list any terms that are used without a definition.
Detecting potential defects¶
An LLM can search for candidate problems such as:
- ambiguous, vague, or subjective expressions;
- undefined terms and inconsistent terminology;
- contradictions between statements;
- missing actors, preconditions, outcomes, or error paths;
- requirements that are not measurable or testable;
- duplicated or overlapping requirements;
- inconsistent units, dates, identifiers, or numerical limits;
- discrepancies between prose, tables, examples, and diagrams;
- possible omissions revealed by a checklist or scenario.
The results are candidate findings. An LLM may overlook defects, misunderstand domain-specific statements, or report a valid design decision as an error. Every finding must therefore be checked against the original passage and the relevant domain knowledge.
Applying review techniques¶
An LLM can support several of the techniques introduced earlier:
- Checklist-based review: Apply every checklist question to the document and cite the passages used as evidence.
- Scenario-based review: Simulate a supplied user or operational scenario and identify steps for which the document provides no defined behaviour.
- Role-based review: Examine the document from the viewpoint of a specified stakeholder.
- Perspective-based reading: Derive test cases, a data model, a process outline, or another intermediate product and identify information that is missing or contradictory.
The reviewer should give each LLM invocation a narrow objective. Separate passes for ambiguity, consistency, security, and testability are generally easier to assess than a single request to “find every problem.”
Retrieval-augmented generation (RAG)¶
Retrieval-augmented generation (RAG) is an approach in which an information-retrieval component first selects relevant material from an external knowledge source and supplies it to the LLM as context for generating an answer. The external source may contain standards, legislation, organizational policies, glossaries, architecture descriptions, earlier specifications, or domain documentation.
In simplified form, the process is:
- The review question and relevant parts of the document are submitted to a retrieval system.
- The retrieval system finds passages that are likely to be relevant.
- The selected passages are added to the LLM's context.
- The LLM analyses the reviewed document using those passages and reports the supporting sources.
- A human reviewer verifies both the retrieved evidence and the resulting finding.
RAG can substantially improve a technically specialized review because the model does not have to rely solely on information encoded during its training. It can compare the work product with project-specific and current authoritative material. RAG does not, however, guarantee correctness. Retrieval may return irrelevant passages, omit an essential source, or use an obsolete document. The generated conclusion may also go beyond what the retrieved evidence supports.
Requirements for trustworthy RAG-assisted review
- Use authoritative, version-controlled sources.
- Record the title and version of each source.
- Require citations to exact sections or passages.
- Distinguish source-supported findings from the model's own suggestions.
- Allow the model to report that the available information is insufficient.
- Verify high-impact findings manually.
Limitations and safeguards¶
LLM-assisted review introduces risks that must be managed:
- Hallucination: The model may invent requirements, rules, references, or explanations.
- Loss of context: Long documents may be truncated, divided poorly, or analysed without important cross-references.
- False confidence: Fluent wording does not demonstrate that a finding is correct.
- Non-determinism: Repeated executions may produce different findings.
- Bias: The model may favour familiar patterns and fail to recognize legitimate domain-specific solutions.
- Confidentiality: Sending sensitive documents to an external service may violate contractual, legal, security, or privacy obligations.
- Traceability: A conclusion without a precise location and supporting evidence is difficult to verify.
Before using an LLM, the organization should determine which documents may be processed, where the data is stored, whether it is retained or used for model training, and which human approval steps are required. Sensitive information should be removed or anonymized when appropriate.
Prompting
The following prompt can be adapted to a requirements specification or another structured technical document:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 | |
Human responsibility
The LLM's output is input to the review process, not its final result. A human reviewer must verify the cited location, reproduce the reasoning, decide whether the finding is valid, assign its actual severity and priority, and approve any resulting modification.
Exercises¶
User Manual Review
Perform a review of the English user manual or the Hungarian user manual.
- Define the intended user role and review objective.
- Tailor the sample checklist to the manual.
- Perform an individual review.
- Record each finding with an identifier, location, category, description, and proposed action.
- Summarize whether the manual is suitable for its intended use.
Requirements Specification Review (Hungarian documents)
Choose one of the following requirements specifications:
Select one review technique and at least one quality criterion. State your selection before beginning the review, then document the findings using the sample finding record.