Quality definition and measurement by JoAnn Hackos in her book on managing doc projects

Hackos, JoAnn. 1994. Managing Your Documentation Projects. New York: John Wiley & Sons.

Discussion about quality and doc quality

• Quality has never been simple to define because quality in technical publications is relative. Quality depends, at least in part, upon the perceptions of the users. 
• Technical communicators have been reluctant to define narrowly what constitutes quality in technical publications. 
• Simple measurements can be called as "manufacturing quality"--spelling, grammar, and formatting errors. This quality is important to the appearance and credibility of publications.
• Tom DeMarco explains in Controlling Software Projects (DeMarco, 1982): "You cannot control what you cannot measure."
• Issue of quality is also economic and emotional.

Definition of doc quality

Crosby defines quality as "conformance to requirements" (Crosby, 1979)

Various definitions of quality within the organization:

- rapid development

- low cost

- complete tech description of the product

- absence of technical errors

- attractive publication

- zero copyediting errors

- measured in usability

Hackos' definition of quality
Quality is meeting the needs of the customer, which makes it a political issue in most orgs because requirements of diverse customer communities must be balanced against each other. This means that group with most power might get to define quality.

Quality should include the needs of diverse customers.
Technical publications that add value have quality.

High-quality publications:

- make information more accessible

- make customer more productive quickl

- reduce training costs

- lower the barriers for discretionary and infrequent users

- foster use by diverse user communities

- reduce the cost of customer support

- can reduce the cost of field maintenance

- can increase sales of a product

Hackos' prescription for achieving quality

- set standards

- hire good people

- use good tools of the trade

To establish a quality metric, Hackos suggests setting levels of quality.

When establishing levels, take into account both product and process measurement.

Product measurement—describes attributes that can be judged by reviewing the finished document; index, organization, style, and accuracy

Estimating metric—hours per page

Defining Quality in Technical Communication: A Holistic Approach--Smart, et al

Smart, Karl, Kristie Seawright, Kristin Bell DeTienne. 1995. Defining Quality in Technical Communication: A Holistic Approach. Technical Communication. 42(3): 474-81.

Article describes existing attempts to define quality and puts forth the authors' approach to defining doc quality.

Discussion about doc quality

• Serious attempt to apply principles of TQ to technical communication (Becker 1993; Fredrickson 1990; Weymouth 1990; Horowitz 1989; Jones 1989) 
• Increased discussion of quality in TC—more than 100 presentations on quality issues in 1992 and 1993 STC conference Proceedings (pp. 185-187). May 1993 issue of Technical Communication featured a special section on quality. 
• Quality in technical communication has come to connote everything from quality standards and customer requirements to personal perceptions of professional communicators. The lack of universally accepted definition of quality remains one of the common concerns expressed in research on quality (Dobyns and Crawford-Mason 1991).

Authors' approach
Authors adapted Garvin’s (1998) framework to TC and used the 2-dimensional model to plot 6 major categories of quality definitions:

• Transcendent: perceived quality 
• Design-based: conformance to specifications, process stability, defect levels, conformance to customer requirements 
• Product-based: product features, performance, reliability, accessibility, usability 
• Customer-based: customer satisfaction and need fulfillment, fitness for use 
• Value-based: customer satisfaction, satisfaction at a reasonable cost 
• Strategic: Predictability and usefulness of readability formulas is limited (Giles 1990; Redish and Seltzer 1985; Clark 1975) 
• Design-based: Quality metrics and criterion-reference measures have become common (Schriver 1993) 
• Product-based: in TC attributes such as usability, accessibility, and reliability used to determine quality (Fredrickson 1992) 

Quality in TC can be defined as a synthesis of those product and service attributes—including accessibility, readability, and usability—that combine to add value for customers, meeting or exceeding their needs and expectations.

Professional recognition and respect through quality--Reilly

Reilly, Annette. 1993. Professional recognition and respect through quality. Technical Communication. 40(2): 231-33.

Here are some questions we bring to the study of quality:

• What features characterize the quality of documentation? (Are there field marks for quality like those that distinguish a scarlet tanager or a Baltimore oriole?) 
• How can we measure quality in communication? Are there objective standards? 
• How are quality communications produced? Is there a standard process? 

How we answer these questions depends on where we expect to find and improve quality:

• In the product 
• In the process 
• In the users' perceptions 
• In the professional communicators 

While the ISO 9000 approach is appropriate for standardized production processes, it may prove unworkable in markets with rapidly changing products, technology insertions, and frequent product obsolescence. These components are mutually exclusive.

A Process for Evaluating Technical Communication Products and Services--Carliner

Carliner, Saul. 1997. Demonstrating Effectiveness and Value: A Process for Evaluating Technical Communication Products and Services. Technical Communication. 44(3): 252-65.

Article provides framework to assess the effectiveness and value of technical communication products. The framework has been adopted from Kirkpatrick’s one developed for trainers.

4 levels: Reaction, Learning, Transfer, Business results

Carliner’s model: User satisfaction, user performance, client performance, client satisfaction

There is no single, unquestionable measure of quality and value, and these terms are not synonymous.

From Redish and Ramey study conclusion is that the value of technical communication is partly demonstrated by the revenues generated or expenses avoided as a result of the communication products we produce.

Usability has emerged as a leading characteristic of quality technical communication, but it does not assess quality, rather it assesses the effectiveness of technical communication products in meeting users’ needs.

Some customers perceive quality and value of TC products not by the products themselves, but on the “customer service” provided to clients by TCs.

Lack of cohesive framework for considering issues of quality and value. Lack of consistently used, widely accepted methodology for assessing quality and value.

Assessing Quality Documents--Smart

Smart, Karl. 2002. Assessing Quality Documents. ACM Journal of Computer Documentation. 26(3): 130-140.

Smart critiques the book Producing Quality Technical Information (PQTI), what he calls one of the first comprehensive discussions of doc quality. He calls it "earliest and best attempt to quantify doc quality characteristics." He takes forward the discussion started by PQTI and offers suggestions to overcome its limitations.

Issues with PQTI: simplifies the quality process, fails to address fluid nature of some aspects of quality, does not account for the larger contextual framing of docs—imp of quality dimensions changes as per audience, context, and purpose of the doc. PQTI offers little grounding as to who developed the dimensions or how. The book provides little empirical or even experiential evidence that the stated dimensions are critical to users’ assessment of quality and corresponding customer satisfaction.

Definition of doc quality

• Example of “acceptable quality”--Aamazon book order 
• “No single definition of quality exists” 
• List of several studies that recognize multiple dimensions of quality. Example of car. 
• Unless you define quality, you cannot replicate, measure, and control it.

Smart suggests two areas for extending the discussion of quality that build upon the limitations of PQTI:

• Clarifying approaches and definitions to quality
• Differentiating among quality dimensions

Smart classifies quality dimensions in three categories:

• Essential (or must-be) quality—basic or expected level of quality; attributes necessary to achieve minimal levels of satisfaction; basic accuracy in spelling or grammar 
• Conventional (or one-dimensional) quality—more traditionally recognized category of quality; results in satisfaction when present and dissatisfaction when absent; completeness in a document; more thorough the manual, more the satisfaction 
• Attractive quality—elements that go beyond customers’ expectations and desires; customers satisfied with absence, but delighted with presence; providing anything that is unexpected; visual communication in docs; more thorough index

Documentation Maturity Model--Huang, Tilley

Huang, Shihong, and Tilley, Scott. 2003. Towards a documentation maturity model. In Proceedings of the 21st annual international conference on Documentation (SIGDOC '03). ACM, New York, NY, USA, 93-99.

There are two aspects to documentation quality: the process and the product. Documentation process quality focuses on the manner in which the documentation is produced (for example, whether or not an organization follows a systematic approach to document development that is closely aligned with the rest of the software process). Documentation product quality is concerned with attributes of the final product (for example, whether or not it uses standardized representations like the Unified Modeling Language (UML) [16] in graphical documentation).

The Documentation Maturity Model (DMM) is specifically targeted towards assessing the quality of software system documentation used in aiding program understanding.

The process component of the DMM is a five-level staged maturity model. It is based on the Software Capability Maturity Model (CMM) created by Carnegie Mellon University’s Software Engineering Institute [3]. The product component of the DMM is centered on a set of key product attributes.

The DMM’s process maturity levels are currently an exact copy of the CMM’s levels. The five process maturity levels of the DMM provide a foundation for continuous documentation quality improvement. The five levels are as follows: Initial, Repeatable, Defined, Managed, Optimizing

A significant amount of work has been done in defining and evaluating documentation quality. However, the focus of much of this work was on guidelines for technical writers and editors producing manuals targeted towards end users; <samir> no source or references given for this statement there are international standards related to documentation products. For example, ISO/IEC 18019 (2000) provides guidelines for the design and preparation of software user documentation.

There is also an ANSI/IEEE standard 1063 (1987) on software user documentation.

In the seminal report IBM produced in 1983, called “Producing Quality Technical Information” (PQTI), they wrote, “technical information that meets all the requirements is quality information” [17]. This concept of documentation quality is illustrated by showing both requirements and examples. At the end of the report, PQTI provides a checklist for reviewers to ascertain whether or not these requirements are met.

In 1998, after several iterations within IBM, a new book called “Developing Quality Technical Information: A Handbook for Writers and Editors” was published [12]. This book expanded the original seven quality attributes into nine, which are grouped into three main categories. The categories are “Easy to Use,” “Easy to Understand,” and “Easy to Find.” The book also provides a procedure for reviewing and evaluating technical information according to these criteria.

Determining and assessing quality is complex, and no single report and checklist can guarantee the development of quality document. A recent article by Karl Smart pointed out one of perceived limitations of the PQTI: its lack of “contextual framing of documents” [23]. Smart classified dimensions of quality into three categories, which suggest their relative importance: essential, conventional, and attractive.

Efficiency:
level 1: manual
2: semi-automatic & static
3: semi-automatic & dynamic
4: automatic & static
5: automatic & dynamic
format: textual, graphical
granularity

How can academia make a difference to document quality research?--Spilka

Rachel Spilka, TCQ, 2000, 9(2), 207-220: The Issue of Quality in Professional Documentation: How Can Academia Make More of a Difference?

This article recommends strategies academics can use to contribute to an issue of great interest in industry: how best to define, measure, and achieve quality documentation. These strategies include contextualizing quality definitions, advocating the use of multiple quality measures, conducting research to identify specific heuristics for defining and measuring quality in particular workplace contexts, and partnering with industry to educate upper management about those heuristics and the benefits of promoting technical communicators to the strategic role of organizational "gatekeepers of quality."

Spilka says:

• Academic research of workplace documentation ignored by industry.
• This is an issue which matters to industry but less to academia is how best to define, measure, and achieve quality in workplace documentation.
• STC has a specialty group on quality; the STC journal Technical Communication has articles and a special column on quality.
• STC funded a year-long study of quality by Janish Redish and Judith Ramey which was published in a special issue of the 1995 journal.

Definition of documentation quality

• The consensus in industry literature is that defining quality of documentation across organizational contexts is an elusive and impossible goal, but, contextualizing quality definitions--that is, customizing them to particular work sites--has the potential to be a useful exercise.
• Authors in the journals TC and STC Quality SIGDOC newsletter DocQment agree that achieving a single definition of quality would be impossible and undesirable.
• Karl Smith's definition.
• Other definitions

Measuring documentation quality

• Multiple quality measures--quantitative (no. of errors) + qualitative (usability testing)
• Using a single measure can be inadequate and detrimental 
• Redish says "numbers don't tell the whole story" and that it is important to examine processes along with products