Review Methodology Section

[smiths]

The next step after #97 and #98 is to review the Methodology Section (Section 1.3, pages 5 and 6).

If it is easier to review, here is the raw tex:

AIMSS/StateOfPractice/QDefOfQualities/QDefOfQualities.tex

Lines 299 to 398 in 75a4a68

	\subsection{Methodology} \label{SecMethodology}
	For each quality we collect as many distinct instances of the definition that we
	can find. Some of the definitions occur in multiple places (for instance the
	definitions from \citet{McCallEtAl1977} appear in many software engineering
	textbooks). Since these instances do not add any new information, they are not
	included; we do not make an effort to identify how frequently the definitions
	reappear in different sources. Once all the definitions are collected, we
	determine the definition that we would like to recommend. The recommended
	definition can either be our preference from the existing definitions, or a new
	definition, which is often found by combining existing definitions.
	To ensure that our recommended definitions are quality definitions, they are
	based on the direct quality measures (Section~\ref{SecDirectQsIntro}). To the
	list of direct qualities, we also add the quality criteria of measurability.
	How the direct qualities are applied to develop quality definitions of qualities
	is described below.
	\begin{description}
	\item[Completeness] To judge the completeness of our definitions, we verify
	that all of the relevant points from the collected definitions have been
	included, or explicitly rejected in the definition's rationale. We also
	verify that all facets of the definition are covered when a quality applies to
	multiple categories of internal, external, product and/or process. Finally,
	we check that all of the qualities listed in this section have been
	considered.
	\item[Consistency] The final list of definitions should be consistent in the
	terminology used. For instance, we do not use synonyms; we use
	the same word or phrase to mean the same concept throughout the final list of
	definitions. Specific decisions on terminology are as follows:
	\begin{itemize}
	\item To represent the idea of cost, effort, time, resources, and
	efficiency, we say \emph{effort}.
	\item To represent ``the degree to which'' or the ``the extent to which'',
	we will say \emph{the extent to which}.
	\item \wss{Fill in additional consistency decisions, as we go through
	definitions.}
	\end{itemize}
	\item[Modifiability] The modifiability of the definitions is aided by the fact
	that the definitions are short. Changes are not difficult to make. The
	modifiability is further aided by the related qualities of traceability and
	abstraction.
	\item[Traceability] Each definition includes a section for the rationale. The
	rationale explicitly links the recommended definition to the previously
	published definitions. The rationale also explains why any aspects of the
	previous definitions were ignored. The rationale also explicitly traces the
	selected definition to the qualities in this section.
	\item[Unambiguity] With natural language it is difficult to remove all
	ambiguity, but every effort is made to keep the definitions simple and to use
	standard and consistent terminology. The rationale given for each recommended
	definition also serves the purpose of making the definitions unambiguous.
	\item[Abstract] Quality definitions should say what the quality achieves, but
	not how to achieve it. We aim for abstract definitions that do not assume any
	specific methodology or tools. For those qualities that fit into multiple
	categories of internal, external, product and/or process, we aim to have one
	definition that will apply to all. When that is not possible, we will clarify
	how the definition is modified between categories.
	\item[Measurability] Although many qualities can only be indirectly measured,
	the definition of the quality should imply a measure that is conceptually
	possible, even if the data needed to complete the measurement is often
	unavailable. The scale type of the units of measure are as follows, in order
	of preference: ratio, interval, ordinal and nominal. These scale types are
	defined in \citet[p.\ 107]{VanVliet2000}. The ratio and interval scale types
	are preferred to the ordinal and nominal types because they can convey more
	information. Specific consequences of aiming for measurability are as
	follows:
	\begin{itemize}
	\item We will avoid the common phrase in definitions of ``the capability
	of'' because this implies a binary measure. By this definition a product
	(or process) is either capable or not; it is a binary measure. Instead of
	measuring capability, we will prefer to use the phrase ``the effort
	required''.
	\item \wss{Add to this list as measurability specific decisions are made.}
	\end{itemize}
	\end{description}
	Our aim is to capture the essence of the quality in an unambiguous way, even if
	it cannot be measured. This is analogous to the definition of true error in
	scientific computing. The true error, which is the difference between the
	calculated and the true solution, can rarely be calculated since the true
	solution is generally unknown. Even though true error can only be estimated, the
	concept of true error is integral to analyzing and understanding the behaviour
	of numerical algorithms.
	After completing each definition we verify that the definition works for all of
	the quality categories (internal, external, product and process) that apply. If
	not, the definition is modified. The rationale section is written to explicitly
	addresses each of these points. Each definition is also checked to verify that
	it is complete, consistent, traceable, unambiguous, abstract and measurable.
	The rational section is written to explain how these qualities, for each quality
	definition, are achieved.