For example, the decision-maker may have indicated in respect of three items A, Band C, that B warrants more priority than C, and C warrants more priority than A. Logically, therefore, he/she should rank B as more important than A, thus:
If, on the contrary, the decision-maker has indicated that A warrants more priority than B (A) B), then he/she has a Level 1 ranking inconsistency.
At this level, the decision-maker is strongly inconsistent. However, of the priority weightings of all three items are insignificantly different, then this ranking inconsistency may not be significant.
Level 2 inconsistencies are those in which there is an inconsistency in relative proportionality as between the items. For example, the decision-maker may have indicated in respect of the three items A, Band C, that B dominated by more than C dominated A. Logically therefore, C should have more priority than A, thus:
If on the contrary, the decision-maker has indicated that C warrants more priority than B (C > B), then he/she has a Level 2 proportionate inconsistency. At this level, the decision-maker is weakly inconsistent. His/her inconsistency lies in the allocation of his/her values on the judgement scale, given that he/she has determined a consistent ranking.
Level 3 inconsistencies are those in which there is inconsistency in weightings implicitly assigned to the items.
For example, the decision-maker may have indicated in respect of three items A, B and C, that B warrants twice as much priority than C, and that C warrants twice as much priority as A. Logically therefore, B should have significantly more priority than A, thus:
If, on the contrary, the decision-maker has indicated that B warrants only marginally more priority than A (B A), then he/she has a Level 3 weighting inconsistency.
At this level, the decision-maker is marginally inconsistent.
His /her inconsistency lies in the choice of points on the judgement scale, given that he/she has determined consistent proportionalities.
Two further levels are relevant in respect of consistency measures, namely Level 4 Coherence and Level 5 Comprehensibility. This constitute, together with levels 1-3, a systematically completeable analysis of the reliability of decision-makers' decisions, and are therefore noted at this point. However, their application is dependent on the decision-maker's use of criteria, as described below.
These five levels of inconsistency may reflect five levels of capability in capability theory (Jaques, 1967, 1976; Gibson and Isaac, 1978; Stamp, 1983).
DECISION STANDARDS
The decision-maker's score on the trace index is evaluated against a tabulated test statistic. This statistic was obtained by using a computer simulation (total items ranking from 5 - 25) which produced nearly normal distributions for the index. These tabulated results for the index indicate the decision-maker's precise 'level' of consistency thence the degree of 'confidence' of 'acceptability' to be accorded his/her priority weightings of budget items.
Empirical standards for 'confidence' or 'acceptability' of results are evolved by collecting data from decisionmakers, using the results to obtain distribution of their inconsistence indices (Henkel, 1977). This may be used to provide confidence measures for:
a) all decision-makers whose RN results are known;
b) all 'informed experts' whose RN results are known;
c) all 'informed experts' in respect of a given specialism.
An "informed expert" is defined for this purpose as a person with appropriate qualifications and experience who is required to pronounce judgement on the issues concerned, and formally accountable for recommendations or decisions arising from such judgements. The "issues" concerned are typically those which entail a significant commitment, reallocation or withdrawal of organisational resources. More rigorous selection of "informed experts" is possible using Delphi-related and other techniques (Dalkey, 1969; Einhorn, 1972).
These standards may be derived on a national basis, an organisation basis, or on the basis of specialism, and may provide a reasonably adequate empirically-based standard for what counts as "good judgement" in a particular field. Perfect consistency, invariably unattainable on a complex issue of 6+ options, is 0.000. Low consistency occurs when a decision-maker's judgements fall within the 1% tail of the distribution of consistency scores as determined either by the simulation or the empirical standard.
This indicates that it would be improbable (i.e. one chance in 100) for such consistency to occur by chance, thus that the measure reflects something significant about the decisionmaker's judgement. Note that the decision standards vary relative to the number of budget items considered.
TEAM PRIORITIES AND CHECKS
The priority weights of a team of decision-makers are determined by taking the geometric mean of each individual decision-maker's weightings, or the weighted geometric mean where the comparative influence of individual decision-makers in respect of the items varies. The geometric mean of individual responses is more reliable than the average response when pooling team results (Dalkey, 1969). The way individual decision-makers' weightings cluster, as indicated by concordance measures, provides an indication of the main views influential in the team's thinking. These team views may also be iteratively defined using a variant of the outranking technique (ROY, 1981).
Team consistency is found by taking the geometric mean of the trace indices of each decision-maker's results. The concordance or degree of agreement between decision-makers is obtained by using the standard Kendall and Spearman concordance measures with tabulated chi-squared test statistic (Blalock, 1972; Tyron, 1952), validated concordance measures with tabulated test statistics and weights. These determine how much confidence can be placed in the pooled priority weights obtained as an adequate representation of the 'team judgement'.
On the other hand, where the concordance measure indicates significant conflict within the team, the precise differences between decision-makers in respect of various items may be specified, together with their range of deviation as expressed in metric form in the n-dimensional Euclidean space in which the weights are located. Again, a variant of the outranking technique (Roy, 1981) may be used to determine more precisely the points of significant conflict and to define the space within which consensual or negotiated agreements are most likely to be found.
As well as team concordance, the relative distance of each decision-maker's judgements from the mean may be calculated. This allows a team 'representative' to be selected in respect of the issue on hand, and/or the polarising viewpoints (e.g. radical, conservative and 'via media') in a team to be pinpointed (Axelrod, 1976). It is often required that influence of various decisionmakers on the decision-outcome be varied. This happens, for example, when it is necessary to reflect the relative accountability, authority, knowledge, capability, experience, or power of decision-makers in a team (Jaques, 1976).
The comparative influence to be assigned to individual decision-makers in respect of the judgement issue has then itself been weighted. The influence weightings are obtained by the same methods as were used to obtain priority weights between items (e.g. by judgement analysis, etc.), except that relative influence of decision-makers participating in RN is judged by the organisational decision-makers instead of the relative priorities between budget items.
Once the relative influence of decision-makers has been determined, this is incorporated in the team priority scale by varying each decision-maker's priorities in proportion to their relative influence. This calculation is made as follows: Let represent the weight given to item i by decision-maker. Then, if the influence of decision-maker j is , and there are n decision-makers, then the composite weight of item i would be:
CRITERIA
It is clear from empirical tests that in making their paired comparison judgements, the specific criteria (or 'constructs') used by the decision-makers became particularly accessible (Miller, 1962). The criteria are elicited simply by asking "why?" they assigned more priority to one item rather than another after each paired comparison judgement made.
Having elicited these decision criteria, a criteria analysis may be undertaken by applying the judgement analysis procedures directly to the criteria, to produce a weighted priority scale of criteria. Thereafter, the same procedures apply as with the classification and prioritization of items.
The prioritized criteria scales may be used in conjunction with the priority weightings of items. Judgement analysis (or a comparable method) is applied to the whole set of items on a criterion-by-criterion basis, using normal techniques of combinatorial analysis to obtain a cumulative priority scale of items (Churchman et aI, 1956; Algie, 1975; Morris, 1968; Hogarth, 1980). The criteria are treated as dimensions.
The priority weight of an item is thus the sum of its relative weight multiplied by its scale value on each dimension in turn. The linear compensatory decision rule embodied in this calculation may be varied by using conjunctive, disjunctive, lexicographic, or elimination-by-aspects rules (Hogarth, 1980; Tversky, 1972; Einhorn and H00arth, 1975; Dawes and Corrigan, 1974).
When decision-makers undertake judgement analysis exercises on the criteria, they may again ask themselves after each paired comparison judgement made "why?" they assigned more priority to one criterion rather than another. This elicits a set of meta-criteria which can themselves be prioritized. Theoretically, this process can continue indefinitely. In practice, a limit to this potentially infinite regress tends to be attained after relatively few interactions. At the conclusion of this process, a cause-and-effect cycle of interlinking criteria is obtained in which the only' statements decision-makers are able to make reduce to assertions of criterion interdependencies already embodied in the prioritized criteria scale (Algie, 1975).