This section describes the process used in Resources Now (RN) for eliciting, deciding, agreeing and adjusting priorities. As usual, several methods may be used, either separately or in combination. The most reliable valid method has thus far proved to be judgement analysis.
A. INTUITIVE PRIORITY RATING The relative priorities of the budget items may be rated intuitively or by intuitive rating procedures ,of the kind commonly used in work evaluation, decision analysis, and Delphi (Otis and Lenhart, 1954; Linstone and Turoff, 1975; Einhorn and Hogarth, 1975; Kahnemann and Tversky, 1979; Tversky and Sattath, 1979; Hogarth and Teboul, 1980). But if priorities are not obtained or checked by judgement analysis, then there is no measure of whether the decisionmaker has been inconsistent or incoherent,and hisjher ratings may have been entirely random.
However, priorities implicit in actual expenditure patterns are obtainable from the Budget Monitoring procedure, which can provide some check on the aptness of intuitively rated priorities.
At a more sophisticated level, intuitive priorities may be evolved through the Budget Constraints section. The decision-maker incrementally adds sets of constraints (e.g. stipulations that the budget of an item/s must expand, or contract, or attain a certain minimum or maximum). He/she observes successive results and makes adjustments accordingly, as in a sensitivity analysis process, until a full allocation is satisfactorily completed. The decision-maker's implied priorities are then calculated through the Budget Monitoring procedure.
B. BASIC JUDGEMENT ANALYSIS
Judgement Analysis is a valid method for obtaining apriority scale for a set of items. The priority scale may be that of an individual decision-maker or a decisionmaking team.
Basic judgement analysis essentially involves a paired comparison exercise on budget items. A valid sampling technique is usually used to save exhaustive comparison of every pair. From this, individual and team priority weightings are calculated for each item by the geometric mean technique. Individual and team consistency is calculated by the trace index technique. Team concordance is calculated using the standard Kendall and Spearman concordance techniques. Confidence levels for consistency and concordance measures are based on tabulated test statistics and on empirical standards. Decisions: the Paired Comparison technique
The decision-maker has or is faced with a set of items (e.g. A,B,C,D,E). His/her priorities between these items are elicited.
The decision-maker successively compares one budget item with another in pairs, for example A with B, B with C, etc. This is the first step of the paired (or pairwise) comparison technique (David, 1963), with which decisionmakers seem at ease (Saaty, 1980). The decision-maker either compares all pairs of items exhaustively, or more usually, a valid sample of pairs of items under the judgement analysis sampling plan.
The decision-maker compares the items in terms of a continuous, 9-point, stimulus-centred judgement scale:
This is a 'linguistic variable scale' (Wang and Chang, 1980; Hersch and Carramazza, 1976). The numbers themselves act as tokens standing for relativity descriptions couched in terms of objectival modifiers like "significantly greater", which implicitly convey metric information (Tannehaus and Murphy, 1981; Toulmin and Goodfield, 1970). The definitions of the scalar calibrations are given in Table 2. The scale provides for judgements or estimations of relative priority, importance, contribution, impact or pertinence of items, as indicated by relative strength of preference in respect of relevant criteria used.
A wide-focus question may be asked, for example, "All things considered, which warrants more priority, item A or B?" In that case, the criteria embody a holistic valuation of the 'optimal-feasible' situation.
The focus of the exercise may be varied to reflect the context within which the decisions are made, the specific standpoint from which they are being made, and the nature and process of the priorities to be evolved. In such circumstances, the context, standpoint, and/or prioritization requirements are specified in the questions asked of decision-makers in place of the qualifying clause "all things considered", The definitions of the scalar points of the judgement scale are correspondingly specified in terms of the "criteria".
Sampled pairwise comparisons: the sampling technique.
Decision-makers often find it impractical, inconvenient and undesirable to compare every item with every other for all possible pairs of items. Exhaustive comparison tends to significantly decrease the consistency of many decisionmakers when there are 15+ items in a group (David, 1963), even among decision-makers who are most consistent most often and who score highly on capability tests. Their cognitive behaviour tends to become mechanical.
A sampling technique is used. This allows priority weightings of items to be produced under the sampling procedure which are not significantly different from those produced by exhaustive decisions on all pairs of items, without perturbing consistency levels. A maximum of 2n decisions are made under the sampling procedure, instead of ½n (n - 1) which are used under the exhaustive comparison procedure (where n is the number of budget items involved in the decisions). Thus, where 10 items are being prioritized, only 20 decisions have to be made instead of 45.
The sampling technique is a standard completable sampling plan, derived from graph theory. Statistical distributions were tabulated for the standard samples. The sample method is performed. Then the entries in the matrix are completed by using the shortest paths in the graph given by the matrix between two indices. The sampling method provides a "safety net", as in practice the methods employed always subsume it. The technique is now detailed.
Under the sampling plan, certain pairs of comparisons are selected which allow the rest of the comparisons to be completed according to a sampling algorithm. The algorithm is used to obtain the shortest paths between the two relevant vertices within the maximal tree. The shortest paths decide the values for the pairs of items on which no decision is made. Where there are several 'shortest paths', the geometric mean of the product of the values of these paths is taken.
A computer simulation was used to produce completed matrices on the 'shortest path' principle. The weights and indices of these matrices were tabulated, and checked against the weights and indices for equivalent matrices produced from exhaustive comparisons. This enabled a standard sampling plan to be devised for n items in which:
- item ordering is randomised
- n comparative decisions are made
- n further comparative decisions are made
- any other comparative decisions are made which decision ¬makers judge to be significant or critical.
Alternative sampling plans are allowable as long as they are 'completable', for example, n + I random comparative decisions in addition to the n - I comparative decisions (A,B), (B,e) , ... , where item-ordering is not randomised.
The sample matrix is then completed from the sample comparisons using the shortest paths. The sample matrix which is completed from the sample comparisons is the most consistent matrix that can be attained, given the constraint that it must concord and cohere in every possible respect with the decision-maker's stated decisions and what can be directly inferred from them under normal logical axioms. The levels of consistency at which the decision-maker operates are fully taken into account. The completed matrix is based on the 'safety net' principle which ensures that a necessary and sufficient number of decisions are made at a sufficiently high level of consistency to preclude any significant discrepancy between the priority weightings inferred from the sample and the priority weights which the decision-maker would have, produced had he made exhaustive comparisons between all pairs of items. The assumption made is that in the process of making exhaustive comparisons, the decision-maker would not generate additional inconsistencies, or introduce a new mode of connecting items other than those already implicit in decisions he/she made in the sampled comparisons. Thus, in a sense, the inferred judgements represent the 'most logically consistent' decisions the decision-maker could have made within the priority framework he/she used when making sampled pairwise comparisons.
Priority weights: the geometric mean technique
From the decision-maker's pairwise comparisons of budget items on the judgement scale, his relative priority weights for each item are validly calculated from a ratio scale using the geometric mean technique. Essentially this involves translating the values first into a reciprocal matrix, then into a skew symmetric matrix. At this point, vector space algebra is used to obtain relative priority weightings (each unique), and normalised to sum to 1.00 (100%). The technique is now detailed.
Each comparison a decision-maker makes between two budget items yields a value on the scale. A reciprocal judgement matrix is formed from his/her pairwise comparisons. The values he/she enters (e.g. A, B = 6) form one half of the matrix. The corresponding half of the matrix (e.g. B, A) is completed by taking the reciprocals of the values (e.g. 1/6 for B, A). A skew symmetrical matrix is obtained from this reciprocal matrix of values by taking the logarithms of the decision-maker's entered values. This converts the problem of finding weights and indices of consistency into a linear one in the category of vector spaces. The weights are obtained by taking sums of this matrix, taking the exponential of these sums~ then normalising to sum to 100% if percentages are required.
Adding and eliminating budget items
Decision-makers often require to know what their adjusted priorities and/or allocations are when items are added or deleted to the original agenda of budget items for judgement, without having to re-execute the whole judgement analysis procedure. Algorithms have therefore been derived which allow adding or deleting items to or from the list, and adjusting the priorities and allocations accordingly. No additional judgement analysis is required when items are eliminated. However, if an item is to be added (e.g. Q), the relative weight of that item to at least two of the other items (e.g. Q,A; Q,B) is obtained through an additional judgement analysis. It is usually wise to make the additional comparisons between the new item Q.and the highest, lowest and medium priority items of those already weighted; or alternatively between Q and the Benchmark budget item where magnitude analysis is used.
The algorithms are only applicable on the condition that the budget item(s) added or deleted do not entail for the decision-maker so significant a redefinition of the judgement context as to significantly change the rankings and/or orders of relativity in the weightings judges originally assigned to items. Such a change may be monitored by means of a check using the a priori intuitive ranking, the magnitude estimation or the fuzzy set techniques.
Consistency: the trace index measure
An index of the decision-maker's consistency is obtained using the trace index technique, and the confidence level established by means of tabulated test statistic and empirical standards. His/her consistency is given by the trace of the cube of the original reciprocal matrix. The consistency index was developed both for exhaustive comparisons and for a sample or subset of the total possible comparisons.
The trace index allows three levels of consistency of judgement to be defined in respect of any judgement analysis exercise, and tabulated test statistics and empirical standards have been derived for each level using computer simulation over 2000 matrices at each level. More stringent test statistics are applied when the sampling plan is used than when exhaustive paired comparisons are made.