Expert methods. Expert analysis
Expert methods are used in solving prognostic, analytical and design problems associated with informality and lack of certainty in ideas about organizational and economic objects. Essence this method: carrying out by experts an intuitive-logical analysis of the problem with a qualitative assessment of the judgment and formal processing of the results. Features of the expert assessment method: the need for a scientifically based organization of examination, the use of quantitative methods to assess the qualitative judgments of experts.
Expert method can be used to determine forecasts for the development of objects; when determining goals and objectives, alternative distribution of resources; when making decisions under conditions of uncertainty and risk.
1st stage using this method - forming a group of experts. Properties that are necessary for a specialist to be included in the expert group.
Competence (degree of qualification in a certain field of knowledge);
Creativity (ability to solve creative problems);
Analytical and broad-mindedness; constructiveness (the ability to form concrete proposals);
Self-criticism of the expert; attitude towards examination.
To form expert groups, testing, documentation and other methods can be used.
Test method consists in the fact that, based on the developed tests, possible candidates are examined, and a group is formed based on the results of the answers.
Documentation method- selection of experts based on their objective characteristics, which are contained in their personal documents (work experience, position, academic degree, number of publications, etc.).
Method of appointment- determination by the head of a group of experts from among the employees. The main disadvantage of this method: the opinion of employees may be consistent, but erroneous, expressing the official position of the organization on this issue (“school effect”). The results of the examination in this case are of interest mainly only for internal use.
2nd stage application of the expert method - conducting an examination. This stage begins with choosing a method for interviewing experts. There are individual, group and Delphic methods.
At individual way from each expert, through questionnaires or interviews, assessments are obtained that are independent of the opinions of others. Then, after their generalization and processing, the overall resulting assessment is determined. It is rational to use individual expertise when it is necessary to develop a point forecast of the condition of an object, when ranking a set of objects, and in other cases when the most important qualities of an expert are his competence and constructiveness.
Group The method involves obtaining a summary assessment or a common decision from all experts at once through joint discussion. Its use is advisable when searching for unconventional solutions, when assessing the characteristics of little-studied objects, i.e. if necessary, obtain a creative solution. A group survey can be carried out through discussions, meetings, conferences, and brainstorming.
Delphi method synthesizes a number of positive features of individual and group examinations. Experts independently express their opinions in writing. The most important component of the method is carefully developed survey programs carried out in several rounds, and the regulation of questions in each subsequent round. At the end of each round, the group of examination organizers analyzes the responses received, summarizes them and prepares a certificate-bulletin based on the results of the round, the text of which is familiarized to all experts. In this case, the information in the certificate is anonymous. During the repeated survey, experts receive questions that clarify the initial answers and formulated conclusions, taking into account the results of the previous round. In the third round, experts are told on which points there is a common opinion; experts who expressed a different opinion from others are asked to justify it. The fourth, most often the last, round repeats the procedure of the third. Thus, the area of divergence of opinions is narrowed and developed common decision. The advantage of the Delphic method is that it reduces or completely eliminates such psychological factors, as ostentatious conviction, unwillingness to refuse to publicly express one’s opinion, the influence of authority.
3 th stage expert methods - processing survey results. To ensure the possibility of formal processing of examination results, a numerical system is required that describes the properties of objects and the relationships between them using quantitative parameters (various scales of names (classifications), orders, intervals, relationships, differences).
The naming scale is used to describe the belonging of an object to certain classes. Order scale - to measure the ordering of objects according to one or a number of characteristics (rank scale). Interval scale - to display the magnitude of differences between the properties of objects. Ratio scale - to reflect the relationship between the properties of objects, for example, their weight. Difference scale - if necessary, determine how much one object is superior to another in one or more characteristics.
The choice of scale is determined by the objectives of the examination, the characteristics of the object, and the capabilities of the group.
When processing the results of the examination, the choice is important measurement method. The most commonly used methods: ranking, paired comparison, direct assessment, sequential comparison.
The regulation must meet the following requirements: provide sufficient diversity of wording; unity of the structure of the formulation (for example, the formulation must consistently answer the questions: what is necessary? over what (with what)? for what?) The resulting formulations must sufficiently fully reflect their most important content, i.e. have significant capacity; the formulation should be done in such a way as to exclude discrepancies.
Problems of improving expert technologies are associated with the development of the following areas: the formation of an expert commission, the organization and conduct of examinations based on the use of modern methods, the use of multi-criteria assessments when interpreting the results.
25. HEURISTIC METHODS
Heuristic methods are based on logic, intuition and experience of decision makers (DMs). These methods allow you to “catch” and use these processes when developing alternatives. Depending on the approach used, heuristic methods are divided into formal heuristic and informal heuristic.
The basis of formal heuristic methods is the formalization of techniques for solving complex problems by a person by modeling his thought processes. They include the method of evolutionary modeling, labyrinth methods, etc.
Evolutionary modeling assumes the presence of initial experience in the process of adopting SD. This experiential, informational material is needed to run the evolutionary model. Based on existing experience, several solution options are being developed that allow us to approach eliminating the problem and achieving the solution goal from different positions. Each option is examined based on predetermined criteria. In test mode, the original, reference solution (“parent”) is changed randomly, resulting in the production of a “child” (generated idea). If the “descendant” is worse than the “parent”, it is discarded, and through another mutation a new “descendant” is born. If the “child” is better, then the “parent” is discarded, and the “child” takes its place, and the procedure is repeated again. The main advantage of this method: the ability to use computer technology, which allows you to search quite quickly. However, you cannot get an outstanding, non-standard, creative solution.
Labyrinth methods are based on a step-by-step search followed by an assessment of the possible continuation of the path to eliminating the problem. If the direction is “dead-end”, a return to the starting point occurs, and the process is repeated again until a path for further movement is found.
Conceptual modeling is based on collecting initial information when analyzing the situation and building a structural model that allows you to isolate the most important elements relationships. The main means to achieve the goal is a structuring method based on the principle of decomposition (separation).
The basis of informal heuristic methods is the management of human intellectual activity. The need for such management is due to the peculiarities of his thinking (informality, ability to generalize, to navigate in an uncertain situation, tendency to dissipation and loss of information). To enhance the first three properties and neutralize the last two, psychointellectual generation of ideas is used.
The process of solving complex problems using idea generation is carried out in the form of a targeted, controlled conversation-discussion between two direct participants: the leader and the decider. The presenter poses questions to the decider, to which the decider must express his judgment. A discussion ensues around these judgments. Opponents and experts can be assigned to help the presenter. The opponents' task is to criticize the decision maker's judgments and involve him in the discussion. The task of the experts is to help the presenter evaluate judgments and outline the consequences of further discussion. Several possible schemes for organizing idea generation sessions:
By the number of leaders: poly-control (several leaders), mono-control (one leader), auto-generation (no leader);
By the number of decisive ones: unilateral schemes (one decisive), multilateral schemes (many decisive);
By type of contact: with direct contact (in one room), with indirect contact (through technical means).
Conditions for ensuring targeted idea generation:
It is necessary to ensure psychological comfort (creating a comfortable workplace, high spirits and a feeling of relaxedness in the decision maker);
Ensure structuring of the process of finding a solution (develop a psychoheuristic program containing a list of issues discussed, goals of discussion and recommendations);
Create information and technical support systems.
As a result of generation, a set of data should be obtained that makes up the main information array or field of possible solutions.
A number of methods of psychological activation are based on the concept of psycho-intellectual generation of ideas. The choice of generation method is made depending on the nature of the task. When solving urgent issues the best methods may be direct brainstorming or business games. In inventive creativity - types of brainstorming and synectic methods. In prognostic tasks - questionnaire methods, morphological analysis and etc.
The nominal group technique is based on the principle of limitations on interpersonal communication, so that all members of the group gathered to develop a solution are initial stage express their proposals in writing independently, independently of others. Then each participant reports the essence of his project, the presented options are considered by group members (without discussion or criticism), and only after that each group member, independently of the others, presents in writing a ranking of the ideas considered. The proposals that receive the highest score are accepted as the basis for the decision. The peculiarity of this method and its advantage is that despite the joint work of group members, there is no restriction on individual thinking.
The smash attack technique can be used to detect flaws, erroneous conclusions, and conclusions in a study that is in the process of being completed. The meeting is attended by up to 50 people who are familiar with the working document that is the subject of discussion in advance. All participants speak in turn. The task of each speaker is to discover what is possible more disadvantages of work. The merits of the work and ways to eliminate the shortcomings are not mentioned. The duration of one performance is 1-3 minutes; it is forbidden to repeat shortcomings noted by other participants. Sometimes it is advisable to hold a discussion in two circles, providing the opportunity for repeated presentations by those who wish to clarify their comments.
The smash attack is a valid means of negative analysis. Therefore, the authors of the document under discussion should not be present at the discussion. Organizing a smash attack is similar to a direct brainstorming attack.
Synectic methods are based on the method of brainstorming, carried out by a special group of specialists distinguished by their flexibility of thinking and broad outlook. Such groups, accumulating techniques and work experience, achieve positive results in finding new technical solutions.
Synectics- this is a method of forecasting by analogy, transferring conclusions regarding certain characteristics from one subject to another. At the first stage, the leader sets the task and answers questions from group members. In the second stage, each participant offers his own ideas, trying to look at problems from different points of view, thus overcoming “psychological inertia.” The leader selects one of them and forms its essence (key ideas). In the third _, participants search for analogies to the key idea, using known facts from other areas of knowledge. In the fourth, the leader tries to apply some of the analogies and ideas proposed by group members to the task at hand. Next, an expert assessment of the chosen direction is carried out, and if the conclusion is positive, the work continues until a decision is made.
Morphological methods in the development of alternatives are an approach to establishing typical logical connections and interdependence. Most often, this group of methods is used to identify possible options for technical, economic, and organizational solutions.
A morphological table is constructed. The entire process of developing solution options in the table is presented in the area of three fields. The first field - informational - serves to designate all operations of collecting, pre-processing, storing and transmitting information. The analytical field represents operations related to the analysis of information and the selection of alternatives. In the target - conclusions, targets, limitations and requirements for the solution are formed. All operations occurring during the development and decision-making process are traced.
In practice, none of the methods is used in isolation from the others; obtaining an effective solution is possible only through an appropriate combination of them.
Expert method- this is the solution of problems based on the judgment (opinion) of highly qualified specialists in the relevant field of knowledge (science, technology, economics, etc.). When making an expert assessment of events (phenomena), it is necessary to: clearly formulate the purpose of the study; correctly determine the time of events; develop the organization of surveys (interviews) and questionnaires; form a group of experts; ensure the mutual independence of their judgments, the absence of the authority of a position or personality influencing the choice of alternatives, and generalize the results obtained.
The essence of expert forecasting methods is in developing the collective opinion of a group of specialists in this field. The method of expert assessments consists of experts conducting an analysis of the problem with a quantitative assessment of judgments and formal processing of the results. The generalized expert opinion obtained as a result of processing is considered a solution to the problem.
Expert ranking. Ranking is used in cases where
when direct assessment is impossible or impractical. Wherein
“The ranking of objects contains only information about which of them
is more preferable, and does not contain information about how much or in
how many times one object is preferred over another.
Method for ranking alternatives: The essence of the method lies in the ordering of the compared alternatives according to the degree of preference of a certain risk. If there are significantly few such alternatives (not>20), then the expert is presented with the entire set of alternative options, and he indicates the most preferred among them. Then the expert will determine the most preferable option from the remaining ones. And so on until all the evaluated alternatives are ranked. If the estimated variations are greater, then first they are divided into ordered groups according to preferred ones using the expert classification method, and then ranked within groups.
25. The essence of expert methods. Alternative ranking methodExpert decision-making methods are methods of collecting and processing the opinions of experts (specialists, respondents) on the problem under consideration in order to make the necessary decisions. Experts depending on the task
can be both specialists and consumers. These methods are used to solve semi-structured and unstructured
structured problems when a mathematical description of the problem is impossible or the necessary statistical information is missing. These methods are based on the experience, knowledge and intuition of specialists (experts); information received from experts is processed using statistical methods.
Expert methods are used to solve the following problems: - development and evaluation of alternatives for solving problems; - ordering a list of alternatives (various objects) and selecting one or more best alternatives according to one or more criteria; - assessing the influence of factors on a certain indicator or situation; - assigning numerical values. values of qualitative indicators; - forecasting the development of the situation under consideration.
Expert methods are used to evaluate the quality and competitiveness of goods, the competitiveness of an enterprise, markets, consumer preferences, competitors and suppliers, risks, factors influencing the decision, etc. To apply the expert method, an expert group and a working group are created to organize work and calculate points. Members of expert groups and working group experts on this issue are included. Optimal
number of experts from 5 to 12 people. Information is collected from experts using questionnaires.
Decision making based on expert methods includes the following:
1. Definition of the object and purpose of the examination;
2. Formation of a working group;
3. Selection of objects and criteria for their evaluation;
4. Development of a scale for rating objects according to criteria and survey questionnaires
5. Formation of a group of experts;
6. Scenario development and examination procedure;
7. Collection and analysis of expert information;
8. Processing of expert information;
9. Analysis of the examination results;
10.Decision making.
Expert methods are classified according to the following criteria:
number of experts: - individual; - collective; number of criteria: - single-criteria; - multi-criteria. number of stages: - single-stage; - multi-stage.
Ranking method applied when you need to organize some
objects (decisions, indicators, factors) according to some criterion and select from
one or more of them are the best or most important.
This method is used if the number of alternatives is not too large
face and uses one criterion to order the alternatives. Method at
also changes to order several factors according to the degree of their influence
on the indicator or situation under consideration if there is a lack of statistical
The essence of the closing method in ordering the compared alternatives according to the degree of preference of a certain risk. If there are significantly few such alternatives (not>20), then the expert is presented with the entire set of alternative options, and he indicates the most preferred among them. Then the expert will determine the most preferable option from the remaining ones, and so on until all the evaluated alternatives are ranked. If the estimated variations are greater, then first they are divided into ordered groups according to the preferred group using expert classification method, and then ranking within the group.
Paired comparison method
Is one of the most common assessment methods comparative preference of alternative options. With the method of paired comparisons, the expert is sequentially offered pairs of alternative options, for which he must indicate the more preferable one. If an expert finds it difficult to do this regarding any pair of objects, he has the right to consider the alternative options being compared to be equivalent or incomparable. After sequential presentation of all alternative options to the expert, their comparative preference is determined according to the assessments of the given expert.
As a result of paired comparisons, if the expert turns out to be consistent in his preferences, all evaluated alternative options may be ranked according to one or another criterion, indicator, or property.
If the expert has recognized some alternative options as incomparable, then the result will be only a partial ordering of them. In the practice of using the method of paired comparisons, one often has to deal with inconsistency and even contradictory assessments of the expert. In these cases, it is necessary to conduct a special analysis of the examination results.
We also note that with a sufficiently large number of alternative options being evaluated, the procedure for pairwise comparison of all possible pairs of them becomes labor-intensive for the expert. In this case, it is advisable to use appropriate modifications of the method of paired comparisons.
For example, if we assume the consistency of the expert’s assessments, then, practically, a single presentation of each alternative option paired with some other option is sufficient. According to it, all features are compared in pairs with each other. Based on pairwise comparisons, scores for each attribute are then found through further processing. To make it more convenient for the expert to make comparisons, the characteristics (A,B,C,...N) are entered into the table both horizontally and vertically.
The expert fills in the cells of such a table. Comparing the attribute with itself gives one. In the first cell the expert writes one, in the second - the result of comparing the first attribute with the second, in the third - the result of comparing the first attribute with the third, etc. Moving on to the second line, the expert writes down in the first cell the result of comparing the second attribute with the first, in the second - one, in the third - a comparison of the second attribute with the third. The half of the table located above the diagonal serves as a reflection of the lower half. To avoid confusion, it is advisable to fill out only one half of the table (above or below the diagonal). Thus, the experts' answers will be presented in the form of the following matrix.
examination credibility tree goal
Expert methods of analysis in the study of socio-economic and political systems occupy a special place in the display and assessment of social processes.
In general, two groups of expert assessments can be distinguished: individual and collective:
Individual expert methods used for forecasting in relatively narrow areas of science and practice. They are based on the use of expert opinions, independent of each other. The information that the customer receives from the expert is unique and focused on a problem of a local nature.
1. Method of garlands and associations.
The scope of application of this method can be a complex problem situation, poorly studied, regarding which there is no established opinion. The method can only be implemented under the condition; a lot of preliminary work, which consists of an in-depth study of the properties of analogues of the object under study, the feasibility of their involvement, the formation of the psychological attitude of experts, etc.
2. Method of paired comparisons.
Based on a simple comparison, the expert of alternative options from which he must choose the most preferable. The method makes it possible to take into account the equivalence or fundamental incomparability of the presented alternatives, and therefore they are excluded from the analysis. During such a comparison, the expert not only selects the best options, but also formulates criteria that allow such a choice, emphasizing the properties and characteristics of the selected alternative.
3. Method of preference vectors.
When used, the expert is presented with the entire set of alternative options being evaluated and is asked to indicate for each of them how many alternative options he is ahead of. The obtained information can be represented as a vector, one of whose components is the number of alternatives that are superior to the first, the second component is the number of alternatives that are superior to the second, etc. This method can also be used during collective examination, presenting a collective view on the relationship between the alternatives under consideration.
4. Method of focal objects.
A distinctive feature of this method is that it completely eliminates quantitative approach and is aimed mainly at concentrating the researcher’s attention on the so-called focal object, which serves as the basis for comparison with the objects that make up the field of directed random search.
5. Individual expert survey.
Possible in the form of an interview or in the form of an analysis of expert assessments.
The interview method means a conversation between the customer and the expert, during which the customer, in accordance with the developed program, poses questions to the expert, the answers to which are significant for achieving program goals. Analysis of expert assessments involves individual filling out by an expert (experts) of a form developed by the customer, based on the results of which a comprehensive analysis of the problem situation is carried out and possible ways to solve it are identified. The expert presents his considerations in the form of a separate document, drawn up on the basis of a thorough examination of the object.
6. Midpoint method.
Used with a large number of alternative solutions. To do this, two alternative solutions are formulated, one of which is the least preferable, the second the most. After this, the expert is asked to select a third alternative, the assessment of which is located between the values of the first and second alternatives. The procedure ends when the comparative preference of all alternative options participating in the examination is determined.
7. Churchman-Ackoff method.
In accordance with this method, all alternative options are ranked by preference, and the expert assigns quantitative ratings to each of them. The advantage of this method is that experts allow adjustments during the discussion of alternative options. If one option is preferred over another, then their values are summed up.
8. Lottery method.
In accordance with the method, available alternatives are distributed in descending order of preference.
Collective expert assessments used when predicting objects and processes of an interdisciplinary nature.
Collective methods are the most effective in terms of achieving maximum objectivity of expert assessment, since they involve the use of a wide and representative range of specialists. In general, methods for organizing collective idea generation can be divided into several types.
1. Brainstorming.
The main goal of brainstorming is to stimulate the creative process of generating ideas, which is possible in a group discussion. The method allows, in an uncertain situation, to develop the maximum number of possible solutions, concentrating the participants’ attention on the problem under discussion. Characteristic feature This method is a procedure for separating the stage of generating ideas from the stage of their evaluation. The main advantage of the brainstorming method is its focus on finding non-standard solutions, capable of being realized with the most open and free mode of discussion of the problem. This mode of discussion allows not only to identify possible directions in the solution current problems, but also to form a group of people with high creative abilities, which is extremely important for any type of organization.
2. Methodology of courts.
The essence of the technique is to organize a discussion of a problem situation in a group, when one side develops independent proposals, and the other gives criticism to all these proposals. The use of this technique reproduces a court scenario in which there is a competition between the prosecution and the defense. The purpose of the methodology is to identify the most reasoned and optimal solution obtained during a thorough examination.
A characteristic feature of the courts’ methodology is its role-playing, which allows the participants in the discussion to express themselves most fully in the organizational process, realizing not only their own place in it, but also the place of other services and job functions.
3. Black box method.
The main advantage of the “black box” method is to minimize the possible influence of stakeholders on making ineffective decisions. The method aims to identify a special analytical center, which makes conclusions exclusively on the developments of independent experts who evaluate the prospects for making decisions on a certain list of problems.
4. Heuristic forecasting method.
The use of this method involves the involvement of highly specialized experts who, based on pre-developed questionnaires and tables, must develop a general model of the object under study. Thus, the heuristic forecasting method is a method for obtaining and specialized processing of forecast estimates of an object through a systematic survey of highly qualified specialists (experts) in a narrow field of science, technology or production. Forecast expert assessments reflect the individual judgment of a specialist regarding the prospects for the development of his field and are based on the mobilization of professional experience and intuition.
5. Synectic method.
One of the advantages of the synectic method is the ability to achieve consolidation between the various levels of management involved in the process of making an expert assessment. The use of the method is advisable due to the fact that during the adoption of expert assessments, problems are discussed between representatives of the same level, which allows them to speak frankly and balancedly.
In this case, it is necessary to take into account the opinion of not only representatives of senior management levels, who can often have a very general idea of the object being assessed, but also ordinary performers, whose knowledge about the object is very valuable and practically significant.
The synectic method is significant in the practice of Japanese enterprises, where the factor of personnel involvement in decision making is given great importance. Thanks to the consolidation of interests between expert levels, not only social stability is achieved, but unique way awareness and resolution of current problems.
6. Diary method.
A characteristic feature of this method is its impersonality and extreme objectivity. The implementation of the method is possible only if there are documentary sources, on which the examination report is exclusively carried out. Most often, the method is used in a management structure with a strictly regulated system job descriptions. The object of expert assessment is, as a rule, the shift log, instructional documents, on the basis of which recommendations are developed for improving the activities of enterprises.
7. “Delphi method”
It is the most popular of the expert methods, and its popularity is directly related to the heuristic capabilities of the method itself, which allow solving complex complex problems.
The essence of the method is a consistent survey of experts’ opinions on a problem of interest to the organizers of the examination.
The method involves a series of interviews with experts who do not have the opportunity to enter into direct contact with each other and receive information about the conclusions of others only from their written reports. The purpose of the method is to make an objective and accurate assessment of existing alternatives in order to make optimal and socially acceptable decisions.
Ministry of Education and Science of Russia
Mari State Technical University
Department of Control and Production of Radio Equipment
on the topic of: Expert assessment methods
Completed: art. gr. BZD-41
Kopylova I.V.
Checked by: Prof. department Cyprus
Skulkin N.M.
Yoshkar-Ola 2012
Introduction
Expert assessment
Stages of expert assessment
Types of expert assessments
Processing the results of the expert survey
Conclusion
Bibliography
Introduction
Examples of expert assessment methods. How will the economic environment change over time? What will happen to the natural environment in ten years? How will the environmental situation change? Will the environmental safety of industrial production be ensured, or will a man-made desert begin to spread around? It is enough to think about these formulations of natural questions, to analyze how ten or even more so twenty years ago we imagined today to understand that there simply cannot be one hundred percent reliable forecasts. Instead of statements with specific numbers, you can only expect qualitative estimates. However, we engineers must make decisions, for example, about environmental and other projects and investments, the consequences of which will be felt ten, twenty, etc. in the future. years. What should I do? It remains to turn to the methods of expert assessments. What are these methods?
1. Expert assessment
Expert assessment- the procedure for obtaining an assessment of the problem based on the opinion of specialists (experts) for the purpose of subsequent decision-making (choice).
Experts(from the Latin "expertus" - experienced) - these are persons who have knowledge and are able to express a reasoned opinion on the phenomenon being studied.
Expert assessment methods - these are methods of organizing work with expert specialists and processing expert opinions.
The essence of expert assessment methods is that the forecast is based on the opinion of a specialist or a team of specialists, based on professional, scientific and practical experience. There are individual and collective expert assessments.
Expert judgments are often used in selection, for example:
one version of a technical device for launching into a series of several samples,
a group of astronauts from many applicants,
a set of research projects for funding from a mass of applications,
recipients of environmental loans from among many who wish,
when choosing investment projects for implementation among those presented, etc.
2. Stages of expert assessment
1. Setting the purpose of the study.
Expert assessment involves the creation of a certain mind that has greater abilities compared to the capabilities of an individual person. The source of multimind capabilities is the search for weak associations and assumptions based on the experience of an individual specialist. The expert approach allows you to solve problems that cannot be solved in a conventional analytical way, including:
· Selecting the best solution option among the available ones.
· Forecasting the development of the process.
· Searching for possible solutions to complex problems.
Before starting an expert study, it is necessary to clearly define its purpose (problem) and formulate an appropriate question for experts. It is recommended to adhere to following rules:
· A clear definition of the conditions, time, external and internal limitations of the problem. * The ability to answer a question with accuracy accessible to human experience.
· It is better to phrase the question as a qualitative statement rather than as an estimate of a number. For numerical estimates, it is not recommended to specify more than five gradations.
· Experts evaluate possible options, and one should not expect them to construct a complete action plan or a detailed description of possible solutions.
2. Choosing a research form, determining the project budget.
Existing types expert assessments can be classified according to the following criteria:
· According to the form of expert participation: full-time, correspondence. The face-to-face method allows experts to focus their attention on the problem being solved, which improves the quality of the result; however, the correspondence method can be cheaper.
· By the number of iterations (repetitions of the procedure to increase accuracy) - one-step and iterative.
· For the tasks to be solved: generating solutions and evaluating options.
· By type of answer: ideological, ranking, evaluating an object on a relative or absolute (numerical) scale.
· According to the method of processing expert opinions: direct and analytical.
· By the number of experts involved: no limit, limited. Typically 5-12 experts are used.
Most known methods expert assessments: Delphi method, brainstorming and hierarchy analysis method. Each method has its own timing and need for experts. After choosing the expert assessment method, you can determine the costs of the procedure, which include paying experts, renting premises, purchasing office supplies, and paying a specialist to conduct and analyze the results of the examination.
3. Preparation of information materials, questionnaire forms, procedure moderator.
Before making a judgment, experts must consider the problem presented in a comprehensive manner. To carry out this procedure, it is necessary to prepare information materials describing the problem, available statistical data, reference materials, application forms, equipment. The following mistakes should be avoided: mentioning materials developers, highlighting one or another solution option, expressing management’s attitude towards the expected results. The data should be versatile and neutral. It is necessary to develop questionnaire forms for experts in advance. Depending on the method, they can be with open and closed questions, the answer can be given in the form of a judgment, paired comparison, ranked series, in points or in the form of an absolute assessment.
The procedure itself is carried out by an independent moderator of the procedure, who monitors compliance with the regulations, distributes materials and questionnaires, but does not express his opinion.
4. Selection of experts.
The problem of selecting experts is one of the most difficult in the theory and practice of expert research. Obviously, it is necessary to use as experts those people whose judgments will most help in making an adequate decision. But how to identify, find, select such people? It must be said frankly that there are no methods for selecting experts that will certainly ensure the success of the examination.
In the problem of selecting experts, two components can be distinguished - compiling a list of possible experts and selecting an expert commission from them in accordance with the competence of the candidates.
Experts must have experience in areas relevant to the tasks being solved. When selecting experts, one should take into account the moment of personal interest, which can become a significant obstacle to obtaining an objective judgment. For this purpose, for example, Schar's methods are used, when one expert, the most respected specialist, recommends a number of others and further along the chain until the necessary team is selected.
5. Carrying out an examination.
The procedure differs depending on the method used. General recommendations:
· Establish and comply with regulations. Increasing the time to make a decision beyond the optimal one does not increase the accuracy of the answer.
6. Statistical analysis of results . After receiving the experts' answers, it is necessary to evaluate them. This allows:
1) Assess the consistency of expert opinions. In the absence of significant expert agreement, it is necessary to identify the reasons for the inconsistency (presence of groups) and recognize the lack of consensus (negligible results).
)Evaluate the research error.
)Build a model of the object’s properties based on the experts’ answers (for analytical examination). The results of the expert assessment are presented in the form of a report. The report indicates the purpose of the study, the composition of the experts, the assessment obtained and the statistical analysis of the results.
7. Preparation of a report with the results of expert assessment.
. Types of expert assessments
Expert assessment methods can be divided into two groups:
§ methods of collective work of an expert group
§ methods for obtaining individual opinions of members of the expert group.
Methods of collective work of an expert group involve obtaining a common opinion during a joint discussion of the problem being solved. Sometimes these methods are called methods of directly obtaining collective opinion. The main advantage of these methods is the possibility of versatile analysis of problems. The disadvantages of the methods are the complexity of the procedure for obtaining information, the difficulty of forming a group opinion based on the individual judgments of experts, and the possibility of pressure from authorities in the group.
Teamwork methods include brainstorming, scenarios, business games, meetings and court.
Brain attack.It is organized as a meeting of experts, whose speeches are subject to one, but very significant restriction - you cannot criticize the proposals of others. You can develop them, you can express your ideas, but you can’t criticize them! During the meeting, experts, “infecting” each other, express more and more extravagant ideas. About two hours later, the meeting recorded on a tape recorder or video camera ends, and the second stage of brainstorming begins - analysis of the ideas expressed. Typically, out of 100 ideas, 30 deserve further development, out of 5-6 they make it possible to formulate applied projects, and 2-3 ultimately bring a useful effect - profit, increased environmental safety, improvement of the natural environment, etc. Moreover, the interpretation of ideas is a creative process. For example, when discussing the possibilities of protecting ships from a torpedo attack, the idea was put forward: “Line up sailors along the side and blow on the torpedo to change its course.” After development, this idea led to the creation of special devices that create waves that knock the torpedo off course.
Method "635"- one of the varieties of brain attack. The numbers 6, 3, 5 represent six participants, each of whom must write down three ideas within five minutes. The leaf goes around in a circle. Thus, in half an hour, everyone will write down 18 ideas, and all together - 108. The structure of ideas is clearly defined. Method modifications are possible. This method is widely used in foreign countries(especially in Japan) to select from a variety of ideas the most original and progressive ones for solving certain problems.
Business gamesbased on modeling of functioning social system management when performing operations aimed at achieving the set goal. Unlike previous methods, where expert assessments are formed during a collective discussion, business games involve the active activity of an expert group, each member of which is assigned a certain responsibility in accordance with pre-drawn rules and a program.
The main advantage of business games is the ability to develop a solution in dynamics, taking into account all stages of the process under study with the interaction of all elements social system management. The disadvantage is the complexity of the organization business games in conditions close to the real problem situation.
Meeting method(“commissions”, “round table”) - the simplest and most traditional. It involves holding a meeting or discussion with the aim of developing a single collective opinion on the problem being solved. Unlike the brainstorming method, each expert can not only express his opinion, but also criticize the proposals of others. As a result of such thorough discussion, the possibility of errors in reaching a decision is reduced.
The advantage of the method is the simplicity of its implementation. However, at a meeting, the erroneous opinion of one of the participants may be accepted due to his authority, official position, perseverance or oratorical abilities.
Commission method- one of the methods of expert assessments, based on the work of special commissions. Groups of experts for " round table"discuss this or that problem in order to harmonize points of view and develop a common opinion. The disadvantage of this method is that the group of experts in their judgments is guided mainly by the logic of compromise.
Script writing methodis based on determining the logic of a process or phenomenon over time under various conditions. It involves establishing a sequence of events that develop during the transition from the existing situation to the future state of the object. A unique scenario can be a description of the sequence and conditions international integration economies of countries, including the following issues:
from what simplest forms to more complex forms this process should go;
how it will affect the national economy and economic ties of countries;
what are the financial, organizational, social, legal problems that may arise during the internationalization of the economy.
The forecast scenario determines the development strategy of the forecasted object. It should reflect the general goal of the development of the object, the criteria for assessing the upper levels of the goal tree, the priorities of problems and resources for achieving the main goals. The scenario displays a sequential solution to the problem and possible obstacles. In this case, the necessary materials for the development of the forecast object are used.
The scenario should be written in such a way that, after reading it, the general goal of the work being carried out in the light of socio-economic objectives for the forecast period becomes clear.
It is usually multivariate in nature and covers three lines of behavior:
optimistic - development of the system in the most favorable situation;
pessimistic - development of the system in the least favorable situation;
working - development of the system taking into account counteraction to negative factors, the occurrence of which is most likely.
As part of the forecast scenario, it is advisable to develop a backup strategy in case of unforeseen situations.
The finished script must be analyzed. Based on the analysis of information recognized as suitable for the upcoming forecast, goals are formulated, criteria are determined, and alternative solutions.
Court methodis a variation of the “meetings” method and is implemented by analogy with the conduct of a trial.
The chosen solution options act as “defendants”;
in the role of “judges” - decision makers;
in the role of “prosecutors” and “defenders” - members of the expert group.
The role of “witnesses” is played by various selection conditions and expert arguments. When conducting such a “trial,” certain decisions are rejected or accepted.
The “court” method is advisable to use when there are several groups of experts adhering to different decision options.
Methods for obtaining individual opinions of members of the expert group are based on preliminary receipt of information from experts interviewed independently of each other, with subsequent processing of the received data. These methods include questionnaire methods, interviews and Delphi methods. The main advantages of the individual expert assessment method are their efficiency, the ability to fully use the individual abilities of the expert, the absence of pressure from authorities and the low cost of examination. Their main disadvantage is the high degree of subjectivity of the resulting assessments due to the limited knowledge of one expert.
Delphi method.The goal is to develop a program of sequential multi-round individual surveys. Individual surveys of experts are usually carried out in the form of questionnaires. Then they are statistically processed on a computer and a collective opinion of the group is formed, arguments in favor of various judgments are identified and summarized. The computer-processed information is communicated to experts, who can adjust the assessments, while explaining the reasons for their disagreement with the collective judgment. This procedure can be repeated up to 3-4 times. As a result, the range of assessments is narrowed and a consistent judgment is developed regarding the prospects for the development of the object. Features of the Delphi method:
a) anonymity of experts (members of the expert group are unknown to each other, interaction between group members when filling out questionnaires is completely excluded);
b) the possibility of using the results of the previous round of the survey;
c) statistical characteristics of group opinion.
This method helps to predetermine the development problem situations, of a long-term nature. Our specialists working in the field of scientific and technical forecasting are also developing methods for processing expert assessments. They are called heuristics.
Interview methodinvolves a conversation between a forecaster and an expert using a question-answer pattern, during which the forecaster, in accordance with a pre-developed program, poses questions to the expert regarding the prospects for the development of the forecasted object. The success of such an assessment largely depends on the ability of the expert to give impromptu opinions on various issues.
Analytical methodinvolves careful independent work by an expert to analyze trends, assess the state and development paths of the predicted object. The expert can use all the information he needs about the forecast object. He draws up his conclusions in the form of a memorandum. The main advantage of this method is the ability to make maximum use of the expert’s individual abilities. However, it is of little use for predicting complex systems and developing strategies due to the limited knowledge of one specialist expert in related fields of knowledge.
. Processing the results of the expert survey
expert collective individual survey
Let's move on to consider the procedures performed at the stage of processing survey results.
Based on expert assessments, generalized information about the object (phenomenon) under study is obtained and a decision is formed, specified by the purpose of the examination. When processing individual expert assessments, various quantitative and qualitative methods are used. The choice of one method or another depends on the complexity of the problem being solved, the form in which expert opinions are presented, and the goals of the examination.
Most often, when processing survey results, methods of mathematical statistics are used.
Depending on the goals of the examination, the following problems can be solved when processing assessments:
· formation of a generalized assessment;
· determining the relative weights of objects;
· establishing the degree of consistency of expert opinions, etc.
1)Formation of a generalized assessment
So, let a group of experts evaluate some object, then x j - assessment of the jth expert, , where m is the number of experts.
To form a generalized assessment of a group of experts, average values are most often used. For example, median (M E ), which is taken to be such an estimate in relation to which the number of large estimates is equal to the number of smaller ones.
A point estimate for a group of experts, calculated as the arithmetic mean, can also be used:
2)Determining the relative weights of objects
Sometimes it is necessary to determine how important (significant) a particular factor (object) is from the point of view of some criterion. In this case, they say that it is necessary to determine the weight of each factor.
One method for determining weights is as follows. Let x ij - assessment of factor i, given by the jth expert, , , n is the number of objects being compared, m is the number of experts. Then the weight of the i-th object, calculated according to the estimates of all experts (wi ), is equal to:
where w ij - the weight of the i-th object, calculated according to the estimates of the j-th expert, is equal to:
3)Establishing the degree of consistency of expert opinions
If several experts participate in a survey, discrepancies in their assessments are inevitable, but the magnitude of this discrepancy is important. A group assessment can only be considered sufficiently reliable if there is good agreement between the responses of individual experts.
To analyze the spread and consistency of estimates, we use statistical characteristics- measures of dispersion.
Variation range (R):
Xmax - x min ,
where x max - maximum assessment of the object; min - minimum assessment of the object.
The standard deviation, calculated using the well-known formula:
where xj is the assessment given by the j-th expert; is the number of experts.
The coefficient of variation (V), which is usually expressed as a percentage:
The approaches to consistency checking used when assessing objects using the ranking method are specific.
In this case, the result of the expert’s work is a ranking, which is a sequence of ranks (for expert j): x 1j , x 2j, …, x nj .
The agreement between the rankings of two experts can be determined using the Spearman rank correlation coefficient:
where xij is the rank assigned to the i-th object by the j-th expert; ik is the rank assigned to the i-th object by the k-th expert; i is the difference between the ranks assigned to the i-th object.
The value can vary from -1 to +1. If the estimates are completely identical, the coefficient is equal to one. The coefficient is equal to minus one when there is the greatest discrepancy in expert opinions.
In addition, the calculation of the rank correlation coefficient can be used as a way to assess the relationship between any factor and the resulting characteristic (reaction) in cases where the characteristics cannot be measured accurately, but can be ordered.
In this case, the value of the Spearman coefficient can be interpreted similarly to the value of the pairwise correlation coefficient. Positive value indicates a direct relationship between factors, a negative one indicates an inverse relationship, and the closer the absolute value of the coefficient is to one, the closer the relationship.
When it is necessary to determine the consistency in the rankings of a large (more than two) number of experts, the so-called concordance coefficient is calculated - the general rank correlation coefficient for a group consisting of m experts:
Note that the subtrahend in parentheses is nothing more than the average sum of ranks (when summed for each object) received by i objects from experts.
The coefficient W varies in the range from 0 to 1. Its equal to one means that all experts assigned the same ranks to objects. How closer value coefficient to zero, the less consistent the expert estimates are.
Conclusion
Experience, intuition, a sense of perspective, combined with information, help specialists more accurately select the most important goals and directions of development, find best options solving complex scientific, technical and socio-economic problems in conditions where there is no information about solving similar problems in the past.
The use of the method of expert assessments helps to formalize the procedures for collecting, summarizing and analyzing the opinions of experts in order to transform them into a form that is most convenient for making an informed decision. But it should be noted that the method of expert assessments cannot replace either administrative or planning decisions; it only allows one to replenish the information necessary for preparing and making such decisions. The widespread use of expert assessments is legitimate only where it is impossible to use more accurate methods to analyze the future.
Expert methods are constantly being developed and improved. The main directions of this development are determined by a number of factors, including the desire to expand the scope of applications, increase the degree of use of mathematical methods and electronic computer technology, and also find ways to eliminate emerging shortcomings. Despite the successes achieved in last years in the development and practical use of the expert assessment method, there are a number of problems and tasks that require further methodological research and practical testing. It is necessary to improve the system for selecting experts, increasing the reliability of group opinion characteristics, developing methods for checking the validity of assessments, research hidden reasons, reducing the reliability of expert assessments. However, even today expert assessments in combination with other mathematical and statistical methods are an important tool for improving management at all levels.
Bibliography
1Orlov A.I. Expert assessments. // Factory laboratory. ? 1996. ? T. 62. ? No. 1. ? pp. 54-60.
2Orlov A.I. Expert assessments. Textbook allowance. - M.: 2002.
Beshelev S.D., Gurvich F.G. Expert assessments in making planning decisions. Textbook allowance. - M.: Economics, 1976. ? 287 p.
Evlanov L.G., Kutuzov V.A. Expert assessments in management. - M.: Economics, 1978. ? 133 p.
Management. Textbook allowance. / Ed. Zh.V. Prokofieva. - M.: Knowledge, 2000. - 288 p.
Beshelev S.D., Gurvich F.G. Expert assessments. - M.: Nauka, 1973. - 79 p.
Statistical methods for analyzing expert assessments. - M.: Nauka, 1977. - 384 p.
Moiseev N.N. Mathematical problems of system analysis. - M.: Nauka, 1981. - 487 p.
Litvak B.G. Expert assessments and decision making. - M.: Patent, 1996.
Characteristics of expert assessment methods [Electronic resource]: #"justify">Expert assessment. / Wikipedia. [Electronic resource]: #"justify">Expert assessments. // StatSoft: SPC Consulting. [Electronic resource]: http://www.spc-consulting.ru/app/expert.htm
Tutoring
Need help studying a topic?
Our specialists will advise or provide tutoring services on topics that interest you.
Submit your application indicating the topic right now to find out about the possibility of obtaining a consultation.
The expert method for determining quality indicators is based on taking into account the opinions of specialist experts. An expert is a specialist competent in solving a specific problem (from Latin word expertus - experienced). This method is used in cases where quality indicators cannot be determined by other methods due to insufficient information, the need to develop special technical means, etc.
The expert method is a combination of several different methods that represent its modifications. Well-known varieties of the expert method are used where the basis of the decision is the collective decision of competent people (experts). For example, decisions of various councils, conferences, meetings, commissions, as well as examiners when assessing students’ knowledge, etc. - These are decisions made by expert methods. When using the expert method, the following factors are important: the competence of experts when forming an expert group; methods of interviewing experts; methods of information exchange; types of scales used for assessment. For expert assessment, it is advisable to use scales with an odd number of gradations, in which, in addition to the lower and upper, there is an average (satisfactory) level.
The expert method can be used in the formation overall assessment(without detail) the level of product quality, as well as when solving many particular issues related to the determination of quality indicators. The areas of application of the expert method are as follows:
assessment of regulatory documents for products;
generalized assessment of product quality;
classification of goods being valued;
determination of the range of quality indicators of the products being assessed;
determination of weight coefficients for quality indicators:
selection of basic samples and determination of the values of basic quality indicators;
measurement and assessment of quality indicators using the senses (organoleptically);
assessment of single indicators, the values of which are determined by measurement or calculation methods;
product identification;
determination of a complex quality indicator based on a set of individual (generalized and group indicators);
assessment of product competitiveness;
product certification.
The expert method is not used if quality can be assessed by other methods with greater accuracy or less cost.
The results of the expert assessment have elements of uncertainty and unreasonableness. The reliability of the assessment results depends on the competence and qualifications of the experts. The expert’s competence consists of professional and qualimetric competence. Professional competence requires knowledge of:
development history of the products being evaluated, changes in their properties and quality indicators;
conditions for design (construction) and production (processing) of products;
values of quality indicators of domestic and foreign analogues;
prospects for product development, reflected in research works, patents, and design developments;
consumer requirements, conditions and nature of consumption (operation) and disposal.
Qualimetric competence requires a clear understanding of the approach to assessing the quality of goods, methods of quality assessment, issues of constructing and applying rating scales, determining subjective probabilities and distinguishing a sufficient number of gradations of the assessed object.
An expert's qualifications are determined not only by knowledge of the subject of discussion, but also by the ability to express clear, unambiguous judgments. The specific capabilities of the expert are taken into account. For example, in the food industry, when assessing the quality of food products, they take into account the expert’s ability to perceive taste, smell, etc., as well as his state of health. Experts assessing aesthetic and ergonomic quality indicators must be knowledgeable in the field of artistic design.
When using the expert method to assess quality, a working and expert group is formed. The working group organizes the procedure for interviewing experts, collects questionnaires, processes and analyzes expert assessments.
The expert group is formed from highly qualified specialists in the field of creation and use of the products being evaluated: commodity experts, marketers, designers, constructors, technologists, etc. The number of experts included in the group depends on the required accuracy of average estimates, the permissible labor intensity of evaluation procedures, the capabilities of group management and organization , in which a group is formed, but the group must have at least 7-12 people. If it is necessary to increase the accuracy of quality assessments, the composition of the group can be increased to 15 - 20 experts.
It is desirable that the expert group is formed not for one examination, but as a permanently functioning body with a fairly stable composition of experts.
During the group’s work with experts, based on analysis of the results previous work General approaches and principles for assessing product quality are being developed, which increases the effectiveness of the assessment.
When assessing quality, experts are asked to build a hierarchical nomenclature of quality indicators, and it is advisable to start the assessment from the level of consideration at which there are single indicators for which objective assessment methods exist.
When constructing a nomenclature of quality indicators, it is advisable to fulfill the following four conditions:
the characteristic by which any complex indicator is divided into n indicators below the level (classification characteristic) must be the same for all n indicators. This provides experts best opportunities to compare indicators when determining weight coefficients (the next operation after constructing the nomenclature);
When determining weight coefficients, the expert compares the importance of various quality indicators included in a homogeneous group. Since this operation becomes difficult with a significant number of indicators, which reduces the objectivity of the results, the number of indicators included in a homogeneous group should not exceed 10;
if a quality indicator is repeated at two or more levels, then its weight coefficient is considered overestimated. Therefore, repetition of indicators is undesirable;
the number of quality indicators included in homogeneous groups: at the same level, the nomenclature of quality indicators should not differ sharply, since an increase in the number of indicators can lead to a decrease in the values of the weight coefficients. For example, in the nomenclature of quality indicators at the first level there are three indicators (S = 1, 2, 3), at the second level each of them consists of a certain number of indicators - respectively q, I, p. It is desirable that the condition q I p be satisfied or that these numbers be sufficiently close.
Possible errors can be reduced by including the necessary explanations in the explanatory note.
The work of experts consists of performing two independent but interrelated operations - adjusting (clarifying) the nomenclature of quality indicators and determining the weighting coefficients of these indicators.
After studying the presented range of quality indicators, each expert comes to one of the following conclusions:
The nomenclature contains quality indicators, the weight of which is insignificant. They can be ignored when assessing quality (the expert crosses out these quality indicators);
the nomenclature does not include sufficiently important indicators (the expert enters these indicators into the nomenclature);
indicators that can be considered unimportant should be deleted and at the same time the nomenclature should be supplemented with indicators that are quite important;
The nomenclature of quality indicators is compiled correctly.
The expert writes down the reasons for his actions on a separate sheet attached to the nomenclature, or states them to the technical worker who fills out this sheet.
After analyzing the opinions of experts by members of the working group, the nomenclature is adjusted and again sent to the experts to determine the weighting coefficients of quality indicators.
The determination by experts of the weighting coefficients of quality indicators begins with ranking, when each quality indicator is assigned a certain rank. If a homogeneous group includes four or more indicators, experts pre-rank them: rank 1 is assigned to the important indicator, rank 2 - the next in importance, etc. If the expert believes that the weight of two or more indicators is the same, then he assigns them the same ranks. If there are less than four indicators, the ranking operation can be omitted.
Each expert becomes familiar with the weight coefficients assigned by other experts and their justifications. If the expert group is formed from employees various organizations, which are difficult to collect for collaboration, then each expert attaches a brief rationale to the completed questionnaire. Since this procedure is very labor-intensive, it is recommended to use it when the number of indicators included in block diagram, is relatively small (about 10... 15). Otherwise, experts are invited to provide justification for only some of the weight coefficient values at their discretion. To simplify the procedure, you can refuse justifications and request them only if necessary.
Each expert gets acquainted with the anonymous opinions of other experts and again puts down the values of the weight coefficients.
At the expert group meeting, an open discussion of all weighting factors is held. All experts have the opportunity to briefly justify their judgments about the significance of the weight coefficient of each indicator and criticize other opinions. To eliminate the possible influence of official position on the opinions of experts, it is desirable that experts speak in sequence from junior to senior (by official position). After the discussion, the experts record in the documents the values of the weight coefficient of the quality indicator and move on to discussing the next indicator.
Based on the results of assessing weight coefficients, the consistency of expert opinions is assessed using coefficients of variation, concordance, etc.
The values of the coefficients of variation are determined by: the number of levels in the nomenclature of quality indicators, the variety of consumer opinions, the competence of experts, etc. When determining weight coefficients, we can recommend the following values of coefficients of variation, obtained based on an analysis of the results of the work of various expert groups: V< 0,10 - согласованность высокая; V= 0,11 ...0,15 - согласованность выше средней; V=0,16...0,25 - согласованность средняя; V= 0,26...0,35 - согласованность ниже средней; V >0.35 - consistency is low.
If the consistency of expert opinions is average or above average, then we can proceed to assessing the consistency of the expert group’s opinions on the weight of all indicators. If consistency is below average, additional analysis is required. The reasons for the low consistency of expert opinions can be subjective and objective. Subjective ones include: insufficient awareness of experts about the quality indicator, the weight coefficient of which is determined; unclear understanding of the problem being solved; arithmetic errors experts, etc.
Objective reasons - the classification of consumers, on which experts rely when determining weight coefficients, is insufficient. Therefore, it is necessary to define the conditions of consumption more clearly.
To identify the cause of low consistency, the weight coefficients of this indicator are re-determined with discussion and the coefficient of variation is again calculated. If after this procedure the coefficient of variation has not improved (or has improved slightly), then the obtained values of the weight coefficients should be discussed with justification of expert opinions and clarification of the consumer group.
The purpose of the discussions is to strive to develop a common understanding among all experts of the nature of the work and issues, to identify the true opinion of each expert, and not to seek to bring together the values of the weight coefficients that experts assign. Thus, the coefficient of variation to a large extent is not a measure of “disadvantage” in opinions, but a measure of the differences in the requirements of individual groups of consumers for the products being evaluated.
