Document Type
Thesis
College
College of Engineering
Department
Industrial and Engineering Management
Degree
MS in Engineering Management
Date Completed
5-2024
First Committee Member
Salmon, Christian
Abstract
This thesis examines the reliability and validity of expert judgment in Multi-Criteria Decision Analysis (MCDA), particularly how judgments differ when experts are informed versus uninformed about context-specific details. The investigation centers on developing and testing performance metrics for expert judgments to evaluate their effectiveness in MCDA settings, where decision-making often depends on the accurate interpolation of missing or incomplete data.
Utilizing empirical research methods, the study systematically tests the accuracy of expert judgments by creating experimental conditions that mimic real-world data scenarios. This involves the preparation of multiple datasets, including basketball statistics and occupational safety metrics, where controlled missing values simulate incomplete information. The research then employs a data elicitation process where both expert and non-expert groups are surveyed to assess their ability to accurately interpret and fill data gaps. These groups are further divided into subgroups of informed and uninformed participants to rigorously test the impact of having contextual knowledge on the accuracy of their judgments.
Initial results indicate a significant enhancement in judgment accuracy when participants have access to contextual information, underscoring the importance of informed decision-making in MCDA. The study also highlights that while expert knowledge can partially compensate for a lack of specific data, the presence of detailed background information substantially improves the quality of decision- making across all participant groups. Moreover, this research contributes to methodological advancements in MCDA by proposing new metrics for assessing the reliability and validity of expert judgments. These metrics aim to standardize the evaluation of expert input and enhance the predictive accuracy of MCDA models, particularly in high-stakes environments where poor decisions can have severe consequences.
In conclusion, the thesis not only bridges significant gaps in empirical research concerning the reliability of expert judgment in MCDA but also proposes actionable strategies to enhance decision-making processes. This study lays the groundwork for future research aimed at refining these methodologies and expanding their application across diverse fields and decision contexts.
Recommended Citation
Gajendra, Vinith, "Reliability of Expert Judgement Elicitation in the field of Multi-Criteria Decision Analysis" (2024). Master’s Theses - College of Engineering. 23.
https://digitalcommons.law.wne.edu/coetheses/23
