Multi Criteria Decision Analysis (MCDA) is a tool to support decision making in health care. It complements other tools. It does not replace them.
There are many guidelines that teach you how to do an MCDA [2,3]. We have not invented it, but we have done many of them [4–8]. Some of this knowledge is summarized in our white paper [9] (Check it out).
Based on the guidelines available, and on the learnings we have been accumulating, here are the 10 steps you need to follow, including some tips.
1) Define the decision
Although it sounds obvious, it is important. Start by defining what decision you need to make.
And here is one of the advantages of the MCDA. It allows you to use it in basically any type of decision making. Buy an apartment. Buy a toaster. Where to go on holidays. In health care, they can also be used in many decision problems. Here are some examples:
- Benefit risk assessment. Usually, to approve a new medicine or a new technology, regulatory agencies have to balance the benefits and risks. See how the European Medicine Agency (EMA) has done that.
- Health technology assessment. There are endless examples here: include health interventions or medicines in the universal coverage, approve, deny or coverage with evidence development type of decisions, integrating patient´s view in the HTA process, etcetera. [10,11]. Several countries use MCDA as part of their HTA processes (France, the Netherlands, etc). See more.
- Resource allocation and priority setting. Managers must make decisions on where to place their resources (money, people, time). This task is complex as there are always many options, and benefits and risks are not always that clear. MCDA can definitely help you with this task [12,13].
- Who is eligible for COVID vaccination. A very recent problem. And yes, MCDA was also used for this [14]. There are at almost 40 publications using MCDA for COVID decision problems (Look it up).
Examples are infinite. Determining the value of a drug in a certain disease. Prioritizing access to patients. Shared decision making between doctors and patients. Defining the vaccine calendar. Etc [1].
 | Tips: Take your time. This is a crucial step. Are you leaving something important out of this decision? For example, if you want to evaluate a drug against existing alternatives. Have you included all alternatives? Are they really used in practice? Why are you including some comparators and excluding others? |
2) Choose the team
Choose who should be part of the decision. Examples are patients, clinicians, nurses, psychologists, pharmacists, general public, managers in health care, member of evaluating bodies, etc.
Their task: to participate in weighting criteria and scoring the alternatives. But, most importantly… they are there to present their point of views. Debate is crucial to understand the different perspectives and to achieve a holistic view of the arguments around the decision.
| | Tips: The committee should have diverse profiles. Not too many people. Not too few either. Coming from different geographies. And… they should be real experts. |
3) Train the team
Before you start the process, give some training to the team. Explain the MCDA method. Provide reading materials. Include examples. Answer questions. Videos also help, especially if they are connected to day to day decisions, such as buying a toaster, an apartment, etc. Want to see an example?
4) Select the criteria
Now, it is time to agree upon the criteria by which the alternatives will be evaluated. They can come from an existing framework. Some examples are EVIDEM [15] and EunetHa. But there are many others. In many cases, you will find a nice one in the literature that fits your decision problem.
You can adapt them to your needs. If a criterion is not relevant to the decision problem, delete it. If you miss a significant criterion, add it. Do this together with the experts and achieve consensus.
If there is no existing framework, create your own. It can include quantitative and qualitative criteria.
Some criteria will be absolute (you are not comparing A vs. B). Some will be relative (you are comparing A vs. B)
- An example of absolute criterion: disease severity.
- An example of relative criterion: efficacy.
Be creative. But follow the following principles:
- Completeness. All criteria that are relevant to the decision are included.
- Non redundancy. Do not duplicate or overlap criteria.
- If you include: cost and efficacy. Do not include cost effectiveness.
- Functionality. You should be able to define them.
| | Tips: You can include as many criteria as you wish. In practice, it goes from 3 to 19 (average: 8) [3]. Do not over complicate. Do not oversimplify. |
5) Collect the evidence
Gather data for each criterion. Prepare a document called evidence matrix. Without this, the decision will be based on opinions. This is not what we want.
Information can be found in various sources. Not only in data from clinical trials, but also in data registries, official institutions for decision making (MoH, EMA), grey literature, etc. There are also several methods to collect information: systematic reviews, narrative reviews, focus groups, etc. This depends on how much time and resources you have. And how important the decision is. You do not need to do a systematic review for a small decision problem. But if it affects the entire population, and it costs hundreds of millions then you should do it.
Before we forget… you will never find evidence for everything that you are looking for. This is part of life and decision making.
| | Tips: Summarize it. Then, summarize it again. Less is more. Present findings in tables. And remember: experts have to read it, understand it, and evaluate alternatives based on the evidence. If clinicians are able to review the evidence matrix, even better. |
6) Weigh the criteria
What importance does each criterion has? This is the question you want to answer in this step.
It does not matter what the alternatives are. Make sure your experts do not mix those two things.
There are several methods to do that. Explicit and implicit methods. It can be done on scales from 1 to 5. Or you can ask experts to distribute 100 points into all criteria. As we said, many ways exist.
Let us say that you are weighting criteria to decide on which flat you will buy. There are five criteria. My need for a flat, size, light, location, and price. This is how it will look like (using a scale 1-5).
| Criteria | Scale 1-5 |
| My need for a flat | 3 (21%) |
| Size | 5 (36%) |
| Light | 2 (14%) |
| Location | 1 (8%) |
| Price | 3 (21%) |
| Total | 14 (100%) |
What it means is that no matter what the apartment is, the amount of square meters will always be the most important thing for you when buying a flat.
7) Score the alternatives
How severe is a disease? Are there alternative treatments? Those are types of questions you will score relating to absolute criteria.
How does alternative A perform versus alternative B? This is the main question you answer on relative criteria.
Using a scale, the committee of experts will answer those questions. Let us see how one of the experts scores in this apartment?
Scores expert 1 (scale of 5 points)
| Criteria | Flat A | Flat B |
| My need for a flat | 5 | |
| Size | 2 | 4 |
| Light | 4 | 1 |
| Location | 4 | 1 |
| Price | 1 | 5 |
8) Calculate the value
Do you still not know what flat to buy? Calculate their values. Aggregate weights and scores (this is the most common, but not the only one. There are many ways of doing this step).
Values from expert 1
| Criteria | Weights | Score A | Value A | Score B | Value B |
| My need for a flat | 0.21 | 1.00 | 0.21 | 1.00 | 0.21 |
| Size | 0.36 | 0.40 | 0.14 | 0.80 | 0.29 |
| Light | 0.14 | 0.80 | 0.11 | 0.20 | 0.03 |
| Location | 0.08 | 0.80 | 0.06 | 0.20 | 0.01 |
| Price | 0.21 | 0.20 | 0.04 | 1.00 | 0.21 |
| Total | 1.00 | – | 0.57 | – | 0.76 |
Well, you would go for flat B.
A few things to consider. Weights and scores must be normalized.
- Weights Look at the % in the section “weight”
- Scores. Here, we have a scale of 5. So, divide the scores by 5 to normalize them. In my need for a flat, 5 divided by 5 equals 1.
Now, do this for all experts, and aggregate it, and will have the final value of your decisions.
9) Deal with uncertainty
To deal with uncertainty, you can do many things. One of them is to re-do the weighting and the scoring. Are the results too different? Then, you could also eliminate outliers and see what it would do to your decision.
If you have reference values from other decisions, use them and see what they do to your MCDA. For example, if other people have weighted the same framework you have, what happens if you replace your weights with their weights?
10) Report the results
Publish your results, so everyone will know how you came to your decision. Hopefully, this will increase the probability of a decision being accepted.
| | Tips: The qualitative aspects are the most important part of a report. What did experts say? Were there differences of opinion? How was data interpreted by them? What is the value of the alternative by different groups of stakeholders? |
References
1. Angelis A, Lange A, Kanavos P. Using health technology assessment to assess the value of new medicines: results of a systematic review and expert consultation across eight European countries. Eur J Health Econ. 2018;19(1):123–52.
2. Thokala P, Devlin N, Marsh K, Baltussen R, Boysen M, Kalo Z, et al. Multiple Criteria Decision Analysis for Health Care Decision Making—An Introduction: Report 1 of the ISPOR MCDA Emerging Good Practices Task Force. Value Health. 2016;19(1):1–13.
3. Marsh K, IJzerman M, Thokala P, Baltussen R, Boysen M, Kaló Z, et al. Multiple Criteria Decision Analysis for Health Care Decision Making—Emerging Good Practices: Report 2 of the ISPOR MCDA Emerging Good Practices Task Force. Value Health. 2016 Mar 1;19(2):125–37.
4. Zozaya N, Martínez-Galdeano L, Alcalá B, Armario-Hita JC, Carmona C, Carrascosa JM, et al. Determining the Value of Two Biologic Drugs for Chronic Inflammatory Skin Diseases: Results of a Multi-Criteria Decision Analysis. BioDrugs Clin Immunother Biopharm Gene Ther. 2018;32(3):281–91.
5. Zozaya González N, Alcalá Revilla B, Arrazola Martínez P, Chávarri Bravo JR, Cuesta Esteve I, García Rojas AJ, et al. Pathway towards an ideal and sustainable framework agreement for the public procurement of vaccines in Spain: a multi-criteria decision analysis. Hum Vaccines Immunother. 2020;16(11):2873–84.
6. Zozaya N, Caballero T, González-Quevedo T, Setien PG, González MÁ, Jódar R, et al. A multicriteria decision analysis (MCDA) applied to three long-term prophylactic treatments for hereditary angioedema in Spain. Glob Reg Health Technol Assess. 2022;9:14–21.
7. Zozaya N, Abdalla F, Alfonso Zamora S, Balea Filgueiras J, Carrascosa Carrillo JM, Delgado Sánchez O, et al. Assessing the value contribution of bimekizumab for the treatment of moderate-to-severe psoriasis using a multidisciplinary reflective multi-criteria decision analysis. Expert Rev Pharmacoecon Outcomes Res. 2022;0(0):1–13.
8. Zozaya N, Basterrechea MIA, Bollo E, Castellví I, Espín J, Ortego N, et al. A multi-criteria decision analysis on the value of nintedanib for interstitial lung diseases. Int J Technol Assess Health Care. 2022;38(1):e64.
9. Multi-Criteria Decision Analasys in Healthcare: its usefulness and limitations for decision making [Internet]. Weber. [cited 2022 Oct 18]. Available from: https://weber.org.es/publicacion/multi-criteria-decision-analasys-in-healthcare-its-usefulness-and-limitations-for-decision-making/
10. Youngkong S, Baltussen R, Tantivess S, Mohara A, Teerawattananon Y. Multicriteria Decision Analysis for Including Health Interventions in the Universal Health Coverage Benefit Package in Thailand. Value Health. 2012 Sep;15(6):961–70.
11. Radaelli G, Lettieri E, Masella C, Merlino L, Strada A, Tringali M. IMPLEMENTATION OF EUNETHTA CORE MODEL® IN LOMBARDIA: THE VTS FRAMEWORK. Int J Technol Assess Health Care. 2014;30(01):105–12.
12. Phillips L, Bana e Costa C. Transparent prioritisation, budgeting and resource allocation with multi-criteria decision analysis and decision conferencing. Ann OR. 2007;154:51–68.
13. Marsh K, Dolan P, Kempster J, Lugon M. Prioritizing investments in public health: a multi-criteria decision analysis. J Public Health. 2013;35(3):460–6.
14. Chaker Masmoudi H, Rhili A, Zamali I, Ben Hmid A, Ben Ahmed M, Khrouf MR. Multi-Criteria Decision Analysis to Prioritize People for COVID-19 Vaccination When Vaccines Are in Short Supply. Front Health Serv [Internet]. 2022 [cited 2023 Mar 29];2. Available from: https://www.frontiersin.org/articles/10.3389/frhs.2022.760626
15. EVIDEM Collaboration. EVIDEM Framework 10th edition- Overview [Internet]. 2017 [cited 2018 Jan 4]. Available from: https://www.evidem.org/evidem-framework/