|Year : 2016 | Volume
| Issue : 1 | Page : 31-33
Evaluation of inter-rater agreement between three causality assessment methods used in pharmacovigilance
Saket J Thaker, Rahul S Sinha, Nithya J Gogtay, Urmila M Thatte
Department of Clinical Pharmacology, Seth GS Medical College and KEM Hospital, Mumbai, Maharashtra, India
|Date of Submission||23-Dec-2014|
|Date of Decision||02-Feb-2015|
|Date of Acceptance||02-Aug-2015|
|Date of Web Publication||29-Mar-2016|
Nithya J Gogtay
1st Floor, New MS Building, Seth GS Medical College and KEM Hospital, Parel, Mumbai - 400 012, Maharashtra
Source of Support: None, Conflict of Interest: None
|How to cite this article:|
Thaker SJ, Sinha RS, Gogtay NJ, Thatte UM. Evaluation of inter-rater agreement between three causality assessment methods used in pharmacovigilance. J Pharmacol Pharmacother 2016;7:31-3
|How to cite this URL:|
Thaker SJ, Sinha RS, Gogtay NJ, Thatte UM. Evaluation of inter-rater agreement between three causality assessment methods used in pharmacovigilance. J Pharmacol Pharmacother [serial online] 2016 [cited 2019 Dec 6];7:31-3. Available from: http://www.jpharmacol.com/text.asp?2016/7/1/31/179361
Causality assessment is the evaluation of a possibility that a particular treatment is the cause of an adverse event. It plays a key role in pharmacovigilance both toward signal generation and risk benefit evaluation. There are several methods and algorithms for causality assessment, but none is considered the gold standard and use varies from country to country. This is due to inter-individual variation in using the methods as also varying sensitivities and specificities of each method., Bayesian methods of causality have greater credence but have lower acceptance as they are rigorous to perform and involve use of complex calculations., In the absence of a universally accepted method the present study was carried out to assess the intra- and inter-rater variations among three widely used causality methods—the WHO-UMC method (recommended by the Pharmacovigilance Program of India (PvPI)), Naranjo's algorithm (a widely quoted method in case reports in literature) and the European ABO method (used in European Union to harmonize decision making). To the best of our knowledge, a study comparing these three methods has never been carried out.
The study protocol was deemed exempt from review by the Institutional Ethics Committee. A total of 401 adverse events collected as part of the surveillance of the PvPI between January and April 2013 were assessed independently by three different raters who had at least 3 years of training in the discipline. The WHO- UMC method classifies causality as certain, probable, possible, unlikely, unclassified and unclassifiable, that is, six categories. Naranjo's algorithm categorizes causality into four categories—definite, probable, possible and doubtful. The European ABO system also categorizes causality into three categories—A, B and O.,, Taking into consideration these differences, agreement was assessed by collapsing the categories and making them uniform. While comparing the agreement between European ABO method with the other two methods, all scales were classified into three ranks as follows: (a) certain/probable/A (rank 1); (b) possible/possible/B (rank 2) and (c) unlikely/doubtful/unclassified/unclassifiable/O (rank 3). However, an agreement between WHO-UMC scale and Naranjo's algorithm was made using four ranks—(a) certain/definite (rank 1); (b) probable (rank 2); (c) possible (rank 3) and (d) Unlikely/unclassified/unclassifiable/doubtful (rank 4). Ranks were compared with each other and agreement was calculated using Fleiss kappa index.,
A majority of the ADRs found by WHO-UMC method were possible (39.06%), followed by probable (34.33%) and certain (26.18%). Naranjo's algorithm found 52.54% of the ADRs as probable followed by possible (47.30%). Causality assessment by European ABO method assessed 56.87% of the ADRs as A followed by B (42.86%). When the three methods were compared, the highest inter-rater agreement was obtained with the Naranjo's algorithm (314/401 = 78%). The overall inter-rater agreement in all three methods was high (kappa statistic = 0.69) [Table 1]. Overall, intra-rater agreement was substantial in two raters (kappa = 0.69 and 0.74) and moderate (kappa = 0.66) in the third rater. The highest agreement was observed between Naranjo's algorithm and European ABO method (kappa = 0.74) followed by WHO-UMC scale and Naranjo's algorithm (kappa = 0.69) on pair-wise comparisons.
|Table 1: Intra-rater and inter-rater agreement in three different causality assessment methods|
Click here to view
The present study found good inter- and intra-rater concordance between the three commonly used causality methods—the WHO causality scale, the Naranjo's algorithm and the European ABO system. Davies et al. observed a fair agreement (kappa = 0.31) between six assessors using the WHO-UMC method, Naranjo's algorithm and Venulet algorithm which was lower than that observed in the present study. A higher agreement between the WHO-UMC method and Naranjo's algorithm was reported by them which has been corroborated by our observation as well (kappa = 0.61). Arimone et al. found poor agreement between five raters in 31 adverse drug reaction pairs using seven criteria for causality assessment and global causality derived from a questionnaire. Most common causality rating in our study with the WHO-UMC method was “possible” which corroborates with Macedo et al., while Naranjo's algorithm assessed most common causality rating as “probable” which is in consonance with Sriram et al.
None of the causality assessment methods is either complete or reproducible. Very few methods take into consideration prior probability of the occurrence of an event with actual causal association. Also, categorization of causality into “possibly related” or “probably related” does not practically offer any additional advantage and leads to poor inter-rater agreement. The causality methods used in the present study have their own merits and demerits [Table 2]. We did find a good agreement between the three raters which could be the result of an actual causal association between the drug and the adverse event or due to individual expertise or commission of same error by all the raters simultaneously. However, the presence of confounding variables like underlying disease, concomitant medications, and absence of de-challenge/re-challenge information could have prevented the achievement of complete agreement.
|Table 2: Strengths and limitations of the three causality assessment methods|
Click here to view
Although the WHO-UMC scale is widely accepted and has been recommended by the National Pharmacovigilance Programme of India, we obtained the highest inter-rater agreement with Naranjo's algorithm. Additionally, it had a very good agreement with the other two methods as well. Hence, Naranjo's algorithm can be considered as a preferred method in individual case causality ascertainment because it is more objective and less dependent on personal expertise.
Financial support and sponsorship
Conflicts of interest
There are no conflicts of interest.
| References|| |
World Health Organization (WHO), Uppsala Monitoring Centre. The use of the WHO- UMC system for the standardized case causality assessment. WHO [online]. Available from: http://www.who-umc.org/graphics/4409.pdf
. [Last accessed on 2014 Nov 14].
Agbabiaka TB, Savović J, Ernst E. Methods for causality assessment of adverse drug reactions: A systematic review. Drug Saf 2008;31:21-37.
Meyboom RH, Hekster YA, Egberts AC, Gribnau FW, Edwards IR. Causal or casual? The role of causality assessment in pharmacovigilance. Drug Saf 1997;17:374-89.
Gallagher RM, Kirkham JJ, Mason JR, Bird KA, Williamson PR, Nunn AJ, et al
. Development and inter-rater reliability of the Liverpool adverse drug reaction causality assessment tool. PLoS One 2011;6:e28096.
Fleiss Joseph L. Measuring nominal scale agreement among many raters. Psychol Bull 1971;76:378-82.
Davies EC, Rowe PH, James S, Nickless G, Ganguli A, Danjuma M, et al
. An investigation of disagreement in causality assessment of adverse drug reactions. Pharm Med 2011;25:17-24.
Arimone Y, Miremont-Salamé G, Haramburu F, Molimard M, Moore N, Fourrier-Réglat A, et al
. Inter-expert agreement of seven criteria in causality assessment of adverse drug reactions. Br J Clin Pharmacol 2007;64:482-8.
Macedo AF, Marques FB, Ribeiro CF, Teixeira F. Causality assessment of adverse drug reactions: Comparison of the results obtained from published decisional algorithms and from the evaluations of an expert panel. Pharmacoepidemiol Drug Saf 2005;14:885-90.
Sriram S, Ghasemi A, Ramasamy R, Devi M, Balasubramanian R, Ravi TK, et al
. Prevalence of adverse drug reactions at a private tertiary care hospital in south India. J Res Med Sci 2011;16:16-25.
Seger D, Barker K, McNaughton C. Misuse of the naranjo adverse drug reaction probability scale in toxicology. Clin Toxicol (Phila) 2013;51:461-6.
Khan LM, Al-Harthi SE, Osman AM, AbdulSattar MA, Ali AS. Dilemmas of the causality assessment tools in the diagnosis of adverse drug reactions. Saudi Pharm J 2015. [Article in press].
[Table 1], [Table 2]