Validity and reliability of an objective structured assessment tool for performance of ultrasound-guided regional anaesthesia
Background: We aimed to examine the validity and reliability of previously developed criterion-referenced assessment checklist (AC) and global rating scale (GRS) to assess performance in ultrasound-guided regional anaesthesia (UGRA). Methods: Twenty-one anaesthetists’ single, real-time, UGRA proced...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Published: |
Elsevier
2018
|
| Subjects: | |
| Online Access: | https://eprints.nottingham.ac.uk/52635/ |
| _version_ | 1848798772704837632 |
|---|---|
| author | Shafqat, Atif Rafi, M. Thanawala, Vishal Bedforth, Nigel M. Hardman, J.G. McCahon, Robert A. |
| author_facet | Shafqat, Atif Rafi, M. Thanawala, Vishal Bedforth, Nigel M. Hardman, J.G. McCahon, Robert A. |
| author_sort | Shafqat, Atif |
| building | Nottingham Research Data Repository |
| collection | Online Access |
| description | Background: We aimed to examine the validity and reliability of previously developed criterion-referenced assessment checklist (AC) and global rating scale (GRS) to assess performance in ultrasound-guided regional anaesthesia (UGRA).
Methods: Twenty-one anaesthetists’ single, real-time, UGRA procedures (total: 21 blocks) were assessed using 22-item AC and 9-item GRS scored on a 3-point and 5-point Likert scales respectively. We used one-way ANOVA to compare assessment scores between 3 groups (group I: ≤30 blocks in the preceding year; group II: 31–100; group III: >100). Concurrent validity was evaluated using Pearson’s correlation (r). We calculated type-A intra- class correlation coefficient (ICC) using an absolute agreement definition in two-way random effects model, and inter-rater reliability (IRR) using absolute agreement between raters. The inter-item consistency was assessed by Cronbach’s alpha (α).
Results: Greater UGRA experience in the preceding year was associated with better AC [F (2,18) 12.01; p <0.001] and GRS [F (2,18) 7.44; p =0.004] scores. There was strong correlation between mean AC and GRS scores [r=0.73 (p <0.001)] and strong inter-item consistency for AC (α = 0.94) and GRS (α = 0.83). The ICC (95% CI) and IRR (95% CI) for AC was 0.96 (0.95 – 0.96) and 0.91 (0.88 – 0.95) respectively and 0.93 (0.90 – 0.94) and 0.80 (0.74 – 0.86) for GRS.
Conclusions: Both assessments differentiated between individuals who have performed fewer (≤30) and many (>100) blocks in the preceding year, supporting construct validity. It also established concurrent validity and overall reliability. We recommend both tools may be used in UGRA assessment. |
| first_indexed | 2025-11-14T20:25:05Z |
| format | Article |
| id | nottingham-52635 |
| institution | University of Nottingham Malaysia Campus |
| institution_category | Local University |
| last_indexed | 2025-11-14T20:25:05Z |
| publishDate | 2018 |
| publisher | Elsevier |
| recordtype | eprints |
| repository_type | Digital Repository |
| spelling | nottingham-526352024-08-15T15:29:51Z https://eprints.nottingham.ac.uk/52635/ Validity and reliability of an objective structured assessment tool for performance of ultrasound-guided regional anaesthesia Shafqat, Atif Rafi, M. Thanawala, Vishal Bedforth, Nigel M. Hardman, J.G. McCahon, Robert A. Background: We aimed to examine the validity and reliability of previously developed criterion-referenced assessment checklist (AC) and global rating scale (GRS) to assess performance in ultrasound-guided regional anaesthesia (UGRA). Methods: Twenty-one anaesthetists’ single, real-time, UGRA procedures (total: 21 blocks) were assessed using 22-item AC and 9-item GRS scored on a 3-point and 5-point Likert scales respectively. We used one-way ANOVA to compare assessment scores between 3 groups (group I: ≤30 blocks in the preceding year; group II: 31–100; group III: >100). Concurrent validity was evaluated using Pearson’s correlation (r). We calculated type-A intra- class correlation coefficient (ICC) using an absolute agreement definition in two-way random effects model, and inter-rater reliability (IRR) using absolute agreement between raters. The inter-item consistency was assessed by Cronbach’s alpha (α). Results: Greater UGRA experience in the preceding year was associated with better AC [F (2,18) 12.01; p <0.001] and GRS [F (2,18) 7.44; p =0.004] scores. There was strong correlation between mean AC and GRS scores [r=0.73 (p <0.001)] and strong inter-item consistency for AC (α = 0.94) and GRS (α = 0.83). The ICC (95% CI) and IRR (95% CI) for AC was 0.96 (0.95 – 0.96) and 0.91 (0.88 – 0.95) respectively and 0.93 (0.90 – 0.94) and 0.80 (0.74 – 0.86) for GRS. Conclusions: Both assessments differentiated between individuals who have performed fewer (≤30) and many (>100) blocks in the preceding year, supporting construct validity. It also established concurrent validity and overall reliability. We recommend both tools may be used in UGRA assessment. Elsevier 2018-07-31 Article PeerReviewed Shafqat, Atif, Rafi, M., Thanawala, Vishal, Bedforth, Nigel M., Hardman, J.G. and McCahon, Robert A. (2018) Validity and reliability of an objective structured assessment tool for performance of ultrasound-guided regional anaesthesia. British Journal of Anaesthesia, 121 (4). pp. 867-875. ISSN 1471-6771 Anaesthetists; Checklist; Educational assessment; Reproducibility of results; Ultrasound https://www.sciencedirect.com/science/article/pii/S0007091218305233?via%3Dihub |
| spellingShingle | Anaesthetists; Checklist; Educational assessment; Reproducibility of results; Ultrasound Shafqat, Atif Rafi, M. Thanawala, Vishal Bedforth, Nigel M. Hardman, J.G. McCahon, Robert A. Validity and reliability of an objective structured assessment tool for performance of ultrasound-guided regional anaesthesia |
| title | Validity and reliability of an objective structured assessment tool for performance of ultrasound-guided regional anaesthesia |
| title_full | Validity and reliability of an objective structured assessment tool for performance of ultrasound-guided regional anaesthesia |
| title_fullStr | Validity and reliability of an objective structured assessment tool for performance of ultrasound-guided regional anaesthesia |
| title_full_unstemmed | Validity and reliability of an objective structured assessment tool for performance of ultrasound-guided regional anaesthesia |
| title_short | Validity and reliability of an objective structured assessment tool for performance of ultrasound-guided regional anaesthesia |
| title_sort | validity and reliability of an objective structured assessment tool for performance of ultrasound-guided regional anaesthesia |
| topic | Anaesthetists; Checklist; Educational assessment; Reproducibility of results; Ultrasound |
| url | https://eprints.nottingham.ac.uk/52635/ https://eprints.nottingham.ac.uk/52635/ |