University of Birmingham > Talks@bham > IMA West Midlands Branch > “Calibrate with Confidence”, Integrated panel assessment

“Calibrate with Confidence”, Integrated panel assessment

Add to your list(s) Download to your calendar using vCal

If you have a question about this talk, please contact David Smith.

Suppose your company advertises for new employees and tasks a panel with scoring the applicants. Each panel member has time to assess only a fraction of the applicants but each applicant’s dossier is evaluated by at least two panel members. How should the panel combine their scores to remove the effects of differences in use of the scale and different matches between expertise of the assessor and skills of the applicant? Appointments are a big investment for a company.

Or suppose you are in charge of an examination board for which students are assessed on their selection of options. How do you combine the marks to take into account differences in marking standards and length between the options?

Or suppose a research assessment panel of 15 people evaluates 3000 publications in applied mathematics. Each panel member can evaluate 400 publications so each publication gets two assessors. How should the panel combine their scores to remove the effects of differences in use of the scale and different matches between expertise of the assessor and subject of the publication? The 2016/7 payout to English universities for the results of the REF (for all subjects) was over £1.5B so one ought to try to get it right.

With an undergraduate project student and colleagues from Coventry University, we devised, implemented and tested a method to achieve the above, that we call “Calibrate with Confidence”. I will explain and demonstrate it.

Thanks to SpectraAnalytics, it is available in Excel, Python and GUI from www.calibratewithconfidence.co.uk

Our paper is available at

http://rsos.royalsocietypublishing.org/content/4/2/160760

An article about it appeared in Times Higher Education

https://www.timeshighereducation.com/news/new-algorithm-hopes-counter-potential-bias-funding-panels

This talk is part of the IMA West Midlands Branch series.

Tell a friend about this talk:

This talk is included in these lists:

Note that ex-directory lists are not shown.

 

Talks@bham, University of Birmingham. Contact Us | Help and Documentation | Privacy and Publicity.
talks@bham is based on talks.cam from the University of Cambridge.