Process Optimisation for Time Series Interviews and Digital Scoring in Medical Dental Recruitment and Selection for NHS-England

The paper focuses on how the efficiency and effectiveness of the Video Interviewing and Digital Scoring (VIDS) system in the NHS recruitment process could be improved. VIDS enables remote participation, reducing logistical costs and providing real-time feedback. However, its implementation is resource-intensive, necessitating significant technical support. The study focuses on optimising the VIDS process, particularly through the standardisation and automation of scoring to ensure consistency and reduce human error. Adopted in 2012, Multiple Mini Interviews (MMIs) were enhanced in 2023 by VIDS, allowing candidates to participate remotely, thus minimising the need for physical assessment centres. The pilot phase of VIDS demonstrated high user satisfaction but highlighted the need for continuous technical support. The study outlines methods for optimising this support, such as diagnostic tools, bypass procedures and streamlined technical assistance. The results confirm that these optimisations contributed to a more efficient recruitment process. While VIDS offers clear benefits, the paper emphasises the necessity of ongoing support and refinement to maximise its effectiveness in future NHS recruitment efforts.

Introduction

Video Interviewing and Digital Scoring (VIDS) have been recognized as reliable, feasible, and well-received by both candidates and evaluators within the NHS. O’Brienet al. (2011) propose that future longitudinal research could further validate the effectiveness of Multi Mini Interviews (MMIs) in evaluating the potential of applicants to succeed as medical professionals. The paper “Your Approach to Hiring Is All Wrong” by Peter Cappelli critiques modern hiring practices, highlighting the inefficiencies and poor outcomes of current recruitment strategies (Cappelli, 2019). Cappelli argues that despite increased hiring activity, companies like large organizations are not necessarily making better hires. This is partly due to the extensive outsourcing of the hiring process to third-party vendors, who rely on superficial metrics and technologies like social media screening and machine learning algorithms. The traditional, rigorous process of evaluating candidates through in-depth interviews and internal promotions has been largely abandoned, replaced by a focus on passive candidates and reliance on unproven technological tools. This shift has led to higher costs, lower retention rates, and less effective hiring practices overall. Cappelli suggests that organizations need to reevaluate their approaches, focusing on developing internal talent, using more reliable evaluation methods, and measuring the success of their hiring practices based on long-term employee performance (Cappelli, 2019).

VIDS being used for MMIs could significantly address many of the inefficiencies discussed by Cappelli in 2019. By integrating VIDS into the hiring process, companies can enhance the accuracy and reliability of candidate assessments through standardized digital scoring, reducing bias and ensuring consistent evaluation criteria across all candidates. Additionally, VIDS allows for a more structured and efficient interviewing process, which can mitigate some of the issues caused by the ad-hoc methods currently in use. The real-time analysis and feedback provided by VIDS also enable quicker decision-making and better alignment with long-term hiring goals, potentially improving both the quality of hires and retention rates. Therefore, VIDS represents a practical tool for optimizing modern hiring practices, aligning them more closely with the needs of organizations seeking to improve their recruitment outcomes. Despite the benefits, the VIDS process is resource-intensive, requiring significant effort in planning, development, implementation, and evaluation (Kropmanset al., 2023). Process optimization and efficiency need further investigation rather than research into the validity and reliability of the original MMIs.

Efficiency and process optimization are pivotal in the integration of VIDS systems within the NHS. The shift from traditional, face-to-face interviews to digital platforms has markedly streamlined the recruitment process by reducing the necessity for physical resources such as assessment centers and travel logistics. The implementation of VIDS has enabled a more efficient allocation of time and financial resources, with candidates now able to participate remotely, thereby lowering logistical costs and minimizing the carbon footprint associated with large-scale recruitment events. Additionally, the digital nature of VIDS enhances efficiency by offering real-time statistical analysis and immediate feedback, which significantly reduces the time between interview completion and decision-making (Cameron & MacKeigan, 2012; Kellyet al., 2014).

Process optimization within VIDS is primarily driven by the standardization and automation of scoring, which minimizes human error and ensures consistency in evaluations. The use of digital scoring tools standardized assessment criteria, resulting in more reliable and unbiased evaluations (Callwoodet al., 2022). This automation facilitates instant analysis and feedback, allowing for the rapid identification of areas needing improvement, whether in the MMI station design or in examiner performance. Moreover, tailored technical support enhances the user experience, ensuring smooth operations during the initial implementation phases (Van Der Wantet al., 2021). By optimizing both the technical and administrative components of VIDS, the NHS can achieve higher efficiency, ultimately contributing to a more effective and equitable recruitment process.

The Medical and Dental Recruitment and Selection (MDRS) unit, responsible for recruiting healthcare professionals across the NHS in England, Scotland, Wales, and Northern Ireland, adopted MMIs as part of its selection process starting in 2012. This approach, pioneered by Kevin W. Eva in 2004, employs a series of stations to evaluate applicants’ non-cognitive skills (Evaet al., 2004; Pauet al., 2013). The strategy indicates that MMIs are becoming increasingly integral to recruitment processes in Medicine, Dentistry, and Nursing and referring to Capelli’s findings (2019), hopefully in other non-medical areas as well.

Substantial support was provided to NHSE/MDRS during the initial rollout of VIDS between January and May 2023. At least one in-house assessment coordinator/administrator and one troubleshooter were required per assessment event during the pilot stage to familiarise users with the new system. The pilot phase saw a user satisfaction rate of 86.7%, surpassing expectations. VIDS also allowed interviewees to participate from their homes or offices, reducing the need for physical assessment centers and associated travel costs (Kropmanset al., 2023).

The adoption of VIDS in recruitment processes is proving advantageous in terms of resource allocation and convenience for both applicants and NHSE. Using MMIs (2–4 stations) has significantly reduced the need for assessment centers and travel costs. However, implementing VIDS also requires additional staffing and technical support, particularly for applicants unfamiliar with the system. Findings suggest that one administrator can manage 30–40 interviews per day per specialty, while one technical support person per specialty is necessary to provide 24/7 support before and during the interview event. These insights are invaluable for future planning of remote MMI events within the NHS but might need further optimisation after its first year of roll-out.

Optimization of technical support for applicants using VIDS is crucial to ensuring a seamless and effective recruitment process. Given that many applicants may be unfamiliar with the digital platforms used in VIDS, tailored technical support is essential to minimize disruptions and enhance the overall user experience. During the initial implementation phases, particularly in the pilot stage, providing real-time assistance through dedicated technical support teams proved invaluable in addressing user concerns and technical challenges promptly. By ensuring that technical support is readily available, especially on the days of interviews, the NHS can significantly reduce the potential for technical failures that could impact the fairness and accuracy of the assessments. There have been firewall issues with local NHS trusts, meaning that a lot of clinicians have been advised to interview from home on personal devices. Furthermore, optimizing this support through well-trained staff and clear communication channels not only improves the user experience but also increases applicants’ confidence in the digital process, leading to higher satisfaction rates and smoother operations overall (Callwoodet al., 2022).

This paper aims to identify ways to optimize the support process provided to NHS England/Medical Dental Recruitment and Selection (NHSE/MDRS) for the use of VIDS and MMI online technology. The focus of this study is on optimizing the support provided after the first year of implementation, spanning May 2023 to May 2024. The report highlights how technical support for VIDS and MMIs can further reduce the need for physical resources while necessitating additional staffing and technical adjustments for participants utilizing the system for the second time. VIDS is deemed reliable, feasible, and well-accepted by both candidates and evaluators within the NHS. O’Brienet al. (2011) suggest that future longitudinal research could provide further insights into the validity of MMIs as an effective tool in assessing the potential of applicants to become successful medical professionals (O’Brienet al., 2011). In addition, a 2019 systematic review by Yusoff confirmed the satisfactory predictive validity of MMI, initially proposed by Eva and colleagues (Yusoff, 2019). Despite its merits, the VIDS process does face some constraints, notably its resource-intensive nature, which necessitates significant efforts in planning, developing, implementing, and evaluating (Kellyet al., 2014; Kropmanset al., 2023) as well as above-mentioned firewall issues at local trusts.

Methods

Similar to the year prior, including a pilot phase between January and May 2023, the medical and dental specialty interviews from July 2023 to July 2024 were analyzed. Each applicant underwent Multiple Mini Interviews (MMIs) that consisted of 2 to 4 stations. The content for each station was varied and was specifically tailored based on job requirements, specialty-dependent competency frameworks, and the scoresheets relevant to each specialty. We refer to the method section published before (Kropmanset al., 2023).

As before, the process involved applicants participating among various specialties across England, Wales, Scotland, and Northern Ireland, with almost all interview events being held on Video Interviewing & Digital Scoring System (VIDS) developed by Qpercom’s Advanced Assessment Solutions, intended for remote live application interviews. The timeframe for the MMIs and accompanying online support spanned similar to the 2022–2023 6-month period, with applicant feedback being gathered immediately upon completion of the online MMI interview. Subsequently, the collected feedback was analyzed over the course of June 2024.

The Implemented Support Process

The support process that was put in place involved:

1. Deployment of a diagnostic web tool for applicants, prompting them to verify their technical setup before interviews.

2. Integration of the Pubble-ASK (customer support web chat) tool into both the applicant web pages and the diagnostic web tool.

3. Activation of Pubble-ASK only when an applicant failed a component of the diagnostic test.

4. Provision of live, dedicated agents to respond to Pubble-ASK queries.

5. In 2023–2024, a ‘bypass’ option was introduced, allowing applicants with poor connections to proceed while making them aware of potential connection problems (such as loss of video/audio or complete disconnection). However, applicants with audio issues were not permitted to proceed to the interview.

6. Creation of a Slack command, allowing dedicated agents to quickly bypass applicants who required immediate support.

The Improved Support Process

In 2022–2023, there was only one bypass option, which was managed by the administrator (Fig. 1). Based on last year’s feedback, the ‘self-bypass request’ was introduced (Fig. 2), which went straight to the administrator rather than the support team to create a more streamlined process (Fig. 3). The support delivery was performed by one support agent between 8.00–16.00, one support agent between 09.00–17.00, and a 24/7 team covering the ‘in-person live support’ to ensure an optimal applicant experience. Furthermore, an ‘applicant status feature’ was introduced within the interview monitor, meaning that a 15-minute manual ‘agent check’ was no longer required. Administrators could now check in ahead of the interviews and meet with upcoming applicants remotely online.

Fig. 1. Project flow chart in 2022–2023 with only one bypass option (source: Kylee Fort).

Fig. 2. Improved support process flowchart 2022–2023 with one bypass options (source: Kylee Fort).

Fig. 3. Stages of a bypass request in 2023–2024 with three bypass options (source: Kylee Fort).

Results

Over the course of one year, 71 professional groups within the NHS conducted all year-round remote interview events across the UK. A total number of 26,390 self-assessment and interview scoresheets were submitted between July 2023 and July 2024 using the self-assessment and digital scoring functionalities of the Video Interviewing and Digital Scoring (VIDS) platform. The median number of applicants was 105 per event or 556 applicants per interview day, recorded between the fourth week and the sixteenth week.

A total of 920 support conversations were conducted, of which 629 were unique users. Of all the conversations, 64% happened a few days prior to the scheduled interview dates. 35% of the conversations took place outside 08.00–17.00 or at the weekend. The overall average response time was 2 min and 9 s, whereas the response time between 08.00–17.00 was 1 min 12 s. The average response time outside of working hours was 3 min 56 s.

The distribution of queries over the hours of the day varied a lot over 24 h, with a peak around 09.00 and 10.00 (Fig. 4). Queries during the night were mostly due to different time zones calling in for support prior to the interviews scheduled.

Fig. 4. Number of queries and their distribution by hour of the day.

Support was required during all weekdays, with peaks on western working days (Monday–Friday) (Fig. 5). Approximately 160 queries or more were asked on Mondays, Tuesdays, Wednesdays, and Fridays.

Fig. 5. Distribution of queries based on the days of the week.

Most queries were dealt with immediately, while some others took a bit longer (Fig. 6). Support agents kept queries open until the interview day had passed, allowing them to continue the same conversation if the applicant inquired again. This practice explains why some queries have extended resolution times.

Fig. 6. Distribution of queries and their resolution times.

Most queries (58.7%) came from applicants applying for a training place in 20 or more different Medical Specialty Training (MST), with an average of 2.9% of queries per specialty. Dental Core Training (DCT) applicants were responsible for 22.9% of the queries (Fig. 7). Query types mostly revolved around RTT-Round Trip Time and Quality (audio and connections) and were the most frequent queries that came into support. A high number of reports of audio issues were largely related to Bluetooth (not recommended). There were spikes in conversations when incidents occurred, and agents were able to reassure applicants live during the F2F video connection. Another 18.4% were related to various other, mostly generic/technical/procedural queries.

Fig. 7. Distribution of queries based on specialty.

Discussion

Throughout the pilot stage in 2022–2023, valuable feedback was gathered from applicants and recruitment staff on the VIDS process and remote MMIs. It was found that 25% of medical applicants and 17% of dental applicants reported experiencing issues during the process. These issues, as reported last year, have led to improvements in the VIDS software and process listed here. Considering the total number of interviewees being seen, NHSE was almost two (1.98) times more efficient compared to the 7225 (14450 PA) applicants interviewed during the first 6 months of the pilot via the Video Interviewing and Digital Scoring (VIDS) platform (Kropmanset al., 2023).

In contrast to last year’s report, this year, 64% of the queries were coming in days before the interview event, and 35% were coming in during out-of-hours support and appeared to be valid for 2024. We have the impression that applicants are better prepared from a technical standpoint, and still, 35% of queries need to be sorted around and close to the interviews scheduled. The in-person support was valid for both the software provider ( Qpercom Ltd) and  NHSE/MDRS. It helped engineers and interview administrators to learn from users how they use the video interviewing software. Moreover, the Observer view feature has helped navigate the platform easier for observers/invigilators, who join the interview(s). This highlights the importance of gathering feedback in order to enhance the efficiency of the platform.

Despite a high user satisfaction rate of 86.7% last year that exceeded expectations, considerations were still made in order to improve the user experience even more. The ability of almost all interviewees, interviewers, observers/invigilators, and administrators to join their interviews from the comfort of their homes or office space added to their overall satisfaction. Moreover, improving the efficiency of the software and support package improved the experience of all involved but still such a remote process with a considerable amount of moving parts in terms of applicants, going through multiple video stations, meeting various interviewers, and completing different tasks using various graphical media still needs an appropriate level of technical support before and during the events.

During the pilot phase of the VIDS system and remote MMIs, it became evident that there were several advantages to this new online recruitment and selection approach. Firstly, the convenience for participants cannot be overstated. Being able to join multiple interviews from the comfort of their own homes or offices eliminated the need for participants to travel to assessment centers and reduced the cost and inconvenience. Moreover, the remote interviewing process allowed the NHSE to avoid the expense of renting additional office space for assessment centers and paying large sums in travel expenses. Last but not least, the amount of data that becomes available in the course of these evaluations might be of crucial value for NHSE/MDRS and other NHSE trusts in the future (Capelli, 2019).

In conclusion, valuable insights were gained through feedback from participants and observations during the second year of the support system in all four nations. As a result, several improvements have been listed, developed, and will be implemented. Applicants will have their own individual electronic portfolio management system to download their personal feedback and the ability to upload and answer Multiple Choice and Essay questions and Situational Judgement Tests (SJTs). Moving forward, the goal is to continue gathering feedback from end-users and applying design thinking principles to further enhance efficiency in remote MMIs as the most efficient strategy for recruitment, selection, and follow-up assessments.