Dr Nicola Taylor

Addressing fairness IN selection

Designing a fair process

Using assessments that have good local evidence for reliability and validity (for the applicants that you are likely to assess) is a prerequisite for being able to apply them fairly to any employee (or applicant).

Download page as pdf

Introduction

There are many reasons why we should consider fairness as an integral part of the selection process in organisations. One of the reasons many organisations in South Africa place emphasis on fairness has to do with the fact that the Employment Equity Act explicitly requires the application of the fairness principle throughout the selection process.

Another legal aspect could be the avoidance of litigation – applicants who experience the selection process as fair are less likely to sue or become litigious if they are not selected (Bauer et al., 2020).

Besides these legal aspects, there are other benefits to the organisation that fairness in the selection process can bring about. Research into the perception of fairness in the US Military’s selection process showed that applicants who experienced the process to be fair were more likely to accept the job offer than those who did not (Harold et al., 2016).

The credibility of the business is also improved when applicants perceive the process to be fair (Bauer et al., 2020). There is some evidence that fairness in the selection process has a positive influence on job performance (Konradt et al., 2016).

And finally, having a fair selection process also improves the likelihood that the organisation will take more diverse employees on board and be more inclusive in their processes (Bonaccio et al., 2019).

Fairness in the selection process holds different meanings depending on the perspective from which a person views the process.

Generally, it is accepted that fairness is a social construct, not a psychometric one (SIOP, 2018), so it has more to do with the process followed and decisions made, than an actual property of any assessment or technique used in the process..

However, the quality of assessments will have an impact on whether the selection process is perceived to be fair or not.

According to SIOP (2018), fairness can be described in a number of ways. One possible meaning of fairness is equal group outcomes or equal passing rates, where, for example there are no group differences on a score to be considered.

The problem is that just because there are differences in scores across groups, these do not necessarily indicate unfairness or bias.

This definition was rejected out of hand by the committee responsible for compiling the Standards for Educational and Psychological Testing, and subsequently the committee that compiled the 5th edition of the Principles for the Validation and Use of Personnel Selection Procedures (SIOP, 2018).

A second meaning has to do with the equitable treatment of candidates and their access to the selection process. The third potential definition has to do with the candidate’s ability to perform on any psychological construct without being disadvantaged in some way, and the last definition is that fairness is a lack of bias.

Equitable Treatment

The APA Dictionary of Psychology defines fairness as “the equitable treatment of test takers in order to eliminate systematic variance in outcome scores among people with different racial or cultural experiences and other background influences”. Equitable treatment has to do with the testing conditions and access to the testing process for candidates (SIOP, 2018).

Some of the principles have to do with inclusivity and making sure that people are not unnecessarily excluded from the assessment process due to factors that are unrelated to their ability to do the job.

This definition covers some elements of procedural justice, which is discussed under the candidate experience below.

All candidates should have the opportunity to access the same practice material.

There should be opportunities to retest where reasonable.

There should be accommodation for people with disabilities to participate in the process.

There should be accommodation for access or the lack thereof to the technology required to participate in the selection process (SIOP, 2018).

access

Accessible testing situations allow all test takers to demonstrate their standing on a given construct without being advantaged or disadvantaged by their individual sociographical characteristics.

This principle highlights the need to ensure that any construct that is measured is not affected by factors such as age, race, gender, socioeconomic status, cultural background, disability, and so on.

This also speaks to the validity of the construct for different groups (SIOP, 2018).

lack of bias

Bias can loosely be defined as systematic error in test scores that affects performance of different groups in a way that is unrelated to the construct being measured (SIOP, 2018).

While a lack of bias certainly doesn’t equate to fairness or even ensure fairness, you cannot have a fair process with biased instruments, so it plays a role in ensuring fairness in selection using assessments.

There are two main types of bias that have bearing on fairness, namely measurement bias and predictive bias.

measurement bias

Measurement bias has to do with systematic error in the actual score obtained, in other words, where different groups receive systematically higher or lower scores on a given item or construct due to factors that should be unrelated to performance on the construct.

This is typically checked using DIF analysis (differential item functioning) and item response theory techniques. It is important to note that measurement bias in and of itself is very difficult to identify, as often item-level differences do not roll up to show differences at the test level.

predictive bias

Predictive bias has to do with error in the predictor-criterion relationship, in other words the slope of the regression lines in predicting outcomes is different for different groups (SIOP, 2018).

There are very few published studies that show the existence of such a bias, but predictive studies are also difficult to come by.

This is mostly because they require a good, reliable criterion variable to predict, and it is difficult to get the sample sizes you would like for generalisability.

publication bias

In many cases as well, you find what we call publication bias – authors don’t always want to publish studies where their assessments demonstrate bias.

Long story short, it is always important to investigate bias in assessments, but also important to remember that the existence of differences does not always mean the test is biased either.

They could well reflect real differences that need to be accounted for (Theron, 2007).

While there is no single definition of fairness, by working with various definitions we gain a comprehensive understanding of how to approach fairness in selection.

While each approach on its own doesn’t guarantee a fair process, if we take them all together, then we have a better chance of compiling a fair selection process.

This paper explores a number of topics relevant to both perceived and procedural fairness that should be taken into consideration when designing a fair selection process.

We will start with designing a fair process, then look at the candidate experience, and then discuss making fair decisions.

designing a fair process

the candidate experience

making fair decisions

designing a fair process

“The objective of personnel selection is to add value to organisations by maximising the performance of employees by regulating the quality of employees moving into, up, and out of the organisation” (Theron, 2007, p. 114).

In order to maximise the performance of employees in an organisation, you need to understand the elements that contribute to that performance, such as the requirements of the job.

By focusing on the requirements of the job, you can work towards a more objective evaluation of applicants for the role, thereby increasing the likelihood of making a fair decision at the end of the process.

Job requirements

To understand what a person needs to be successful in the job, you first need to break the job down into its basic elements and identify those aspects that are critical to success.

Once you have done that, you need to decide what it is that you need to assess or measure, and map these back to the job requirements.

Now that you know what to assess, you need to select the right tools for the process.

job analysis

The first step in any selection process is understanding what behaviours, knowledge, skills, and abilities (or competencies) are required for a person to be successful in the role (Meiring & Buckett, 2016).

Job analysis involves the collection of information regarding the tasks, behaviours, working conditions, and skills required for the job in order to create a comprehensive job description for the role. There are many ways in which job analysis can be done – the type of process followed and the subsequent analysis may depend on the kind of job and position in the organisation (Meiring & Buckett, 2016).

The importance of this procedure is to gain an in-depth understanding of what the job entails and what are the determinants of success in the role. More recent research (Parker et al., 2017) suggests that how the role is conceptualised should include the team and the organisation, as this plays out in person-organisation fit, which is also predictive of performance (Kristof-Brown et al., 2005).

What to assess

Finding the right person for the job and being able to retain them is key to an organisation’s success (Furnham, 2019).

The future performance of any applicant is the key focus of the selection process. However, we do not have direct information on the candidate’s future performance in the position at the time of the selection decision, so we have to use other methods that have been shown to predict performance (Theron, 2007).

Examples of these methods are interviews, reference checks, and assessments of cognitive ability, personality, integrity, interests, and emotional intelligence.

Once the job analysis is complete, the next step is to determine what should be assessed during the selection process in order to distinguish the likely successful candidates from those less likely to be successful (Meiring & Buckett, 2016).

For many, this will be to identify the knowledge, skills, and abilities (or competencies) required for the job, followed by the underlying psychological constructs that speak to the competencies.

It is important to have a clear map of the constructs that are measured to the competencies and behaviours required in order to perform on the job.

Once you know which constructs you would like to assess, then you can decide on the assessments and techniques that best measure those constructs in your context.

Not only is it illegal to use poor quality psychometric assessments in any organisational process, it is also unethical and can lead to unfairness in the selection process.

Select assessments

Section 8 of the Employment Equity Act 55 of 1998 states that “psychological testing and other similar assessments of an employee are prohibited unless the test or assessment being used (a) has been scientifically shown to be valid and reliable; (b) can be applied fairly to all employees; and (c) is not biased against any employee or group.”

It goes without saying that using assessments that have good local evidence for reliability and validity (for the applicants that you are likely to assess) is a prerequisite for being able to apply them fairly to any employee (or applicant). It is critical to note that the issue of bias is not a simple one.

Just because an assessment demonstrates differences between groups on a particular characteristic does not make the assessment itself biased.

It is entirely possible that the assessment reflects issues in the wider society, such as access to quality education and opportunities to gain experience in the field (Theron, 2007). However, if you have not done the proper evaluation of the properties of the assessment to start off with, you will not be able to make an informed decision regarding the appropriateness of the assessment.

“Psychological tests that report […] differences between ethnic groups […] should therefore not be characterised as villains responsible for the problem but rather as unbiased messengers relatively accurately conveying the consequences of a tragic social system” (Theron, 2007, p. 114).

The point of using assessments that have evidence for reliability and validity across groups, even if there are differences, is that what you know, you can manage and plan for. There are many ways in which you can treat assessment results to ensure that they result in being used in a fair and equitable manner (Huysamen, 1995).

If you use an assessment without understanding how it functions across various groups, you will be unable to ensure the fair application of the assessment in the selection process.

Science of selection

In South Africa, if you are a registered psychology professional, you are duty-bound to stay up to date with the latest scientific developments in your field, and to make sure that your work is based on the science of psychology. While this may not be a professional or legal requirement in countries outside of South Africa, the principle should still apply to anyone who has to make decisions about the lives and livelihood of others.

Audit

It is important to regularly determine to what extent your existing selection procedures contribute to predicting success on the job.

Once a selection process has been in place for a while, this is no longer as simple as correlating the initial selection scores with future performance, because you are working with a preselected sample.

Choose an experienced research partner to help you to regularly check the validity of your process, as well as the fairness of the outcomes and the return on investment for your organisation.

One of the most important tests is to ensure that the assessments you use predict performance in your organisation and do so in the same way for different groups (Barnard, 2012; Kriek et al., 1994).

Remember that the fairness of the process does not only depend on the assessment results, but on the entire selection process (Theron, 2007). Huysamen (1994) has reviewed a number of techniques in a helpful way to test the fairness of outcomes of a selection process. There are many methods that can be used to increase the fairness of a selection process, and the method chosen should reflect that way in which the final decision is made (Huysamen, 1995).

validate

It is critical to do research when implementing new processes.

It is easier to determine the effectiveness of a given process when implementing a brand-new process than trying to evaluate the effectiveness of a process that has been in place for some time.

Insist on piloting new processes either with your incumbent employees, or alongside your existing process. While this may carry an additional cost, it is beneficial to know that something doesn’t work at the beginning of the process rather than to discover it further down the line.

There are many exciting developments in the field of psychological assessment and selection. Implementing techniques such as machine learning, artificial intelligence, and neural networks for selecting people for jobs is being bandied about in the popular media as a more objective selection procedure due to the fact that no people are involved in making the selection decision (Melda, 2018; Aon, 2020).

The problem comes in when the algorithms are trained to select on the basis of previously discriminatory practices (IBM Research, 2018).

An example is at Amazon, when training AI algorithms to select “objectively” based on previously successful hires, even though they excluded race and gender, the machine found things like university sport activities that would still recommend the male candidates for the job (Dastin, 2018). So, if you’re basing new selection methods on old biased methods, they will continue to discriminate unfairly (Yankov et al., 2020).

The good news is that, generally, the field is committed to finding ways to reduce bias in decision-making algorithms.

Societies like the Association for the Advancement of Artificial Intelligence (aaai.org) and the Association of Computing Machinery (acm.org) support conferences such as the Artificial Intelligence, Ethics, and Society conference and AI for Good Global Summit that further the responsible and transparent use of AI for the benefit of humankind.

There are also guidelines related to organisational and procedural justice called algorithmic justice that are in place to help improve the fairness of decisions made using modern predictive techniques (Yankov et al., 2020).

Psychologists like Oswald et al. (2020) are also making valuable contributions to the ethical use of big data methods in this space.

review

If you don’t have the data, time, or money to do your own research in-house, one of the most effective ways to inform the decisions you need to make is to study existing and reputable published research.

Make sure that you rely on trusted sources of information whose findings can be verified.

Meta-analyses are the ideal source of information, as they consolidate the findings of multiple other studies in a single place. There are a number of meta-analyses that have been done looking at the impact of various selection methods and their ability to predict performance in the workplace (e.g. Schmidt & Hunter, 1998; Schmidt et al., 2016).

The methods that fall in the top 10 most predictivemethods include assessments of:

cognitive ability
integrity
interests and personality
interviews
reference checks
biographical data

Combining these with assessments of emotional intelligence, indicators of fit to the organisation, and certain assessment centre exercises can help you to build a defensible selection process (Schmidt et al., 2016).

plan

In order to be able to do research in your organisation and to make decisions based on data, you need to have your employee data available to you in such a way that it is easy to use.

Working with data in fragmented worksheets, across different HR systems and databases has to be a way of the past. In fact, the various pieces of legislation regarding the protection of personal information require organisations to have good control over the data that they have access to (Protection of Personal Information Act 4 of 2013; Regulation (EU) 2016/679).

In a recent survey, Accenture found that over 90% of employees were willing for their organisation to collect and use data on them in their work, on condition that they benefit from this in some way (Shook et al., 2019).

The better you can track, monitor, and combine different sources of data, the more informed decisions you will be able to make regarding the needs of the organisation. Plan to manage your employee data effectively to make better decisions.

candidate experience

There is a vast body of work dedicated to the candidate’s experience of fairness during the selection process.

Research shows that a candidate’s experience can have an impact on how they view the company, whether they accept the job offer or not, how they take the rejection, and how they perform once employed (Bauer et al., 2020; Harold et al., 2016; Konradt et al., 2016).

Ployhart and Harold (2004) demonstrated that when candidates have a negative assessment experience, it can result in subsequent negative attitudes towards tests and the selection process, as well as potential withdrawal from the selection process or even litigation.

Additional consequences could include a loss of the organisation’s existing customer base, lack of engagement on hiring, and the impact on the candidate’s well-being (Bauer et al., 2020).

Procedural justice

The organisational justice literature was used as a framework to inform the way in which candidates perceive the selection process to be fair.

Organisational justice has to do with the extent to which employees experience the way in which organisations make decisions and apply them to be fair (Gilliland, 1993). It is typically categorised into two aspects: procedural justice and distributive justice.

Procedural justice is when employees perceive the way in which the organisation makes decisions to be fair and distributive justice is when employees perceive the way in which people are rewarded for their input to be fair (Gilliland, 1993). There is a wide body of research for each of these two concepts.

Gilliland (1993) came up with a set of ten rules that all selection procedures should follow to enhance the perception of fairness with applicants, known as procedural justice. While these are decades old, they still carry relevance today, perhaps even more so with virtual assessment techniques on the rise (Bauer et al., 2020).

His ten guidelines suggested for ensuring a good candidate experience are described briefly below:

Tests and information requested of the candidate should be related to the job requirements.

The candidate should have the opportunity to demonstrate their skills, knowledge, and abilities, and standing on various psychological constructs during the process.

The candidate should have the opportunity to challenge decisions or be allowed a second chance to be considered for the position.

The process should be consistent over time and across candidates.

The candidate should receive timely feedback on their performance during the selection process.

Explanations for why certain procedures are used should be provided.

Administrators should be honest when communicating with the candidate.

The candidate should be treated with warmth and respect.

The candidate should have the opportunity to ask questions and engage in discussion during the process.

The questions or information asked of the candidate should be appropriate and legal.

In local research on the perception of fairness in selection, Visser and Schaap (2017) reported that job applicants typically have a positive view of the use of cognitive and personality assessments in the selection process in South Africa.

This is in line with older research (Visser & De Jong, 2001) that also showed that South Africans viewed the use of ability and personality assessments in the selection process more positively than French and US job applicants. South Africans reported that the face validity and job-relatedness of the selection procedures were most important to them in the determination of the fairness of the selection process (Visser & De Jong, 2001).

Inclusivity

Diversity and inclusion are important topics in the human resource management literature at the moment. While they are related constructs, they are also distinct.

Definitions of diversity typically have to do with the biographic or demographic characteristics of a person and their representation in an organisation (Shore et al., 2020).

Inclusion definitions typically have to do with a culture in the organisation where all employees are able to fully participate and contribute, no matter what their biological or sociological characteristics (Shore et al., 2020).

Diverse companies are 35% more likely to outperform the national average in their industry (Hunt et al., 2015). Inclusivity in the selection process is therefore not only important in terms of fairness principles, but can have an impact on the success of the organisation.

Most of the principles set out in the next section will enhance the candidate’s experience of inclusivity in the selection process. The topics set out below are three areas that in our experience tend to be overlooked outside of age, gender, and ethnicity in the selection process, so will receive special attention here.

Meta-analyses show that while diversity practices may have both positive and negative outcomes, organisations that promote a climate of inclusivity tend to produce more positive outcomes

Mor Barak et al., 2016

Pay attention to language ability.

If being able to read or speak English is not a requirement for the job, then assessments that require this as part of the selection process

would immediately and unnecessarily exclude people based on their English literacy. Some assessments would allow for the items or instructions to be verbally translated, but it is always wise to determine this with the test provider beforehand. The danger of an unstandardised translation is that it could lead to inaccurate results, which would increase the likelihood of an unfair decision being made.

Check technology requirements

As of the beginning of 2020, 62% of the South African population had access to the Internet, which is slightly higher than the global

rate of 59% (Kemp, 2020). While this number is likely to have increased since then, access to the Internet does not always guarantee that all applicants will be able to participate in the selection process. Of the population between the ages of 16 and 64 who have access to the internet, 96% of people access the internet via mobile device, and 83% of mobile connections are prepaid (Kemp, 2020). That means that the majority of users are likely to be concerned about the amount of data used during the application process.

You can make your selection process more inclusive by doing things like zero rating or subsidising the data on your site, providing the amount of data typically used during the selection process upfront, or providing a location for candidates to be able to go to complete the selection process.

With virtual selection processes, you will also need to ensure that there is sufficient support for candidates who experience technical challenges during the process.

The basic principles when using assessments should be applied in virtual settings as well. If the test requires supervision, this can be done using video conferencing techniques, or using security verification procedures. Candidates should have the same opportunity to perform well in the selection process whether they apply virtually or in person (SIOP, 2018).

Accommodate disabilities

The national rate of people with disabilities in South Africa is 7.5%, according to the last national census, but the labour market

absorption is low (Statistics South Africa, 2011). While recruiters may feel that a very low percentage of candidates with disabilities actually apply for positions in their organisation, it may well be that these disabilities are invisible or not disclosed in order to avoid potential discrimination, or that certain aspects of the recruitment process unnecessarily exclude people with disabilities (Bonnaccio et al., 2020).

Take the time to carefully review the demands of your selection process and whether it is likely to exclude candidates with disabilities based on those demands. If the disability is unlikely to impact on performance in the role, excluding them by virtue of your process is an unfair practice. You can also partner with advocacy groups that can advise you on how to improve the inclusivity of your selection process.

Making fair decisions

The pointy end of the selection process is coming to making the final decision. At this stage, the integration of all the information gathered needs to happen, and the recommendations made.

Check your biases

These are not your prejudices (although you should check those too), but cognitive biases or systematic errors in thinking that we may not even be aware that we have when making decisions (Dwyer, 2018).

There are several posts and publications on this topic, and the list of potential biases or “thinking shortcuts” is too long to cover here. One of the seminal works on the topic is Daniel Kahneman’s (2011) book, “Thinking fast and slow”. Dwyer (2018) also provides a good description of a number of these cognitive biases.

One cognitive bias that relates to practitioner competence is the Dunning-Kruger effect (Kruger & Dunning, 1999).

This bias is essentially the researched outcome of the well-known proverb: “A little knowledge is a dangerous thing”.

This effect describes the tendency of people who have only a little bit of knowledge in an area to grossly overestimate their ability, as well as the tendency of experts to underestimate their knowledge (Dwyer, 2018).

Humility and self-reflection are key to keeping yourself honest as a professional – rather assume that you don’t know if you are not sure of your facts.

Stay up to date with research in the field

One of the ethical requirements of practicing in the field of psychology is to ensure that you maintain high levels of competence and standards of best local and international practice.

While you may be competent in the techniques that you use regularly, it is critical to ensure that they are still relevant, and that they conform to best practice in the modern workplace. If your organisation doesn’t have learning circles, or you are an independent practitioner, joining professional societies, community groups, and forums is a great way to keep track of the latest developments in your field.

Many journals also are now available open access, so you do not have to pay to have access to peer-reviewed research.

Make sure you are trained to use the assessments or techniques you use

Make sure you are trained to use the assessments or techniques you use (or at least have access to the manual where training is not required).

While you may be widely read and have a good understanding of the assessments that you use, make sure that you are trained in the use and interpretation where training is required or recommended.

The reason for this is to ensure that the standards in using the assessment are upheld, and that the results are not used inappropriately. If you are not trained, you jeopardise the fairness of the process.

Use as much data as possible from different sources

The decision to select someone cannot be based on a single assessment, nor can it be based on assessments alone. The use of good, reliable, valid psychological assessments on its own is not sufficient to ensure a fair selection process (Theron, 2007).

The most appropriate approach is to use a combination of techniques and assessments that have been shown to be good predictors of work performance.

The research done by Schmidt et al. (2016) on the most predictive tools and techniques in the selection process should inform you in terms of the most valuable elements in the selection process.

Kuncel et al. (2013) also found that often expert predictions of success were less accurate than mechanical predictions (using algorithms to calculate final scores), so removing the influence of human judgement error in decision-making is one of the best ways to increase fairness in the selection process.

Make sure that you make decisions on aspects required for the person to be successful in the job.

This is somewhat a repetition of the first section on job requirements, but it certainly bears repeating here. It is important to roll all the information gathered during the process back up to the basic requirements of the job. The key differentiators between candidates should always be on those aspects most likely to result in success in the role.

conclusion

Fairness is a complex construct, but it is not impossible to achieve fairness in the selection process.

By understanding what is required of the individual to be successful in their role, you can identify objective elements to measure as part of the selection process.

You can design a selection process that the candidate perceives to be fair by paying attention to a few key guidelines. And finally, though good, considered decision-making and keeping yourself up-to-date and upskilled, you can make sure that the decisions you make are also fair.

While this paper deals with several broad topics related to fairness in the selection process, each one of these topics consists of a large repository of research and many useful resources. The intention is not to overwhelm, but to provide some key points for consideration when evaluating the fairness of your selection process.

references

American Psychological Association. (n.d.). Fairness. In APA dictionary of psychology. Retrieved July 24, 2020, from https://dictionary.apa.org/fairness

AON. (n.d.). Artificial Intelligence (AI) in assessment: The 8 questions you need to know the answers to. https://assessment.aon.com/en-us/online-assessment/ai-in-assessment

Barnard, A. (2012). Psychological assessment: Predictors of human behaviour. In M. Coetzee & D. Schreuder (Eds.), Personnel psychology: An applied perspective (p. 134-167). Oxford University Press South Africa.

Bauer, T. N., McCarthy, J., Anderson, N., Truxillo, D. M., & Salgado, J. F. (2020). What we know about the candidate experience: Research summary and best practices for applicant reactions [White paper]. Society for Industrial and Organizational Psychology. https://www.siop.org/Portals/84/docs/White%20Papers/candidate%20experience.pdf?ver=2020-07-02-073420-397