As a market leader in the candidate evaluation industry, Coderbyte takes its responsibilities seriously. We are audited continuously by Assurance Lab, as attested in our SOC 2 reports. Our approach to test validation is rigorous in order to verify that Assessments are accurate and fair.
Overview of validation
The U.S. Equal Employment Opportunity Commission provides a fact sheet for the use of tests and other evaluation methods during employment processes.
In short, candidate evaluation methods should be “validated”, meaning that there is evidence that the test results mean what they are intended to mean. For hiring, the intended meaning of scores is that they are predictive of the qualifications necessary to perform the jobs for which they are deployed to evaluate candidates.
Within Coderbyte’s library, each challenge is assigned a difficulty and estimate time frame which is validated as follows:
- Easy: The challenge can be solved perfectly within the estimated time frame by a developer with a Bachelors of Computer Science degree at an accredited US-based University with deep familiarity of the challenge coding language and/or a developer with at least 1 year of hands-on experience with the designated technology at a competitive technology firm.
- Medium: The challenge can be solved perfectly within the estimated time frame by a developer with a Masters of Computer Science degree at an accredited US-based University with deep familiarity of the challenge coding language and/or a developer with at least 3 years of hands-on experience with the designated technology at a competitive technology firm.
- Hard: The challenge can be solved perfectly within the estimated time frame by a developer with at least 5 years of hands-on experience with the designated technology at a competitive technology firm.
Similarly, Assessment templates can be configured for any difficulty level with the same validation, so long as the default settings are not adjusted.
Results for candidates across our customer base are available on a per challenge basis.
Results for your candidates are available on a per report basis.
Our validation does not extend to custom challenges developed by clients.
Validation evidence
Coderbyte demonstrates validity evidence during the task creation process outlined below. Coderbyte has an experienced customer support team who can help manage local validation studies to collect and document validity evidence at your organization.
Validation process
Each challenge within Coderbyte’s library is developed in a rigorous multi-step, multidisciplinary process to validate the accuracy and fairness of the challenge, while mitigating adverse impact. The process is overseen by our Head of Assessment Science.
Step 1: Challenges and Assessment templates are requested by leading employers using our in-app form that requires a level of difficulty and job description for the requested challenge.
Step 2: Our Content Team, which consists of experienced full-stack engineers, drafts a challenge or Assessment to meet the requirements. Challenges are written in plain-English and avoid cultural references that are not widely understood.
Step 3: The challenge or Assessment is tested by a sample of developers in roles that match the job title to ensure that they can solve it perfectly in the allotted time frame. If relevant, developers in roles less senior are also given the challenge or Assessment to ensure that they cannot solve it perfectly in the allotted time frame, thus indicating an appropriateness of difficulty.
Step 4: The challenge or Assessment is then released in a limited beta to our client base. Candidate results are monitored and feedback is solicited.
Step 5: For candidates that perform poorly, demographic data is collected to evaluate Adverse Impact. If no Adverse Impact occurs, the tests are released to our entire client base.
We encourage candidates and employers with concerns to contact support.