How are competency levels (beginner, intermediate, advanced) determined?

Whether a user has a beginner, intermediate, or advanced competency levels for a particular skill depends on the user's skill score. A skill score is a value between 0 and 1000. Users with a skill score is in the bottom part of the range are beginner, those with skill scores in the middle part are intermediate, and those with skill scores in the top part of the range are advanced.

Users can find their own skill scores on their user profile. Administrators and team managers in an enterprise account can find a summary of the skill scores for users in their enterprise on the Skill Profiles dashboard. The image below shows an excerpt from a sample Skill Profiles dashboard:

Skill scores derive primarily from two data sets:

• The difficulty of exam or quiz questions that a user completes, combined with the correctness of the user responses
• The complexity of the hands-on lab that a user completes, combined with the results of the validation step (Note: Not all hands-on labs affect the user's skill scores.)

The skill scores in QA are not simple counts or percentages of correct answers. Instead, the difficulty level of the questions and hands-on labs that the user complete correctly indicates the user's competency level. Just like users can have  beginner, intermediate, or advanced competency in a skill, questions and hands-on labs are have a beginner, intermediate, or advanced difficulty rating.

Note: Users' performance on a question or hands-on lab actually helps to inform its difficulty rating. For example, if a question about a certain skill is intermediate-level, then the algorithm expects users who are intermediate- and expert-level in that skill to get it right, but not beginner-level users. If a lot of unexpected users are getting the question right or wrong, then its difficulty level might need adjustment. This approach focuses and verifies the level of the question and helps keep the assessments in QA consistent.

Our corporate website contains a page with even more detail about QA Skill Assessment Technology.

To understand the distribution of skill competencies on your Skill Profiles dashboard, it is helpful to understand the assessment design.

Beginner-level questions gauge users' familiarity with the fundamentals of a given service, its place among other services and how it connects with them, and its general use cases. The beginner level is the one level where questions can solely focus on what the service is, whereas Intermediate and Advanced questions focus on how to best use the service(s) based on different business and technical objectives. Beginner level questions are useful for people with some computing experience, but who are new to a specific service or cloud computing.

Intermediate-level questions gauge the prerequisite and fundamental knowledge of distributed computing and cloud service architectures, as well as the basic understanding of how a given service functions. Intermediate level questions focus on assessing the user's understanding of how to use the service to achieve basic tasks such as planning and designing cloud architecture, creating applications for the cloud, and maintaining a secure, stable and cost-effective cloud service. Intermediate level questions are useful for people who understand how cloud computing works, and have some familiarity with services, but are expanding their knowledge to new areas.

Advanced-level questions assess knowledge that comes with extensive experience with how a service or related service works, as well as how to use the service to achieve specific tasks. Advanced questions assess users on complex concepts and often provide scenarios with multiple (sometimes competing) variables that require the user to weigh trade-offs and think critically to analyze a given situation.

Detailed reference knowledge is not required to correctly answer advanced level questions. It is not the goal of advanced questions to test encyclopedic knowledge of a given service. Rather, advanced questions are designed to gauge the complex model and critical thinking that correlates with the experience levels of expert-level practitioners in a given domain.