NBME Self Assessments: Ultimate Guide for the USMLEs and Shelf Exams

by Alec Palmerton, MD in Plan

8 comments

The NBME Self-Assessment Services are the cornerstone of evaluating your USMLE progress. Also known as the NBME Practice Exams, they predict your Step 1 and Step 2 CK scores. Versions also exist for Shelf exams, which can also be used to predict your all-import Shelf scores.

In my work as a USMLE tutor, they are critical to:

• Establish a baseline,
• Track your progress over time
• Assess the effectiveness of our studying

In this article you will learn:

• The different kinds of NBME practice exams, and when to use them;
• How much NBMEs may over-/underpredict your final score;
• Basics like where to sign up, how much they’ll cost, and how many exams there are;
• The surprising time you should take your first NBME;
• Whether you should take your test self- or standard-paced;
• Which NBMEs don’t give you an actual predictive score
• Why some students score lower than their predicted score (and what to do about it);
• Much more

Updated: April 5, 2020 to reflect explanations being added to NBMEs in 2020

Table of Contents

Don’t Take Your USMLE Without Taking an NBME Self Assessment

The National Board of Medical Examiners (NBME) writes the USMLE and Shelf exams. To help students prepare, the NBME also release practice exams. These NBME practice exams are the official practice exams for the USMLEs and Shelf Exams. (The “NBME Self-Assessments” are often referred to as simply “NBMEs.” You will often see them referred to by their number, as in “NBME 18,” or “NBME 20”).

Currently, there are:

• 6 NBME Self Assessments for Step 1
• 3 NBME Self Assessments for Step 2 CK
• 26 NBME Subject Exams for Shelf exams
• 1 NBME Self Assessment for Step 3

So what’s the big deal with the NBME Self-Assessments? The NBME tracks students’ scores and measures them against their actual scores. As such they are the most accurate predictors available of what your USMLE/Shelf exam scores will be.

Your USMLE scores are a critical part of your residency application. As such, you shouldn’t take your Step 1 or Step 2 CK without having taken at least one NBME Self-Assessment. (And ideally multiple).

The Basics: Cost, Where to Sign Up

• All NBME Self-Assessments cost $60 (see below for more).
• Subject (Shelf) exams cost $20 each.

To sign up:

• Go to the NBME Self-Assessment Services website here.

Anyone Can Sign Up

Essentially, anyone can register for the NBME Self-Assessment Services practice exams.

You do NOT need to have:

• Registered for any of the USMLEs,
• ECFMG-certification (for IMGs), or
• Have any verifiable form of credentialing.

The Self-Assessments Are Shorter Than the Real Exams (But Each Block is Longer)

Each USMLE has 40-question blocks that are 1 hour each. Note, however, that the blocks for each NBME are longer.

Below are the names for each NBME and what USMLEs they correlate with:

Predictive Value of NBMEs

Officially, these are the predictive value of the NBMEs for:

• Step 1: 2/3 of students will score +/- 13 points of their NBME from within 1 week
• Step 2 CK: 2/3 of students will score +/- 15 points of their NBME from within 1 week
• Step 3: No predictive score given

These are the official figures. However, in my experience, I’ve found that:

Step 1 NBMEs Are Highly Predictive

As discussed above, officially 2/3 of students’ Step 1 scores should be +/- 13 points of an NBME taken within a week. Here is data from the National Board of Medical Examiners:

Actual Step 1 Scores Minus Score Predicted by NBME Within 1 Week of Exam (Source: NBME)

What do you notice? The highest frequency group is the “0-9” point group. This means the largest group of students saw a 0-9 point improvement from their most recent NBME. It also implies that the Step 1 NBMEs (slightly) underestimate students’ final scores.

This histogram and personal experience show that Step 1 NBMEs are quite accurate. It is uncommon to see a student score more than +/- 5 points differently than their last 2 NBMEs. The difference of +/- 10 points is even rarer.

What does all this mean? It’s unwise to take your test if your NBMEs aren’t close to your desired Step 1 score. This is doubly important for those who are at risk of failing. Most students who fail their USMLE have had fair warning. However, when they explain their decision to take the test, they often say they were:

• Hoping to have a good day, or
• They felt the need to meet a particular deadline (usually self-imposed)

Step 1 NBMEs are very accurate. You should expect your final Step 1 score to be close to your most recent NBMEs.

(To read Are You Ready to Take Your USMLE or Need More Time?, click here).

Step 2 CK NBMEs May (Substantially) Underpredict Scores

The Step 2 CK NBME Assessments are less predictive than the Step 1 Assessments. The NBME’s own data support this. 2/3 of students will score within 13 points of their Step 1 NBME but within 15 points of their Step 2 CK NBME.

Dig deeper, however, and the story gets even murkier. Here is the histogram provided from the NBME:

Actual Step 2 CK Scores Minus Score Predicted by NBME Within 1 Week of Exam (Source: NBME)

Take a moment to compare it to the Step 1 histogram above. What do you notice? There is MUCH greater spread in scores for Step 2 CK relative to the NBME than for Step 1. Specifically, Step 2 CK scores seem to skew even more positive from the predicted scores.

Side-by-Side Analysis Shows Degree of Step 2 CK NBME Underpredicton of Final Score

Here are the data presented side-by-side:

Actual Step 1 and Step 2 CK Scores Minus Score Predicted by NBME Within 1 Week of Exam

Viewed side-by-side, it appears Step 2 CK NBMEs likely underpredict your final score. (And probably by a lot). Here are some interesting statistics, according to the NBME’s own data:

Likelihood that final score is equal/greater than last NBME within 1 week:

• Step 1: 68%
• Step 2 CK: 77%

Likelihood that final score is ≥ 10 than NBME within 1 week:

• Step 1: 31%
• Step 2 CK: 50%

Likelihood that final score is ≥ 20 than NBME within 1 week:

• Step 1: 7%
• Step 2 CK: 23%

Those statistics are pretty remarkable, particularly for Step 2 CK. It implies ~1 in 4 students will score at least 20 points higher on their actual test than predicted by the Step 2 CK NBME. (By comparison, many fewer Step 1 students will accomplish this feat).

~1/3 Chance Actual Step 1 Score Will Be Lower. ~1/4 Chance Step 2 CK Score Will Be Lower.

We can do the same analysis for the likelihood that your score will be lower than predicted.

Likelihood that final score is ≤ 1 lower than last NBME within 1 week:

• Step 1: 32%
• Step 2 CK: 23%

Likelihood that final score is ≤ 11 lower than last NBME within 1 week:

• Step 1: 9%
• Step 2 CK: 8%

Likelihood that final score is ≤ 21 lower than last NBME within 1 week:

• Step 1: 3%
• Step 2 CK: 3%

Here, we can see that the likelihood that there is < 1/3 chance your NBME will overpredict your Step 1 score. (It’s even lower for Step 2 CK).

That’s not to say that you won’t have a negative surprise with Step 2 CK. However, based on the NBME’s data, self-assessments likely underpredict Step 2 CK scores.

This squares with my experience. It is uncommon to see someone score +/- 10 points on Step 1 relative to their previous practice exams. However, swings of +/- 10 points are routine to see for Step 2.

Although imperfect, NBME Self Assessments are as close to a crystal ball as you’ll find in predicting your USMLE score.

Step 2 CK Score Volatility: Due to Subject Matter?

Why is there such a difference in the predictive power of the Step 1 vs. Step 2 CK NBMEs? Remember, there are fewer NBMEs for Step 2 CK (only 3, as opposed to 6 for Step 1). This may hurt their predictive power.

I suspect, however, that the difference in predictive power is due to the exams themselves. Much of Step 1 involves mastery of content. Having a good/bad day, won’t affect my overall knowledge. Thus, my Step 1 score is likely going to be similar to my prior NBMEs.

(To read How Are USMLE Questions Written? 9 Open Secrets for Impressive Boards Scores, click here).

Doing well on Step 2 CK, however, involves much more question interpretation. Knowing what each sentence means in context – in a timed setting – is a huge challenge. If you don’t sleep the night before, your ability to interpret the questions will take a hit. This likely causes both the NBMEs and the final scores to be much more volatile.

(To read The Secret to Scoring 250/260+ You Can Learn Right Now: Question Interpretation, click here).

Shelf Exam Predictive Value Questionable (But Better Than Nothing)

The NBME doesn’t offer data on the predictive power for the Shelf Self-Assessments. There is no official explanation given for why, although we can speculate. The cynical view is that the Shelf NBMEs likely aren’t very predictive. (This has been my experience).

It is not uncommon to see students score +/- 10 from their Subject Exam to their final Shelf score. Why such large swings?

Like with Step 2 CK, Shelf exams depend much more heavily on question interpretation (QI). As we discussed above, QI ability fluctuates day-to-day much more than content knowledge. Thus, you would expect performance on Shelf exams to be much more volatile.

The bigger issue, however, is likely due to sample size. NBME Self-Assessments for USMLEs have ~200 questions each. The Shelf-specific assessments have only 50 questions.

Even the Shelf exams themselves are shorter than the USMLEs. Each Shelf exam is 110 questions, whereas Step 1, which has the fewest questions, has 280 items. Like with all small sample sizes, there will be much more variability.

The Lone Step 3 NBME Doesn’t Give a Predicted Score

There is no score given for the lone NBME. Unlike Step 1 and Step 2 CK, Step 3 has an additional portion: the Clinical Case Simulations. The Step 3 Self Assessment doesn’t have a CCS portion, and so the NBME likely isn’t able to offer a predicted score.

Why Some Students’ Scores Drop Dramatically On Their Real Test

It is rare to score far below your predicted score (see above). You wouldn’t necessarily know that from looking through message boards, though.

It seems like every other post is someone wailing about:

• How their score was so much different than predicted, or
• How the test was NOTHING like any of the UW/NBME questions, etc.

Hysterics aside, there are several reasons why students’ scores may underperform their NBMEs.

Address Anxiety Issues and Don’t Repeat NBMEs

First, remember you should never take an NBME more than once. Repeating the same NBME reduces their predictive power. (Even if it’s been up to a year). I have seen significant (10-15+ points) score overprediction on previously-taken NBMEs. Recalling questions, even vaguely, can change how you take the test, and can inflate your score.

Second, boot-legged copies of NBMEs are unlikely to give accurate scores. Every NBME has a different scale for how much each question is worth. Even if you knew how many items you got wrong, it wouldn’t lead to an accurate score.

Third, statistically, a small number will experience large swings in scores. This is true for normal distributions, although as we saw above, this is rare.

Finally, I’ve known a few students who panicked on their exam. They changed their entire approach (i.e., guessed blindly). Their scores plummeted below their predicted (up to a 70 point drop in one case).

NBMEs Show Each Question’s Answer (But Without Any Explanations)

Over the years, the NBME has given more and more information out for each exam. Previously, they only told students which questions they had gotten wrong. Subsequently, they gave students the correct answer. (But only for the items they got wrong).

Currently, the NBME Self-Assessments will show you the correct answer for each question. However, you will NOT see any explanation of why other responses were incorrect.

Update April 2020: the NBME has already begun adding explanations to some forms, beginning with the Clinical Mastery Series Medicine Forms 5 + 6. They announced that we should expect explanations for other NBMEs later in 2020.

How Should Your Review Your NBME Self-Assessment?

How useful is it to know the answers to the questions if you don’t see why that answer is correct? It depends.

On the one hand, as a way of learning, reviewing an NBME is frustrating. What’s the point of looking over a question if all you know is the right answer? After all, there was a reason you got the question wrong in the first place.

What’s worse, there is no official guide with explanations for each NBME. You may find answers on message boards. However, the quality of those explanations may be suspect.

(To read NBME Self Assessments: Guide to Reviewing Your USMLE Practice Test, click here).

Look for How Many Questions You Miss Because of Interpretation

That said, there is one thing you should do when reviewing an NBME:

Be sure to assess how many questions you missed on topics where your knowledge is already good.

Up to 50% of students’ wrong answers are for “silly” reasons. In other words, they:

• Misinterpret a lab/physical exam finding (or just ignore it),
• Don’t read carefully,
• Misunderstand what the question is asking, or
• Generally, don’t fully apply everything that they know.

Don’t make the common mistake of assuming every wrong question is due to a lack of knowledge. Many of the wrong answers have nothing to do with your depth/breadth of knowledge.

These mistakes of question interpretation are most dangerous if you’re aiming for 240-260+. There is only so much you can increase your knowledge. Much of scoring 240+ deals involves limiting your unforced errors.

Regardless of your goal score, many wrong questions are not due to a lack of knowledge. You can become aware of this from doing the UWorld or Kaplan QBanks. However, it becomes even more evident when you take an NBME exam. This is especially true since you get an estimate of your score afterward.

(To read The Secret to Scoring 250/260+ You Can Learn Right Now: Question Interpretation, click here).

Standard- vs. Self-Paced?

There are two “Timing Modes” for taking the NBMEs. You can either choose “Standard-Paced” or “Self-Paced.” Generally speaking, the “Self-Paced” exams allow you to take up to 4x as much time per section.

Generally, I recommend doing Standard-Paced, even if you’ve been practicing doing untimed questions. Your exam will be at a standard-pacing. With the Self-Assessments, your goal is to predict your final score. The best way to do is to have your practice approximate the real thing as much as possible.

Source: “Administration Period.”  NBME Website: https://nsas.nbme.org/nsasweb/doc/administration_periods.htm

Your First NBME Should Be Before Your “First Pass” Through First Aid/UWorld

One of the biggest mistakes I see students make is that they wait to take their first NBME. Specifically, they wait until they’ve done a “First Pass” though First Aid +/- UWorld. (In other words, they want to review everything in First Aid and UWorld first before doing their first NBME. I don’t recommend this).

It’s understandable. Students often want to wait to take their first NBME because:

• They’re worried it won’t be “accurate” because they haven’t reviewed enough
• A low score would damage their confidence, or
• They already “know” it will be a low score

“What’s the point?” they ask.

Let me be blunt: you should take your first NBME as early as possible. Here are the reasons to take an NBME practice exam at the beginning:

• You need to establish a baseline, but also
• You must know how effective your studying is

Without a Baseline NBME, You Have No Idea How Well You’re Studying

Let’s say that you didn’t plan to take an NBME until 1 month before your exam, and the score on it was 220. Is that good? You have no idea.

Let’s say you’d taken an NBME 2 months before that, and had scored a 180. You would have improved by 40 points in 2 months to a 220: that’d be good progress!

But what if you’d scored a 220 on your NBME 2 months ago, and were still scoring 220 now? In this latter scenario, your score wouldn’t have improved at all. This would indicate an inadequate approach that you should address.

Let’s be clear: you should NOT feel wholly prepared by the time you take your first NBME. That’s not the point! Instead, your first NBME establishes a baseline and gauges your studying effectiveness.

First-Years: Consider Taking an NBME (Long) Before Dedicated Studying

A special piece of advice for first-years: consider taking an NBME long before your dedicated study begins. Some students may even take a practice test as early as the summer between first and second year. As we’ll discuss below, taking an NBME early will help you tailor your studying to the unique demands of the USMLEs.

Note, this may seem like counterintuitive advice. By the summer, you’ve likely studied less than half of the material. A practice test so early will be minimally predictive, so why bother?

We’ve discussed before how the USMLEs test the application of key concepts. In contrast, most med schools often test minutiae – the opposite of what you’ll find on Step 1. As a consequence, many students’ studying is poorly adapted to their future USMLEs. It’s a sad irony that while cramming facts may lead to high med school exam scores, it is a poor method for USMLE studying, let alone clinical practice.

So how does an early NBME fit into all of this? It’s not obvious how different med school exams are from the USMLEs until you take a practice test. However, once you’ve taken your first NBME self-assessment, the differences become apparent. After taking the practice test, you’ll have a better idea how to prepare for Step 1.

If you’ve been cramming PowerPoint slides for two years, you can’t expect to flip a switch and begin to master/understand concepts. Taking an NBME early will help you start the transition to mastery sooner, which will be better for your Step scores – not to mention your future patients.

Do UWSAs Overestimate One’s Score?

There is ongoing controversy whether the UWorld Self-Assessments (UWSAs) overestimate one’s score. The UWSAs are UWorld’s equivalent of the NBME Self-Assessments.

Generally, more often than not, UWSAs seem to overpredict students scores. (You can see students’ actual UWSAs and final scores here). I took both Step 1 UWSAs, which were an underestimate. (Then again they only predicted scores up to 265). More students have seen their scores overestimated than the reverse. That said, I assume that UWorld adjusts their scaling over time, so the answer may change.

My advice: take the UWSA scores with a grain of salt. Instead, you can use them to help simulate an 8-block test, which I recommend. To simulate an 8-9 hour exam, you can do one UWSA first, followed by an NBME. Of the two, I’d use the NBME as an estimate of your score.

(To read 1-2 Weeks Before Your USMLE? Read This, click here).

How Often You Should Take NBME Self Assessments

We’ve already discussed why you should take an NBME early to establish a baseline. But you’re probably wondering: how often should you do an NBME Self-Assessment? The short answer: it depends on your exam and timeline.

Remember, your goal with NBMEs is to track your progress. You do NOT want to run out of them. There is nothing worse than taking your test without knowing what you might score.

That said, here are my recommendations for how often to take NBME practice tests.

Step 1: Every 2-4 Weeks, Depending on Timeline

I recommend students take an NBME every 2-4 weeks. Why? The general goal is to see if you’re making progress, and if so, how quickly.

Peoples’ scores don’t go up week by week. It takes 2+ weeks to see significant progress.

As such, I don’t recommend taking an NBME more frequently than every other week. If your timeline is more than 6 months, I will wait 1-2 months in-between practice exams. Remember, it’s terrible (and avoidable) to run out of NBME Self-Assessments. Space out your practice tests accordingly.

(To read Want to Add 20-60+ Points to Step 1? Here’s How Long It Takes (16 Experiences), click here).

Step 2 CK: Full Self-Assessments Every 1-2 Months, with Subject Exams Every 1-2 Weeks

Remember, the accuracy of the Step 2 CK Self-Assessments is suspect. This is doubly true for the NBME Subject Exams. As such, don’t read into any single score.

Because there are only 3 Self Assessments for Step 2 CK, you need to be cautious with their use. Since the scores are less accurate, your goal should be to track your progress.

To do so, you can use the NBME Subject Exams (e.g., Medicine, Surgery, or Family Medicine) every week. While the individual scores don’t tell you much, the trend and overall scores are useful. Step 2 CK students who scored 260+ I’ve tutored often score in the high 80s/low 90s on the Subject Exams.

Which NBMEs Should You Take First and Last?

So, which NBME Self-Assessment should you take? Which gives the best score prediction? And which exam best simulates the real exam?

The short answer is that there is no clear evidence that one NBME is better than the others. Let me explain.

It Doesn’t Matter Which Exam is “Hardest” Because Each Exam Has Its Own Scale

Each exam has its own scale. This means that questions are worth different amounts of points for various assessments. The upshot? Some tests will be “harder” than others. However, each exam’s scale should take that into consideration.

Sometimes an exam gets a reputation (sometimes deserved) for being a better predictor. Among the old set of Step 1 NBMEs, NBME 18 was thought of as the most “predictive.” However, in March 2019, all but one of the Step 1 Self-Assessments were replaced.

Currently, there is no evidence to suggest one NBME is better than another. Which leads me to my next point: you should take the lowest numbered NBME first.

Take the Lowest Numbered Exam First (It Will Be Retired First)

I recommend starting with the lowest-numbered exam first. (E.g., start with NBME 18, then move to NBME 20, etc.). Why? Because typically, the NBME replaces Self-Assessments beginning with the oldest first. If you’ve taken an NBME that will be replaced, you get a “bonus” NBME to do. If not, you’ve replaced one unseen NBME with another one. (See more below).

Should You Search Message Boards for Question Explanations?

All NBME Self Assessments now give you each question’s correct/incorrect responses. However, should you search for the explanations for those questions?

This is a personal decision. However, I don’t recommend trying to find every question’s explanation.

Time is a scarce resource, particularly in Boards prep. It might help you to reason through each answer in your Self Assessment. However, your time would likely be better spent by doing more Kaplan and UWorld questions.

If you decide to review every answer in-depth, I recommend:

• Setting a time-limit for yourself so you don’t go overboard, and/or
• Discussing the test with someone you trust to know the answers

Also, as a reminder, searching online message boards is a risky proposition since you can’t be sure of the quality of the answers.

When/How to Re-Start NBME Self Assessments If You’ve Had a Large Gap in Your Preparations

Life happens, and plans change. Some students have to delay their dedicated studying/Step 1 exam, sometimes by up to a year. Others may have attempted a USMLE previously, and failed or stopped their preparations. If you are in this position, when should you re-start taking your NBMEs?

You have likely already used a significant number of practice tests. As such, it’s important to space out the remaining exams, so you don’t run out.

Remember, if you can avoid it, you do NOT want to repeat old NBMEs. Even up to a year afterward, I’ve seen NBMEs overpredict scores by 10-20 points. As such, assume you’ll take an NBME 2 weeks before your final test. With any remaining exams you haven’t done, space them out accordingly.

As for the previous NBMEs you took, they will serve as your new baseline. You can measure your progress in your further dedicated studying relative to them.

When Does the NBME Release New Self Assessments?

Typically, the NBME releases one new Step 1 Self-Assessment every year in early March. That said, there are exceptions:

• There were no Step 1 Self-Assessments released in 2018
• 5 of the 6 Self Assessments were replaced in March 2019

The NBME seems to release Step 2 CK Self Assessments less frequently. There isn’t a clear pattern.

There is still only a single USMLE Step 3 NBME.

Taken All the NBMEs Already? Here Are Your Options.

If you’re in the unfortunate scenario of having already used all of the NBMEs, you have a few options.

1. UWSAs Are Your Second Best Option Behind the NBME Self Assessments

First, you can use the UWSAs. Again, these likely overestimate your score, but they’re better than nothing.

2. Repeat NBME Practice Exams with Caution

Second, you can try to repeat the NBME Self Assessments. I only recommend this if you have at least 6 (and ideally 12) months since you took the exam last. It’s even better if you never reviewed the NBME.

Let’s say you took NBMEs 22 and 23, and they initially predicted 160 and 170. If you repeated them 6 months later and scored 200 and 220, I wouldn’t trust the actual predictions.

However, I would take off 20 points as a conservative estimate of the score. (In this case, the scores would be between 180 and 200).

3. Beware of Online Score Converters/Calculators, Especially If You’ve Done Multiple Passes

Third, you can try various online score converters/calculators. These take multiple variables (e.g., UWorld scores) and predict your actual score.

In my experience, it is the least accurate method. It would be my last resort, especially if you have repeated UW or your other QBanks.

It’s fine to repeat QBanks. And QBank scores might predict your final score. However, having already studied the questions will likely limit their predictive power.

“Offline” Bootlegged NBME Practice Exams: Accurate Predictor or Waste of Time?

The NBME Self Assessments aren’t cheap. $60 for a practice test ($20 for the Subject exams) may seem steep. However, should you use the bootlegged NBME copies floating around the internet?

I recommend using the official NBME Self Assessments and NOT using “offline” NBMEs. (I have no financial gain from this recommendation). Why? As mentioned before, each exam has a different scale. A 90% correct on one will convert to a different score than 90% on another.

I know of no score converters for each NBME exam. As such, you’ll never know what your score is from an “offline” NBME. The primary point of taking an NBME is to estimate your final score. Taking a test without getting a predicted score is of limited utility.

I’m all for saving money, but as someone once told me, “The USMLEs aren’t the place to be cutting corners.” This advice applies to the NBME Self-Assessment Services.

Concluding Thoughts: Don’t Take Your Test Until You’re Ready

The biggest mistake I see students make: they take their test and hope for a miracle.

Don’t Take Your Test If You Might Fail

One prominent example is students who fail their USMLE. While some students have scores tank on their exam, they’re in the minority. What’s more common is students feeling an urgency to take their exam. Typically the student feels external pressure to take their test, either because:

• Their school has set an arbitrary deadline (and they don’t ask for an extension)
• Friends/family keep asking them, “will you just take your test already?”
• They’ve already filed for an extension of their test permit. (And delaying further would cost extra money).

None of these reasons justify taking your test if you have a high risk of failing. Yes, failing a USMLE isn’t the end of the world. However, it will shut the door at many residency programs.

(To read Fail Step 1, Step 2 CK or CS? Do This, click here).

Ask Yourself: “Would I Be OK With My Score?”

Don’t take your test if you’re at high risk of failing. Just don’t.

If you’re above passing, but below your goal score, it’s a personal decision whether to take your test or not. In other words, determine if your score is within the range of your goals. You can find the average USMLE scores by specialty for US students and IMGs here.

If you’re not sure whether you should take your test, consider these articles:

Are You Ready to Take Your USMLE or Need More Time?
1-2 Weeks Before Your USMLE? Read This

What do you think? Let us know in the comments!

Photo by Elijah Hiett.