Have you ever heard of people who “just did UWorld,” and did fantastic on Step 1 or Step 2 CK? Ever wonder how someone might do 80-100+ UWorld questions in a day? Are you considering (or have already tried) doing lots of QBank questions as your primary/sole approach to preparing for USMLE? If so, you are not alone.
Question banks are an integral part of preparing for the USMLEs. As such, it is natural to wonder: if QBanks are good, then more must be better, right? If practicing 30-60 usmle questions/day is a good thing, why not do even more? And if I know that I have 2000+ questions left in UWorld, and only a few weeks left to study. Shouldn’t I do 100+ questions per day?
(Similarly, if you’re wondering if the famous UFAP – or its variants – is “enough,” check out this article here).
We will address all of these questions (and more). Specifically, we will discuss:
- The evidence for the importance of QBank questions in USMLE studying,
- Why so many people claim you should “just do questions,”
- The way to evaluate whether such advice makes sense,
- When “just do questions” may make sense,
- The necessary condition for USMLE and clinical excellence, and
- Much, much more
Table of Contents
Why Doing QBank Questions is Critical for USMLE Studying: The Evidence
The proponents of the “just do lots of questions” approach to studying for the USMLEs have a lot of ammunition. After all, QBank questions are an essential part of USMLE studying.
First of all, doing practice questions has a solid research base. Here you can see the key table from a review article comparing the relative utility of various techniques.
Evidence-Based Techniques to Maximize Learning
| Technique | Description | Utility |
|---|---|---|
| Summarization | Writing summaries (of various lengths) of to-be-learned texts | Low |
| Highlighting/underlining | Marking potentially important portions of to-be-learned materials while reading | Low |
| Keyword mnemonic | Using keywords and mental imagery to associate verbal materials | Low |
| Imagery for text | Attempting to form mental images of text materials while reading or listening | Low |
| Rereading | Restudying text material again after an initial reading | Low |
| Elaborative interrogation | Generating an explanation for why an explicitly stated fact or concept is true | Moderate |
| Self-explanation | Explaining how new information is related to known information, or explaining steps taken during problem solving | Moderate |
| Interleaved practice | Implementing a schedule of practice that mixes different kinds of problems, or a schedule of study that mixes different kinds of material, within a single study session | Moderate |
| Practice testing | Self-testing or taking practice tests over to-be-learned material | High |
| Distributed practice | Implementing a schedule of practice that spreads out study activities over time | High |
Notice that “Practice testing” is one of two techniques to achieve “High” utility. (Along with distributed practice, which includes spaced repetition).
Second, your USMLE/Shelf will be a series of multiple-choice questions asking you to apply clinical principles to various scenarios. Thus, it would be logical for your practice to be as close to the “real thing” as possible.
Finally, many people who score high on the USMLEs have done lots of questions. Some may even have followed the “just do lots of questions” approach. Little research exists on the topic for USMLEs. However, the limited evidence also supports that the number of MCQs correlates with final USMLE scores. (In this case, Step 1).
In this retrospective study, students at Washington University School of Medicine in St. Louis completed an anonymous survey asking for their Step 1 scores. They also recalled their MCAT score and their pre-Step 1 study period. Items included the number of questions they completed, the use of spaced repetition flashcards, and many other things.
| Standardized β | 95 % CI | p-value | |
|---|---|---|---|
| Honours | 0.347 | (0.136, 0.559) | 0.002 |
| MCAT | 0.28 | (0.0973, 0.463) | 0.003 |
| MCQ (# of questions) | 0.298 | (0.139, 0.457) | < 0.001 |
| Anki (# of reviews) | 0.195 | (0.0265, 0.3630) | 0.024 |
| Test anxiety | − 0.326 | (− 0.512, − 0.141) | < 0.001 |
The take-away? The more questions the students did, the greater their Step 1 score.
Questions Are Important. But Should You “Just Do Lots of QBank Questions”?
Doing practice questions should be a considerable part of preparing for USMLE. Some even go so far as to say that it should be the SOLE focus of Step 1 and Step 2 CK prep. On the other hand, we’ve all seen/heard from people who say, “all you need is UWorld,” with the sort of self-assured confidence of a high-scorer.
You may wonder, “if this person scored so well by just doing QBank questions, why shouldn’t I do it, too? Who am I to question success?”
The issues with following the “just do questions” crowd is two-fold:
1) Correlation ≠ Causation
As is rightly pointed out by the Washington University study on Step 1 scores, correlation does not equal causation. They state specifically, “For example, students who complete more practice questions or flashcards may simply study more in general…Similarly, students with attributes associated with improved test performance, such as concentration or self-directedness, may be more likely to use more practice questions or Anki flashcards and also be more likely to score higher on Step 1 via other mechanisms. More direct causal inferences on the effects of these self-testing resources require a prospective randomized trial.”
In other words, maybe good students just study more in general – because they study more, they will do more questions and Anki cards.
2. “One-Size Fits All” Approaches Ignore One’s Foundation (or Lack Thereof) and Are Therefore Dangerous
Similarly, logic dictates that students with stronger foundations will have less to review and thus cover more questions.
For example, imagine a student with excellent preparation before their dedicated study. They had mastered every vital concept – and had made fantastic Anki cards, so they never forgot anything. By the time they started to ramp up their QBank blocks, they had little remaining to learn. Rather than each QBank block uncovering a host of things they needed to spend time mastering, they could breeze through questions with minimal time spent on review.
Contrast that with someone who has been cramming through med school to get by. They may have a weak foundation. Consequently, every practice question will reveal many things they need to review. To do more questions, this student with a shakier foundation would have to SKIP learning essential items they would have missed before. And what would this cramming get them? They could achieve a vanity metric like the number of questions completed in a day over authentic learning.
Silent Evidence: The “Rats in New York” Problem
OK, so maybe it doesn’t make sense to sacrifice mastery of concepts to do more questions by the end of the day. In theory, that may make all the sense in the world. But how do we square this with the reality that some people “do just questions” and end up scoring quite well?
To answer this, we have to consider the phenomenon of “silent evidence.” In a brilliant description of the power of silent evidence, Nassim Taleb demonstrates how it’s crucial to consider the drop-out of participants.
Let’s say we take 100 rats from New York City, all with varying strength levels. We can graph their strengths on a bell curve. Now, let’s do something to try to get rid of the rats, like irradiate them to the point of killing some of them. (These are imaginary rats, mind you).
The radiation would weaken all rats, but the first rats to die would be the weakest. So now we have a truncated Bell curve, where the frailest rats are eliminated.
Let’s say that we repeated the irradiation until the point where only the top 5% of rats by strength remained. If we were to compare those rats with, say, the general rat population of New York City, the average power of the surviving 5% would be much higher.
You Can “Prove” Just About Anything If You Ignore Enough Evidence
In other words, the surviving rats would be STRONGER than the average pre-radiation rat. Of course, this would be an illusion, since the radiation had the effect of creating silent evidence – eliminating all of the weaker rats.
The allure of the strong irradiated rats would be too difficult for some to resist, however. Some people would be tempted to argue that the radiation caused the rats to be stronger. This is because, again, the (surviving) irradiated rats were stronger to begin with.
However, does that PROVE that radiation made them stronger? Of course not! In fact, we can see that the “rats made stronger by radiation” were made weaker by it.
How do irradiated rats relate to whether you should do many UWorld questions? Because any time someone tells you, “just do this, and you will succeed,” you must consider the silent evidence. Specifically, numerous people “just did questions” and got high scores.
However, how many people have tried – and failed – with the same approach? How easy would it be to talk to them? We all know that those with high scores are much more likely to broadcast their study methods than those who do poorly. But if everyone “just did questions,” we would only hear from the top scorers – and thus come away believing practice questions were all we needed.
Who Should NOT “Just Do Lots of UWorld Questions” as Their Main Study Method?
So, what can we conclude so far? With few exceptions, my view is that:
QBank questions are necessary – but not SUFFICIENT – for improving your USMLE score
The insufficiency of QBank questions is PARTICULARLY true for those with a weak foundation. If you’ve been resorting to cramming to get through med school, there is little sense in trying to hit a high number of questions per day.
Note that this advice will go directly against the advice of many well-meaning administrators. For example, one common thing to hear from (some) academic advisors is that “our most successful students all are doing 100+ questions per day.” The implication – or sometimes outright demand – is that students just need to do lots of questions.
As we’ve seen, the evidence for this question-heavy one-size-fits-all approach is flawed. It is most flawed for those who never took the time to master things correctly the first time. Ironically, the students who are most in need of slowing down are often the exact students who try to speed up.
When is “Just Do Lots of QBank Questions” Reasonable?
Note that I am NOT saying that QBank questions are bad. As mentioned before, the evidence supports the idea that questions are necessary for USMLE studying. However, that is a far cry from saying that doing all and/or mainly practice questions is sufficient.
That said, “just do lots of questions” is good advice sometimes! As you might guess, question-heavy approaches work best for those who already have a strong foundation. In these cases, the student will have already learned a lot of the necessary material. Questions will simply be a means to improve the application and integration of that knowledge.
And remember – while it may be possible to do well with a question-heavy approach, consider the long-term opportunity cost. Namely, what if you can make – or use – excellent Anki/spaced repetition cards to improve long-term retention? Not only can it help solidify your foundation, but it can also set you up for long-term success since you’ll have less to re-learn in the future.
Concluding Thoughts
Asking successful people what they did seems like a fool-proof way to achieve similar success. Want to know how to score high on the USMLEs? Just ask people who did well, and do what they did! Right?
While it seems wise to seek counsel from those who have achieved what you wanted to in life, there are obvious caveats. The biggest, in this case, is to consider the context – and the likelihood that the given advice was the sole (or even primary) determinant of success.
What if I asked a marathon runner what they did to prepare the week before their race? I’d miss out on the countless hours of preparation in the months/years leading up to that week. Similarly, simply looking at what someone did during the weeks leading up to their test ignores the crucial context of what they did BEFORE that.
More Important Than the Resource is the Approach
The most important element missing in the “just do questions” mantra is what you do afterwards. You could do 100 questions, but review them poorly or not at all.
One observation made by many educators is that a necessary condition for academic success is the mastery/application of principles. Have you ever met a master clinician who didn’t have a mastery of their craft? Conversely, have you ever found someone who had crammed their way through school who had reached the heights of their field?
A student in the Online Course recently described mastery as taking a thick textbook and making it thin. In other words, you learn the material so well that you boil down a lot of information into its core principles. This “reasoning from first principles” approach is endorsed by numerous luminaries, including Elon Musk.
We believe so strongly that medicine can – and should – be boiled down to essential principles that we have based the entire Course around a handful of such principles. For a free consultation where we walk you through how to use core concepts to enjoy AND excel at your USMLE studies, click here.
Whether you find them on your own – or use us as guides – we encourage you to take a thick textbook and make it thin. QBank questions are an essential part of this – as long as the aim is the mastery and application of the core principles of medicine.
