Let’s Put the Coding Quiz in Perspective: 3 Reasons Why the Coding Quiz Is Not a Good Interview Tool
Coding quizzes are touted as a staple of developer interviews, ostensibly to filter out the silver from the dross, sift the wheat from the chaff. For interviewers, coding quizzes represent a cheap, readily available assessment of prospective job-seekers’ programming abilities. And not only are they great for assessing the coder’s style, leading coding quiz sites like Hackerrank even save snapshots of the state of the code as it is written, making it great for insights into how the coder thinks as he codes.
However, for non-developers or the otherwise uninformed, it is tempting to treat the coding quiz as an assessment of the programmer in general. After all, a programmer’s bread-and-butter should be how to code, right? But as we will see here, there are various aspects of programming and software development that coding quizzes either cannot cover, or will simply fail at.
If you’re looking to use a coding quiz as part of your interview but don’t have programming experience of your own, my advice here basically is to get someone who’s experienced in reviewing code to look at it on your behalf.
To preface this article with my experience with the coding quiz, when I interviewed for a tech company, they gave me a Hackerrank test consisting of 2 questions. The first was for string manipulation (which was fairly straightforward), but the second question was a tough one, dealing with optimization of an algorithm. With limited time for the quiz, needless to say I scored poorly on that one.
After reviewing the results with my interviewer, however, I discovered that almost everyone else who interviewed before also scored equally poorly for that question in particular. Not only that, in the weeks after I was hired, only one person would get a full score for that quiz out of dozens of interviewees (he was also hired). Fundamentally during hiring discussions, the decision to hire or not was not based on the results of the quiz.
And with that in mind, let’s look at the first reason.
1: It follows the standardised test framework.
You see, the coding quiz in its commonly encountered form suffers from the same problem as any other standardised test, and the metric in this case is whether the test cases hardcoded into the quiz succeed. Like any standardised testing, they can be trained towards. Most test questions revolve around more generic stuff like string manipulation (and not the quantum physics kind) and algorithms, which is small enough of a subset to study/train for.
For example, that second coding quiz question I took? Given a list of points with x-y coordinates, find the 2 coordinates closest to each other. It’s easy enough for most programmers to answer, but the optimization is a little tricky to achieve. The fact that it is common, however, means that people can train to answer this type of question, and remember that one person who scored full marks for it? Yeah, he trained specifically for that question.
You may say that he was clever enough to prepare for it, but this is a feature/bug inherent to standardised testing, that it tends to be game-able this way. Don’t get me wrong; standardised testing has a long history of use as a benchmark for figuring out all sorts of aptitudes. But only if they test a good representation of the subject at hand and test it well, and that is by no means easy to achieve.
Whatever metric the test uses to measure aptitude will almost never be truly representative of the actual subject matter; it can’t be, because demonstrating said knowledge is usually too wide-ranging to cram into a single test session. Ask too many questions to try and cover everything, and the interviewee may tap out halfway through. Ask too few questions and your test coverage of the subject matter may have too many holes.
I shan’t go further about standardised testing as I’m no pedagogical expert; all I want to highlight is the fact that the coding quiz tests based on a common marking scheme. There are plenty of resources online that talk about why standardized testing is good and bad. They’re all but a Google search away.
And speaking of Google…
2: Knowledge is getting easier to Google nowadays.
You might wonder why companies out there can’t just make more tests to cover domain-specific knowledge, like machine learning or web development. And they could, but they normally don’t. Why? Because there’s not much point, honestly. A coding quiz that is, say, 45 minutes or an hour long would only be able to cover one or two basic aspects of the tested domain. Aspects that (nowadays) can be easily Googled. And the more basic the aspect, the more readily available it usually is via your search engine of choice.
Should Googling be punished? After all, a good developer should have the knowledge at his fingertips, right? Well, not really. Show me a good developer, and I will show you a developer with strong Google-Fu. And why should that be a bad thing? The cleverest developer still has to learn new things, and what better resource to learn new things with than the Internet? Needless to say, the Internet is wonderful for obtaining domain-specific knowledge at all levels, and anyone would be remiss to ignore this immense resource. Heck, you’re reading this on the Internet right now.
But you might think that the coding quiz is typically short enough to discourage Googling for the answer anyway, and you’d be right. The good news is that this produces a snapshot of the interviewee’s current knowledge of the subject. However, if the subject matter can be learned quickly anyway, what good is that benchmark? The coding quiz cannot capture how quickly the interviewee can pick up new technologies or frameworks, which is arguably a more important skill than knowing a certain skill at present. A good indicator of this is if the interviewee has a track record of working in positions where he has handled new technologies.
3: Assessing code quality requires trained eyes, not a code compiler.
Now the difference between good and great developers can be summed up with this quote: Good a̵r̵t̵i̵s̵t̵s̵ developers copy, great a̵r̵t̵i̵s̵t̵s̵ developers steal. Not only is it important to make solutions for problems, it’s even more important to understand why those solutions work, and put them together in a way that not only solves the problem, but solves it cleanly.
What is “clean” code, then? Clean code generally has a few qualities, but just to give an overall sense of it…
If you’re working to make code for someone, you have to write code at least for other developers to review. Good code is often written with the consideration that another developer is going to read it in the future, even if that someone is yourself. If you have to extend some functionality you wrote a couple of weeks ago that you now cannot decipher, then it doesn’t matter how effectively it worked, it’s still a problem. Now you’ll either have to treat that chunk of code you don’t understand like a black box and work around it instead of making what could have been a minor code change inside that chunk, or rewrite large portions of the code from scratch (and risk breaking everything!)
Fortunately, the coding quiz can give you insight into the programmer’s code quality, but you won’t see it in the test score. Rather, you’ll have to read the code to judge its quality for yourself. But like I just mentioned, good code is often written for other developers to read, and should be done with that in mind. I‘m not suggesting that good code not meant for non-developers’ eyes, because if it’s readable to Average Joe, it’s absolutely readable to developers too. But again, the code is for other developers to read because it impacts them first (and after that maybe those guys in middle management when they ask to review the code for some reason, but I digress).
Add-on: There are other tools better suited for the task.
If you wanted to assess the interviewee’s code quality, the coding quiz isn’t the only way to assess it. Sure, it might be a decent way to assess it in a fixed setting, but there are other less intrusive ways to get a handle on it.
If the interviewee has worked on an open-source project (even a public Github/Gitlab /whatever in his name will do), a quick look on his public work, well, works. It is common for developers to have source code posted on a public repository, even if it was for a school project back from college days.
An online coding interview assignment (basically a coding quiz, but not timed) would also give a snapshot of the interviewee’s code quality just fine as well, and it helps that the interviewee can do it in his spare time AND have an opportunity to pick up the skills needed for the job in his spare time. This also primes them for the sort of work that they will be touching upon, helping them decide whether this position is in their wheelhouse. This cannot be underestimated because the question of capability and motivation is best answered by the developer himself. You won’t get that with a generic coding quiz.
To end off…
That was my 3 reasons for why the coding quiz is not a good interview tool.
I know I’ve been bashing the coding quiz this entire article, so let me leave you with a slightly more balanced summary on this. A tl;dr, which I’m sure if you’ve scrolled to the bottom to see, you’ll be happy to read.
The coding quiz is good for assessing general programming skills. Want to see how well the interviewee can program in a limited-time setting? Sure. Wanna see an interviewee’s code quality? You got it.
But there are details that make the coding quiz’s effectiveness questionable. It follows a standardised test format, which can be trained for, possibly subverting the usefulness of the metric. Search engines make any coding quiz that tests domain-specific skills less relevant, especially for interviewees who learn quickly. And finally, if you wanted to test for code quality or inform the interviewee of the work required in the position, there are other, way better tools suited for the task.
If you enjoyed this article and want to see more of my writing, check out this piece about why the 2020 pandemic has enlightened the world about the joys of remote work. Or if you want an algorithm to becoming a good programmer, this piece has got you covered!
