From 2017 to 2023 I interviewed and evaluated over 100 C++ software engineering candidates, with the final three years also covering Python. In this article I’ll cover key lessons I learned from this experience, including:<\/p>\n\n\n\n
The TL\/DR<\/strong> for the time poor reader: Test your software engineering candidates in the exact types of problems they’re likely to face, with take-home exercises, i.e. no<\/strong> Codility-like logic puzzles that must be solved within 2 hours.<\/p>\n\n\n\n But why? Are there exceptions? How costly might it be to develop that take home test? For that, I recommend the full article below…<\/p>\n<\/div>\n\n\n\n In general, our interview approach consisted of three stages:<\/p>\n\n\n\n I’m glad to say that in all my time evaluating candidates, we never made a hire that didn’t work out. In this regard, it is important to note three things:<\/p>\n\n\n\n The initial short interview is an essential step to make sure you’re not wasting the candidate’s valuable time, as much as they’re not wasting yours. Ideally, this shouldn’t be more than 15-20 minutes to…<\/p>\n\n\n\n I intentionally add this emphasized box, as I’ve often seen this go side-ways, particularly with technically-minded interviewers.<\/p>\n\n\n\n The phone-interview is not <\/strong>the time to have the candidate find your favorite <\/em>answer to a technical question (spoiler alert: It is almost never <\/strong>that time), but instead to get any generally correct notion of an answer, and ideally probe how deep that correct answer goes.<\/p>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n Hopefully, the short interview concluded with a promising outcome and you’re confident that the candidate at least has a solid shot at passing the take home test.<\/p>\n\n\n\n A former colleague of mine once said…<\/p>\n\n\n\n Test what you ship, ship what you test…<\/p>\nTim Crawley<\/cite><\/blockquote>\n\n\n\n He’s unlikely the first to coin the phrase, but it’s an excellent adage and is somewhat applicable to hiring programmers. <\/p>\n\n\n\n Test your candidates with actual problems they’ll counter, or at least very similar ones.<\/p>\nMe (and likely many others)<\/cite><\/blockquote>\n\n\n\n This is precisely what the take home test is intended to do. In our case, we often hired computer-vision capable software engineers, and thus provided a take home test which specifically involved simple computer vision related problems, along with one or two very generic algorithmic problems.<\/p>\n\n\n\n Each candidate was asked to implement solutions to a small set of tasks which came with a function signature we defined, such as:<\/p>\n\n\n\n Along with some details about what the candidate could assume regarding the incoming data, very little is needed to ensure their code can plug neatly into your own custom-built testing framework. By only providing a set of sparse headers, the candidates are free to implement their solutions in whatever Integrated Development Environment (IDE) they’re comfortable with, along with whatever unit-tests they deem necessary, if any.<\/p>\n\n\n\n Although the tasks themselves had already been determined by the time I joined the company, and barely changed in the 6 years of my employment, I spent about 6-8 weeks creating and modernizing a C++ framework to test and evaluate all the candidates solutions reducing the overall time to analyze a solution from about 14 hours down to about 3-4 hours per candidate.<\/p>\n\n\n\n Admittedly, 6 weeks – even over the course of 6 years – is costly developer time. Depending on what your senior developers cost per week, and you do <\/strong>want a senior developer building the take home test, it can easily run you 11000 USD or more. But if your hiring needs are on-going, and comparing against popular testing platforms (like Codility), 11000 USD amortized over 6 years suddenly doesn’t seem so bad.<\/p>\n\n\n\n Furthermore, I’d argue that Codility and similar platforms often come with some serious drawbacks, compared to a custom in-house take home test. Such as:<\/p>\n\n\n\n Having read all those points, if you’re prone to black and white thinking, you may also be thinking…<\/p>\n\n\n\n So you’re saying Codility is absolutely awful and should never be used, despite the fact that it’s clearly commercially very succesful given all its clones, how stupid are you?<\/p>\nDichotomous thinker<\/cite><\/blockquote>\n\n\n\n Contrary to popular belief on the internet, it is actually possible to hold the belief that something is both good and<\/strong> bad in different ways. So while I’m giving Codility and similar platforms here the business, by pointing out that their ways of testing candidates aren’t very representative of real-life software engineering work, and <\/strong>come with significant caveats, their approach is still effective. But just because something is effective, that doesn’t mean that it’s the best approach. In a later section, I’ll denote when I would recommend using these platforms.<\/p>\n\n\n\n One topic I will address here which often comes up when allowing any test taker to complete a test while not under observation and without a hard time constraint. The topic of cheating<\/strong>. First, I recommend the reader accept the fact that fully eliminating cheating is impossible. No matter how you test or what constraints you add, there will always be ways around them. The goal isn’t to eliminate cheating entirely, but rather to reduce it as best as possible. Even Codility accepts this impossibility, given that the platform has candidates agree to their ‘Code of Honour<\/a>‘ promising that you don’t cheat when taking one of their tests.<\/p>\n\n\n\n So we just throw our hands up in the air and give up?!<\/p>\nDichotomous thinker<\/cite><\/blockquote>\n\n\n\n No, dear dichotomous thinker. In the in-person interview, we use the candidate-provided solutions as a jumping-off-point to discuss mistakes, coding style, implementation alternatives, etc. So while the take home test undoubtedly widens the candidates options to cheat by allowing them ample time to coordinate other people to solve the given tasks, they will ultimately be expected to have an understanding of the code produced, as well as why it built the way it was.<\/p>\n\n\n\n The take home test provides exceedingly fertile ground to get an insight into how the candidate prefers to program and the follow-up in-person interview is a great place to discuss how open they are to alternate ideas and implementations. <\/p>\n\n\n\n Assuming the candidate passed the first two phases with promising colors, the final step consists of:<\/p>\n\n\n\n One of the great benefits of starting the interview with discussing their take home test is the candidate’s familiarity with a piece of code they’ve made themselves. Presumably it was made in their preferred environment, on their own time with as little artificial pressure as possible, and similar to a real working environment as possible.<\/p>\n\n\n\n While I cannot share explicit examples of discussions we had in this phase, I can comment on my recommended approach to discussing the candidates provided solution. <\/p>\n\n\n\n One important detail to discuss first, is your interviewing mindset. Consider that for the candidate, an interview scenario is very dissimilar <\/strong>to any day-to-day tasks or discussions they are likely to have at your company, should you hire them. Unless your candidates are often going to be seated in tense discussions with clients who are likely to challenge their solutions or suggestions, I advise keeping in mind how stressful an interviewing scenario can be for some candidates. Of course, it is still your job to evaluate promising candidates, but I advise starting with a mindset of:<\/p>\n\n\n\n How can I help this candidate showcase the best answers they can come up with?<\/strong><\/p>\n\n\n\n Rather than:<\/p>\n\n\n\n Let’s see if this candidate is really as good as the first two phases seemed to indicate.<\/strong><\/p>\n\n\n\n So how do we approach discussing the take home test with the former mindset?<\/p>\n\n\n\n Based on the candidates solution’s and errors found within, I suggest having an increasing set of hints ready to narrow the problem down and see when the candidate is able to jump in and solve any problems you’ve found in their solution.<\/p>\n\n\n\n For example, let’s look at a hypothetical – but common – iteration deletion issue in C++:<\/p>\n\n\n\n This code is supposed to remove all values of 10 in a vector. The veteran C++ coder may have noticed the problem, namely that calling .erase() on an iterator will invalidate it, causing the code to crash.<\/p>\n\n\n\n Assuming this code is a part of a larger solution by the candidate, an approach to discussing this iterator issue could be:<\/p>\n\n\n\n The benefit of this approach is that the candidate is provided a figurative ladder from which to make the final leap to the solution on their own. Of course, the more steps – in the form of hints – they’re provided on the ladder, the less impressive the leap. By motivating a gradient of outcomes, as opposed to 100% correct or 100% incorrect, you’ll get a more nuanced idea of where approximately the candidates depth of knowledge lies, as well as providing tension relief as the candidate ideally always makes the jump to the correct solution, even if most of the steps were provided.<\/p>\n\n\n\n I suggest informing the candidate upfront about your intention to give them hints during the discussion along with the fairly obvious fact that the sooner an answer occurs to them, the better. There are only benefits to your candidates knowing how the interview will be conducted, and it’s not really possible for them to ‘game the system’. Note that actively providing a candidate with hints does occasionally lead to a manageable side-effect. Specifically, that some candidates will be inclined to suggest solutions or ideas as questions, e.g., “Is the issue X?<\/em>“, “Perhaps a solution could be Y?<\/em>“, etc.<\/p>\n\n\n\n In fairness to all candidates, I respond by asking candidates what they think, when they pose these types of “question answers”. The goal is to find the candidates committed answers, and then have a dialogue about it. Even an incorrect answer can lead to a dialogue that yields valuable knowledge. Ultimately, candidates posing questions is great, but when it comes to specific problems you expect them to understand and solve, you do want to make sure they can stand on their own two feet.<\/p>\n\n\n\n Note, that while a candidate may muddy the waters regarding standing on their own two feet, they’re not the only <\/strong>source of potential ambiguity.<\/p>\n\n\n\n You, or an eager co-worker may equally be inclined to start nodding before the answer has barely begun to leave the candidates lips. Something I’ve witnessed on more than one occasion myself. This is to be expected, since you and your colleagues will see multiple candidates step into the exact same figurative potholes.<\/p>\n\n\n\n Fight the habit of becoming complacent <\/strong>and do your best to keep both verbal and non-verbal communication reserved until you have hear the complete and unique answer from a candidate. It’s possible you may already be convinced they’re a good fit, but that may not be the case for your co-workers.<\/p>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n A benefit I cannot overstate in regards to having a conversation about implementation mistakes is seeing how well a candidate handles simply having made a mistake. We all make mistakes.<\/strong> All of us. No exception.<\/p>\n\n\n\n The vast majority of candidates who made it to the third phase would – given the opportunity – quickly realize their error and offer a solution.<\/p>\n\n\n\n Anecdotally, I will never forget a particular candidate who – when faced with an issue in the proposed solution – seemed physically incapable of recognizing a problem we’d found, and consequently also unable to come up with any solutions. In hindsight, I rationalize the experience with this candidate as them being most familiar with always being the expert in the room and rarely challenged on any of their solutions.<\/p>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n In 2012, Valve’s employee handbook<\/a> leaked on to the internet. A salient point it makes is:<\/p>\n\n\n\n […] hiring is the single most important thing you will ever do at Valve.<\/p>\nValve handbook for new employees – Page 6<\/cite><\/blockquote>\n\n\n\n I tend to agree, that it is among the absolute most important activities for any company, which is why I’d be wary of modifications to the outlined interview approach, born out of an interest to lower costs. Hiring is one of the last places you’ll want to cut corners given the catastrophic consequences a bad hire can cause a company.<\/p>\n\n\n\n That being said, here are some possible scenarios for deviation.<\/p>\n\n\n\n If you find yourself unable to convince management that hiring really isn’t the place to spare resources, or find that your hiring interval easily exceeds a year or more between hires, then I suggest either one of two method variations:<\/p>\n\n\n\n The outlined approach focuses on determining a candidates technical skills. It’s worth at least a single paragraph to emphasize that there are times where the best hire for the company, isn’t the person who excels the most in a technical capacity. Sometimes, the best hire you can make is the employee that can help shift troublesome company culture in a positive direction.<\/p>\n\n\n\n Simultaneously, it is also a grave error to hire an exceptionally technically talented individual who’s personality clashes with the existing team, or simply has difficulty recognizing issues in their own work, as an earlier anecdote described.<\/p>\n\n\n\n But culture-related hiring concerns is beyond the scope of this article, and probably deserving a wholly separate article, on their own.<\/p>\n\n\n\n Our interview approach was refined over the course of five years, but fundamentally remained the same. The biggest refinements include:<\/p>\n\n\n\n No matter how clear and explicit you are in your task instructions, some candidates will end up overlooking some details.<\/p>\n\n\n\n Our internal policy was to fix minor formality issues if necessary, but if anything took more than a couple of minutes or pertained to the implemented solution itself, we’d simply push the solution back to the candidate with a note on what remains unfinished.<\/p>\n\n\n\n In terms of technical limitations, we very intentionally did our best to only require the minimum it took for a candidates solution to neatly plug in to our in-house testing framework. Our reasoning being that we deliberately want to see a candidates design approach to solving a problem, in addition to the actual problem solution itself.<\/p>\n\n\n\n Admittedly, by eventually requiring candidates to inherit from our abstract C++ object, we’ll never again see submissions with code written and submitted in a Word (.doc) file. I do miss those days. On the bright side, it didn’t prevent us from receiving a submission containing so many syntax errors that nothing could compile. Upon inquiring about the errors, the candidate told us they felt using a compiler was cheating.<\/p>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n With soft problems like hiring, no approach will ever be perfect. Here are the caveats the outlined interview approach:<\/p>\n\n\n\n Finally, I thought it prudent to think of some concerns I might have, had I not spent all this time using the approach:<\/p>\n\n\n\n Phew…<\/em><\/p>\n\n\n\n If you got to the end, I appreciate your attention and thank you for taking the time to read my ramblings. I hope you’ve found some value in the article and all the best with your interviews and interviewing!<\/p>\n","protected":false},"excerpt":{"rendered":" From 2017 to 2023 I interviewed and evaluated over 100 C++ software engineering candidates, with the final three years also covering Python. In this article I’ll cover key lessons I learned from this experience, including: The TL\/DR for the time poor reader: Test your software engineering candidates in the exact types of problems they’re likely […]<\/p>\n","protected":false},"author":1,"featured_media":706,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_FSMCFIC_featured_image_caption":"","_FSMCFIC_featured_image_nocaption":"","_FSMCFIC_featured_image_hide":"","footnotes":""},"categories":[27,26],"tags":[28,31,30,29],"_links":{"self":[{"href":"https:\/\/www.lasselaursen.com\/wp-json\/wp\/v2\/posts\/621"}],"collection":[{"href":"https:\/\/www.lasselaursen.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.lasselaursen.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.lasselaursen.com\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.lasselaursen.com\/wp-json\/wp\/v2\/comments?post=621"}],"version-history":[{"count":42,"href":"https:\/\/www.lasselaursen.com\/wp-json\/wp\/v2\/posts\/621\/revisions"}],"predecessor-version":[{"id":715,"href":"https:\/\/www.lasselaursen.com\/wp-json\/wp\/v2\/posts\/621\/revisions\/715"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.lasselaursen.com\/wp-json\/wp\/v2\/media\/706"}],"wp:attachment":[{"href":"https:\/\/www.lasselaursen.com\/wp-json\/wp\/v2\/media?parent=621"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.lasselaursen.com\/wp-json\/wp\/v2\/categories?post=621"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.lasselaursen.com\/wp-json\/wp\/v2\/tags?post=621"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}Our interview approach<\/h2>\n\n\n\n
\n
\n
1. Initial phone-interview<\/h3>\n\n\n\n
\n
2. Take home test<\/h3>\n\n\n\n
\n
\n
void ModifyImage( int* data, int imageWidth, int imageHeight );<\/code><\/pre>\n\n\n\n\n
\n
\n
\n
3. In-person interview<\/h3>\n\n\n\n
\n
std::vector<int> values;\nfor (auto iter = values.begin(); iter != values.end(); ++iter)\n{\n\tif (*iter == 10)\n\t{\n\t\tvalues.erase( iter );\n\t}\n}<\/code><\/pre>\n\n\n\n\n
\n
\n
\n
When to deviate<\/h2>\n\n\n\n
\n
Lack of resources or rare hiring<\/h3>\n\n\n\n
\n
\n
\n
Culture hire<\/h3>\n\n\n\n
How our approach evolved<\/h2>\n\n\n\n
\n
Caveats and concerns<\/h2>\n\n\n\n
\n
\n
\n