Lasse Laursen

From 2017 to 2023 I interviewed and evaluated over 100 C++ software engineering candidates, with the final three years also covering Python. In this article I’ll cover key lessons I learned from this experience, including:

• Which interviewing approach we used and why I’d recommend it over Codility or similar platforms.
• Rare cases where I would suggest deviating from our standard interviewing approach.
• How the approach evolved over time.
• Caveats and concerns with the approach.

Our interview approach

In general, our interview approach consisted of three stages:

1. An initial phone-interview to make a quick ~20 minute screening of a promising candidate.
2. A take home test which typically took 4-8 hours to solve thoroughly.
3. A final in-person interview which typically lasted 2-3 hours.

I’m glad to say that in all my time evaluating candidates, we never made a hire that didn’t work out. In this regard, it is important to note three things:

1. Chance is a significant factor. You can’t test for everything, and part of hiring is rolling a die. But you can try to make that die-roll as predictable as possible.
2. Both the initial and in-person interview involved either two or three company employees, so credit definitely goes to my former co-workers in helping assess promising candidates.
3. The ‘perfect hiring record’ could simply mean the bar was set too high, rather than ‘just right’, which is bad.

1. Initial phone-interview

The initial short interview is an essential step to make sure you’re not wasting the candidate’s valuable time, as much as they’re not wasting yours. Ideally, this shouldn’t be more than 15-20 minutes to…

• …build a bit of familiarity. Cover basics you wouldn’t even think of as a source of potential problems, like how well they speak the language you speak within the company. We had a few candidates that were very difficult to understand over the phone. Not enough to dismiss the notion of hiring them there and then, but enough to weigh this into the general consideration of whether to proceed to the second interview phase.
• …ask the most basic of software engineering questions, and I do mean basic. In C++, a basic question could be whether they’re familiar with smart pointers, and if they know what they’re useful for. Good basic questions address functionality that nearly every developer is almost guaranteed to have come across, using a particular language.

Hopefully, the short interview concluded with a promising outcome and you’re confident that the candidate at least has a solid shot at passing the take home test.

2. Take home test

A former colleague of mine once said…

Test what you ship, ship what you test…

Tim Crawley

He’s unlikely the first to coin the phrase, but it’s an excellent adage and is somewhat applicable to hiring programmers.

Test your candidates with actual problems they’ll counter, or at least very similar ones.

Me (and likely many others)

This is precisely what the take home test is intended to do. In our case, we often hired computer-vision capable software engineers, and thus provided a take home test which specifically involved simple computer vision related problems, along with one or two very generic algorithmic problems.

Each candidate was asked to implement solutions to a small set of tasks which came with a function signature we defined, such as:

void ModifyImage( int* data, int imageWidth, int imageHeight );

Along with some details about what the candidate could assume regarding the incoming data, very little is needed to ensure their code can plug neatly into your own custom-built testing framework. By only providing a set of sparse headers, the candidates are free to implement their solutions in whatever Integrated Development Environment (IDE) they’re comfortable with, along with whatever unit-tests they deem necessary, if any.

Although the tasks themselves had already been determined by the time I joined the company, and barely changed in the 6 years of my employment, I spent about 6-8 weeks creating and modernizing a C++ framework to test and evaluate all the candidates solutions reducing the overall time to analyze a solution from about 14 hours down to about 3-4 hours per candidate.

Admittedly, 6 weeks – even over the course of 6 years – is costly developer time. Depending on what your senior developers cost per week, and you do want a senior developer building the take home test, it can easily run you 11000 USD or more. But if your hiring needs are on-going, and comparing against popular testing platforms (like Codility), 11000 USD amortized over 6 years suddenly doesn’t seem so bad.

Furthermore, I’d argue that Codility and similar platforms often come with some serious drawbacks, compared to a custom in-house take home test. Such as:

• Needless logical reasoning – Codility and similar platforms often present coding challenges wrapped in complex logic puzzles, e.g. how do you use 10 servants to find the single barrel out of 1000, which contains slow-acting-poison, before the king’s next big ball? Unless you’re hiring developers to literally solve puzzles in 101 brain teaser books, please do not do this. I suggest problems where the general solution is readily apparent and the main focus is on the implementation itself. Our take home test did contain a small set of gotchas (one of which I failed upon being initially hired), but they were all implementation-related curve-balls, such as arithmetic overflow or iterator invalidation.
• Unrealistic time pressure – Codility and similar platforms often provide a timed environment in which to solve problems as fast as possible, but most development work occurs without any immediately-looming running clock. It is a detrimental constraint on the test and will skew results.
• Encourages lazy inaccurate interviewing – Codility and similar platforms appeal to clients by promising the allure of boiling every candidate down to a single comparable number, e.g. Developer 1: 72%, Developer 2: 66%. I’m sure Codility has some wording somewhere encouraging their customers to inspect candidate solutions, rather than just rely on numbers, but their landing page sure pushes the notion of simplifying your developer hiring task down to “72 > 66, so I guess we hire Developer 1?”. You may wonder why this is problematic?
  • In short, because this overall-developer-score is a questionable representation of overall performance. To the best of my knowledge, the Codility scores are (still) largely informed by two factors: Code run-time efficiency, and code correctness. Examining these factors is useful, but distilling an overall candidate score from it, is problematic. I’ve hired developers who, due to a small oversight (addressed in the follow up interview), provided a solution resulting in near 0 code correctness. I’ve also declined to hire developers who delivered nearly unreadable code that passed both run-time and code correctness requirements.
• Unintelligible task descriptions – I’ve witnessed first-hand how some task descriptions on Codility and similar platforms are so poorly formulated, that I’d argue it was actually more difficult to understand the problem to be solved, rather than figuring out how to actually solve the problem. Mind-boggling. I’m certain this does not apply to the majority of tasks, but I believe the reason it happens at all, is because ‘hard to understand‘ can look indistinguishable from ‘difficult to solve‘, for Codility in testing. If the percentage of correct answers is low, it’s easily explained away by assuming the problem itself was just hard, rather than difficult to understand. Given that our take-home test contained the same set of tasks, we quickly took notice when two or more candidates appeared to have the same misunderstanding.
• Environment familiarity – Codility and similar platforms all have their own user-interface that the developer must familiarize themselves with. These can, and do, get in the way of testing the candidate. Codility (and likely other platforms) know this, as they strongly encourage candidates to do a practice test before proceeding to the real one. Admittedly, this is a minor issue compared to the first four.

Having read all those points, if you’re prone to black and white thinking, you may also be thinking…

So you’re saying Codility is absolutely awful and should never be used, despite the fact that it’s clearly commercially very succesful given all its clones, how stupid are you?

Dichotomous thinker

Contrary to popular belief on the internet, it is actually possible to hold the belief that something is both good and bad in different ways. So while I’m giving Codility and similar platforms here the business, by pointing out that their ways of testing candidates aren’t very representative of real-life software engineering work, and come with significant caveats, their approach is still effective. But just because something is effective, that doesn’t mean that it’s the best approach. In a later section, I’ll denote when I would recommend using these platforms.

One topic I will address here which often comes up when allowing any test taker to complete a test while not under observation and without a hard time constraint. The topic of cheating. First, I recommend the reader accept the fact that fully eliminating cheating is impossible. No matter how you test or what constraints you add, there will always be ways around them. The goal isn’t to eliminate cheating entirely, but rather to reduce it as best as possible. Even Codility accepts this impossibility, given that the platform has candidates agree to their ‘Code of Honour‘ promising that you don’t cheat when taking one of their tests.

So we just throw our hands up in the air and give up?!

Dichotomous thinker

No, dear dichotomous thinker. In the in-person interview, we use the candidate-provided solutions as a jumping-off-point to discuss mistakes, coding style, implementation alternatives, etc. So while the take home test undoubtedly widens the candidates options to cheat by allowing them ample time to coordinate other people to solve the given tasks, they will ultimately be expected to have an understanding of the code produced, as well as why it built the way it was.

The take home test provides exceedingly fertile ground to get an insight into how the candidate prefers to program and the follow-up in-person interview is a great place to discuss how open they are to alternate ideas and implementations.

3. In-person interview

Assuming the candidate passed the first two phases with promising colors, the final step consists of:

• Discussing the solutions provided in the take home test.
• Discussing 1-2 simple in-house developed coding problems.
• Discussing a customer requirements situation for a in-house issue we’ve worked on for years.

One of the great benefits of starting the interview with discussing their take home test is the candidate’s familiarity with a piece of code they’ve made themselves. Presumably it was made in their preferred environment, on their own time with as little artificial pressure as possible, and similar to a real working environment as possible.

While I cannot share explicit examples of discussions we had in this phase, I can comment on my recommended approach to discussing the candidates provided solution.

One important detail to discuss first, is your interviewing mindset. Consider that for the candidate, an interview scenario is very dissimilar to any day-to-day tasks or discussions they are likely to have at your company, should you hire them. Unless your candidates are often going to be seated in tense discussions with clients who are likely to challenge their solutions or suggestions, I advise keeping in mind how stressful an interviewing scenario can be for some candidates. Of course, it is still your job to evaluate promising candidates, but I advise starting with a mindset of:

How can I help this candidate showcase the best answers they can come up with?

Rather than:

Let’s see if this candidate is really as good as the first two phases seemed to indicate.

So how do we approach discussing the take home test with the former mindset?

Based on the candidates solution’s and errors found within, I suggest having an increasing set of hints ready to narrow the problem down and see when the candidate is able to jump in and solve any problems you’ve found in their solution.

For example, let’s look at a hypothetical – but common – iteration deletion issue in C++:

std::vector<int> values;
for (auto iter = values.begin(); iter != values.end(); ++iter)
{
	if (*iter == 10)
	{
		values.erase( iter );
	}
}

This code is supposed to remove all values of 10 in a vector. The veteran C++ coder may have noticed the problem, namely that calling .erase() on an iterator will invalidate it, causing the code to crash.

Assuming this code is a part of a larger solution by the candidate, an approach to discussing this iterator issue could be:

• Question: It appears there’s a problem with the solution, causing it to crash. Do you have any idea what it might be?
  • Always start by giving the candidate the space to show off the most, i.e. with little to no guidance at all. It’s possible that they’ve looked at their solution again after submitting it and found the problem in-between the take home test and the in-person interview. If nothing comes to mind, and especially if the submitted solution is large, I suggest not dwelling here for too long. Assuming no answer is forthcoming, provide…
• Hint 1: There seems to be an issue with a part of this for-loop. Any ideas?
  • How much to help in terms of narrowing down the problematic area is subjective and also dependent on the issue itself. My general advice here is try and hit a happy medium. Narrow too much and you’re practically giving the candidate the answer. Narrow too little and the candidate may remain clueless and feel heightened anxiety by your possible expectation that the narrowing should have helped enough. If the answer continues to elude the candidate, we provide…
• Hint 2: When we ran the code with this set of values ([3,6,10,3,4]), it crashed midway when the iterator pointed at 10. Thoughts?
  • Notice how this hint is really narrows down the issue, and still doesn’t give away the technical reasons as for why the crash occurs. If this also fails to yield insight I recommend…
• A renewed approach: Stepping through the code together and have the candidate explain to you what is happening step-by-step.

The benefit of this approach is that the candidate is provided a figurative ladder from which to make the final leap to the solution on their own. Of course, the more steps – in the form of hints – they’re provided on the ladder, the less impressive the leap. By motivating a gradient of outcomes, as opposed to 100% correct or 100% incorrect, you’ll get a more nuanced idea of where approximately the candidates depth of knowledge lies, as well as providing tension relief as the candidate ideally always makes the jump to the correct solution, even if most of the steps were provided.

I suggest informing the candidate upfront about your intention to give them hints during the discussion along with the fairly obvious fact that the sooner an answer occurs to them, the better. There are only benefits to your candidates knowing how the interview will be conducted, and it’s not really possible for them to ‘game the system’. Note that actively providing a candidate with hints does occasionally lead to a manageable side-effect. Specifically, that some candidates will be inclined to suggest solutions or ideas as questions, e.g., “Is the issue X?“, “Perhaps a solution could be Y?“, etc.

In fairness to all candidates, I respond by asking candidates what they think, when they pose these types of “question answers”. The goal is to find the candidates committed answers, and then have a dialogue about it. Even an incorrect answer can lead to a dialogue that yields valuable knowledge. Ultimately, candidates posing questions is great, but when it comes to specific problems you expect them to understand and solve, you do want to make sure they can stand on their own two feet.

When to deviate

In 2012, Valve’s employee handbook leaked on to the internet. A salient point it makes is:

[…] hiring is the single most important thing you will ever do at Valve.

Valve handbook for new employees – Page 6

I tend to agree, that it is among the absolute most important activities for any company, which is why I’d be wary of modifications to the outlined interview approach, born out of an interest to lower costs. Hiring is one of the last places you’ll want to cut corners given the catastrophic consequences a bad hire can cause a company.

That being said, here are some possible scenarios for deviation.

Lack of resources or rare hiring

If you find yourself unable to convince management that hiring really isn’t the place to spare resources, or find that your hiring interval easily exceeds a year or more between hires, then I suggest either one of two method variations:

• Ditch the in-house testing framework – Source your take home tasks from online sources, possibly even Codility or other testing platforms. I still recommend providing the tasks as take home tests rather than online timed tests to avoid the negatives listed earlier. While this approach has a lower upfront cost, it will instead take more time to properly analyze and evaluate each candidate, as you’ll be doing more manual work there.
• Ditch the take home task – Save even more resources and rely fully on a cheap start-up Codility knock-off variant. If you absolutely must use such a platform, I recommend keeping all the previously mentioned downsides in mind:
  • Needless logical reasoning – Generally pick the easiest problems they provide so you focus on your candidate’s implementation skills, as opposed to logic puzzle solving skills.
  • Unrealistic time pressure – Turn off the running clock if possible.
  • Encourages lazy inaccurate interviewing – In addition to the platforms performance evaluation, make sure you inspect the solutions your candidates to create yourself. You have much to gain from not relying on the single-digit evaluation of the candidates you test.
  • Unintelligible task descriptions – Familiarize yourself with the problems you picked and try solving them yourself.

Culture hire

The outlined approach focuses on determining a candidates technical skills. It’s worth at least a single paragraph to emphasize that there are times where the best hire for the company, isn’t the person who excels the most in a technical capacity. Sometimes, the best hire you can make is the employee that can help shift troublesome company culture in a positive direction.

Simultaneously, it is also a grave error to hire an exceptionally technically talented individual who’s personality clashes with the existing team, or simply has difficulty recognizing issues in their own work, as an earlier anecdote described.

But culture-related hiring concerns is beyond the scope of this article, and probably deserving a wholly separate article, on their own.

How our approach evolved

Our interview approach was refined over the course of five years, but fundamentally remained the same. The biggest refinements include:

• Automation – When I initially assumed my duties as candidate evaluator, it was mostly a manual task and took 1-2 days to fully test and analyze a candidate’s solution to the take home test. Building an in-house test framework reduced this time down to about 3-6 hours per analysis, depending on the solution. Well worth the 4-6 week investment of developer time to streamline this critical and ever re-occurring task.
• Standardization – Automating the testing of the take home test also required more standardization of the test itself. While we generally preferred to give candidates as wide a berth as possible when it came to implementing their solutions, I would recommend some structural limitations such as requiring the implemented solution to fit a specific function signature.
• Task description evolution – We encouraged all candidates to ask for clarification if something was unclear, and multiple candidates asking the same question was a dead giveaway that our task description required further clarification.

Caveats and concerns

With soft problems like hiring, no approach will ever be perfect. Here are the caveats the outlined interview approach:

• High use of candidate’s time – For any candidate who passes the initial phone interview, it can require a full day of their time to solve the take home test. I’d suggest offering candidates compensation for this time investment. Most companies are unlikely to do so.
  • Note that this caveat also applies to Codility and similar platforms. Although most Codility tests I’ve taken set the time-limit to two hours, considering the amount of prep-work I’d recommend anyone taking a Codility test to do, it quickly becomes comparable to take home tests.
• High use of resources – If I worked for Codility, this would be the drum I would bang. “Your developers time is too valuable to spend on interviewing! Let us handle this tricky task and you can just focus on whatever it is your awesome company does!“. Do not fall for this tempting line of thought. Hiring is one of the most important tasks you’ll ever do. Yes, the outlined approach requires actually investing time in the candidates solutions, but that’s where an in-house testing framework can cut that work down to a manageable size.
• Limit the number of judges – A tangential point to the outlined approach, but I still feel strongly enough about to include. I recommend limiting the number of people you involve in the hiring decision, as my impression from interviews with other companies, is that there’s a trend towards the opposite.

Finally, I thought it prudent to think of some concerns I might have, had I not spent all this time using the approach:

• Cheatability of take home test – Given that we rarely varied the tasks given to hopeful candidates, a point can be made that we open ourselves up to rampant cheating. Keeping in mind the aforementioned assertion that eliminating all cheating is a pipe-dream, we intentionally had the initial phone screening as a barrier to avoid needlessly providing our test kit. Furthermore, the in-person interview dug so deep in the provided solution that if the candidate hadn’t written it, they’d have to have spent a significant portion of time familiarizing themselves with it and why it was built they way it was, making the whole effort to avoid time spent writing it in the first place irrelevant.
• Not enough judges – As hinted early on in the article, a maximum of typically three people interviewed the developers we hired. Other people, including Joel Spoolsky – who’s article on interviewing I largely agree with and highly recommend – would assert that three is too small a number. My impression, from being interviewed at other companies, is that there is a trend towards a higher number and I generally caution against it. Personally, I think both diffusion of responsibility (“If things go side-ways no one person is at fault.”), and the thought that each additional interviewer is additive (“Best get everyone’s input on this!”), attracts companies to this decision of increasing the number of interviewers. I’d do not agree with the latter, and at the risk of sounding like a broken record player, I caution against increasing the number of involved individuals in a hiring decision. I’ll leave finding the exact right number for you, to you, and possibly a future article.

Phew…

If you got to the end, I appreciate your attention and thank you for taking the time to read my ramblings. I hope you’ve found some value in the article and all the best with your interviews and interviewing!