A Real Developer Challenge

Spotify is having a coding challenge to find "top-notch talent to join our NYC team". The challenge is to solve the most algorithmic puzzles in four hours... alone. "You may not cooperate with anyone, and you may not publish any discussion of solutions." What sort of developer will win this competition? Someone who is quick, dirty, has a mathematical mindset and lucky enough to write something that happens to work for the test data set. The "rockstar". Is this somebody you want on your team? Would you want to maintain their code?

Last year while on contract, the company in question was passing around their coding problem they used to test new hires. It was pretty typical stuff: give the data going in, the data they want out, and write a little program to do the transform. They even supplied most of the program, including a test; the prospective hire just needed to write one sort subroutine which could deal with "Low", "Medium" and "High" as well as numbers.

Predictably, this halted all coding in the office for a solid half day while everyone figured out the most clever way to sort the data. My opus was to observe that the input data was already sorted, so I redefined the shuffle() routine. The best one was from a co-worker who observed that "Low", "Medium" and "High" sort correctly by their last letter in reverse order. It was fun for us, but it wasn't very useful.

This is a pretty typical coding problem used to judge potential hires, and it sucks. All it tells you is the candidate is not completely incompetent. Why do we keep using them? They're easy. They're easy to think up, easy to judge and easy to administer. They're also the sort of clean, algorithmic problems a stereotypical programmer loves to solve. Do they have anything to do with detecting a good developer? No. Can we fix it? Yes!

Before this can be fixed, first we have to work out what a project wants in a developer. What do developers do all day? We can work this out by reversing every contrived element of our typical programming contest: clean, well defined inputs; clean, well defined expected output; a clear description of the algorithm; pre-existing template code; pre-existing acceptance tests. When's the last time you were handed a problem like that in the real world?

Instead, we get poorly defined inputs, sample input that has anything to do with reality being a luxury, inputs riddled with mistakes. Expected behavior and output are vaguely defined. As a developer we're presented with a clean sheet, no template, no tests, just a blinking cursor and a blank page.

How does one solve the contrived example? First the specification, inputs and outputs are carefully examined. If there's any ambiguity it's discussed up front with the person who probably wrote the problem and understands it perfectly. Then the candidate goes off and writes some code until it passes the tests. There is little or no interaction with people, everything is handed on a silver platter in unambiguous terms.

How does one solve the real example? First you have to find somebody who understands the problem, usually not a programmer, and discuss the problem. Then you drag some samples out of them, converting them into a format you can actually use. The user probably doesn't know what they really want, so the behavior/output will be ill defined. Pressed, the user will make up something that you know will be wrong as soon as you show it to them. Armed with this "information" you hammer it into some sort of algorithm and now need to write some code.

But you don't just hammer out a script. You need to do it in a way that matches the team's coding style. You'll probably want to make the meat of it reusable, so you need to write it as a library not just a one-off script. It'll have to deal with the inevitable bad input, which means good error handling and recovery. Other people will have to use it, which means good documentation. It needs tests written in a way which works with whatever integration server the team is using. And, of course, it should all be checked into version control with well defined and logged commits.

And then, when you've written all that and got it working, you take it back to the user and they tell you it's not what they wanted. Or they show you some new input that doesn't match what they originally said. If you're good... and lucky, your code is robust enough to handle it. If you're not... back to coding with you! Repeat until dead.

In that light, a good developer is one who works well with others, but also can make a multitude of small, detailed decisions about a problem they know very little about. They need to take vague requirements and work with a user to turn them into something a computer can do repeatedly. A good developer looks beyond the immediate requirements and thinks ahead ensuring the code is flexible enough to withstand future change. A good developer writes code not for themselves, but for everyone else on the team.

So... how do you test all that? And in less than an hour? Turns out it's pretty easy with some simple modifications to the classic example. You keep the same basic algorithmic problem, but you give it to the candidate the way a user would. It can be as simple as this:

Could you sort this data please? [Excel document attached]

You can be a bit more clever, feeding the candidate what seems like enough information to do the work but is full of subtle ambiguities, tempting them to take the easy route and just get to coding, weeding out those with the undesirable tendency to program without questioning what they're doing and why, but that's the crux of it.

Now sit back and see what the candidate does with that. Respond to their questions, but remain in the persona of a user. What you're looking for here is how they push back. What sort of questions to do they ask to gather and clarify the requirements? What sort of assumptions do they make? How well do they bridge the communications gap between customer and programmer?

Given a blank page to code on, what do they do with that? Do they write tests and documentation? Do they write a quick script or a library? Do they use version control? Do they ask about your code standards? Do they use pre-existing libraries? Do they write to the letter of the requirements or leave some flexibility? And again, what are their communications like through this process?

Once they've submitted their first solution, change the requirements on them subtly by offering a second set of inputs. These will be the same but subtly different. Maybe throw in some Unicode or deliberately malformatted lines. Maybe change the sort criteria. Does their code fail gracefully? Messily? Silently? Do they notice the changes? What shape does the discussion over the change take? Are they indignant about the changing requirements, do they take them in stride? How much work does it take them to adapt their code to the new requirements?

This test will take more time than a traditional puzzle test, but it can be administered easily enough over email and doesn't involve more than a few minutes attention at any given time. A dud developer can sink your team and cause far more damage than good, so it's worth the little bit of extra effort to find out if they can actually do what developers do, and not just solve clever puzzles.


It's funny to me to read posts about how to test candidates. As a Perl developer in Europe in literally all cases it was me who had to interview the company to find out whether they have any development standards.

I wish it were that simple. It's not just that companies skip mentioning such things. Out of 8 companies i talked to, there was only one that recognized the value of CPAN enough to agree to letting me put things in place to use it in production. None even knew testing or used VCS in such a way as to not be a complete failure. Instead i've got 8 distinct and unique horror stories.

Finding a company that actually asks me to write a test would be the holy grail for me.

"My opus was to observe that the input data was already sorted..."

Your "opus"? What the heck does that even mean? It's an anagram of "soup". It's the name of a penguin. These things do not help.

You can be a bit more clever, feeding the candidate what seems like enough information to do the work but is full of subtle ambiguities...

This is actually a terrible way to interview people. From a position of power (you're the one with the money), you pretend to ask someone a question, while actually wanting them to dispute your question. If you want to test the candidate's ability to tease out testable requirements, tell him that up front.

The thing about puzzle/trivia type interview questions is they tend to select candidates who have a particular kind of mindset. I suppose that may be a proxy for good developers, but I'd say it's rough at best. I like your ideas a lot better.

The developer's job is to wriggle through the interview process, fellating as necessary, until he gets the job. He can then figure out how Management wants him to behave. If he performs this behavior well enough, he's a good hire.

If he finds himself stuck in your interview process, he can do one of three things: (1) be pedantic and techincally correct ("the best kind"!), and hope that makes you think he looks smart; (2) be agreeable and ingratiating; (3) interview like he doesn't need the job.

"Guess how you want me to act" is a terrible command to give someone who you can hire or fire at will.

@educated_foo: I understand your point, but if I hand a candidate a spreadsheet and say "imagine this is another day on the job" and then I tell them I want the data returned to me, sorted, that should be enough. If a candidate is so unimaginative that they can't "imagine this is another day on the job", I don't think I'd trust them. There is so little imagination required to understand that sentence that I can't possibly imagine (cough) that they will show any creativity on the job.

@shwern: one company I worked for always asked a "business question". The candidate would be given a totally non-technical scenario and asked to discuss the pros and cons. They weren't expected to necessarily give a "correct" answer, but they were expected to show enough common sense to think about things from a business point of view rather than a technical one. Candidates who could not do that were generally not offered a job, regardless of the strength of the rest of their answers. I quite liked that. As a result, we tended not to hire "dogmatic" people who couldn't accept that their perfect technical solution might be a bad business solution.

Case in point: one company I worked for had two choices for some software they had to buy. The technical people were upset because they unanimously agreed that solution A was the best technical solution, but the company went ahead and chose solution B was the only one the the technical people said could be implemented by a legally mandated deadline that would cost the company a ton of money if they missed. Technical considerations had to come after business ones in this case.

Leave a comment

About Michael G Schwern

user-pic Ya know, the guy who tells you to write tests and not use MakeMaker.