Want To Recruit Better Developers? Give Them Broken Code

Hiring the right programmers is key for development, but it can be a nightmare to find time to administer coding tests and interview candidates. So rather than relying on a scripted phone round that potential hires can easily game, the mobile consulting firm Mutual Mobile is implementing more pragmatic screening in which candidates have to fix a broken app and run it in order to get to the next round of interviews. That’s it.

Hiring the right programmers and designers is crucial for any software business. Without a cohesive team, projects may never come together. But hiring a qualified developer can be such a long process that it can sometimes literally be endless: Just when one position is finally filled another may be opening up.

Big companies fix this problem by staffing up giant HR departments that take care of initial candidate screening and organize the company’s developers to administer coding tests and other specialized evaluations. But startups and smaller companies don’t have the same resources, even if they have a few HR people. With 325 total employees and about 60 iOS engineers, Mutual Mobile is large enough to have an HR "talent team," but when they first opened their HR department, they found that hiring iOS devs was draining all of its resources.

"We started originally with the traditional phone screening," says Ron Lisle, Mutual Mobile’s director of iOS. "And we found that folks were getting pretty good at fooling us on the standard list of questions. We’ve had some folks that have applied to engineering jobs and I’ve gotta believe they’ve never programmed at all."

The iOS team found that the only way to get a clear sense of someone’s skills was to fly them to Mutual Mobile’s Austin offices for in-person interviews and code tests. But relying on on-site meetings to sort through everyone who got past the phone screen was costly and inefficient, especially because the candidates coming in weren’t always prepared.

"About a year ago we were still doing the phone screen, but we got some folks where it was just really hard to tell," Lisle says. "It was like, ‘okay this person might be really good, but I’m just uncomfortable about this or that. I can’t tell if they really know coding.’ And so we thought, well, instead of just writing the person off, if we had some sort of test we could send to them then we could actually see what they would do with coding up something. We could verify that they can actually compile and build code."

Asking developers to go through a pre-interview code test isn’t new: Startups like Codility even offer automated testing as a product. But Mutual Mobile took their tests a step further. To get extra insight into whether a developer was right for the company (Lisle claims the company wants developers to "literally be crazy about mobile"), the company asked a few of its engineers working on the programming test to start adding run-time issues and other bugs to the code they would send to potential new hires. To decrease the impact on the company’s business, Mutual Mobile turned to what it calls "bench resources," or employees that have finished one client’s project and are waiting to begin something new, to write the tests.

"We thought, okay, so let’s send them some code that they’ll have to build, and let’s throw a few bugs into it while we’re at it and we’ll kind of get a sense for are they able to find and fix bugs," Lisle says.

To see what the experience is like, Co.Labs asked veteran iOS and Android developer Chris White (who works with Co.Labs editor Chris Dannen) to download the test and give it a shot.

"It’s interesting to engineer problems into the test, things that are broken" he says. "You don’t want them to be too onerous, but you don’t want them to be too obvious either. So I thought they did a reasonable job. If you’ve been using the iOS development tool exchange and XCode for a while then you should be able to fix these in a reasonable amount of time and already have provisioning profiles to run it on a device."

Mutual Mobile usually hires senior engineers, but the iOS team intentionally built three levels of bugs into the code so they can use it to evaluate a range of experience levels. Lisle emphasizes that the goal is not to hire people based on the test, but simply to use it as a productive screening tool so that candidates who come to Austin for interviews and problem-solving tests already pass the company’s basic requirements.

"We’re almost always looking for really senior people and we hope this test will be such that somebody who gets everything would be at that level," Lisle says. "However, when we’re hiring the middle experience level folks, they would get most of the questions but not all of them."

In thinking about the test a few weeks after taking it, White notes that coding tests which throw a candidate into the deep end are becoming more common. For example, White recalls that recently while applying for a job he was asked to do a pair programming project using a company’s own code during an on-site interview. But sending a test like Mutual Mobile’s out to first-round candidates, before they are even offered an interview, is still unusual.

"It’s a pretty interesting concept because oftentimes you’re new on a project, you don’t know the code at all, and you’re given assignments so you’ve just got to basically jump in," White says.

Mutual Mobile is beginning to implement the test in hiring and Lisle hopes that the result will be a much more streamlined process. He took the test himself and said that it was "a bit of a challenge," but should be fun for engineers who love what they do. White says he enjoyed the test.

"I hope they were honest with me in telling me how well I really did," Lisle says, laughing. "The intent was it would not be this big burden, like do this work to prove yourself, but more here’s something to play with and show us what you can do."

[Image: Flickr user Michael Himbeault]

Add New Comment


  • Hugh Fullerton

    This seems unethical to me.  If you promise to hire when the candidate fixes the app, fine.  But otherwise, it is expecting something for nothing.  The employer risks only an hour conducting an interview, and gets something of serious value.    Business should not take advantage of the unemployed like this.

  • oscode

    I expect the bugs are intentionally created for the purpose of validating a candidate's skill. They won't be unsolved problems which are still in their products.

  • Dan Sutton

    Debugging's only really one aspect of programming. I've met plenty of people who are brilliant at debugging but who couldn't write a single line of original code to save their lives. Debugging doesn't involve nearly the level of lateral thinking that writing an application from scratch requires: it's the difference between fixing a car and designing a new engine: the two, while obviously related, are worlds apart in terms of the level of understanding required.

  • Mario Zorica

    Dan you missed the whole point of this bug-fixing tests. As stated in the article: " ... simply to use it as a productive screening tool so that candidates who come to Austin for interviews and problem-solving tests already pass the company’s basic requirements.". In other words this is just a sort of negative filter, I believe it is not intended to indicate who to hire, but rather who is just not a good fit and by eliminating these participants you save time. Additionally it is also mentioned that there are some sites that offer automated testing, but that their tests go a step further. I really do not see that ... For example I'm familiar with this site: http://www.testdome.com/ It also offers coding tests and among them there are problem solving tests that just say "Fix a bug" and that's it, so I don't see any improvements or difference at all.

  • benjamin

    I don't think Dan totally misses the point. You are right the article is about presenting an initial problem-solving test but why stop at just letting candidates debug. Just like Dan mentions writing entire applications includes and surpasses a simple debugging test.

    However I agree with you Mario, websites such as www.codility.com, www.hackerrank.com and www.kattis.com already support this. It is interesting that companies are intent on building their own tests to better represent the work they do when those test usually take time and effort to develop and still can't hold the same standard as these testing websites.

  • Iain Collins

    For developers this is a huge time sink for a company you may decide you don't want to work for when you get to meet them - and if you're looking around you'll probably be checking out quite a few potential employers. 

    Sometimes you may not get meet the person who reviews your work during the interview process. They may not even check it out at all. It might be reviewed by someone with much less experience, or who isn't even a developer (or, who is a manifestly bad developer).

    It's far more efficient to just only hire developers who have already published code (e.g. on GitHub or a private repo) in the first place - and to make that a condition for applications.

    It's a better use of their time, as it's a one-time exercise, even if they don't have anything published already - at most you might ask for a version of something in a specific language.

    You learn a lot more more from doing this, like how they document their code without guidance, if their application checks for dependancies properly and/or builds in one step, if they commit early and often and if they appropriately format and word their commits.

    The community gets something out of it too (and so, usually will the developer).

    It's mind boggling to me that this isn't the norm.

  • mutinda boniface

    I greatly agree with this idea of giving a developer something that has already been done to perfection then introduce some bugs so that they may fix them.

    One big point i should raise is, you should not always hire the candidate who actually fixes the bugs but the coding practices and standards should also play a major role in the whole hiring process.

    Take for example:

    An application developed using OOP can clearly be fixed by a developer who actually understands the OOP structure which in most cases is a nightmare to non-OOP developers. So what i mean at this point is the standards of the ported in code will be totally different between the two developers. Take for instance the non-OOP developer fixes the bugs using try and error but the OOP guys somehow gets stuck in the way your code was originally written but introduces some better practices to your code.. whom will you employ???

  • kplcjl

    I find it hard to believe someone who can't code, can fix it by finding and correcting bugs in the code when they aren't even told how to reproduce the problem, that's like an infinite number of monkeys put close to typewriters producing the entire works of Shakespeare. Obviously the bugs aren't business logic errors unless they are also handed the business logic documentation.

  • mutinda boniface

    I get it but of course fixing bugs you must first get whats the app is intended to do otherwise you will spend hours and days starring at the code.

  • kplcjl

    I didn't come from the OOP world, so I definitely don't have the respect for it that you do. These are bugs that slipped by code review and made it into the code base until they were found and fixed. One time I was looking over code to understand what they were doing. I saw something that was a self-contained bug. I went to the manager expecting her to say, "Oh, yea. Thanks for spotting that" Instead I spent a week getting the specification document, reading the section, agreeing that the code does exactly what was asked for. She was ready to walk away vindicated, I'm about ready to slink away, but I just have to add... "It's just a pity, it doesn't do what it is intended to do." I did know OOP concepts, but the problem had nothing to do with that and that comment caused her to listen to what I was saying. This had passed through several experienced hands and hadn't been spotted. It wasn't an immediately fatal problem and the fatal problem was endemic, so pointing at one piece of code wouldn't help. This doesn't require a knowledge of threads, nor OOP design, you just need to be able to read one line in hundreds and spot the flaw. A simple test of the app would never find the bug. You don't need anything but a basic understanding of mathematics and a basic assumption to spot the flaw:

    if ((Maximum_threads_allowed + Number_of_threads_to_get) >= Current_thread_count ) Start_new_threads();

    The basic assumption is that all fields are positive numbers, usage elsewhere confirmed that was true.

    PS I had to edit the code because even intending to write it wrong, I wrote it the correct way!
    PPS I correctly wrote it wrong, then incorrectly fixed it to work correctly and then recorrected. This is the correct bad logic. if max is 20, step is 10 and current is 30, it would raise the thread count to 40> if((20+10) >= 30) get_steps();