Assessing Software Skill Sets
We formally documented, measured and assessed hiring 50 software development candidates for one single skill set. Guess what? They lied… Read our results.
One of our projects required an additional Telerik expert for UI (user interface) custom software development for some 60 UI pages required by a client. Our guys were busy with a long term Oil & Gas mobility project so we needed to hire a new person.
Skill Set Assessment Process
To assess the competency of applicants for this specific skill, we created a short requirements document (a grid and calendar with a few lesser known Telerik functionalities). Our most experienced Telerik developer coded the requirement in 45 minutes. We advertised the position world-wide, stating we required ONLY this single skill. We attracted 52 applicants. After the initial interview (which involved reference checks, verification of work history and experience with Telerik), the pool of applicants was short-listed to 29 due to misrepresentation (many had lied about their Telerick experience). Note we did not short list any of the candidates based upon their hourly rate. As you will read below, short-listing based upon their rate per hour is a critical mistake.
One by one, we provided the 29 remaining short-listed candidates with a clean remote development workstation with VS 2012, Telerik, Resharper and SQL installed and configured. The requirements document was located on the workstation. We reviewed the explicit written instructions and verbally reviewed the requirements with each applicant. The timer was started, the recorder was turned on and we watched each applicant perform the assigned task.
Of the 29 people, 9 dropped out of the running after 30 minutes, they could not even create a Telerik based project. This is a task that takes 2 minutes. In 30 minutes, they could not even accomplish a common process related to using all Telerik modules.
Of the 20 remaining applicants, after creating an empty project, 18 of them went directly to the Telerik website and copy/pasted example code (Telerik has excellent code examples). But try as they might, they could not progress past their copy/paste “achievement”. They never actually wrote a single line of code themselves. Only 2 of the applicants directly coded the requirement based upon retained knowledge, without referring to the Telerik website. And only 1 applicant completed 90% of the requirements in the 2 hour time allotment (that our developers completed in 45 minutes).
Skill Set Assessment Summary
In summary, the 52 applicants were short-listed to 29 (45% cut rate). Of the remaining, 93% produced nothing in 2 hours and only 2 of the applicants had a semblance of retained knowledge for ONE SINGLE SKILL.
Yet if their rate was $5 or $10 more per hour, most employers would have not have considered them for the short-list, even though they were the only people that could actually deliver.
Based upon this real world test, without this assessment process, you could have chosen to hire one the 27 different developers to produce nothing in 2 hours. Or, if you performed a similar assessment (unlikely), you could have selected one developer (of the 2) who was capable of completing 70% of the effort. Or you could have hired us and we would have completed the task in 45 minutes.
Basing a hiring decision due to a lower hourly rate, or excluding developers who charge a higher rate is ill-advised to the extreme, and yet it’s common place in the software development industry. DO NOT MAKE THIS COMMON MISTAKE.
Based upon our 130 YEARS of combined management experience in vetting and hiring developers, people need to seriously reconsider hiring “freelancers” rather than a professional North American based custom software development firm which provides a legal, enforceable guarantee backed up with liability insurance.
Freelancers use their computer for personal email (malware), they rarely have antivirus installed, they often use a “free” version (called community editions) or a “personal use” edition of software tools which EXPLICITLY prohibits commercial development use. As such, they illegally use software to develop software for you. Freelancers do not usually provide an MSA, nor do they legally and explicitly assign the IP you paid them to create for your company. They most likely do not have a separate commercial firewall. They do not have liability insurance. And while they may state a guarantee, such a guarantee is nearly impossible to enforce or get judgement upon (the court in India is backlogged 10 years).
By hiring them, you risk them introducing viruses in the code. If they work remotely on your development system, their infected PC can infect your network. The risk of intellectual property theft is real. And what about their ability to infect your customer data residing in the database they have access to? What if they unintentionally expose your customer data? Or sell your customer data? All of these risks are real, they have occurred, most often unknowingly, but sometimes purposely. Too often, you read news articles about major corporations whose data has been breached.
Hiring a freelancer exposes you to risks which you are liable for. Have you ever deeply examined the computer a freelancer is using? Have you browsed their Internet history? Have you confirmed their development environment is secure and the tools they use are legal for commercial development? Do they utilize a commercial firewall appliance? Have you done a criminal check? Did you ask if they have liability insurance? Have you even attempted to ensure they are who they say they are? Have you seen their passport or drivers license? Probably not. Utilizing a trusted commercial development firm located in North America eliminates these risks.
Most organizations do not utilize such an intensive and elaborate vetting process to assess individual skills for each applicant. Why not? The answer is usually that the company itself does not have the knowledge to assess a candidate beyond the common employment verification and reference checks. Compounding this, organizations demand a skill set scope that is far too wide, involving vastly different technologies which are different careers in their own right. This means you are probably hiring a person who is 70% incompetent. And most companies that hire freelancers do not consider the freelancer risks I identified above (for which a company can be held liable for or at the very least, you lose your job for). This is where we are today.
Do EVERYTHING possible to assess candidate skills. Assess EACH skill. Determine where they fit best (UI layer, Logic layer or Data layer). And force them to focus their knowledge, education and training to where they best fit and their passion resides. Otherwise, your hire will cost you dearly, your timeline will be screwed and you will be held responsible. If you hire a freelancer, examine the system they will be using to develop code and access your customer data. Remember: Just one copy / paste by a developer from their PC can infect your entire network. And always do a criminal check. Otherwise, you and your company can be held liable. Especially if your customer data is exposed.
Campbell Software is a commercial development house. Each developer is insanely vetted for EACH skill they claim to know. After passing this muster, we perform a criminal check to ensure our hires can be trusted. We then identify their strengths and utilize their strengths as part of a development team. Our developers work at our site, using our locked down, secure environment. Personal use of corporate assets is a firing offence (albeit we provide them with access to a separate DMZ separate from our network for their personal use devices.) These processes and policies are CRITICAL for us to deliver our promise of providing higher quality software faster for less in a secure and safe environment that protects you. And we back up our promise with a legally enforceable guarantee.
About Gary Campbell
Gary Campbell is the founder of Campbell Software, a Custom Software Development Company. He and his company provide commercial custom software development to USA and Canadian clients across North America. His developers and management have nearly 500 years of COMMERCIAL experience. Commercial experience is different than in-house experience. For example, a person who was employed by 3 companies over 10 years knows 3 tech stacks. But a team of commercial developers who have worked for 200 clients over 10 years have experience with 200 unique tech stacks. They also ramp up extremely quick and they have a much deeper and wider expertise than an in-house developer. To view all content at Campbell Software, view our Sitemap.