Mining for Computing Jobs

Starting in the late '80s, various researchers began studying advertisements to determine what skills most IT jobs demand. In the early '90s, researchers such as Chuck Litecky and Kirk Arnett expanded previous work to include systematic nationwide sampling of US newspaper ads. In the late '90s and early 2000s, various studies began mapping the migration of IT job skills ads to the Internet, generally showing that most ads were migrating and that demanded skills were changing with the adoption of new technologies. Other researchers' methods included interviews and surveys, with many focusing on the importance of managerial and technical skills in computing jobs. Occupation analysts initially created the O*NET database and supplemented it with annual surveys. However, it's often unrepresentative of actual jobs and the complete skill sets that employers might require. Additionally, research methods often don't attempt to determine which combinations of skills businesses frequently desire.

Similarities among the skills required in each of the clusters suggest that we can abstract the 20 job definitions into five larger general job classifications: Web developers, software developers, database developers, managers, and analysts.

In any discipline, and especially in a discipline with a dynamic, highly competitive technology environment, professionals should periodically review the skills sets in high demand and identify industry trends in which their skill sets might be falling behind. Where downsizing and outsourcing is common, keeping up with current skill sets is critical.

Microsoft is a dominant presence in software. However, our research indicates that few organizations focus exclusively on Microsoft-oriented technologies. Although five of 20 clusters require skills specifically in Microsoft technologies, they account for only about 14 percent of all job ads. Open source and Java jobs are very competitively positioned, and they comprise about 12 percent of the ads. Open source technologies might soon match the demand for Microsoft technologies in terms of the number of jobs. Microsoft development skills and open source skills should constitute a significant component of skill sets for both first-time job seekers and established computing professionals.

The Web developers group accounts for 12 percent of all jobs, yet when considered among all development groups, Web development jobs account for almost a quarter of all programming work. This finding highlights the surprisingly low number of jobs whose only focus is Web-based programming. Additionally, although Microsoft isn't a dominant force in overall Web development, its technologies account for a third of all such work, whereas Java and open source development account for over 40 percent. It appears that Web development is an arena without a dominant technology but rather with focused niches from all platforms and that many different skills are marketable in this arena.

In the database administrator cluster, 91 percent of all jobs require Oracle. There's no corresponding cluster with such a high preference for any other database management system. The strong preference for Oracle skills is significant for computing professionals' career planning.

From the 1970s through the 1990s, the programmer's role expanded to include not only technical programming but also an increasing knowledge of business, communication skills, and critical thinking. The programmer became a programmer/analyst. Through 2003, programmer/analyst was the job term occurring most frequently in job ads. In a study of ads placed online between 2001 and 2003, ads for programmer/analysts specifically requested skills in software development and software (98 percent), but also business (83 percent), social (83 percent), problem-solving (77 percent), management (67 percent), architecture/network (67 percent), and hardware (42 percent) skills.

Our study determined job titles composed of collections of skill sets that appeared together in on-line ads. The results indicate that a split is occurring between programmer jobs and analyst jobs. There are three different developer job types: Web, software, and database developer. Each leans heavily on technical skills. The other two job types, managers and project analysts, have more general technical skills but also include skills such as leadership, strategy, and security.

Development jobs requiring considerable technical knowledge but needing relatively little business or management knowledge are more easily outsourced than jobs requiring a great deal of business domain knowledge. Perhaps organizations are consciously or unconsciously preparing to outsource more development jobs by separating the programmer from the analyst. This has clear implications to people seeking to upgrade their skill sets. For organizations, this result might indicate where the industry is headed, and they might wish to prepare accordingly.

Methodology: Using a Web content data mining application, we extracted almost a quarter million unique IT job descriptions from various job search engines and distilled each to its required skill sets. We statistically examined these, revealing 20 clusters of similar skill sets that map to specific job definitions. The results allow software engineering professionals to tune their skills portfolio to match those in demand from real computing jobs across the US to attain more lucrative salaries and more mobility in a chaotic environment.

We developed software that systematically searched Monster.com, HotJobs.com, and SimplyHired.com daily between July 2007 and April 2008 for jobs requiring a degree in computer science, management information systems, computer information systems, and other computing programs. We extracted 244,460 unique job advertisements. We analyzed the data using cluster analysis, a statistical technique for classifying cases in groups that maximizes differences between groups and minimizes those within a group.

Source: IEEE Computer Society; Chuck Litecky, Andrew Aken, Altaf Ahmad, and H. James Nelson; 1/2010