For CS171 by Adam Scarborough
Motivation:
The idea was essentially just to find out where the work was distributed around the Commonwealth, and to find coorelations between where the jobs are, where the most/least money was being paid for those jobs, and where the lowest cost of living was that was still close to where the money was. The reason for choosing this was pretty selfish. The job market is pretty bad right now, and I figured that having a simpler means of finding work in the off hand chance that I find my self suddenly unemployed. I also figured that if it helped me, that it would likely help others as well.
Data Sources:
The data came from two sources. The Massachusetts Dept. of Employment and Training (see references below), and SimplyHired.com. All of the data collected can be found here, or by going to the links provided below in the references section. For the MA DET data I collected wage data for Worker Improvement Areas (WIA); which, segmented primarily by county and metro area (e.g. Lower Merrimack, North Worcester County, Boston, Berkshire, etc...). The wage data is then listed by how the state denotes the diffferent industry categories for those regions. For example, the "Roofing" job category that is represented under the "Construction" industry there is the following data for Q3 2008 (the most recent data available from the site at the time):
Number of Employees,,,Total Wages,Average Monthly Employment,Average Weekly Wages,
"58","654","645","604","$7,512,851","634","$912"
, and there is something similar for each of the other industry categories for WIA districts. The original source data that was pulled for the inital release of this application for wages by WIA can be found in the source here.
The employment rate data also came right from the MA DET site (see references below). It was the only data set that I could find on their site that all of the data per town/city for the entire state in one file. It's a tab delimited file, that contains data for almost every town/city in the Commonwelath on number of employed, number of unemplyed, and the percentages for both. None of the data from the DET was easily accessible as there was no easily identifiable public web service to just grab the data from. You have to go to their site, make the selections, and click a button to down load a .csv or excel file. The city/town coordinates for the map were plotted by myself using processing. The map came from the MA DET as well a link to the original pdf can be found below in the references section. The version used of this was generated with photoshop, unfortunately processing has a hard restriction with respect to .png dpi -- it has to be 72dpi or all you get is a big black square. So, some of the detail does get lost with the dpi reduction.
All of the listing data comes from scraping SimplyHired as their site is based around a simple restful api; which, made it easy to scrape. Unfortunately, as I could not become a partner of SimplyHired with out spending a lot more money then I could manage I can't get real time updates for the application, so it has to be scraped for now. I decided to do this just with python, and beautiful soup. It takes 30-40 minutes to scrape all fo the data for the couple of job categories I selected. Some pre-scraped data is available to use to take a look at the application while you wait for the latest data to be scraped. The scraping application source is located right here, and is licensed under the MIT license.
The Application:
When the applciation is loaded select a job type from the drop down menu, and check the show listings check box,
Hover over the points on the map to see where is being looked at, and some listing quantity, and unemployment information. Check the unemployment checkbox to rener that data on the map.
When you've found a location that you'd like to apply for the jobs in "click" on the location, and the browser opens up thanks to BrowserLauncher2.
All of the source is freely available under an MIT license right here, and a zipped version here. Its made up of two parts. The front end swing application in which Processing is embedded into it, and the python data collector. The precompiled executable jar file is located in the dist folder. To run the application with the existing data download the jar file, and run the following:
java -jar "<path to where you downloaded>\WitCUI.jar"
or just load the source in netbeans, and run from there.
To run the python data collector use python 2.6.2, and while in the PythonDataScraper folder run,
python DataCollector.py
References:
Massachusetts Deptartment of Employment and Training:
http://www.mass.gov/?pageID=elwdhomepage&L=1&L0=Home&sid=Elwd
Unemployment by town: http://lmi2.detma.org/Lmi/lmi_ces_a.asp
Wage by Worker Improvement Area: http://lmi2.detma.org/Lmi/lmi_oes_a.asp
Map of Commonwealth: http://www.mass.gov/mgis/ix_cnty.pdf
SimplyHired.com: http://www.simplyhired.com
BrowserLauncher2: http://browserlaunch2.sourceforge.net/
Processing: http://www.processing.org