One common question related to customer data is "where are our customers"? This type of data can show us trends for sales, as well as opportunities for future sales. One great way to show this is by creating a heat map:
I created the heatmap as a set of tiles using a few open source software components.
TileCache for generating the tiles based on x/y/z coordinates from the Google Maps API
The first thing I did was import all the addresses into a Postgres table with an additional column defined to hold the Geometry column (which PostGIS provides). Raw addresses don't really help put points on the map, so I had to geocode the addresses. This can be accomplished in a handful of ways. I found the Ruby Geocoder gem to be quick and painless.
From there I fired up TileMill and designed my map layer the way I wanted it. You can think of TileMill as a kind of a map design studio. Import geospatial data from a variety of formats, and then style the data using CSS-like scripting. Here is the interface:
Here you can see changing the heatmap color palette to blue for the most dense areas of the heatmap
You can do something relatively simple like this example, but get as complex as creating a customized street map from Open Street Map data.
When you've designed your map just the way you like, you can export the map in a variety of formats (PDF/PNG/etc.) for publishing. I wanted to serve this map as a layer on top of a Google map, so there are a few more steps involved. Instead of exporting to PNG or PDF, I exported the map as "Mapnik XML". Mapnik is a tool that renders maps, and you create these maps by using XML. TileMill is a means to create this XML in a visual way.
This XML includes everything necessary to connect to the data source and specify all the visual styles necessary to render the map image.
With the Mapnik XML, I can use TileCache to read that XML and have it generate tiles based on the x/y/z coordinates from the Google map API. The x is the horizontal, y is vertical, and z is the zoom level the user is zoomed to (1-20). This xyz scheme is based on the TMS specification by osgeo.org.
TileCache is a web application that receives HTTP GET requests and returns a 256x256 pixel PNG file. For instance, this URL: http://localhost:8080/1.0.0/customer/4/4/5.png produces this PNG:
(Border added to show dimensions)
In tilecache.cfg you can define one or more layers according to the docs.
One nice feature of TileCache is that when a tile is requested, it is rendered and then cached on disk (or another cache that you can configure). The next time that tile is requested it is served from cache instead of regenerated.
Once the tile server is set up, it is just a matter of using the mapping platform of your choosing (such as Google Maps) and finding the correct API to include a new layer onto your map.
This isn't a comprehensive guide, but I hope it gives you a good overview of one set of tools that enables you to quickly create a really nice looking map to print or overlay on a map like Google Maps.
Codemash 18.104.22.168 was a great time. I always run into some of my favorite people in the development community. This year was no different, and to add to the fun I was able to share some of my experience using Free and Open Source GIS tools by giving one of the talks.
I am making the slide deck available here for anyone who wanted to snag a few of the links. In case you don't want to go through the powerpoint looking for them, here they are:
A couple years ago my wife bought me an electric smoker. Really nothing fancy or complicated; it's essentially a metal drum with an electric heating element and a few racks. Now that I've had it for a while I recommend it to everyone. You can get some great tasting meat from some really cheap cuts, and on top of that it's fun to experiment with flavors.
So far I've smoked 5 cuts: beef brisket, pork shoulder, chicken breasts, a whole turkey (Thanksgiving!) and a whole chicken. The pork shoulder is what I started out with because it's got so much fat interwoven throughout the cut that it's really hard to dry out and ruin. The brisket took a few tries to get right, and I've yet to successfully smoke a chicken (more on that later).
I got the smoker for Christmas, so I had thought that it might be a little impractical in the cold weather. So I waited until the weather warmed up a little - into the 50s maybe (if that). It was fine. Since then I've used the smoker during light rain and 40s. I think it will add some to the cook time, but it will work.
Yesterday we had a record breaking 101 degrees in Columbus, Ohio where we live and I smoked a brisket for 20 hours. It really didn't seem to cook much differently than when I smoked one a few weeks ago at around 75-80 degrees.
One thing I recommend spending a little money on is a probe thermometer. Keeping the lid on the smoker is crucial to keeping the heat in. Many probe thermometers have remotes that you can read from indoors to keep you from having to continually open the smoker and check temperature. Another reason for a probe thermometer is that the smoker colors the meat so much that it's easy to think the meat is done when it's still got some time left to cook. With turkey and chicken the smoke can add a pink tinge to the meat color which makes it appear under-done.
The model of smoker I have does not have a thermostat. I've often thought that this might be a nice feature - but keeping an eye on the meat temperature to keep from getting over or underdone seems to be enough to come out with a solid product.
The first couple times I tried to smoke a whole chicken I did so without a thermometer. The first time I undercooked the meat and the second time it dried out so much it wasn't really any good to eat.
Many smokers come with a bowl for water that sits somewhere above the heat source. It is really important to keep water in the bowl during cooking. It makes a noticeable difference in the juiciness of the meat.
The first few times I smoked I put the meat on the bare rack and let the drips fall wherever they happened to go. This lets a lot of smoke into the meat, but the cleanup is not really worth it. Instead I get disposable aluminum cooking trays and put the meat inside. The smoke is still able to penetrate through the open top, and I can also add a couple cups of water in the pan for added moisture. For briskets and pork shoulder I take the meat out after the first hour and a half and cover it with aluminum foil. Since 90% of the smoking is already done at this point, it just serves to keep moisture locked in.
Experimenting with wood flavor is a lot of fun. I've tried apple, mesquite, hickory, and peach. For pork shoulder my favorite is hickory and apple for brisket. I didn't care at all for the mequite. The key with the wood is to not use too much. The smoker will really only "smoke" for about the first hour and a half. The rest of the time is cooking. Getting too much smoke will make the meat taste like a hot dog. Of course another way to get different flavors into the meat is by concocting rubs. What I do is get a bowl and start dumping stuff in. I like a combination of cayenne, black pepper, ground cumin, salt, garlic powder, onion powder, and paprika. I've read that leaving the rub on for a day in the fridge helps get that flavor locked in - I tried it and it really doesn't make much difference.
For thanksgiving turkeys we smoke with bacon laid out over the top of the bird. This keeps the turkey basted the majority of the cook time. My suggestion here is to find unsmoked bacon. The smoked bacon gives the outside layer of meat an over-smoked flavor that is actually somewhat unpleasant.
Any of the meat I've smoked I enjoy eating straight - no buns or sauces (needing sauce is a sign of failure!). As soon as the meat comes out of the smoker I prepare it for storage in the fridge. With the pork shoulder and brisket I pull it apart and remove the fat, but then at the end pour the juices it was cooking in over the meat. The juice is full of the flavors of the rub you used as well. Keeps it nice and juicy when warmed up in the microwave or over the stove.
For visualizing spatial queries there are several open source tools that can do the job. The sweet spot for these tools seems to be when you have the need to view an entire "layer". Layer here is synonymous with either a spatial database table, or a shapefile (http://en.wikipedia.org/wiki/Shapefile). This worked for a majority of the visualization I needed, but other times I just wanted to run a spatial query and see the result on a Google map:
This is a screenshot of a simple tool I created to input a SQL query and get a map vector overlayed on the Google map. As long as the column being returned in the SELECT is wrapped with ST_AsGeoJSON, the vector will render. GeoJSON is just JSON in a specific format, and it is awesome. As you can see, PostGIS has ST_AsGeoJSON() in there ready to go.
The dropdown in the screenshot shows each of the connection strings defined in the web.config.
The code is written for ASP.Net MVC and PostGIS, and you can get it here.
The past 6 or 8 months have had me deep into geospatial technology. Going from knowing nothing to releasing a production grade application has taken me through the spectrum of fun and frustrating. There is so much that I've learned I want to document it here in hope that it will help someone else.
From the starting line, one of the first decisions to make is a geospatially aware database system. Unless you're only dealing with a few small spatial queries, your DBMS is going to be doing the bulk of the heavy lifting. In the case of the application I am writing, the database wound up being my primary abstraction layer. 80% of the application layer is relegated to parsing HTTP requests and turning them into SQL commands.
In the mainstream there are many choices:
PostgreSQL with PostGIS
Microsoft SQL Server
Plan on paying to play for numbers 3 and 4. The rest are Free and Open Source Software (FOSS).
The shop I'm at develops primarily with SQL Server 2008 so it was a natural first choice, but quickly became frustrated at the lack of support from some of the open source GIS tools. Many didn't support drivers for SQL Server and some that did did not support it well. I'm sure this will change in the future. There are some other deficiencies such as lack of support for some aggregate spatial functions such as STUnion (http://msdn.microsoft.com/en-us/library/bb933914.aspx). This particular functionality was addressed in SQL Server 2012.
PostgreSQL/PostGIS is a defacto standard in FOSS communities for it's performance and reliability. I'll go out on a limb and say that any FOSS GIS product will support PostGIS. This was critical for my project as it was a goal to use as little commercial software as possible. Since beginning development with PostGIS 8 months ago, there has never been any regret. Because GIS is an extremely fun and rewarding niche of computer science, there are huge numbers of enthusiasts that are motivated to contribute. Since these folks are so passionate about this technology, I have found that they build perfection into their work.
Since just releasing our first GIS application this week I now have a much better appreciation for cartography and the work that goes into creating beautiful mapping applications. It has been a joy to become involved in the open source GIS community and interact with some extremely talented and passionate people. I owe these folks a ton of props for their contributions to open source projects, and helping me come up to speed.