Gav's Blog

I’ve been mostly quiet on the geospatial front recently as I’ve been busy with work, but there are a couple of things worth commenting on.

1. Environment Canterbury GIS Beta blog. Not the first to the council GIS blogging game, but certainly early, is my local regional council in Canterbury. They’ve started a blog to engage with the community, and built their first mashup. This is a great step, and I’m excited to see more of the public GIS practitioners provide a means of direction communication and engagement with the communities they represent. Very exciting.
2. A reminder that whilst we’ve still got a long way to go, things could actually be worse.

Getting back to mashups, I believe that they are a nice simple and cost effective means of communicating simple geospatial needs to users – ECan’s swimming water quality mashup is an example of that – a simple means of seeing visually the quality of swimming places around Canterbury.

However, the real benefits are going to come when that information is freed from the encumberance of being tied to an Internet connection. Lots of recreational activities that we undertake are not always near Internet connectivity, and mobile data access is still prohibitively expensive to be checking these sort of data sources over summer when away from your computer and Internet connection.

As a very keen recreational user of handheld GPS for geocaching and other activities, I am ever hopeful that councils such as Environment Canterbury will start to consider making some of their underlyign data available for download so that it can be bundled into offline maps that can be used in car, auto and mobile phone, GPS devices.

This will complete the full loop – councils create our communities (geospatially at the very minimum), they record geospatial data about our communities, and with mashups – we are finally starting to get some of this geospatial data back.

I look forward to the day when councils are extremely open with the geospatial data and make it available so that we can put in our choice of portable mapping device

Always a sucker for nice aerial imagery – NASA has a nice image for Aqua/MODIS of a plankton bloom near Chatham Island that was taken a couple of days ago. Hard to believe that without satellite imagery, we’d have little idea of the scale of these phenomena.

Back in July, I posted about dc.gov releasing some data. I was a bit slow replying to a comment made by Nat Torkington then, and felt that a reply actually required a new post to elaborate further on why I’m so supportive of governments – be they local or national – releasing data that has been paid for by the rate/tax-payer. Nat said:

“Isn’t it the case that the USA doesn’t have an authoritative roading database, either? That’s why Navteq, TeleAtlas, and Google have to drive the roads.”

Whilst the US doesn’t have an authoritative roading database either, the release of the TIGER line shapefiles has spurred the development of free and open maps – e.g. the inclusion of Tiger data in OpenStreetMap, and the production of free and open maps for GPS units. This mirrors what has occurred in New Zealand with the likes of the NZ Open GPS Maps project, utilising the free information made available from Land Information NZ.

However, this leaves us with two broad types of maps both with their problems – commercial datasets with restrictive usage conditions and free datasets maintained by volunteers that may not be sustainable in the long term. In New Zealand, the commercial dataset providers are primarily Terralink, Critchlow’s and Eagle Technology, with some more affordable sets made available by Kim Ollivier. The free maps are primarily catered for by the New Zealand OpenStreetMap project and the NZ Open GPS Maps project.

My problem is that there is a lot of inefficiency in the current way that mapping data is managed in New Zealand (and this probably applies internationally). Why do we have four+ commercial sources for roading data and two volunteer driven projects all duplicating each other, as well as Government agencies that have legislative responsibilities for roading infrastructure?

Well, it is because LINZ is not currently funded to provide a centralised repository for all this information – they are too busy focusing on the cadestral database where they make their money. Instead we are producing inefficient silos of information, that are all subtly different. I have been prodding at a few people to try and get the NZ OpenStreetMap and Open GPS Maps projects to try and consolidate the underlying database to OSM, and I believe that this will occur over the long term, but there are a number of issues to work through before this will happen.

As Nat indicated in the original post – in the US Navteg, TeleAtlas and Google drive the roads there, and we’ve got at least Terralink, Google and probably others driving the roads here. In addition we have active volunteers also driving roads and correcting errors in OpenStreetMap and the Open GPS Maps project – I personally provide GPS tracklogs to OSM, and have also placed the 2007/8 High Speed Data Survey in there. The interesting part is that all of the errors are being corrected from the original LINZ roading dataset. So, because the New Zealand Government has not funded LINZ to maintain the roading dataset, make it widely available under permissive licensing terms, and allow feedback and corrections to be suggested for review and possible inclusion, we now have a massively inefficient approach to mapping roads in New Zealand.

All of these projects have sprung up because LINZ is not funded to provide the correct road dataset in the first place.

We can’t support that in a small country in New Zealand where only corporates, local authorities, and central government agencies can afford the commercial roading datasets due to expense. I know at least one of the commercial datasets costs over $100,000 to license. What this means is that small-and-medium sized businesses are being left out in the cold from using geospatial information to improve the way they do business as it is too expensive, and rate/tax-payers do not have affordable access to the information for tourism, recreational and safety purposes.

As the Immediate Past President of the NZ Recreational GPS Society, I’ve seen people balking in our forums at having to pay extra for decent road or topographical maps. Some of these are expensive because the GPS map vendor has needed to license the underlying data from a commercial provider. In addition to the cost, vendors also have to implement measures to stop the reverse-engineering and redistribution of this licensed data. However, like most forms of Digital Rights Management (some may say Restrictions), the technical mechanisms cause their own problems. I’ve just been helping with one person that has been suffering through Garmin’s Map Unlock process that is poorly communicated to customers, and provides nothing but roadblocks in an effort to set up the maps on the user’s computer and GPS. And even when he hopefully does have the maps unlocked, he will only be able to install them on one GPS!

Perhaps as a comparison, I am not able to download and install a copy of the Yellow Pages on my iPhone so that I can use it in a disconnected manner, but I can download the free and open Zenbu iPhone application that bundles all the data – so if for whatever reason I am out of mobile coverage, I can still use this data as it is stored locally on the device. I don’t believe that commercial directory services would be very comfortable about releasing their datasets to be installed on mobile devices, as they would risk the loss of their database in which the perceived value of their business resides. So having data released under permissive liceneses is also essential for new applications such as storing massive geospatial resources in our pockets.

That said, I’m not really in favour any more of the Government attempting to build a single massive dataset any more, as I think Government has proven that it cannot build these IT things effectively because there is too much management by committee, and the commercial vendors that provide the infrastructure are just looking for a jackpot if they win the tender (e.g. tender prices of $9-48 million for the failed National Address Register (NAR) project). I don’t see the need for the Government to build what is effectively their own OpenStreetMap infrastructure when we can just use something like OSM. Honestly, NZ Govt should just approach OpenStreetMap and look at an arrangement where Government can publish geospatial datasets into OSM with the ability to set some layers (such as say electoral and property boundaries which shouldn’t be editable) as read only, and the rest as editable – e.g. roads and walking tracks that can be maintained by everyone. If the publisher of a layer doesn’t want the original layer edited, then in some circumstances editable child layers should be allowed – e.g. so I can add a new walking track to a layer that hasn’t yet been updated to reflect it, and the owner of the original dataset can then look at whether they want to accept the change back into their layer.

Commercial geospatial datasets put nothing but roadblocks in the way for new and creative uses of geospatial data. I have no problem with commercial datasets providing value-add to the data, but the fundamental data such as roads and the like should be made as open and accessible as possible to encourage adoption and standardisation upon that dataset – this will also consolidate feedback and error correction. If I find an error now, I can’t report it to LINZ – they won’t listen. What benefit do I have in reporting a roading error to a commercial provider? Indeed the only benefit I get is if I report the error to a free and open project.

Adoption and standardisation of fundamental datasets are important to ensure consistency between map sets. Right now on my GPS I have two maps sets that both provide roads and you don’t have to look far to find discrepancies between the two datasets – but guess what, they are both derived from the LINZ road centrelines.

If left to commercial providers, geospatial data will be left as an expensive tool that only large organisations can afford.

The sooner governments in general recognise this, start funding the publishing and maintenance of fundamental datasets, the sooner we will see a real renaissance in how spatial information is used by the average organisation and individual. That is why I am so supportive of dc.gov releasing all their data.

I originally wrote this article for The Box, the Tuesday Technology section of The Press in Christchurch, New Zealand – it appeared on the 23rd September 2008. It also appeared on Stuff.co.nz.

Have you ever wanted to quickly find all the photos taken at your family bach? Chances are that unless you’ve been meticulous in filing your photos or tagging them with keywords, this could take quite a bit of time. Wouldn’t it be easier if you could click on your bach on a map, and bring up all of your photos within 1km, or display all of your holiday photos on a map? This is the promise of geotagging.

Simply put, geotagging records the latitude and longitude of the camera at the time the photo was taken and stores it in the image file.

Geotagging is not a new technology. It has been possible to geotag images for several years now, but previously only enthusiasts or professionals geotagged their photos, as it added extra steps to processing photos. It also required a GPS receiver that could record tracks – a breadcrumb trail of where the GPS had been. By matching the time in the GPS track log with the time that photos were taken, it is possible to reasonably estimate where the photo was taken. This took extra time and effort, and except for a dedicated few it was not worth the effort. The GPS receivers also added extra weight and bulk to carry around.

Recently GPS functionality has greatly shrunk in size and power demands, making it more friendly for the photographer by enabling the technology to be directly embedded in cameras and mobile phones.  Already a number of camera phones support geotagging photos – including much of the Nokia N series, and the recently released Apple iPhone 3G. Nikon has embedded a GPS receiver in their new Coolpix P6000 compact, and provide an optional GP-1 GPS attachment for recent Nikon digital SLR cameras. As more devices support geotagging, especially more affordable cameras and mobile phones, the possibilities (and the risks) are going to grow exponentially.

Combining GPS receivers with other devices makes the whole geotagging process transparent and automatic, and requires no effort from the user. This is going to rapidly open up opportunities for all sorts of geotagged data. But is it just technology for technologies sake? Not really, there are existing applications for geotagging, and even more to be come.

Travel photography just begs for geotagging. It is a great means of recording where holiday snapshots were taken, as you can easily show people where you took the photos. Not only that, but as people upload geotagged photos, they become a great travel planning tool as you can see photos that other have taken in a location that you are travelling to, and find sights nearby that you otherwise may have missed.

Real estate also stands to gain from geotagged photos – by being able to quickly load photos of properties for sale into an online searchable map, it will be easier to browse location and appearance at the same time. Councils and infrastructure companies have been using geotagged images for a number of years now to assist with managing assets – imagine being able to take a photo of a pothole and send the image to the council without having to try and explain where it is. Geotagging even has applications after a disaster – teams performing reconnaissance of an affected area can take geotagged photos whilst they are there, and when they return to an operations centre, the images and their exact location can be loaded into a mapping system to help authorities gain a better understanding of the extent of damage.

Location-based technology does come with inherent risks – mostly privacy related. Although many people are comfortable posting photos online, they may not be comfortable allowing people to determine the location where the photo was actually taken. This may be particularly relevant in the case of photos taken at home. No doubt we will see tools evolve to help people manage the privacy associated with geotagged photos, but in the meantime it is worth thinking about the content of a photo before uploading geotagged photos online.

The benefits of geotagging for the most outweigh the risks, and will likely lead to novel applications. The Apple iPhone 3G already has interesting applications taking advantage of geotagged images. Exposure provides mobile to Flickr – a photo sharing website. The ‘Near Me’ function will get your current GPS co-ordinates from the iPhone, and use that to display geotagged photos from Flickr that were taken near your current location. You can then view a photo, plot its location on a map, and if you desire, Google Maps will give you directions on how to get there.

Geotagged images don’t have to be shared online to reap the rewards – it may be that the biggest benefit is just providing another means of managing the vastly expanding data in your own photo library!

Always being one keen to live on the bleeding edge, I’ve been using the beta software for my Garmin Colorado. It has been great for driving around and recording tracklogs for the OpenStreetMap project.

One of the downside of using the beta software though, means that you can be exposed to bugs. I discovered that recently when attempting to open some of my recent GPX tracklogs and the software just refused to open them. After a bit of hunting, I found a relatively easy means of detecting errors and fixing them.

The tool to use is an XML validator called Xerces produced by the Apache Foundation. On a Mac, I download the appropriate binary package, and I copied the binary files in xerces/bin to /usr/local/bin, and the libraries from xerces/lib to /usr/local/lib. You can then run the program SAXCount that counts the number of elements in an XML file – the side benefit that we’re after is that it is good at detecting and reporting errors that many GPX applications are not capable of.

After working through a few minor problems on the NZ GPS forums, I had Xerces up and validating GPX – including with Garmin’s extensions. Note that if you get an error about trying to connect to Garmin’s server to download the schema, e.g. an error like…

Fatal Error at file , line 0, char 0
  Message: An exception
occurred! Type:NetAccessorException, Message:Could not open file:

http://www.garmin.com/xmlschemas/GpxExtensions/v3/GpxExtensionsv3.xsd

I believe this is a combination of Garmin redirecting the original link to a new location, and SAXCount not handling the redirect very well. If you strike this problem, this post in the forums has the fix. I’d basically recommend keeping a version of the fixed Garmin header ready to cut and paste into each GPX so that SAXCount can actually download each xsd. I’ve been using this one…

<?xml version="1.0" encoding="UTF-8" standalone="no"?><gpx
xmlns="http://www.topografix.com/GPX/1/1"
xmlns:gpxx="http://www.garmin.com/xmlschemas/GpxExtensions/v3"
xmlns:gpxtpx="http://www.garmin.com/xmlschemas/TrackPointExtension/v1"
creator="Colorado 300" version="1.1"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.topografix.com/GPX/1/1

http://www.topografix.com/GPX/1/1/gpx.xsd

http://www.garmin.com/xmlschemas/GpxExtensions/v3

http://www8.garmin.com/xmlschemas/GpxExtensions/v3/GpxExtensionsv3.xsd

http://www.garmin.com/xmlschemas/TrackPointExtension/v1

http://www8.garmin.com/xmlschemas/TrackPointExtensionv1.xsd">

From there it is a simple step to validate.

SAXCount -v=always -n -s -f test.gpx

All going well you’ll get something back similar to.

test.gpx: 24 ms (7478 elems, 2498 attrs, 0 spaces, 38613 chars)

This means that everything checked out ok. Otherwise, it will let you know the lines that have errors, making it quick and easy to open in a text editor to edit or delete the corrupt elements. One trick I’ve noted is that by default, a Colorado GPX has no line breaks in it. A trick here is to search for /trkpt><trkpt and replace it with /trkpt>\r<trkpt – this will insert a linebreak, ensuring that each trkpt element starts on a new line and SAXCount can refer to it by line number for easier identification.

The New Zealand Geospatial Office quietly sprung onto the Internet in the last 24 hours. Good to see they provide a couple of RSS feeds that will be aggregated here in due course. The content management system for the website appears to be the free and open source Silverstripe. Even better is a news release promoting our feed aggregation that we recently set up here

Also noted is a significant update to the LINZ website (press release) and NZTopoOnline has also seen a refresh – although it is not immediately obvious what changes have been made to NZTopoOnline.

The Ministry of Health recently released the Altas of Socioeconomic Deprivation (NZDep2006). PDF maps are available from this Atlas’s homepage on the Ministry’s website, but shapefiles aren’t directly available from the website. A free CDROM is available that contains shapefiles – as well as all the reports. I received my CDROM a couple of days ago and it contains the following layers – CensusAreaUnit, DistrictHealthBoard, NZ_Outline,NZDep2006, and TerritorialAuthority.

As someone that once spent a summer internship in Washington DC working for the DC Government, I really enjoyed living there. I was absolutely stoked to see this post on the Google  Lat Long blog outlining why they are releasing over 84,000 3d buildings to Google Earth. I love the key points that Barney Krucoff, the GIS Manager for DC makes.

1. It is the right thing to do.
2. Because every neighborhood can benefit from 3D.
3. We get better 3D performance from the cloud and we don’t pay for it.
4. We want to communicate with our residents.

It kinda makes you sad to live in a country that doesn’t even see the value in being able to produce and maintain a national authoritative roading dataset, and here is local government saying – here is more than 200 geospatial datasets. Go for it. Download in ESRI or Google formats (I assume shapefile and KMZ). Not only that, but they appear to have a nice data warehouse for attribute data, as well as subscribable feeds for updates.

This is what is required for eGovernment – Government making the information available, in formats that support geospatial applications, with permissive licensing terms. It is also key to having residents take data and mash it up, and give it a life of its own. But that won’t happen until all the fundamental data about the places we live are released out into the electronic wilds.

This time it is BusinessWeek promoting the “Do-Good Imperative” – including free and open source software. There is also a sister article on collaborative map-making during emergencies using the likes of OpenStreetMap. Naturally, this is an area that we are working hard on building these geospatial capabilities into Sahana as well.

Open source and collaborative approachs are certainly starting to get the mainstream attention that they deserve. Now we just need funding to support these developments.

Having just been through the process of obtaining and publishing some Government data, I was interested to see Dennis McDonald covering an Australian blog post on eGovernment on ‘Making data freely available‘.

Craig makes some very good points in his conclusions, and I’ve got one that I would like to add based on recent experience with the State Highway trackpoint data I sourced in NZ and uploaded to OpenStreetMap.

In the followup to uploading the trackpoints to OSM, a group of NZ OSM mappers have been discussing how to classify roads in New Zealand – basically how the road hierarchy is classified. Turns out that this is a hard problem and there is not a single standardised approach taken to classifying roads in New Zealand. After asking for guidance, I became aware of a number of issues and capture these on the gis.org.nz wiki.

The simple fact was that there was not a nationally consistent approach to classifying the road hierarchy in New Zealand. I expect it is similar in Australia, and possibly worse with the additional layer of State government.

This highlights an additional key role that I think Government has to play in national data collection.

Government should set the standards for data collection to ensure that datasets are nationally consistent to enable simple aggregation of disparate datasets.

It doesn’t have to actually perform the aggregation, although that would be nice. It just has to ensure that standards are used to enabled aggregation. Using the road hierarchy classification example from above, this means that a road of a certain class in one part of the country, means exactly the same in another. This would probably occur by an engagement-based approach that determines a controlled vocabulary to define types of roads.

Without this responsibility, citizens are doomed to a million-and-one datasets that cannot be easily aggregated to produce coherent and consistent national datasets.