Gav's Blog

Back in July, I posted about dc.gov releasing some data. I was a bit slow replying to a comment made by Nat Torkington then, and felt that a reply actually required a new post to elaborate further on why I’m so supportive of governments – be they local or national – releasing data that has been paid for by the rate/tax-payer. Nat said:

“Isn’t it the case that the USA doesn’t have an authoritative roading database, either? That’s why Navteq, TeleAtlas, and Google have to drive the roads.”

Whilst the US doesn’t have an authoritative roading database either, the release of the TIGER line shapefiles has spurred the development of free and open maps – e.g. the inclusion of Tiger data in OpenStreetMap, and the production of free and open maps for GPS units. This mirrors what has occurred in New Zealand with the likes of the NZ Open GPS Maps project, utilising the free information made available from Land Information NZ.

However, this leaves us with two broad types of maps both with their problems – commercial datasets with restrictive usage conditions and free datasets maintained by volunteers that may not be sustainable in the long term. In New Zealand, the commercial dataset providers are primarily Terralink, Critchlow’s and Eagle Technology, with some more affordable sets made available by Kim Ollivier. The free maps are primarily catered for by the New Zealand OpenStreetMap project and the NZ Open GPS Maps project.

My problem is that there is a lot of inefficiency in the current way that mapping data is managed in New Zealand (and this probably applies internationally). Why do we have four+ commercial sources for roading data and two volunteer driven projects all duplicating each other, as well as Government agencies that have legislative responsibilities for roading infrastructure?

Well, it is because LINZ is not currently funded to provide a centralised repository for all this information – they are too busy focusing on the cadestral database where they make their money. Instead we are producing inefficient silos of information, that are all subtly different. I have been prodding at a few people to try and get the NZ OpenStreetMap and Open GPS Maps projects to try and consolidate the underlying database to OSM, and I believe that this will occur over the long term, but there are a number of issues to work through before this will happen.

As Nat indicated in the original post – in the US Navteg, TeleAtlas and Google drive the roads there, and we’ve got at least Terralink, Google and probably others driving the roads here. In addition we have active volunteers also driving roads and correcting errors in OpenStreetMap and the Open GPS Maps project – I personally provide GPS tracklogs to OSM, and have also placed the 2007/8 High Speed Data Survey in there. The interesting part is that all of the errors are being corrected from the original LINZ roading dataset. So, because the New Zealand Government has not funded LINZ to maintain the roading dataset, make it widely available under permissive licensing terms, and allow feedback and corrections to be suggested for review and possible inclusion, we now have a massively inefficient approach to mapping roads in New Zealand.

All of these projects have sprung up because LINZ is not funded to provide the correct road dataset in the first place.

We can’t support that in a small country in New Zealand where only corporates, local authorities, and central government agencies can afford the commercial roading datasets due to expense. I know at least one of the commercial datasets costs over $100,000 to license. What this means is that small-and-medium sized businesses are being left out in the cold from using geospatial information to improve the way they do business as it is too expensive, and rate/tax-payers do not have affordable access to the information for tourism, recreational and safety purposes.

As the Immediate Past President of the NZ Recreational GPS Society, I’ve seen people balking in our forums at having to pay extra for decent road or topographical maps. Some of these are expensive because the GPS map vendor has needed to license the underlying data from a commercial provider. In addition to the cost, vendors also have to implement measures to stop the reverse-engineering and redistribution of this licensed data. However, like most forms of Digital Rights Management (some may say Restrictions), the technical mechanisms cause their own problems. I’ve just been helping with one person that has been suffering through Garmin’s Map Unlock process that is poorly communicated to customers, and provides nothing but roadblocks in an effort to set up the maps on the user’s computer and GPS. And even when he hopefully does have the maps unlocked, he will only be able to install them on one GPS!

Perhaps as a comparison, I am not able to download and install a copy of the Yellow Pages on my iPhone so that I can use it in a disconnected manner, but I can download the free and open Zenbu iPhone application that bundles all the data – so if for whatever reason I am out of mobile coverage, I can still use this data as it is stored locally on the device. I don’t believe that commercial directory services would be very comfortable about releasing their datasets to be installed on mobile devices, as they would risk the loss of their database in which the perceived value of their business resides. So having data released under permissive liceneses is also essential for new applications such as storing massive geospatial resources in our pockets.

That said, I’m not really in favour any more of the Government attempting to build a single massive dataset any more, as I think Government has proven that it cannot build these IT things effectively because there is too much management by committee, and the commercial vendors that provide the infrastructure are just looking for a jackpot if they win the tender (e.g. tender prices of $9-48 million for the failed National Address Register (NAR) project). I don’t see the need for the Government to build what is effectively their own OpenStreetMap infrastructure when we can just use something like OSM. Honestly, NZ Govt should just approach OpenStreetMap and look at an arrangement where Government can publish geospatial datasets into OSM with the ability to set some layers (such as say electoral and property boundaries which shouldn’t be editable) as read only, and the rest as editable – e.g. roads and walking tracks that can be maintained by everyone. If the publisher of a layer doesn’t want the original layer edited, then in some circumstances editable child layers should be allowed – e.g. so I can add a new walking track to a layer that hasn’t yet been updated to reflect it, and the owner of the original dataset can then look at whether they want to accept the change back into their layer.

Commercial geospatial datasets put nothing but roadblocks in the way for new and creative uses of geospatial data. I have no problem with commercial datasets providing value-add to the data, but the fundamental data such as roads and the like should be made as open and accessible as possible to encourage adoption and standardisation upon that dataset – this will also consolidate feedback and error correction. If I find an error now, I can’t report it to LINZ – they won’t listen. What benefit do I have in reporting a roading error to a commercial provider? Indeed the only benefit I get is if I report the error to a free and open project.

Adoption and standardisation of fundamental datasets are important to ensure consistency between map sets. Right now on my GPS I have two maps sets that both provide roads and you don’t have to look far to find discrepancies between the two datasets – but guess what, they are both derived from the LINZ road centrelines.

If left to commercial providers, geospatial data will be left as an expensive tool that only large organisations can afford.

The sooner governments in general recognise this, start funding the publishing and maintenance of fundamental datasets, the sooner we will see a real renaissance in how spatial information is used by the average organisation and individual. That is why I am so supportive of dc.gov releasing all their data.

Over at the In Development blog at eGovt, they are asking for comments on the state of progress for eGovernment in New Zealand. I’ve copied my comment here.

Location, Location, Location!
A quick skim and keyword search indicates that the report has almost entirely neglected one of the most important aspects of information, and that it location – its spatial component. It is not surprising then that the Government is struggling with how to handle spatial information. Some of the most important aspects of community anywhere are distance and connectivity. However with Government doing a poor job in making spatial information available to the citizen, it is no wonder that individuals are struggling to find out about consents from two properties over.

Even more of a shame is the fact that Government has failed to deliver on a project to produce an authoritative National Address Register of addresses, roads and placenames. This dataset is one of the most fundamental to being able to place information in space under truly understand its context. eGovernment will never take off under the core location issues are worked out. Whilst the recent release of significant amounts of Statistics New Zealand information is a great step, the true benefits and insights contained in the released census data is being held back by the lack of an authoritative National Address Register.

Organisational Inertia and Champion Individuals
Government organisations themselves are often hesitant to change or try something new. This is not to say it doesn’t happen, and there have been some excellent examples recently of organisations stepping up and trying something new – such as using wikis for consultation and engagement. I am fairly certain that most of these projects will have been championed internally by a small group of individuals, and I’ll bet they had quite a struggle to see their project through. Here’s hoping that these recent projects are the catalyst for more risk taking, and acceptance of exciting IT projects within Government.

Not enough Champions
As a consultant that has worked  a lot within Government agencies, much of the information projects I’ve seen successfully delivered have been almost solely on the back of one or two key individuals. The agencies themselves have often been more of a hindrance to delivery of exciting new initiatives, and the Champions have had to fight an uphill battle against their own organisations, and other Government agencies when ‘engaging stakeholders’. Even worse, a number of these Champions actually run up so many brick walls in their organisation, and within Government, that they end up choosing to leave in frustration, or even worse are forced out by unenlightened superiors. In some circumstances they make their way to an organisation that is supportive, but that fate does not appear to wait all Champions. You need some way to support and encourage these key individuals that lie within Government agencies – there may only be one or two, but they are your key instrument of change from within.

Start Small, Release Early, Upgrade Often
Despite the fact that the National Address Register is such a necessary component of eGovernment. I don’t believe that the project as tendered should have gone ahead – not for something where tender prices ranged between $9 and $48 million. The project would have been so large and complex that it would likely have not been delivered on time, would have been over budget, and would have lacked the desired capabilities.

I think Government has to learn to try the small and simple things first and work up.

Surely, for the price of one or two Geospatial Professionals and supporting hardware, the New Zealand Government should be capable of aggregating all roading information from the 74 Road Control Authorities and Transit New Zealand, and publishing it as a single national roading dataset under permissive licensing. This should be able to be done now for a couple of hundred thousand dollars. But just produce a dataset and get it out there. Get feedback about how it is used, and look at improving the process, and accepting feedback. If you don’t, then the New Zealand Government is going to look pretty ridiculous when volunteers have created their own national roading dataset using OpenStreetMap because Government wasn’t capable.

Here are some simple actions that Government can undertake to encourage more exciting use of Government information.

Find data. Release it under permissive licensing. Release it in formats that make it readily accessible to manipulation in software (e.g. don’t release maps only as pdfs, make the underlying spatial data available). Announce it through a simple clearing house – nothing flash, it only needs to be  WordPress blog pointing to the relevant source. It won’t be until this happens that the more exciting concepts such as entirely unexpected but useful mashups occur.

And that is when things will get really interesting. At that point, we will have citizens building mashups and services about ‘Our Place In Space’ – and they won’t be constrained by the organisational inertia inherent in most Government agencies that are tied back by accountability and liability that makes them hesitant to take bold steps.

Remember eGovernment is not just about Government developing systems and solutions. It does include citizens, communities and organisations building systems to meet their own needs.

I have just received my original OIA request to the Police back that outlines the range of tender prices for the National Address Register. I’ve transposed the text from the scanned pdf below.  I’ve got other work to be doing so I’ll comment on this later. The original pdf is available here.

• NAR OIA Request to NZ Police (pdf, ~550KB)

(New Zealand Police Header)

10 June 2008

Gavin Treadgold

Dear Mr Treadgold

Thank you for your Official Information Act request of 12 May 2008 about the National Address Register (NAR) project.

You have asked for a range of tendered prices and would like to know, so as to get a better understanding of the economics, what the lowest and highest price tenders were for the project. E.g. the range of prices that were tendered. You state that you believe that this is not commercially sensitive information as no tender has been awarded, and no vendors are being associated with the tendered price range.

I can advise that the prices received ranged between $9M and $48M for conforming tender responses.

You have asked for the number of tenders and I can advise that there were four responses to the RFP. One tender was evaluated as being non compliant to a mandatory requirement.

Yours sincerely

Jill Barclay
Technology Manager: GIS
Information, Communications and Technology Service Centre

I just recieved this document today under some of my ongoing Official Information Act (OIA) requests following the recent failure of the National Address Register (NAR) tender process. This is the first of a few I have requested – there will hopefully be more to come.

It was interesting watching the request being passed from agency to agency (although it was understandable). I submitted the request to Police as they were leading this stage of the NAR. The request was passed then to the State Services Commission, and finally on to LINZ, who released the document quite quickly.

Of interest in this document will be comparing government’s estimate of the cost of implementing the NAR over a five year period – versus the submitted tenders. I, of course, am trying to find out the price range of the tenders for comparison.

It is still really disappointing that this project has failed, it had so much potential to deliver a fantastic  authoritative dataset to New Zealanders.

• National Address Register – Moving Forward August 2007 (pdf, ~200KB)

I wanted to share some extremely disappointing news that I received today. The National Address Register project has been terminated.

This project had the intention of providing a single national authoritative dataset for roads, addresses and placename information. The potential of this project was to deliver a free dataset that all organisations and individuals in New Zealand were free to use. This would have made a fantastic resource, and had the potential to consolidate a number of mapping projects, and could have greatly simplified the work associated with project such as the NZ Open GPS Maps project, as the NAR would have provided a single national focal point for feedback and correction of road and address information.

The cynic in me says that the reason this project failed was because of the commercial interests in existing roading datasets. Currently there are multiple roading datasets from different providers, and they are making very good money from these. Some roading datasets sell for six figure sums on an annual basis. Naturally, very few organisations can afford these prices, so only large Government agencies tend to be able to purchase them. Suffice to say, these datasets are different, and there is not a single authoritative dataset amongst them.

The NAR had the potential to create a single, free and authoritative road, address and placename dataset. Tenders were invited for the project, and there was going to be only one organisation to win the tender. As a result, all but one of the current commercial providers stood to lose their revenue streams from their roading datasets. As you will see in the notification below, the tenders were too expensive. I believe that it was in all the commercial vendors interests to put in high tender prices to ensure that the NAR did not go ahead, and that they could protect their existing revenue streams rather than risk missing the tender and losing it all.

The upshot of this is that my faith in the Government to provide geospatial information to its citizens is now close to zero. If they are not capable of producing a single authoritative roading dataset (arguably one of the most important sets of spatial information as it defines most of our physical connectivity) then there is little hope of them being able to deliver any useful spatial information to citizens.

As the NZ Open GPS Maps, Zenbu and NZ Open Street Map projects have shown us, a volunteer community can develop products faster and cheaper than commercial or government organisations, and over time they will have better quality as well.

I believe the time has come for us to build more volunteer communities to provide spatial information that our Government is failing to provide to us. No longer can we wait upon them, rather we must build it ourselves. There are four key areas that we need to focus on.

1. Raw data collection – taking our GPS units out into the real world and collecting and sharing data. Collecting track logs and uploading these to the OpenStreetMap or NZ Open GPS Maps projects. Providing waypoints to OSM and Zenbu. Please – if you haven’t already, consider donating some time and information to these projects so that they have raw data to work with. This ‘field survey’ work is essential to creating our own spatial information resources. (I would particularly encourage geocachers to contribute their tracklogs if at all possible as we tend to travel a little more than others)

2. Mapping – converting the data collected in the field to information. Creating vectors for road lines, adding street name, directions, speeds. Using your local knowledge to map the community around you.

3. End products – converting the spatial information into a form suitable for others to use, for example the NZ Open GPS Maps project producing Garmin map files that can be loaded into GPS units.

4. Distribution – due to the large quantities of information involved, we may need to look at creating an ad-hoc network of individuals and websites to share the vast quantities of information about our country via torrents or similar P2P mechanism.

I believe the time has come for all those that want better access to spatial information to go out there and be a part of collecting, and building it. We can’t wait for Government to build it for us, so we will have to do it ourselves.

Let’s get started.

SUBJECT: NAR PROJECT TERMINATEDThe National Address Register (NAR) project is a cross-government initiative set up to develop infrastructure to improve the provision of address, road and place name information for government agencies, businesses and the wider community.

The project is over-seen by a Steering Committee comprised of representatives of key stakeholders from central government, local government and emergency services agencies.

An integral part of developing a business model and business case for the National Address Register (NAR) was to assess whether there was a supplier able to provide the relevant services and to identify the likely costs. An RFP process was chosen as the most effective way of identifying both of these.

Following assessment of the tender proposals, the Steering Committee has decided to terminate the project. Despite the project showing considerable potential to reduce duplication across government and reduce costs, it is too expensive to proceed with in its current form.

Further investigation into the need for, and the most cost-effective way of providing address, road and place information, will be led by the New Zealand Geospatial Office, within their mandate under the NZ Geospatial Strategy. This work will include determining the optimal role for the Crown, local government and the private sector. Brendon Whiteman, Director New Zealand Geospatial Office - brendon.whiteman@linz.govt.nz; will be happy to answer any queries that you may have in relation to these activities in the context of the overall work programme of the Geospatial Office.

It is expected that agencies will continue with existing arrangements they have for the purchase of this location data, from the commercial sector.

Nancy McBeth of the State Services Commission, is preparing a Lessons Learnt report on the NAR project. If you have some views that you would like considered in that report, please contact her at nancy.mcbeth@ssc.govt.nz by 31 May 2008.

On his return from Annual Leave next week, Laurence Millar, Chair of the NAR Steering Committee will formally write to your Chief Executive to advise of the decision.

Regards
Jill Barclay
Operational Owner NAR project
NZ Police

Received from a public email list I subscribe to.