Looks like hell has frozen over — there’s now an Ontario open buildings layer that’s compatible with the OSM licence: The Open Database of Buildings. Thanks go to StatCan for aggregating all the data sets into one huge database.
It looks like the Bitcoin Map website adds points illicitly to OSM through a Google Maps interface. This is rather bad.
Update: they’re fixing this …
Here’s a test run I took to see if the data was really being added to OSM:
Here’s the POI data, in raw OSM XML:
<?xml version="1.0" encoding="UTF-8"?>
<osm version="0.6" generator="CGImap 0.3.3 (28262 thorn-02.openstreetmap.org)" copyright="OpenStreetMap and contributors" attribution="http://www.openstreetmap.org/copyright" license="http://opendatacommons.org/licenses/odbl/1-0/">
<node id="3383877893" visible="true" version="1" changeset="29257856" timestamp="2015-03-05T02:56:47Z" user="BitcoinMaps" uid="2135320" lat="43.7298277" lon="-79.2721787">
<tag k="addr:city" v="Bemidji"/>
<tag k="addr:housenumber" v="134"/>
<tag k="addr:street" v="Woodfern Drive"/>
<tag k="contact:email" v="name@example.com"/>
<tag k="contact:phone" v="+1 416 555 1234"/>
<tag k="contact:website" v="http://example.com/"/>
<tag k="description" v="i have totally made this up to see if it will be added to OSM even though the location was derived from Google Maps"/>
<tag k="name" v="Entirely FictitiousName"/>
<tag k="payment:litecoin" v="yes"/>
<tag k="shop" v="books"/>
</node>
</osm>
It was Doors Open Toronto last weekend, and the city published the locations as open data: Doors Open Toronto 2013. I thought I’d try to geocode it after Richard suggested we take a look. OpenStreetMap has the Nominatim geocoder, which you can use freely as long as you accept restrictions on bulk queries.
As a good and lazy programmer, I first tried to find pre-built modules. Mistake #1; they weren’t up to snuff:
Perl’s Geo::Coder::Many::OSM would only read from OSM‘s server. MapQuest run their own mirror as part of their great MapQuest Open Platform Web Services suite, and they have almost no limitation on query volume. OSM runs their operation on a shoestring, and too many queries gets you the disapproval face, or worse.
Python’s geopy gave spurious results amid copious whiny error messages.
(Standard operational procedure for python, then… ☺)
Much as I love XML, it’s a bit hard to read as a human, so I smashed the Doors Open data down to simple pipe-separated text: dot.txt. Here’s my code, ever so slightly specialized for searching in Toronto:
#!/usr/bin/perl -w
# geonom.pl - geocode pipe-separated addresses with nominatim
# created by scruss on 02013/05/28
use strict;
use URI::Escape;
use LWP::Simple;
use JSON::XS;
# the URL for OpenMapQuest's Nominatim service
use constant BASEURI =&gt; 'http://open.mapquestapi.com/nominatim/v1/search.php';
# read pipe-separated values from stdin
# two fields: Site Name, Street Address
while (&lt;&gt;) {
chomp;
my ( $name, $address ) = split( '\|', $_, 2 );
my %query_hash = (
format =&gt; 'json',
street =&gt; cleanaddress($address), # decruft address a bit
# You'll want to change these ...
city =&gt; 'Toronto', # fixme
state =&gt; 'ON', # fixme
country =&gt; 'Canada', # fixme
addressdetails =&gt; 0, # just basic results
limit =&gt; 1, # only want first result
# it's considered polite to put your e-mail address in to the query
# just so the server admins can get in touch with you
email =&gt; 'me@mydomain.com', # fixme
# limit the results to a box (quite a bit) bigger than Toronto
bounded =&gt; 1,
viewbox =&gt; '-81.0,45.0,-77.0,41.0' # left,top,right,bottom - fixme
);
# get the result from Nominatim, and decode it to a hashref
my $json = get( join( '?', BASEURI, escape_hash(%query_hash) ) );
my $result = decode_json($json);
if ( scalar(@$result) &gt; 0 ) { # if there is a result
print join(
'|', # print result as pipe separated values
$name, $address,
$result-&gt;[0]-&gt;{lat},
$result-&gt;[0]-&gt;{lon},
$result-&gt;[0]-&gt;{display_name}
),
&quot;\n&quot;;
}
else { # no result; just echo input
print join( '|', $name, $address ), &quot;\n&quot;;
}
}
exit;
sub escape_hash {
# turn a hash into escaped string key1=val1&amp;key2=val2...
my %hash = @_;
my @pairs;
for my $key ( keys %hash ) {
push @pairs, join( &quot;=&quot;, map { uri_escape($_) } $key, $hash{$key} );
}
return join( &quot;&amp;&quot;, @pairs );
}
sub cleanaddress {
# try to clean up street addresses a bit
# doesn't understand proper 'Unit-Number' Canadian addresses tho.
my $_ = shift;
s/Unit.*//; # shouldn't affect result
s/Floor.*//; # won't affect result
s/\s+/ /g; # remove extraneous whitespace
s/ $//;
s/^ //;
return $_;
}
It quickly became apparent that the addresses had been entered by hand, and weren’t going to geocode neatly. Here are some examples of the bad ones:
200 University Ave St W — it’s an avenue, not a street, and it runs north-south
2087 Davenport Road (Rear House) Rd Unit: Rear — too many rears
21 Colonel Sameul Smith Park Dr — we can’t fix typos
0 Construction Trailer:Lower Simcoe at Lakeshore Blvd — 0? Zero??? What are you, some kinda python programmer?
Curiously, some (like the address for Black Creek Pioneer Village) were right, but just not found. Since the source was open data, I put the right address into OpenStreetMap, so for next year, typos aside, we should be able to find more events.
Now, how accurate were the results? Well, you decide:
Pierre Béland reminded the Canadian OSM group of the GeoBase WMS, which has all the NRN road names in it. Here’s my neighbourhood’s OpenStreetMap data with the road names overlaid.
Looking at the OSM Inspector for Toronto, there are still a lot of nodes that need to be remapped (or OpenStreetMap users that need to be advised) when the big licence change comes.
(and yes, I blogged this just so I’d remember the link for OSM Inspector.)
You might notice that there’s now a Ham Radio QSO Map lurking on the front page. Thanks to the WordPress OpenStreetMap plugin (which I’ve slightly abusedbefore). Here’s a small piece of Perl which will take your ADIF log and convert it to a WP-OSM marker file.
Note that this program assumes you’ve downloaded your log from QRZ.com, as it requires the locator field for both inbound and outbound stations.
#!/usr/bin/perl -w
# adif2osm - convert ADIF log to OSM map file
# scruss.com / VA3PID - 2011/06/19
use strict;
use constant MARKERDIR =>
'https://glaikit.org/wp-content/plugins/osm/icons/';
use constant QRZURL => 'http://qrz.com/db/';
sub maidenhead2latlong;
my ( $temp, @results ) = '';
### Fast forward past header
while (<>) {
last if m/<eoh>\s+$/i;
}
### While there are records remaining...
while (<>) {
$temp .= $_;
### Process if end of record tag reached
if (m/<eor>\s+$/i) {
my %hash;
$temp =~ s/\n//g;
$temp =~ s/<eoh>.*//i;
$temp =~ s/<eor>.*//i;
my @arr = split( '<', $temp );
foreach (@arr) {
next if (/^$/);
my ( $key, $val ) = split( '>', $_ );
$key =~ s/:.*$//;
$hash{ lc($key) } = $val unless ( $key eq '' );
}
push @results, \%hash;
$temp = '';
}
}
# generate OSM plugin file
my @data = ();
my ( $mygrid, $station_callsign ) = '';
# output header
print
join( "\t", qw/lat lon title description icon iconSize iconOffset/ ),
"\n";
foreach (@results) {
next unless ( exists( $_->{gridsquare} ) && exists( $_->{call} ) );
$mygrid = $_->{my_gridsquare}
if ( exists( $_->{my_gridsquare} ) );
$station_callsign = $_->{station_callsign}
if ( exists( $_->{station_callsign} ) );
push @data, $_->{freq} . ' MHz' if ( exists( $_->{freq} ) );
$data[$#data] .= ' (' . $_->{band} . ')' if ( exists( $_->{band} ) );
push @data, $_->{mode} if ( exists( $_->{mode} ) );
push @data, $_->{qso_date} . ' ' . $_->{time_on} . 'Z'
if ( exists( $_->{qso_date} ) && exists( $_->{time_on} ) );
my ( $lat, $long ) = maidenhead2latlong( $_->{gridsquare} );
print join( "\t",
$lat,
$long,
'<a href="' . QRZURL . $_->{call} . '">' . $_->{call} . '</a>',
join( ' - ', @data ),
MARKERDIR . 'wpttemp-green.png',
'0,-24' ),
"\n";
@data = ();
}
# show home station last, so it's on top
my ( $lat, $long ) = maidenhead2latlong($mygrid);
print join( "\t",
$lat,
$long,
'<a href="'
. QRZURL
. $station_callsign . '">'
. $station_callsign . '</a>',
'Home Station',
MARKERDIR . 'wpttemp-red.png',
'0,-24' ),
"\n";
exit;
sub maidenhead2latlong {
# convert a Maidenhead Grid location (eg FN03ir)
# to decimal degrees
# this code could be cleaner/shorter/clearer
my @locator =
split( //, uc(shift) ); # convert arg to upper case array
my $lat = 0;
my $long = 0;
my $latdiv = 0;
my $longdiv = 0;
my @divisors = ( 72000, 36000, 7200, 3600, 300, 150 )
; # long,lat field size in seconds
my $max = ( $#locator > $#divisors ) ? $#divisors : $#locator;
for ( my $i = 0 ; $i <= $max ; $i++ ) {
if ( int( $i / 2 ) % 2 ) { # numeric
if ( $i % 2 ) { # lat
$latdiv = $divisors[$i]; # save for later
$lat += $locator[$i] * $latdiv;
}
else { # long
$longdiv = $divisors[$i];
$long += $locator[$i] * $longdiv;
}
}
else { # alpha
my $val = ord( $locator[$i] ) - ord('A');
if ( $i % 2 ) { # lat
$latdiv = $divisors[$i]; # save for later
$lat += $val * $latdiv;
}
else { # long
$longdiv = $divisors[$i];
$long += $val * $longdiv;
}
}
}
$lat += ( $latdiv / 2 ); # location of centre of square
$long += ( $longdiv / 2 );
return ( ( $lat / 3600 ) - 90, ( $long / 3600 ) - 180 );
}
You’ll need to update MARKERDIR to reflect your own WP-OSM installation. Mine might move, so if you don’t change it, and you don’t get markers, please don’t blame me.
The basic code to include a map is like this:
You’ll need to change the marker_file URL, too.
Note that, while this script generates links into the QRZ callsign database, it doesn’t hit that site unless you click a link.