Tag Archives: osm

osm vs maps.me

(Hey! I’m not a lawyer. Read a few paragraphs of this, and it’ll be clear I shouldn’t even try to give any kind of opinion. The screenshots I include with maps.me, booking.com and Google data are presented merely for review and critique of the situation.)

An interesting discussion showed up on the OSM Legal mailing list a couple of weeks back: MAPS.ME combining OSM data and non-OSM data? OSM has some restrictions on how its data can be used. At a grossly simple level, if you mix your proprietary data in with OSM data and present a single layer map, you must either:

  1. provide the proprietary data you mixed in under the same licence as OSM, or;
  2. stop doing that.

The OSMF Licence/Community Guidelines/Horizontal Map Layers – Guideline has some example scenarios that I will quote here:

Examples of where you DO NOT need to share your non-OpenStreetMap data

  1. You use OpenStreetMap as a base topographical map and make your best reasonable efforts to exclude ALL restaurants. You then add a layer of your own restaurant data.

Examples of where you DO need to share your non-OpenStreetMap data

  1. You add restaurants in one area from non-OpenStreetMap data based on comparison with OpenStreetMap data in other layers.

So for this article, please bear in mind that where the above example says restaurants, I’m looking for an analogous case with hotels.

MAPS.ME (which they want you to pronounce as “maps with me”, even though it says “maps dot me”; I guess “maps  me” wouldn’t make such a nice URI) is an offline map app that shows a mix of POIs over a custom-styled OSM basemap. Its publisher mixes OSM POIs with proprietary ones, and publishes a list (local copy: skipped_nodes-20160731.zip) of OSM nodes that it has skipped from its POI list. To satisfy my curiosity, I wanted to see if the required separation of data was in place with maps.me.

First, I used overpass turbo to return all the hotel and motel nodes in and around Toronto (the query I used was “type:node and (tourism=hotel or tourism=motel) in bbox”, saved as TorontoHotelNodes.zip). This list contains 50 hotels and motels.

Next, I used maps.me’s skipped nodes list to find the OSM hotels and motels that maps.me had omitted from their app. This list should closely match the one from booking.com, where the app company said they got their hotel data. For completeness, I queried the OSM hotels extract to see what hotels maps.me had possibly not skipped.

Finally, I overlaid those OSM nodes on a georeferenced screendump from booking.com’s Google map. Here’s what it looks like (click to enlarge):

to_hotels-OSM_vs_maps_me

So OSM nodes not skipped (which is not necessarily the same as used) by maps.me are marked  , while those excluded by maps.me are marked . At first glance, there’s clearly not a 100% correlation between booking.com hotels, but a few clues emerge that hint strongly that maps.me are mixing in some OSM nodes with some proprietary nodes. It’s therefore likely that maps.me has not abided by the OSMF community guideline.

Scrolling around and zooming in on the booking.com hotel map shows that they don’t show you all of the listed properties. There’s always some filter based on location, price or availability. This makes sense if you’re managing a valuable proprietary data set, as there will always be someone trying to scrape your data and take your business with slightly cheaper referrals. One example of a hotel available on booking.com is The Old Mill (booking.com link: Old Mill Toronto) even though it’s not shown with a booking.com pin on my screenshot above. The Old Mill is also shown as not skipped on maps.me, so let’s take a look at it in the app:

2016-08-01_12.09.00Ah; there’s the booking.com property, with the OSM node off to the east under a slightly different name, and the OSM restaurant node somewhere in between. (When you search maps.me for restaurants, in Toronto at least, there are no referral links: missed opportunity there, folks!)

Since OSM is community-generated data, sometimes our spelling isn’t so great. One hotel node that leaps out from the maps.me node set is the (now fixed) “Sharaton”:

2016-07-31_23.35.19aka Sheraton Toronto Airport Hotel to booking.com, or Sheraton Toronto Airport Hotel & Conference Centre to its friends:

2016-07-31_23.36.11I wonder if maps.me will exclude the OSM node for this Sheraton from now on? Now that its name has been edited, if they do skip it, it means that they’re comparing OSM against a proprietary list, so would be breaking the guidelines. On the other hand, if they don’t exclude it, they really need to get a better fuzzy match algorithm, and post haste.

We can’t know the algorithm that maps.me uses to choose hotels to include referral links and skip the OSM data. The details of that are their trade secret, and no open licence can compel them to disclose it. But the cynic in me has found a possible clue in the budget motels that line Kingston Road on the eastern Toronto lake shore. Both the Days Inn – Toronto East and the Park Motel are listed on booking.com. Only the slightly more expensive Days Inn gets a referral link on maps.me. Could it be that the value of the cheaper motel’s referral isn’t worth including? (Update: this is wrong; I found another reason, which I will write about later.)

In addition, I wonder if maps.me’s skipped nodes list could be considered a “derived work” of booking.com? As the choice of nodes to skip is informed by booking.com’s data, it does give us a small insight into their database.

GTALUG OSM Talk last week

Title OpenStreetMap: Building a great map while everyone tells you you’re doing it wrong.

frontpageNotes/Links:

  1. ‘The Map’ — http://www.openstreetmap.org . It supports routing now, too.
  2. QGIS, an open GIS manager. It’s rather good — http://qgis.org/
  3. The OSM Wiki; ridiculously complete documentation: https://wiki.openstreetmap.org/wiki/
  4. OSM Help Stack Exchange-style question/answer: https://help.openstreetmap.org/
  5. All of the OSM stats! — https://wiki.openstreetmap.org/wiki/Stats
  6. Toronto map growth animation — http://www.geofabrik.de/gallery/history/index.html#toronto
  7. Crowdsourced geocoding (+ lawsuit from Canada Post) — http://geocoder.ca/
  8. Open Data Commons Open Database License (ODbL) — http://opendatacommons.org/licenses/odbl/
  9. Canada’s new Open Government portal — http://open.canada.ca related: Toronto Open Data — http://toronto.ca/open
  10. CIPPIC Open Licensing Project (CLIP) — http://clipol.org/
  11. Humanitarian OpenStreetMap Team [HOT] — http://hotosm.org/
  12. OpenCycleMap — http://www.openstreetmap.org/#map=13/43.6666/-79.3785&layers=C
  13. The rather wonderful /uMap/https://umap.openstreetmap.fr/en/

Original GTALUG note: OpenStreetMap links from the other night.

The Invisible City

Now you see it …

There’s not a whole lot north of North Bay. Highway 11 winds through some extensive geometry, but few habitations. Until something wonderful (and quite a bit wrong) happened at 46° 31′ 21″ N, 79° 33′ 9″ W on OpenStreetMap. A whole new town about 25 minutes out of North Bay sprung up overnight — and was as quickly deleted as mappers caught and reverted the vandalism.

Now you don’t.

The misguided mapper put quite a lot of work into this ephemeral town. It’s got urban rail, parklands, residential areas and more. You can see the detail on this large image of the town (650 KB PNG). If you want to play with the data, here’s a zipped OSM XML file of the area before it was reverted. But please, don’t re-upload it; OpenStreetMap should be for real features on the ground, and not for confusing map data consumers.

Thanks to Bootprint for noticing this, and to rw__ for reverting the edits.

Awesome USGS orthoimagery … in Canada?

Update, 2015: I don’t think this coverage works any more. The WMS data was carefully trimmed to the US border last time I looked.

Detail of Erie Shores Wind Farm showing USGS orthophotos from WMS

I was trying to do some OpenStreetMap edits in rural southern Ontario, but the default Bing background aerial imagery is very poor. Clicking around in semi-plokta mode to find better images, I happened upon “OSM US USGS Large Scale Aerial Imagery”, which loaded up some beautiful and recent pictures. I guess that since we’re close to the border, the USGS doesn’t bother to trim its images too close.

Ian Dees, who admins the particular OSM server, confirmed that these are from The National Map‘s TNM_Large_Scale_Imagery service; thanks, Ian! These images are at least as good for my purposes as the SWOOP data — except not $50/km2, and without elaborate usage restrictions.

Updating maps … to outdated data

I caved, and bought a Garmin nüvi 1490. Not that my GPSMap 60CSx wasn’t great at routing, but remembering to update maps before travel and its fiddly mounting requirements were a pain.

So, two hours of downloading map updates, and I fire it up … to find that the three year old hotel in Dartmouth, NS we were staying in was in the wrong place on the map:

Compare with OpenStreetMap:

So now, back near home, I ask it to find a post office. Here’s me parked outside one; do you see it on the screenshots?

Again, compare with OpenStreetMap:

So I thought I’d help Garmin out, but their Report a Map Error page needs me to know the type of my GPS, its serial number, the type and revision of my map, my name and e-mail address, and the coordinates of the error. OSM has me spoiled: at best, I can go in and edit; at second best, I can drop markers on OpenStreetBugs to flag errors for others to fix.

Garmin already has my name, e-mail address, GPS type, serial number and map revision through myGarmin™. The company could just as easily have a web-based map correction system that would be point-and-click. Follow the Leader is one of Garmin’s mottoes. In terms of user correctability of maps, however, they’re only the leader because they don’t know they’ve been lapped.

Toronto Data really open

Looks like the data sets at toronto.ca/open might finally actually be open; that is, usable in a way that doesn’t bind subsequent users to impossible terms. The new licence (which unfortunately is behind a squirrelly link) basically just requires you to put a reference to this paragraph somewhere near your data/application/whatever:

Contains public sector Datasets made available under the City of Toronto’s Open Data Licence v2.0.

and a link to the licence, where possible.

Gone are the revocation clauses, which really prevented any open use before, because they would require you to track down all the subsequent users of the data and get them to stop. Good. I think we can now use the data in OpenStreetMap.

While commenting on the licence’s squirrelly URL — I mean, could you remember http://www1.toronto.ca/wps/portal/open_data/open_data_fact_sheet_details?vgnextoid=59986aa8cc819210VgnVCM10000067d60f89RCRD? — I stumbled upon the comedy gold that is the City of Toronto Comments Wall log. There goes my planned reading for the day.

Ham Radio log to interactive OpenStreetMap

You might notice that there’s now a Ham Radio QSO Map lurking on the front page. Thanks to the WordPress OpenStreetMap plugin (which I’ve slightly abused before). Here’s a small piece of Perl which will take your ADIF log and convert it to a WP-OSM marker file.

Note that this program assumes you’ve downloaded your log from QRZ.com, as it requires the locator field for both inbound and outbound stations.

#!/usr/bin/perl -w
# adif2osm - convert ADIF log to OSM map file
# scruss.com / VA3PID - 2011/06/19

use strict;
use constant MARKERDIR =>
  'http://glaikit.org/wp-content/plugins/osm/icons/';
use constant QRZURL => 'http://qrz.com/db/';
sub maidenhead2latlong;

my ( $temp, @results ) = '';

### Fast forward past header
while (<>) {
  last if m/<eoh>\s+$/i;
}

### While there are records remaining...
while (<>) {
  $temp .= $_;

  ### Process if end of record tag reached
  if (m/<eor>\s+$/i) {
    my %hash;
    $temp =~ s/\n//g;
    $temp =~ s/<eoh>.*//i;
    $temp =~ s/<eor>.*//i;
    my @arr = split( '<', $temp );
    foreach (@arr) {
      next if (/^$/);
      my ( $key, $val ) = split( '>', $_ );
      $key =~ s/:.*$//;
      $hash{ lc($key) } = $val unless ( $key eq '' );
    }
    push @results, \%hash;
    $temp = '';
  }
}

# generate OSM plugin file
my @data = ();
my ( $mygrid, $station_callsign ) = '';

# output header
print
  join( "\t", qw/lat lon title description icon iconSize iconOffset/ ),
  "\n";
foreach (@results) {
  next unless ( exists( $_->{gridsquare} ) && exists( $_->{call} ) );
  $mygrid = $_->{my_gridsquare}
    if ( exists( $_->{my_gridsquare} ) );
  $station_callsign = $_->{station_callsign}
    if ( exists( $_->{station_callsign} ) );

  push @data, $_->{freq} . ' MHz' if ( exists( $_->{freq} ) );
  $data[$#data] .= ' (' . $_->{band} . ')' if ( exists( $_->{band} ) );
  push @data, $_->{mode} if ( exists( $_->{mode} ) );
  push @data, $_->{qso_date} . ' ' . $_->{time_on} . 'Z'
    if ( exists( $_->{qso_date} ) && exists( $_->{time_on} ) );
  my ( $lat, $long ) = maidenhead2latlong( $_->{gridsquare} );
  print join( "\t",
    $lat,
    $long,
    '<a href="' . QRZURL . $_->{call} . '">' . $_->{call} . '</a>',
    join( ' - ', @data ),
    MARKERDIR . 'wpttemp-green.png',
    '0,-24' ),
    "\n";

  @data = ();
}

# show home station last, so it's on top
my ( $lat, $long ) = maidenhead2latlong($mygrid);
print join( "\t",
  $lat,
  $long,
  '<a href="'
    . QRZURL
    . $station_callsign . '">'
    . $station_callsign . '</a>',
  'Home Station',
  MARKERDIR . 'wpttemp-red.png',
  '0,-24' ),
  "\n";

exit;

sub maidenhead2latlong {

  # convert a Maidenhead Grid location (eg FN03ir)
  #  to decimal degrees
  # this code could be cleaner/shorter/clearer
  my @locator =
    split( //, uc(shift) );    # convert arg to upper case array
  my $lat      = 0;
  my $long     = 0;
  my $latdiv   = 0;
  my $longdiv  = 0;
  my @divisors = ( 72000, 36000, 7200, 3600, 300, 150 )
    ;                          # long,lat field size in seconds
  my $max = ( $#locator > $#divisors ) ? $#divisors : $#locator;

  for ( my $i = 0 ; $i <= $max ; $i++ ) {
    if ( int( $i / 2 ) % 2 ) {    # numeric
      if ( $i % 2 ) {             # lat
        $latdiv = $divisors[$i];    # save for later
        $lat += $locator[$i] * $latdiv;
      }
      else {                        # long
        $longdiv = $divisors[$i];
        $long += $locator[$i] * $longdiv;
      }
    }
    else {                          # alpha
      my $val = ord( $locator[$i] ) - ord('A');
      if ( $i % 2 ) {               # lat
        $latdiv = $divisors[$i];    # save for later
        $lat += $val * $latdiv;
      }
      else {                        # long
        $longdiv = $divisors[$i];
        $long += $val * $longdiv;
      }
    }
  }
  $lat  += ( $latdiv / 2 );         # location of centre of square
  $long += ( $longdiv / 2 );
  return ( ( $lat / 3600 ) - 90, ( $long / 3600 ) - 180 );
}

You’ll need to update MARKERDIR to reflect your own WP-OSM installation. Mine might move, so if you don’t change it, and you don’t get markers, please don’t blame me.

The basic code to include a map is like this:

You’ll need to change the marker_file URL, too.

Note that, while this script generates links into the QRZ callsign database, it doesn’t hit that site unless you click a link.