@MaptimeTO asked me to summarize the brief talk I gave last week at Maptime Toronto on making maps from the Technical and Administrative Frequency List (TAFL) radio database. It was mostly taken from posts on this blog, but here goes:
One of the many constraints in building wind farms is allowing for radio links. Both the radio and the wind industries have agreed on a process of buffering and consultation. Here’s how I handled it in Python: Making weird composite shapes with Shapely.
The format is a real delight for all legacy-data nerds: aka a horrible mess of conditional field widths and arcane numeric codes. I wrote a SpatiaLite SQL script to make sense of it all: scruss/taflmunge. This (kind of) explains what it does: TAFL — as a proper geodatabase.
Here’s a raw dump (very little metadata, sorry) from 2013 in the wonderful uMap: Ontario Microwave Links.
In a fabulous piece of #opendatafail, Industry Canada have migrated all the microwave data (so, all links ≥ 960 MHz) to a new system which doesn’t work yet, and also stripped out all of the microwave data from recent TAFL files. They claim to be fixing it, but don’t hold your breath. If you want data to play with, here’s Ontario’s data from October 2013 (nb: huge) — ltaf_ont_tafl-20131001.
Update, 2017: TAFL now seems to be completely dead, and Spectrum Management System has replaced it. None of the records appear to be open data, and the search environment seems — if this is actually possible — slower and less feature-filled than in 2013.
Update, 2013-08-13: Looks like most of the summary pages for these data sets have been pulled from data.gc.ca; they’re 404ing. The data, current at the beginning of this month, can still be found at these URLs:
I build wind farms. You knew that, right? One of the things you have to take into account in planning a wind farm is existing radio infrastructure: cell towers, microwave links, the (now-increasingly-rare) terrestrial television reception.
Wrestle with the Spectrum Direct website, which can’t handle the large search radii needed for comprehensive wind farm design. At best, it spits out weird fixed-width text data, which takes some effort to parse.
Download the Technical and Administrative Frequency Lists (TAFL; see update above for URLs), and try to parse those (layout, fields). Unless you’re really patient, or have mad OpenRefine skillz, this is going to be unrewarding, as the files occasionally drop format bombs like
Yes, you just saw conditional different fixed-width fields in a fixed-width text file. In my best Malcolm Tucker (caution, swearies) voice I exhort you to never do this.
So searching for links is far from obvious, and it’s not like wireless operators do anything conventional like register their links on the title of the properties they cross … so these databases are it, and we must work with them.
That’s why I wrote taflmunge. It currently does one thing properly, and another kinda-sorta:
For all TAFL records fed to it, generates a SpatiaLite database containing these points and all their data; certainly all the fields that the old EXE produced. This process seems to work for all the data I’ve fed to it.
Tries to calculate point-to-point links for microwave communications. This it does less well, but I can see where the SQL is going wrong, and will fix it soon.
taflmunge runs anywhere SpatiaLite does. I’ve tested it on Linux and Windows 7. It’s just a SQL script, so no additional glue language required. The database can be queried on anything that supports SQLite, but for real spatial cleverness, needs SpatiaLite loaded. Full instructions are in the taflmunge / README.md.
TAFL is clearly maintained by licensees, as the data can be a bit “vernacular”. Take, for example, a tower near me:
The tower is near the top of the image, but the database entries are spread out by several hundred meters. It’s the best we’ve got to work with.
Ultimately, I’d like to keep this maintained (the Open Data TAFL files are updated monthly), and host it in a nice WebGIS that would allow querying by location, frequency, call sign, operator, … But that’s for later. For now, I’ll stick with refining it locally, and I hope that someone will find it useful.
Update: thanks to André, this works! See the updated ogr2ogr line.
All I really wanted to do is make a map like this:
This is an Azimuthal Equidistant projection, with me at the centre (of course) and the rest of the world spread out in a fan by distance and bearing. It’s somewhat surprising to find that South Africa is almost directly east of Toronto, and New Zealand to the southwest.
If I had a directional antenna and a rotor, this map would show me where I would have to point the antenna to contact that part of the world. I can’t rotate my dipole (unless I commit some unauthorized local plate tectonics) so I’m stuck with where my antenna transmits and receives best.
The above map was made with AZ_PROJ, a PostScript program of some complexity for plotting world maps for radio use. The instructions for installing and running AZ_PROJ are complex and slightly dated. I got the above output running it through Ghostscript like this:
The format of the az_ini.ps file is complex, and I’m glad I’m an old PS hacker to be able to make head or tail of it.
For all its user-hostility, AZ_PROJ is powerful. Here’s a version of the map I wanted all along:
This shows my furthest QSO in each of the 16 compass directions. (You might note that North is empty: my furthest contact in that direction is some 13km away, whether by lack of folks in that sector or dodginess of my antenna.) Contrast that with my Mercator QSO map, and you’ll see that Azimuthal Equidistant is a much better projection for this application.
To show how radically different the world looks to different people, here’s the world according to my mate Rob in Hamilton, NZ:
I’d been trying to use OGR to transform arbitrary shapefiles into this projection. For maps entirely contained within the same hemisphere (so having extent less than ±90° in any cardinal direction), this works:
The lat_0 and lon_0 parameters are just where you want the centre of the map to be. Things get a bit odd if you try to plot the whole world:
The antipodes get plotted underneath, and everything looks messed up. I may have to take my question to GIS – Stack Exchange to see if I can find an answer. Still, for all its wrongness, you can make something pretty, like my whole world Maidenhead locator grid projected this way turns into a rose:
Automatic Packet Reporting System — APRS — is rather clever. It’s a way of reporting position, status or messages via the amateur radio 2m band. Data is relayed via digipeaters, and routed to/from the internet APRS-IS system to any user worldwide.
It’s a little fiddly to set up, even with a very polished (read: $$) handheld radio like the Kenwood TH-D72A. I’m a bit disappointed that the purported SiRFstar III GPS in this radio takes forever to get a lock, but it’s a nice radio despite this.
The screenshot above shows aprs.fi‘s tracking of my handheld (VA3PID-7) last night as I walked to Toronto Mappy Hour.
You might notice that there’s now a Ham Radio QSO Map lurking on the front page. Thanks to the WordPress OpenStreetMap plugin (which I’ve slightly abusedbefore). Here’s a small piece of Perl which will take your ADIF log and convert it to a WP-OSM marker file.
Note that this program assumes you’ve downloaded your log from QRZ.com, as it requires the locator field for both inbound and outbound stations.
#!/usr/bin/perl -w
# adif2osm - convert ADIF log to OSM map file
# scruss.com / VA3PID - 2011/06/19
use strict;
use constant MARKERDIR =>
'https://glaikit.org/wp-content/plugins/osm/icons/';
use constant QRZURL => 'http://qrz.com/db/';
sub maidenhead2latlong;
my ( $temp, @results ) = '';
### Fast forward past header
while (<>) {
last if m/<eoh>\s+$/i;
}
### While there are records remaining...
while (<>) {
$temp .= $_;
### Process if end of record tag reached
if (m/<eor>\s+$/i) {
my %hash;
$temp =~ s/\n//g;
$temp =~ s/<eoh>.*//i;
$temp =~ s/<eor>.*//i;
my @arr = split( '<', $temp );
foreach (@arr) {
next if (/^$/);
my ( $key, $val ) = split( '>', $_ );
$key =~ s/:.*$//;
$hash{ lc($key) } = $val unless ( $key eq '' );
}
push @results, \%hash;
$temp = '';
}
}
# generate OSM plugin file
my @data = ();
my ( $mygrid, $station_callsign ) = '';
# output header
print
join( "\t", qw/lat lon title description icon iconSize iconOffset/ ),
"\n";
foreach (@results) {
next unless ( exists( $_->{gridsquare} ) && exists( $_->{call} ) );
$mygrid = $_->{my_gridsquare}
if ( exists( $_->{my_gridsquare} ) );
$station_callsign = $_->{station_callsign}
if ( exists( $_->{station_callsign} ) );
push @data, $_->{freq} . ' MHz' if ( exists( $_->{freq} ) );
$data[$#data] .= ' (' . $_->{band} . ')' if ( exists( $_->{band} ) );
push @data, $_->{mode} if ( exists( $_->{mode} ) );
push @data, $_->{qso_date} . ' ' . $_->{time_on} . 'Z'
if ( exists( $_->{qso_date} ) && exists( $_->{time_on} ) );
my ( $lat, $long ) = maidenhead2latlong( $_->{gridsquare} );
print join( "\t",
$lat,
$long,
'<a href="' . QRZURL . $_->{call} . '">' . $_->{call} . '</a>',
join( ' - ', @data ),
MARKERDIR . 'wpttemp-green.png',
'0,-24' ),
"\n";
@data = ();
}
# show home station last, so it's on top
my ( $lat, $long ) = maidenhead2latlong($mygrid);
print join( "\t",
$lat,
$long,
'<a href="'
. QRZURL
. $station_callsign . '">'
. $station_callsign . '</a>',
'Home Station',
MARKERDIR . 'wpttemp-red.png',
'0,-24' ),
"\n";
exit;
sub maidenhead2latlong {
# convert a Maidenhead Grid location (eg FN03ir)
# to decimal degrees
# this code could be cleaner/shorter/clearer
my @locator =
split( //, uc(shift) ); # convert arg to upper case array
my $lat = 0;
my $long = 0;
my $latdiv = 0;
my $longdiv = 0;
my @divisors = ( 72000, 36000, 7200, 3600, 300, 150 )
; # long,lat field size in seconds
my $max = ( $#locator > $#divisors ) ? $#divisors : $#locator;
for ( my $i = 0 ; $i <= $max ; $i++ ) {
if ( int( $i / 2 ) % 2 ) { # numeric
if ( $i % 2 ) { # lat
$latdiv = $divisors[$i]; # save for later
$lat += $locator[$i] * $latdiv;
}
else { # long
$longdiv = $divisors[$i];
$long += $locator[$i] * $longdiv;
}
}
else { # alpha
my $val = ord( $locator[$i] ) - ord('A');
if ( $i % 2 ) { # lat
$latdiv = $divisors[$i]; # save for later
$lat += $val * $latdiv;
}
else { # long
$longdiv = $divisors[$i];
$long += $val * $longdiv;
}
}
}
$lat += ( $latdiv / 2 ); # location of centre of square
$long += ( $longdiv / 2 );
return ( ( $lat / 3600 ) - 90, ( $long / 3600 ) - 180 );
}
You’ll need to update MARKERDIR to reflect your own WP-OSM installation. Mine might move, so if you don’t change it, and you don’t get markers, please don’t blame me.
The basic code to include a map is like this:
You’ll need to change the marker_file URL, too.
Note that, while this script generates links into the QRZ callsign database, it doesn’t hit that site unless you click a link.
After yesterday’s post, I went a bit nuts with working out the whole amateur radio grid locator thing (not that I’m currently likely to use it, though). I’d hoped to provide a shapefile of the entire world, but that would be too big for the format’s 2GB file size limit.
What I can give you, though, is:
A Perl program that will generate a shapefile of an entire Maidenhead grid field, down to the subsquare level: make_grid.pl. You’ll need Geo::Shapelib to make this work. 324 (= 182) of these files would cover the whole world, and at 8MB or so a pop, things get unwieldy quickly.
If anyone would like their grid square in Google Earth format, let me know, or read on …
Making KML Files
Several people have asked, so here’s how you convert to KML. You’ll need the OGR toolkit installed, which comes in several open-source geo software bundles: FWTools/osgeo4w/QGis. Let’s assume we want to make the grid square ‘EN’.
Industry Canada publishes the locations of all licensed radio spectrum users on Spectrum Direct. You can find all the transmitters/receivers near you by using its Geographical Area Search. And there are a lot near me:
While Spectrum Direct’s a great service, it has three major usability strikes against it:
You can’t search by address or postal code; you need to know your latitude and longitude. Not just that, it expects your coordinates as a integer of the format DDMMSS.
It’s very easy to overwhelm the system. Where I live, I can pretty much search for only 5km around me before the system times out.
The output formats aren’t very useful. You can either get massively verbose XML, or very long line undelimited text, and neither of these are very easy to work with.
Never fear, Perl is here! I wrote a tiny script that glues together Dave O’Neill‘s Parse::SpectrumDirect::RadioFrequency module (which I wonder if you can guess what it does?) to Robbie Bow‘s Text::CSV::Slurp module. The latter is used to blort out the former’s results to a CSV file that you can load into any GIS/mapping system.
Here’s the code:
#!/usr/bin/perl -w
# spectest.pl - generate CSV from Industry Canada Spectrum Direct data
# created by scruss on 02010/10/29 - for https://glaikit.org/
# usage: spectest.pl geographical_area.txt > outfile.csv
use strict;
use Parse::SpectrumDirect::RadioFrequency;
use Text::CSV::Slurp;
use constant MINLAT => 40.0; # all of Canada is >40 deg N, for checking
my $prefetched_output = '';
# get the whole file as a string
while (<>) {
$prefetched_output .= $_;
}
my $parser = Parse::SpectrumDirect::RadioFrequency->new();
# magically parse Spectrum Direct file
$parser->parse($prefetched_output) or die "$!\n";
my $legend_hash = $parser->get_legend(); # get column descriptions
my @keys = ();
foreach (@$legend_hash) {
# retrieve column keys in order so the output will resemble input
push @keys, $_->{key};
}
# get the data in a ref to an array of hashes
my $stations = $parser->get_stations();
my @good_stations = ();
# clean out bad values
foreach (@$stations) {
next if ( $_->{Latitude} < MINLAT );
push @good_stations, $_;
}
# create csv file in memory then print it
my $csv = Text::CSV::Slurp->create(
input => \@good_stations,
field_order => \@keys
);
print $csv;
exit;
The results aren’t perfect; QGis boaked on a file it made where one of the records appeared to have line breaks in it. It could filter out multiple pieces of equipment at the same call sign location. But it works, mostly, which is good enough for me.