Sometimes I have to put text on a path

Friday, September 2, 2011

List of popular file formats and filter encoding for Map mashups and Geographic information systems; converters of waypoints, tracks, routes



Some popular file formats:

Latitude and longitude are expressed in decimal degrees using the WGS 84 datum. Elevation is recorded in meters. Dates and times are not local time, but instead are Coordinated Universal Time (UTC) using ISO 8601 format.
Find your references in any number of formats:
http://spatialreference.org/
http://spatialreference.org/ref/ it's a listing of spatial references (EPSG) and an interface to search for spatial references. EPSG codes are numeric codes associated with coordinate system definitions. For instance, EPSG:4326 is geographic WGS84.
Projections (a mathematical transformation of the surface of a sphere (3D) onto a 2D plane) are ordered roughly chronologically by type: http://www.radicalcartography.net/?projectionref
Spherical Mercator (http://en.wikipedia.org/wiki/Mercator_projection) is used to describe the PROJECTION used by many commercial API providers.
Proj4 is a library for projecting map data (used by MapServer, GDAL and a multitude of other Open Source GIS libraries): http://trac.osgeo.org/proj/

for all standards: http://www.opengeospatial.org/standards


APR – ESRI ArcView 3.3 and earlier project file
MXD – ESRI ArcGIS project file, 8.0 and higher

http://resources.arcgis.com/
http://www.arcgisblog.com/content/2010/01/arcgis-10-package-mxd-and-all-its-data-map-package

http://support.esri.com/en/downloads/samples-and-utilities/view/productid/78/metaid/1682

SHP – ESRI shapefile
http://en.wikipedia.org/wiki/Shapefile

Shapefiles spatially describe geometries: points, polylines, and polygons and associated attribute information. A shapefile is a digital vector storage format. It was introduced with ArcView GIS version 2 in the beginning of the 1990s.

DEM – USGS DEM file format:
http://en.wikipedia.org/wiki/USGS_DEM
Sources for USGS DEMs:
http://www.geobase.ca/geobase/en/data/cded/index.html Review (in english and french; download and view) a detailed description of the Canadian Digital Elevation Data. Product specifications, metadata and other supporting documentation are also available.
http://dds.cr.usgs.gov/pub/data/DEM/250/
obtain DEM’s via the National Eelevation Dataset(NED) server at: http://ned.usgs.gov

E00 – ARC/INFO interchange file format
http://freegeographytools.com/category/e00

GeoTIFF – Geographically located raster data
http://en.wikipedia.org/wiki/GeoTIFF
GeoTIFF is a public domain metadata standard which allows georeferencing information to be embedded within a TIFF file (6.0). http://trac.osgeo.org/geotiff/
World TIFF – Geographically located raster data: text file giving corner coordinate, raster cells per unit, and rotation


TAB – MapInfo Table file format
http://en.wikipedia.org/wiki/MapInfo_TAB_format
It's  a geospatial vector data format.
http://mitab.maptools.org/

DTED – Digital Terrain Elevation Data
http://en.wikipedia.org/wiki/DTED
It's a standard of digital datasets which consists of a matrix of terrain elevation values. This standard was originally developed in the 1970s to support aircraft radar simulation.
http://www.fas.org/irp/program/core/dted.htm
-----------------
-geoJSON
The GeoJSON format specification was finalized in June 2008.
http://en.wikipedia.org/wiki/GeoJSON
http://geojson.org/geojson-spec.html
GeoJSON is a geospatial data interchange format based on JavaScript Object Notation (JSON).
 In fact, every GeoJSON data structure is also a JSON object, and thus JSON tools can also be used for processing GeoJSON data.
http://www.geowebguru.com/articles/97-technical-overview-geojson
JavaScript Object Notation (JSON), and the terms object, name, value, array, and number, are defined in IETF RTC 4627, at http://www.ietf.org/rfc/rfc4627.txt
http://en.wikipedia.org/wiki/JSON
JSON is simply a mixture of JavaScript object and array literals without variable definitions.
JSON, is a text-based open standard designed for human-readable data interchange. It is derived from the JavaScript scripting language for representing simple data structures and associative arrays, called objects. Despite its relationship to JavaScript, it is language-independent, with parsers available for most languages.The JSON format is used primarily to transmit data between a server and web application (serving as an alternative to XML;  it is generally more compact than XML).

Spatial data format types supported in GeoJSON include points, polygons, multipolygons, features, geometry collections, and bounding boxes, which are stored along with feature information and attributes. The geometries and their properties will have a parent object.
GeoJSON also allows specifying a geographic coordinate system, using the OGC crs (coordinate reference system) property (preferred) or with an EPSG code. If a crs is not defined, GeoJSON will use the WGS84 geoid by default.
GeoJSON is supported by numerous mapping and GIS software packages. Google maps, google maps api...use JSON


-geoRSS
http://en.wikipedia.org/wiki/GeoRSS
http://www.georss.org/Main_Page

As RSS and Atom become more prevalent as a way to publish and share information, it becomes increasingly important that location is described in an interoperable manner so that applications can request, aggregate, share and map geographically tagged feeds.

There are currently two encodings of GeoRSS, Simple and GML (GeoRSS Geography Markup Language (GML)):
GeoRSS-Simple is meant as a very lightweight format that developers and users can quickly add to their existing feeds. It supports basic geometries (point, line, box, polygon) and covers the typical use cases when encoding locations.
GeoRSS GML is a formal GML Application Profile, and supports a greater range of features, notably coordinate reference systems other than WGS-84 latitude/longitude.
Both formats are designed for use with Atom 1.0, RSS 2.0 and RSS 1.0.


Blogger and geoRSS: http://bloggerindraft.blogspot.com/2008/12/new-feature-geotagging.html


-XML: SVG - Scalable Vector Graphics is an XML format. I think that this file format must be in this GIS list because may map servers use this format and all modern browsers use this format. 
http://www.mccurley.org/svg/
http://en.wikipedia.org/wiki/File:USA_Oregon_location_map.svg
http://commons.wikimedia.org/wiki/Category:SVG_maps
better SVG graphics rendering engine: http://antigrain.com/svg/index.html
(MapServer 5.0+ can use this engine; it supports sub-pixel anti-aliasing, as well as many more features).
look also the "svg" category of this blog ;)



-XML-based interchange format: GPX or GPS eXchange Format
GPX – XML-based interchange format
http://en.wikipedia.org/wiki/GPS_eXchange_Format
It describes waypoints, tracks, and routes. 

A collection of points (with no sequential relationship)  is deemed a collection of individual waypoints.
An ordered collection of points may be expressed as a track or a route.  Tracks are a record of where a person has been (with timestamp (because someone is recording where and when they were there)) and routes are suggestions about where you might go in the future...
View the official site: http://www.topografix.com/gpx.asp

-XML-based interchange format: KML
KML – Keyhole Markup Language, XML-based
http://en.wikipedia.org/wiki/Keyhole_Markup_Language
The KML file specifies a set of features (place marks, altitude, images, 3D models, polygons, textual descriptions, camera view...).  Each place always has a longitude and a latitude.
KML files are distributed in KMZ files, which are zipped files with a .kmz extension.
http://schemas.opengis.net/kml/2.2.0/
referencehttp://code.google.com/apis/kml/documentation/kmlreference.html
Developer's Guide: http://code.google.com/apis/kml/documentation/topicsinkml.html
http://kml-samples.googlecode.com/svn/trunk/interactive/index.html


Google Maps supports the following KML elements (that is, KML queries in the Search box and GGeoXml objects in the Maps API):

  • Placemarks
  • Icons
  • Folders
  • Descriptive HTML
  • Entity replacement via and
  • KMZ (compressed KML, including attached images)
  • Polylines and polygons
  • Styles for polylines and polygons, including color, fill, and opacity
  • Network links to import data dynamically
  • Ground overlays and screen overlays

This list does not apply to Google Earth's export to Google Maps, My Maps import, or My Maps export to Google Earth.
differences between maps and earth for KML: http://code.google.com/apis/kml/documentation/kmlelementsinmaps.html


Maximum uncompressed KML file size=10MB.
Maximum number of Network Links=10.

convert GIS data to KML files
You can import GIS Data and file (ESRI Shape, MapInfo .tab data; GeoTIFF, or other raster dataset) into Google Earth Pro so that it can be used as part of your Google Earth visualizations: http://earth.google.com/outreach/tutorial_importgis.html. If you have Google Earth Pro, you can import many kinds of GIS data directly using the methods described in this tutorial. If you are using the free version of Google Earth, there are other tools for converting GIS data to KML.


ESRI ArcGIS and MapInfo, have tools to export GIS data into KML format (for use in Google Earth).

ogr2gui (http://sourceforge.net/projects/ogr2gui/): a free utility that converts many formats, including shapefiles to KML.
shp2kml (http://www.zonums.com/shp2kml.html): a free utility that converts shapefiles to KML.
MapWindow GIS has Shape2Earth (http://shape2earth.com/default.aspx), a tool for converting GIS data to KML.
ESRI ArcGIS contains tools in the ArcToolbox (Conversion Tools > To KML) for converting vector and raster GIS data to KML.
Arc2Earth (http://www.arc2earth.com/) is an extension for ArcGIS that converts GIS data to KML. The Arc2Earth Community Edition (http://www.arc2earth.com/communityedition/) is a free tool that has lower limits than the full version.
MapInfo Professional has a MapInfo Professional Google Earth Link Utility: http://mapinfo-to-google-earth-tm-export-utilit.software.informer.com/


KML and Spreadsheet Mapper
How to enter data in an on-line spreadsheet to generate a set of placemarks in Google Earth and Maps. Google Docs' web-based, collaborative editing allows your team members to simultaneously enter data and instantly publish updates:
http://earth.google.com/outreach/tutorial_spreadsheet.html

KML and Fusion Table
Because Fusion Tables also supports visualizing your data on maps, it is an alternative to Spreadsheet Mapper: http://earth.google.com/outreach/tutorial_fusion_sample.html
http://earth.google.com/outreach/tutorial_fusion_yourowndata.html#comparemappers
Biking and Hiking trails from MTBGuru.com:
http://blog.mtbguru.com/2010/02/24/mtbguru-tracks-as-seen-through-google-fusion-tables/

a scientific paper: Hector Gonzalez, Alon Halevy, Christian S. Jensen, Anno Langen, Jayant Madhavan, Rebecca Shapley, Warren Shen (2010). "Google Fusion Tables: Data Management, Integration and Collaboration in the Cloud". SoCC'10. ACM:
http://www.cse.ohio-state.edu/~agrawal/788-au10/Papers/Oct28/google-fusion-socc10.pdf

Fusion Tables supports a number of data formats:
Comma-separated files (.csv) - Up to 100 MB
Microsoft Excel files (.xls, .xlsx) - Up to 1 MB
OpenDocument Spreadsheet (.ods) - Up to 1 MB
Keyhole Markup Language (.kml) - Up to 100 MB
Data already in a Google Spreadsheet
If you have datasets in Excel or Open Office Spreadsheets that are larger than 1MB, save them as CSV files to take advantage of the larger 100MB limit!

Manage your Data with Open Data Kit Aggregate:
http://earth.google.com/outreach/tutorial_odk_aggregate.html
Open Data Kit (ODK) is a suite of tools that allows data collection using mobile devices and data submission to an online server, even without an Internet connection or mobile carrier service at the time of data collection. Once you've collected data in the field with ODK Collect, you can upload and manage your data using ODK Aggregate. ODK Aggregate is the intermediary server storage platform that accepts the data and can send it on to external applications, if desired. ODK Aggregate also allows you to download your datasets in aggregated formats, such as one .csv file. ODK Aggregate allows you to use Google's AppEngine hosting platform (see many post of this blogfor managing your collected data online.

------------other links:
http://en.wikipedia.org/wiki/Geography_Markup_Language (GML)
http://en.wikipedia.org/wiki/GIS_file_formats

------------freeware and set of library (converters; filters)
"data conversion" database:
http://freegis.org/database/?cat=11
"geodata" database:
http://freegis.org/database/?cat=1


MapTools Packaged Tools and Utilities:
http://maptools.org/


a list of bookmarks:
http://www.spatialhydrology.com/freedownload.html


------GDAL
GDAL (Geospatial Data Abstraction Library) is a multi-format raster reading and writing library (it is used as the primary mechanism for reading raster data in MapServer).
It provides a powerful set of libraries for working with vector data:
http://trac.osgeo.org/gdal/
http://code.google.com/apis/kml/articles/vector.html
http://trac.osgeo.org/gdal/wiki/DownloadingGdalBinaries


------GPSBabel
GPSBabel converts waypoints, tracks, and routes between popular GPS receivers and mapping programs. It also has powerful manipulation tools for such data.
It contains extensive data manipulation abilities making it a convenient for server-side processing or as the backend for other tools. It does not convert, transfer, send, or manipulate maps. It processes data that may placed on a map, such as waypoints, tracks, and routes...
GPSBabel runs on Microsoft Windows 95, 98, ME, 2000, XP, and Vista plus POSIX OSes such as Linux, UnixWare, OpenServer, Solaris, FreeBSD, and OSX.

File FormatWaypointsTracksRoutes

ReadWriteReadWriteReadWrite
Alan Map500 tracklogs (.trl)
alantrl
yesyes
Alan Map500 waypoints and routes (.wpr)
alanwpr
yesyesyesyes
Brauniger IQ Series Barograph Download
baroiq
yes
Bushnell GPS Trail file
bushnell_trl
yesyes
Bushnell GPS Waypoint file
bushnell
yesyes
Cambridge/Winpilot glider software
cambridge
yesyes
CarteSurTable data file
cst
yesyesyes
Cetus for Palm/OS
cetus
yesyesyes
CoastalExplorer XML
coastexp
yesyesyesyes
Columbus/Visiontac V900 files (.csv)
v900
yesyes
Comma separated values
csv
yesyes
CompeGPS data files (.wpt/.trk/.rte)
compegps
yesyesyesyesyesyes
CoPilot Flight Planner for Palm/OS
copilot
yesyes
cotoGPS for Palm/OS
coto
yesyesyes
Data Logger iBlue747 csv
iblue747
yesyes
Dell Axim Navigation System (.gpb) file format
axim_gpb
yes
DeLorme .an1 (drawing) file
an1
yesyesyesyesyes
DeLorme GPL
gpl
yesyes
DeLorme PN-20/PN-30/PN-40 USB protocol
delbin
yesyesyesyesyesyes
DeLorme Street Atlas Plus
saplus
yesyes
DeLorme Street Atlas Route
saroute
yes
DeLorme XMap HH Native .WPT
xmap
yesyes
DeLorme XMap/SAHH 2006 Native .TXT
xmap2006
yesyes
DeLorme XMat HH Street Atlas USA .WPT (PPC)
xmapwpt
yesyes
Destinator Itineraries (.dat)
destinator_itn
yesyes
Destinator Points of Interest (.dat)
destinator_poi
yesyes
Destinator TrackLogs (.dat)
destinator_trl
yesyes
EasyGPS binary format
easygps
yesyes
Embedded Exif-GPS data (.jpg)
exif
yesyes
Enigma binary waypoint file (.ert)
enigma
yesyesyesyes
FAI/IGC Flight Recorder Data Format
igc
yesyesyesyes
Franson GPSGate Simulation
gpssim
yesyesyes
Fugawi
fugawi
yesyes
G7ToWin data files (.g7t)
g7towin
yesyesyes
Garmin 301 Custom position and heartrate
garmin301
yesyes
Garmin Logbook XML
glogbook
yesyes
Garmin MapSource - gdb
gdb
yesyesyesyesyesyes
Garmin MapSource - mps
mapsource
yesyesyesyesyesyes
Garmin MapSource - txt (tab delimited)
garmin_txt
yesyesyesyesyesyes
Garmin PCX5
pcx
yesyesyesyesyesyes
Garmin POI database
garmin_poi
yesyes
Garmin Points of Interest (.gpi)
garmin_gpi
yesyes
Garmin serial/USB protocol
garmin
yesyesyesyesyesyes
Garmin Training Center (.tcx)
gtrnctr
yesyesyes
Geocaching.com .loc
geo
yesyes
GeocachingDB for Palm/OS
gcdb
yesyes
Geogrid-Viewer ascii overlay file (.ovl)
ggv_ovl
yesyesyesyesyesyes
Geogrid-Viewer tracklogs (.log)
ggv_log
yesyes
GEOnet Names Server (GNS)
geonet
yesyes
GeoNiche .pdb
geoniche
yesyes
GlobalSat DG-100/BT-335 Download
dg-100
yes
Google Earth (Keyhole) Markup Language
kml
yesyesyesyesyesyes
Google Maps XML
google
yes
Google Navigator Tracklines (.trl)
gnav_trl
yesyes
GoPal GPS track log (.trk)
gopal
yesyes
GpilotS
gpilots
yesyes
GPS TrackMaker
gtm
yesyesyesyesyesyes
GPSBabel arc filter file
arc
yesyes
GpsDrive Format
gpsdrive
yesyes
GpsDrive Format for Tracks
gpsdrivetrack
yesyes
GPSman
gpsman
yesyes
GPSPilot Tracker for Palm/OS
gpspilot
yesyes
gpsutil
gpsutil
yesyes
GPX XML
gpx
yesyesyesyesyesyes
HikeTech
hiketech
yesyesyesyes
Holux (gm-100) .wpo Format
holux
yesyes
Holux M-241 (MTK based) Binary File Format
m241-bin
yes
Holux M-241 (MTK based) download
m241
yes
Honda/Acura Navigation System VP Log File Format
vpl
yes
HSA Endeavour Navigator export File
hsandv
yesyes
HTML Output
html
yes
Humminbird tracks (.ht)
humminbird_ht
yesyesyesyes
Humminbird waypoints and routes (.hwr)
humminbird
yesyesyesyesyes
IGN Rando track files
ignrando
yesyes
iGO2008 points of interest (.upoi)
igo2008_poi
yesyes
IGO8 .trk
igo8
yesyes
Jelbert GeoTagger data file
jtr
yesyes
Jogmap.de XML format
jogmap
yes
Kartex 5 Track File
ktf2
yesyes
Kartex 5 Waypoint File
kwf2
yesyes
Kompass (DAV) Track (.tk)
kompass_tk
yesyes
Kompass (DAV) Waypoints (.wp)
kompass_wp
yesyes
KuDaTa PsiTrex text
psitrex
yesyesyesyesyesyes
Lowrance USR
lowranceusr
yesyesyesyesyesyes
Magellan Explorist Geocaching
maggeo
yesyes
Magellan Mapsend
mapsend
yesyesyesyesyesyes
Magellan NAV Companion for Palm/OS
magnav
yesyes
Magellan SD files (as for eXplorist)
magellanx
yesyesyesyesyesyes
Magellan SD files (as for Meridian)
magellan
yesyesyesyesyesyes
Magellan serial protocol
magellan
yesyesyesyesyesyes
MagicMaps IK3D project file (.ikt)
ik3d
yesyes
Map&Guide 'TourExchangeFormat' XML
tef
yes
Map&Guide to Palm/OS exported files (.pdb)
mag_pdb
yesyes
MapAsia track file (.tr7)
mapasia_tr7
yesyes
Mapopolis.com Mapconverter CSV
mapconverter
yesyes
MapTech Exchange Format
mxf
yesyes
Memory-Map Navigator overlay files (.mmo)
mmo
yesyesyesyesyesyes
Microsoft AutoRoute 2002 (pin/route reader)
msroute
yes
Microsoft Streets and Trips (pin/route reader)
msroute
yes
Microsoft Streets and Trips 2002-2007
s_and_t
yesyes
Motorrad Routenplaner (Map&Guide) .bcr files
bcr
yesyes
MS PocketStreets 2002 Pushpin
psp
yesyes
MTK Logger (iBlue 747,...) Binary File Format
mtk-bin
yes
MTK Logger (iBlue 747,Qstarz BT-1000,...) download
mtk
yes
National Geographic Topo .tpg (waypoints)
tpg
yesyes
National Geographic Topo 2.x .tpo
tpo2
yes
National Geographic Topo 3.x/4.x .tpo
tpo3
yesyesyes
Navicache.com XML
navicache
yes
Navigon Mobile Navigator .rte files
nmn4
yesyes
Navigon Waypoints
navigonwpt
yesyes
NaviGPS GT-11/BGT-11 Download
navilink
yesyesyesyesyesyes
NaviGPS GT-31/BGT-31 datalogger (.sbp)
sbp
yes
NaviGPS GT-31/BGT-31 SiRF binary logfile (.sbn)
sbn
yes
Naviguide binary route file (.twl)
naviguide
yesyes
Navitel binary track (.bin)
navitel_trk
yesyes
Navitrak DNA marker format
dna
yesyes
NetStumbler Summary File (text)
netstumbler
yes
NIMA/GNIS Geographic Names File
nima
yesyes
NMEA 0183 sentences
nmea
yesyesyesyes
Nokia Landmark Exchange
lmx
yesyes
OpenStreetMap data files
osm
yesyesyesyesyes
OziExplorer
ozi
yesyesyesyesyesyes
PalmDoc Output
palmdoc
yes
PathAway Database for Palm/OS
pathaway
yesyesyesyesyesyes
PocketFMS breadcrumbs
pocketfms_bc
yesyes
PocketFMS flightplan (.xml)
pocketfms_fp
yesyes
PocketFMS waypoints (.txt)
pocketfms_wp
yesyes
Quovadis
quovadis
yesyes
Raymarine Waypoint File (.rwf)
raymarine
yesyesyesyes
Ricoh GPS Log File
ricoh
yesyes
See You flight analysis data
cup
yesyes
Skymap / KMD150 ascii files
skyforce
yesyesyesyesyesyes
SkyTraq Venus based loggers (download)
skytraq
yesyes
SkyTraq Venus based loggers Binary File Format
skytraq-bin
yesyes
Sportsim track files (part of zipped .ssz files)
sportsim
yesyes
Suunto Trek Manager (STM) .sdf files
stmsdf
yesyesyesyes
Suunto Trek Manager (STM) WaypointPlus files
stmwpp
yesyesyesyesyesyes
Swiss Map 25/50/100 (.xol)
xol
yesyesyesyes
Tab delimited fields useful for OpenOffice, Ploticus etc.
openoffice
yesyes
Teletype [ Get Jonathon Johnson to describe
teletype
yesyes
Textual Output
text
yes
TomTom Itineraries (.itn)
tomtom_itn
yesyes
TomTom Places Itineraries (.itn)
tomtom_itn_places
yesyes
TomTom POI file (.asc)
tomtom_asc
yesyes
TomTom POI file (.ov2)
tomtom
yesyes
TopoMapPro Places File
tmpro
yesyes
TrackLogs digital mapping (.trl)
dmtlog
yesyesyesyes
U.S. Census Bureau Tiger Mapping Service
tiger
yesyes
Universal csv with field structure in first line
unicsv
yesyesyesyesyesyes
Vcard Output (for iPod)
vcard
yes
VidaOne GPS for Pocket PC (.gpb)
vidaone
yesyes
Vito Navigator II tracks
vitosmt
yesyesyesyesyesyes
Vito SmartMap tracks (.vtt)
vitovtt
yes
WiFiFoFum 2.0 for PocketPC XML
wfff
yes
Wintec TES file
wintec_tes
yesyes
Wintec WBT-100/200 Binary File Format
wbt-bin
yes
Wintec WBT-100/200 GPS Download
wbt
yesyes
Wintec WBT-201/G-Rays 2 Binary File Format
wbt-tk1
yes
XAiOX iTrackU Logger
itracku
yesyes
XAiOX iTrackU Logger Binary File Format
itracku-bin
yesyesyesyes
Yahoo Geocode API data
yahoo
yes



Filter Encoding is an OGC standard which defines an XML encoding for filter expressions to allow for spatial and attribute querying: http://www.opengeospatial.org/standards/filter

some filters:


Filter FunctionFilter NameWaypointsTracksRoutes
Points along an arcarcyes

Remove unreliable points with high hdop or vdopdiscard
yes
Remove duplicatesduplicateyes

Manipulate altitudesheightyes

Interpolate between trackpointsinterpolateyes

Remove waypoints, tracks, or routes from processing.nuketypes
yes
Remove points around polygonpolygonyes

Remove points within relative distancepositionyes

Include only points within radiusradiusyes

Reverse a pathreverse
yesyes
Simplify routessimplify
yesyes
Stack operationsstackyesnono
Sort waypointssortyes

Manipulate track liststrack
yes
Transform waypoints, tracks, routes into each othertransformyesyesyes