## Friday, September 2, 2011

### List of popular file formats and filter encoding for Map mashups and Geographic information systems; converters of waypoints, tracks, routes

Some popular file formats:

Latitude and longitude are expressed in decimal degrees using the WGS 84 datum. Elevation is recorded in meters. Dates and times are not local time, but instead are Coordinated Universal Time (UTC) using ISO 8601 format.
Find your references in any number of formats:
http://spatialreference.org/
http://spatialreference.org/ref/ it's a listing of spatial references (EPSG) and an interface to search for spatial references. EPSG codes are numeric codes associated with coordinate system definitions. For instance, EPSG:4326 is geographic WGS84.
Projections (a mathematical transformation of the surface of a sphere (3D) onto a 2D plane) are ordered roughly chronologically by type: http://www.radicalcartography.net/?projectionref
Spherical Mercator (http://en.wikipedia.org/wiki/Mercator_projection) is used to describe the PROJECTION used by many commercial API providers.
Proj4 is a library for projecting map data (used by MapServer, GDAL and a multitude of other Open Source GIS libraries): http://trac.osgeo.org/proj/

for all standards: http://www.opengeospatial.org/standards

APR – ESRI ArcView 3.3 and earlier project file
MXD – ESRI ArcGIS project file, 8.0 and higher

http://resources.arcgis.com/
http://www.arcgisblog.com/content/2010/01/arcgis-10-package-mxd-and-all-its-data-map-package

SHP – ESRI shapefile
http://en.wikipedia.org/wiki/Shapefile

Shapefiles spatially describe geometries: points, polylines, and polygons and associated attribute information. A shapefile is a digital vector storage format. It was introduced with ArcView GIS version 2 in the beginning of the 1990s.

DEM – USGS DEM file format:
http://en.wikipedia.org/wiki/USGS_DEM
Sources for USGS DEMs:
http://www.geobase.ca/geobase/en/data/cded/index.html Review (in english and french; download and view) a detailed description of the Canadian Digital Elevation Data. Product specifications, metadata and other supporting documentation are also available.
http://dds.cr.usgs.gov/pub/data/DEM/250/
obtain DEM’s via the National Eelevation Dataset(NED) server at: http://ned.usgs.gov

E00 – ARC/INFO interchange file format
http://freegeographytools.com/category/e00

GeoTIFF – Geographically located raster data
http://en.wikipedia.org/wiki/GeoTIFF
GeoTIFF is a public domain metadata standard which allows georeferencing information to be embedded within a TIFF file (6.0). http://trac.osgeo.org/geotiff/
World TIFF – Geographically located raster data: text file giving corner coordinate, raster cells per unit, and rotation

TAB – MapInfo Table file format
http://en.wikipedia.org/wiki/MapInfo_TAB_format
It's  a geospatial vector data format.
http://mitab.maptools.org/

DTED – Digital Terrain Elevation Data
http://en.wikipedia.org/wiki/DTED
It's a standard of digital datasets which consists of a matrix of terrain elevation values. This standard was originally developed in the 1970s to support aircraft radar simulation.
http://www.fas.org/irp/program/core/dted.htm
-----------------
-geoJSON
The GeoJSON format specification was finalized in June 2008.
http://en.wikipedia.org/wiki/GeoJSON
http://geojson.org/geojson-spec.html
GeoJSON is a geospatial data interchange format based on JavaScript Object Notation (JSON).
In fact, every GeoJSON data structure is also a JSON object, and thus JSON tools can also be used for processing GeoJSON data.
http://www.geowebguru.com/articles/97-technical-overview-geojson
JavaScript Object Notation (JSON), and the terms object, name, value, array, and number, are defined in IETF RTC 4627, at http://www.ietf.org/rfc/rfc4627.txt
http://en.wikipedia.org/wiki/JSON
JSON is simply a mixture of JavaScript object and array literals without variable definitions.
JSON, is a text-based open standard designed for human-readable data interchange. It is derived from the JavaScript scripting language for representing simple data structures and associative arrays, called objects. Despite its relationship to JavaScript, it is language-independent, with parsers available for most languages.The JSON format is used primarily to transmit data between a server and web application (serving as an alternative to XML;  it is generally more compact than XML).

Spatial data format types supported in GeoJSON include points, polygons, multipolygons, features, geometry collections, and bounding boxes, which are stored along with feature information and attributes. The geometries and their properties will have a parent object.
GeoJSON also allows specifying a geographic coordinate system, using the OGC crs (coordinate reference system) property (preferred) or with an EPSG code. If a crs is not defined, GeoJSON will use the WGS84 geoid by default.
GeoJSON is supported by numerous mapping and GIS software packages. Google maps, google maps api...use JSON

As RSS and Atom become more prevalent as a way to publish and share information, it becomes increasingly important that location is described in an interoperable manner so that applications can request, aggregate, share and map geographically tagged feeds.

There are currently two encodings of GeoRSS, Simple and GML (GeoRSS Geography Markup Language (GML)):
GeoRSS-Simple is meant as a very lightweight format that developers and users can quickly add to their existing feeds. It supports basic geometries (point, line, box, polygon) and covers the typical use cases when encoding locations.
GeoRSS GML is a formal GML Application Profile, and supports a greater range of features, notably coordinate reference systems other than WGS-84 latitude/longitude.
Both formats are designed for use with Atom 1.0, RSS 2.0 and RSS 1.0.

-XML: SVG - Scalable Vector Graphics is an XML format. I think that this file format must be in this GIS list because may map servers use this format and all modern browsers use this format.
http://www.mccurley.org/svg/
http://en.wikipedia.org/wiki/File:USA_Oregon_location_map.svg
http://commons.wikimedia.org/wiki/Category:SVG_maps
better SVG graphics rendering engine: http://antigrain.com/svg/index.html
(MapServer 5.0+ can use this engine; it supports sub-pixel anti-aliasing, as well as many more features).
look also the "svg" category of this blog ;)

-XML-based interchange format: GPX or GPS eXchange Format
GPX – XML-based interchange format
http://en.wikipedia.org/wiki/GPS_eXchange_Format
It describes waypoints, tracks, and routes.

A collection of points (with no sequential relationship)  is deemed a collection of individual waypoints.
An ordered collection of points may be expressed as a track or a route.  Tracks are a record of where a person has been (with timestamp (because someone is recording where and when they were there)) and routes are suggestions about where you might go in the future...
View the official site: http://www.topografix.com/gpx.asp

-XML-based interchange format: KML
KML – Keyhole Markup Language, XML-based
http://en.wikipedia.org/wiki/Keyhole_Markup_Language
The KML file specifies a set of features (place marks, altitude, images, 3D models, polygons, textual descriptions, camera view...).  Each place always has a longitude and a latitude.
KML files are distributed in KMZ files, which are zipped files with a .kmz extension.
http://schemas.opengis.net/kml/2.2.0/

Google Maps supports the following KML elements (that is, KML queries in the Search box and GGeoXml objects in the Maps API):

• Placemarks
• Icons
• Folders
• Descriptive HTML
• Entity replacement via and
• KMZ (compressed KML, including attached images)
• Polylines and polygons
• Styles for polylines and polygons, including color, fill, and opacity
• Network links to import data dynamically
• Ground overlays and screen overlays

This list does not apply to Google Earth's export to Google Maps, My Maps import, or My Maps export to Google Earth.
differences between maps and earth for KML: http://code.google.com/apis/kml/documentation/kmlelementsinmaps.html

Maximum uncompressed KML file size=10MB.

convert GIS data to KML files
You can import GIS Data and file (ESRI Shape, MapInfo .tab data; GeoTIFF, or other raster dataset) into Google Earth Pro so that it can be used as part of your Google Earth visualizations: http://earth.google.com/outreach/tutorial_importgis.html. If you have Google Earth Pro, you can import many kinds of GIS data directly using the methods described in this tutorial. If you are using the free version of Google Earth, there are other tools for converting GIS data to KML.

ESRI ArcGIS and MapInfo, have tools to export GIS data into KML format (for use in Google Earth).

ogr2gui (http://sourceforge.net/projects/ogr2gui/): a free utility that converts many formats, including shapefiles to KML.
shp2kml (http://www.zonums.com/shp2kml.html): a free utility that converts shapefiles to KML.
MapWindow GIS has Shape2Earth (http://shape2earth.com/default.aspx), a tool for converting GIS data to KML.
ESRI ArcGIS contains tools in the ArcToolbox (Conversion Tools > To KML) for converting vector and raster GIS data to KML.
Arc2Earth (http://www.arc2earth.com/) is an extension for ArcGIS that converts GIS data to KML. The Arc2Earth Community Edition (http://www.arc2earth.com/communityedition/) is a free tool that has lower limits than the full version.

How to enter data in an on-line spreadsheet to generate a set of placemarks in Google Earth and Maps. Google Docs' web-based, collaborative editing allows your team members to simultaneously enter data and instantly publish updates:

KML and Fusion Table
Because Fusion Tables also supports visualizing your data on maps, it is an alternative to Spreadsheet Mapper: http://earth.google.com/outreach/tutorial_fusion_sample.html
Biking and Hiking trails from MTBGuru.com:

a scientific paper: Hector Gonzalez, Alon Halevy, Christian S. Jensen, Anno Langen, Jayant Madhavan, Rebecca Shapley, Warren Shen (2010). "Google Fusion Tables: Data Management, Integration and Collaboration in the Cloud". SoCC'10. ACM:

Fusion Tables supports a number of data formats:
Comma-separated files (.csv) - Up to 100 MB
Microsoft Excel files (.xls, .xlsx) - Up to 1 MB
OpenDocument Spreadsheet (.ods) - Up to 1 MB
Keyhole Markup Language (.kml) - Up to 100 MB
If you have datasets in Excel or Open Office Spreadsheets that are larger than 1MB, save them as CSV files to take advantage of the larger 100MB limit!

Manage your Data with Open Data Kit Aggregate:
Open Data Kit (ODK) is a suite of tools that allows data collection using mobile devices and data submission to an online server, even without an Internet connection or mobile carrier service at the time of data collection. Once you've collected data in the field with ODK Collect, you can upload and manage your data using ODK Aggregate. ODK Aggregate is the intermediary server storage platform that accepts the data and can send it on to external applications, if desired. ODK Aggregate also allows you to download your datasets in aggregated formats, such as one .csv file. ODK Aggregate allows you to use Google's AppEngine hosting platform (see many post of this blogfor managing your collected data online.

http://en.wikipedia.org/wiki/Geography_Markup_Language (GML)
http://en.wikipedia.org/wiki/GIS_file_formats

------------freeware and set of library (converters; filters)
"data conversion" database:
http://freegis.org/database/?cat=11
"geodata" database:
http://freegis.org/database/?cat=1

MapTools Packaged Tools and Utilities:
http://maptools.org/

a list of bookmarks:

------GDAL
GDAL (Geospatial Data Abstraction Library) is a multi-format raster reading and writing library (it is used as the primary mechanism for reading raster data in MapServer).
It provides a powerful set of libraries for working with vector data:
http://trac.osgeo.org/gdal/

------GPSBabel
GPSBabel converts waypoints, tracks, and routes between popular GPS receivers and mapping programs. It also has powerful manipulation tools for such data.
It contains extensive data manipulation abilities making it a convenient for server-side processing or as the backend for other tools. It does not convert, transfer, send, or manipulate maps. It processes data that may placed on a map, such as waypoints, tracks, and routes...
GPSBabel runs on Microsoft Windows 95, 98, ME, 2000, XP, and Vista plus POSIX OSes such as Linux, UnixWare, OpenServer, Solaris, FreeBSD, and OSX.

Filter Encoding is an OGC standard which defines an XML encoding for filter expressions to allow for spatial and attribute querying: http://www.opengeospatial.org/standards/filter

some filters:

 Filter Function Filter Name Waypoints Tracks Routes Points along an arc arc yes Remove unreliable points with high hdop or vdop discard yes Remove duplicates duplicate yes Manipulate altitudes height yes Interpolate between trackpoints interpolate yes Remove waypoints, tracks, or routes from processing. nuketypes yes Remove points around polygon polygon yes Remove points within relative distance position yes Include only points within radius radius yes Reverse a path reverse yes yes Simplify routes simplify yes yes Stack operations stack yes no no Sort waypoints sort yes Manipulate track lists track yes Transform waypoints, tracks, routes into each other transform yes yes yes