Conferences and Workshop Materials

At FOSS4G 2011, I gave a workshop on Deploying Map Services on Amazon Web Services. The materials are here: https://github.com/spara/aws_foss4g_2011


I'll be presenting "Mapping: What in the Wide, Wide World of Sports Is Going On?" at the 2011 Texas GIS Forum. The presentation is here: http://prezi.com/mc7wstzgky5-/mapping-what-in-the-wide-wide-world-of-sports-is-going-on/

Loading shapefiles into Elasticsearch

Elasticsearch is an open source (Apache 2), distributed, RESTful, search engine built on top of Lucene. Elasticsearch supports point data as a geo_point data type; which provides several types of spatial search:

• distance filter
• distance range filter
• bounding box filter
• polygon filter
• distance facets

Elasticsearch provides a REST API for configuring, loading, and querying data. It also has a bulk loading interface. To load shapefiles into elastic search I wrote a ruby script to convert shapefiles to Elasticsearch's bulk loading format. Here's the process for loading multiple shapefiles as multiple types into a single index (think of an index as a 'database' and types as 'tables').

If you haven't installed Elasticsearch, download it, unzip it, and read the README file to start the node.


1. Create the index

curl -XPUT 'http://localhost:9200/geo/'

2. Create a mapping file (place.json) in your favorite editor. This example is for a shapefile called place

{
    "place" : {
        "properties" : {
            "geometry": {
                "properties": {
                    "coordinates": {
                        "type": "geo_point"
                    }
                }
            }
        }
    }
}

curl -XPUT http://localhost:9200/geo/place/_mapping -d @place.json

3. Convert the shapefile to the ES bulk format.

ruby shapefile2elasticsearch.rb > place_data.json

4. Bulk load the data.

curl -XPUT 'http://localhost:9200/_bulk/' --data-binary @place_data.json

5. Test query.

curl -XGET 'http://localhost:9200/geo/place/_search' -d '{
    "query": {
        "filtered": {
            "query": {
                "match_all": {}
            },
            "filter": {
                "geo_distance": {
                    "distance": "20km",
                    "geometry.coordinates": {
                        "lat": 32.25,
                        "lon": -97.75
                    }
                }
            }
        }
    }
}'

6. Add a second type definition for zipcodes:

{
    "zipcode" : {
        "properties" : {
            "geometry": {
                "properties": {
                    "coordinates": {
                        "type": "geo_point"
                    }
                }
            }
        }
    }
}

curl -XPUT http://localhost:9200/geo/place/_mapping -d @zipcode.json

7. Convert and bulk load the data.

curl -XPUT 'http://localhost:9200/_bulk/' --data-binary @zipcode_data.json

8. Test query.

curl -XGET 'http://localhost:9200/geo/zipcode/_search' -d '{
    "query": {
        "filtered": {
            "query": {
                "match_all": {}
            },
            "filter": {
                "geo_distance": {
                    "distance": "20km",
                    "geometry.coordinates": {
                        "lat": 32.25,
                        "lon": -97.75
                    }
                }
            }
        }
    }
}'

Give us THUNDERDOME!

The FOSS4G WMS Shootout is bikeshedding. I have the utmost respect to the WMS Shootout/Benchmarking participant teams, but the exercise is pointless. There should be a new and more relevant competition for web map servers. There should be Thunderdome.


First things first, this is lower case wms — web map server, not the deceased Web Map Service specification. Similar to the Programming M*therf*cker manifesto, we do one thing in Thunderdome: we serve maps.


Second, there is no such thing as a level playing field in the real world. Assuming all things being equal and spherical cows is fine for theory, but the world doesn't work that way. So the first rule is no limits on technology or architecture, if a vendor wants to spend a million dollars, no problem! The catch is that the architecture, tweaks, and costs have to be disclosed. To make this a little more realistic, the infrastructure has to be deployed on approved Amazon Machine Images (AMIs) and the billing has to be publicly available. So deploy one machine or deploy a cluster of a 100 machines, doesn't matter. We want to see the price/performance ratio.


Third, all participants must serve the same data set. You choose the backend: file, database, or NoSQL — your choice, doesn't matter. Again your architecture is fully documented, including indexes built, connection pooling, etc. We're after configuration and maintenance information.


Fourth, the prescribed set data and maps must be served in a format or protocol supported by OpenLayers. This means no plugins. WMS, tiles, GML, GeoJSON, ArcGIS, whatever works best. Have at it, just as long as it can be retrieved by OpenLayers. Why OpenLayers? It has the best support for a variety of formats and protocols of all the javascript mapping libraries.


Fifth, users will be able to download a tool similar in design to Anonymous' LOIC (Low Orbit Ion Canon) to fully exercise each system for a week. This geoLOIC, as it were, will send the appropriate requests to the targeted system. This tool will gather real world performance across a variety of platforms and network conditions, the results will be sent back to the organizers at the conclusion of the test.


Sixth, logs for each server will record down time, response time, and other metrics. If a geoLOIC shuts down a server, the team has to get it back up. Uptime is crucial.


All, participant data will be presented across a standard set of metrics such as uptime, total response time, response time at the server and cost. All documentation, logs, etc will be publicly available.


The goal of wms Thunderdome is develop a set of statistics that implementors can use to evaluate web map servers across a variety of axis whether it be price/performance, availability/cost, maintenance/performance, etc.


Now go listen to the creepy man's intro.

Summarizing why WMS is Dead

At the urging of @wonderchook I did an Ignite talk at WhereCampDC titled: WMS is Dead

I feel that I've flogged this issue enough but nothing like putting in finishing nails in the coffin to tidy it up.


Architecture and Scaling
WMS was designed in the late 1990's, when we really didn't understand how the web worked at scale. I think that the dominant mental model of web services at that time was RPC and it's evident in the architecture of OGC W*S. The architecture was designed to mimic GIS systems, so WMS requests are standardized RPC calls to an abstract model of a GIS. Where WMS went wrong was that it ignored existing Web standards and that the Web's architecture and infrastructure was optimized for delivering documents via links. Ignoring URIs to retrieve standard well known mime types such as pngs contributes to it's clunky interface. Retrieving tiles via URIs is a Web native operation; whilst sending long requests to create a single document is not. That being said, I believe that layering a REST interface over WMS is a futile exercise in trying to hang with the cool kids. As Bruce Sterling often says, "Good luck with that."¹


Cartography
In WMS, you make make maps with SLD. I can't say enough that SLD is a bad idea. XML is for data, it should not be perverted in a single configuration, query, and pseudo programming language.


Projections
Projections have been cited as one reason WMS is superior to tile based mapping. My response is that Web mapping is a different medium from traditional cartography. In traditional cartography, the map maker has only the X and Y dimensions to convey information; which makes projections one of the most important tools for the cartographer. In web mapping, we have n dimensions to present information from info bubbles to interactive tools. Projections are still important for operations that require measurement, but that occurs on the backend and is less important for presentation.


Interoperability
WMS fails at interoperability because it is implemented at the application level. Chaining together a bunch of applications through a series of client requests does not result in a performant web application. In addition, WMS stateless (request/response) operations bypass web native optimizations such as HTTP caching; which has led to software such a MapProxy to overcome this shortcoming. Again this is a bandaid solution to a deeper architectural problem.


Interoperability should take into consideration how the Web is designed rather than providing an overlay that implements loosely-coupled client-server like architecture over the Internet. This means that interoperability should occur at a much lower level than application to application communications, such as data and document types.


I think that the prevalence of tile servers proves the point that URIs plus well known mime types trumps Web RPC. Data is out of the scope of WMS since it is primarily designed as a presentation interface (with lots of kruft), but I think it is important to the future of Web mapping as well as interoperability. 


When the W*S standards were under initial development, OGC made a bet that XML would rule the day; which is understandable since XML was the new hotness at the time. Implicit in this assumption was that XML parsers would improve, especially in the browser. However, as this blog explains, developers are tired of writing XML parsers and use JSON instead. JSON is a well supported and compact way to serialize data that trusts that the developer will use good taste to unpack the data instead of mandating the download of XML schema. The rise of fast and powerful Javascript engines as well as server side Javascript such as node.js was certainly not predictable. This does not bode well for W*S standards, which will be overcome by events even more quickly. For the INSPIRE folks, "Good luck with that."


I feel that the direction of Web mapping will be tied to Javascript on both the client and server side. We are already seeing this with the popularity of TileMill which uses node.js and client scripting libraries such as Leaflet and ModestMaps-JS. Like it or not, this direction in Web mapping is organic instead of top down from a standards body.


¹In all fairness, WMS was designed to retrieve a map dynamically based on a user's request. The use case was not, "I want a fast background map with data sprinkles"; which is the real world use case.

Recovering the Google I/O Samsung Galaxy Tab 10.1 on OSX

Short Story


Several days after an OTA update to Honeycomb, my Google I/O Galaxy Tab crashed and went into a constant reboot state. These are the steps I used to recover. Note that this will delete all your personal data in the /data directory

1. Download fastboot, you will need this tool to get to the recovery screen. Unzip it and make sure it is executable: 
   
   
   

   chmod 755 fastboot-mac
   

   
   
2. Put the Tab into recovery mode by holding the Power button and Volume Down. The screen should look like this: 
   
   
   
   
   
3. Select USB by clicking on Volume Down, click on Volume Up to confirm. Plug the Tab into your computer
4. Download recovery.zip and unzip it
5. Fastboot the Tab (I used the parameters from here):
   
   

   ./fastboot-mac -p 0x0700 -i 0x0955 boot recovery.img
   

   
   
6. On the Tab, use the Volume control to a factory reset. This will reformat /data — note that all your data will be deleted.
   
   
   
7. Reboot the Tab.

Long Story

I tried a number of methods to recover the Tab without losing data. I tried to use Droid Basement's Guide to recover the Tab, but I was stymied by /data. When the Tab booted using fastboot, the /data directory was mounted read only.  I tried mounting /data rw numerous times via adb shell but every time I tried to write to /data it immediately reverted to a ro state. Reformatting /data seemed to the remaining choice.

If you want to do more with the Galaxy Tab, I highly recommend Droid Basement for hacks and images.


UPDATE 4/19/2012

If you need to recover files, you can used the Clockwork Recovery image instead of the stock recovery image.

./fastboot-mac -p 0x0700 -i 0x0955 boot recovery-cwm_4.0.0.4-sam-tab-10.1.img

Mount the /data directory from the Clockwork Recovery meny and navigate to the directory containing the files you want to retrieve. You can then use adb from the Android SDK to retrieve the files.

./adb pull /data/media/DCIM/Camera/20120415_191707.mp4

How-To: Install the Locker Project

The Locker Project collects your personal data into a single platform. You can read more about its core values and goals at the Locker Project blog.


This how-to covers installing on OSX. It assumes you have gcc, git, python, and setuptools

1. Install node.js

$ git clone --depth 1 https://github.com/joyent/node.git
$ cd node/
$ export JOBS=2
$ mkdir ~/local
$ ./configure --prefix=$HOME/local/node
$ make
$ make install
$ export PATH=$HOME/local/node/bin:$PATH

2. Install npm

$ curl http://npmjs.org/install.sh | sh

3. Install the locker

$ git clone https://github.com/quartzjer/Locker.git
$ cd Locker
$ npm install
$ python setupEnv.py

4. Start the locker and go to http://localhost:8042 in your browser

$ node lockerd.js

If you see "TypeError: Invalid argument to getAgent" you will need to change the following in node-http-proxy.js 

$ cd ./Locker/node-modules/http-proxy/lib

# using your favorite editor, change node-http-proxy.js to look like this:

function _getAgent (host, port, secure) {
//var agent = !secure ? http.getAgent(host, port) : https.getAgent({
// host: host,
// port: port
//});
var opts = {
host: host,
port: port
};
var agent = !secure ? http.getAgent(opts) : https.getAgent(opts);

agent.maxSockets = maxSockets;
return agent;
}

Weekend Hack: Adding an Auxiliary Input to VW Radio with Satellite and iPod Adapter

I've bought cars over the internet for the past decade without ever seeing then, so when I bought a 2009 VW Jetta TDI with an iPod adapter I thought, "that would be a nice option." However, VW's iPod interface is terrible. For example, albums and playlists are treated as CDs and even basic mp3 tags such as album or song names are displayed as Folder N or Track N. The iPod option also removes the auxiliary audio input. After trying to hack an auxiliary input on to the iPod adapter and failing because the the adapter needs an iPod attached to work, I decided to try a different route.

The car came with a Sirius satellite radio (pre XM acquisition) that I don't use for a number of reasons, but mainly because Sirius pissed me off with the way they handled pre-merger XM contracts. It occurred to me that I could use the satellite radio audio output as an auxiliary input.

The satellite radio box is mounted in the trunk. The speaker outputs from the radio are on a plug with a group of three purple wires and two orange striped wires.

I originally thought about running a line with 3.5mm mini stereo plug to the center console, but I decided to add a blue tooth receiver instead. This offers more flexibility and I can play music from multiple devices including Pandora on my Android phone.


Parts list:
Bluelink Bluetooth Music Receiver
USB Car Charger
A to A USB Cable


The Bluelink Bluetooth Music receiver is an inexpensive bluetooth receiver that can run on either 110 volts or through a USB cable. The Jetta has a convenient 12 volt socket in the trunk so the Bluelink can be powered with a USB car charger connected via a male A to A USB cable.

The Bluelink comes with a 3.5mm mini stereo to RCA cable which I used to make a new cable connecting the audio wires fron the satellite radio to the Bluelink input.

The wiring schematic from the satellite radio is:

• purple w/ blue stripe > ground (black)
• purple w/ white stripe > right channel (red)
• purple w/ green stripe > left channel (white)

After the wires were soldered, they were covered with heat shrink tubing and wire connectors were added. I've found that electronics in cars are constantly vibrating and using connectors improves the reliability of the installation.


The Bluelink was ziptied to the rear deck frame and the cable from the satellite radio and USB power cable were tucked away and ziptied as needed.

Install script for the OpenGeo Suite Community Edition on Ubuntu

OpenGeo released a packaged version of the OpenGeo Suite that can be installed through apt or yum. The install requires some familiarity with Linux, which maybe a barrier if you are coming from the Windows world. I've posted an install script for OpenGeo Suite Community Edition with the Apache2 proxy configured. The script is available on gist. 


The script does the following:

• update the list of repositories to include OpenGeo's repository
• install the OpenGeo Suite Community Edition
• install apache2
• configure apache2 to proxy the Suite web applications to port 80 

The script was tested on Ubuntu Desktop 10.10 Desktop and the Canonical Ubuntu 10.10 64-bit AMI ami-cef405a7.

How-To: Spatial and Ruby on OSX

Notes on getting started using Ruby and rgeo on OSX to manipulate spatial data. 


Setup:
1. Download and install ruby 1.92

$ curl ftp://ftp.ruby-lang.org//pub/ruby/1.9/ruby-1.9.2-p180.tar.gz -o ruby-1.9.2-p180.tar.gz 
$ tar xvfz ruby-1.9.2-p180.tar.gz
$ cd ruby-1.9.2-p180
$ ./configure
$ make
$ sudo make install

2. Download and install homebrew package manager

$ ruby -e "$(curl -fsSLk https://gist.github.com/raw/323731/install_homebrew.rb)"

3. Download and install geos and proj libraries. The ruby gem rgeo requires geos and proj. While it is possible to build rgeo against  the Kyng Chaos Frameworks, it's far easier to install these libraries using brew.

$ brew install geos
$ brew intall proj

4. Install rgeo. RGeo is a ruby gem based on the OGC Simple Features Specification.

$ sudo gem install rgeo

5. Install supporting rgeo gems.
Shapefile:

$ sudo gem install dbf
$ sudo gem install rgeo-shapefile

PostGIS:

$ gem install pg
$ gem install rgeo-activerecord
$ gem install activerecord-postgis-adapter

GeoJSON:

$ gem install rgeo-geojson

Overcome by Events: Chicago Tribune shows how to map

Clint Waggoner¹ (an old boss) once told me, "If you wait just a little bit longer, things tend to resolve themselves." After saying WMS is OBE and baiting people with the #WMSisdead hashtag on Twitter, I felt that I should prove my point by using the tools I mentioned and documenting the process to create custom tiles and display them on the web. 


Project deadlines, a long weekend at the beach, telecons, and a cruise to Cozumel :) resulted in putting it off. As usual Clint was right.


The Chicago Tribune published a five part series on how to make custom maps using PostGIS, TileMill, Mapnik and display it on Google maps. Their code is also available on github.



¹I have undying respect for Clint because he used to dynamically allocate memory by wiring magnetic-core memory as his program was running. He did this on a PDP11. In a sub. Next to a nuclear reactor.

and the shift away from W*S architecture continues

On the Safe blog, Paul Nalos writes about Iowa DOT's data publishing strategy:

Second – and this is where it gets really interesting – they presented an approach where multiple data sources are fused together in the visualization stage. The idea is that it is easier to query multiple data sources (e.g., using web services) in the visualization layer than to keep composite or centralized data stores in sync. Merging data this late in the game likely doesn’t make sense in all cases (e.g., if complex analysis or transformation is required), but the basic advantages are clear: users benefit from an integrated display of data in one system when they would previously have consulted two.

"An integrated display of data" sounds a lot like a tile cache.

 

OBE: Where we last left off ...

Blogger's default commenting system is pathetic and I can't be arsed to find a better solution. So here are the comments to Overcome by Events formatted for easier reading. My responses are indented.

Regina Obe said...

I think you are wrong that WMS is dead.
Aside from the fact you used my name in vein -- I'll let that piece slide.

Both Tile Servers and WMS have there place and for different reasons.

Tile Servers are great for standard stuff like a general overlay when you don't care about being able to feed in different projections, or don't care about other datasets. This is because they are easy to scale and are fast and face it are static (well as far as the viewer is concerned anyway).

They suck when you are talking about very custom layers that few people use or people need in different projections are are changing constantly (by constantly I mean by seconds and you are feeding a lot of it you can't just push with JSON/XML/take your pick ajax overlay).

This is where you need WMS. Both taste great and work great together. Just don't use the wrong tool for the job.

February 19, 2011 1:36 PM 

sophia said... 

I'm afraid WMS is dead, it's just lurching around like a zombie waiting for pickaxe to the brain. At least, WMS did have some life unlike the majority of OGC specs which were still births. 

Your line of argument is correct if we assume the point of WMS is Web based GIS. However, that isn't how web mapping is used in practice. Serving light weight vector data formats over static maps is the dominant pattern for web mapping. So if you want a custom map, why not use Tile Mill to generate a custom layer? Sure only a couple of projections are available, but in practice do the majority of users care? If your require custom projections why not use a desktop solution which has better cartographic support? 

As for dynamic maps, tile servers are evolving to use different algorithms to account for rapidly changing data. Frankly, having seen the amount of effort expended on the FOSS4G WMS Shootout, I think that a code sprint improving tile servers would be a better expenditure of time. 

Thanks for letting the acronym slide, overuse of acronyms is a nasty habit I picked up in DC and OGC meetings.
February 19, 2011 4:45 PM  

sgillies said...

In tiles you have what are essentially positional arguments (http://twitter.com/mnot/status/34854552700194816), which suck for programmers, but you get cachability, scalability, all the goodness of URIs. In WMS you have keyword arguments - more programmer friendly, but no consideration for caching, none of benefits of URIs. Trade-offs.

February 19, 2011 5:59 PM  

Regina Obe said...

Sophia,

I'm with Sean on this. I agree that the majority of web mapping is dishing up common variety stuff best handled with tile servers. TielMill I'm not familiar with which gathering from Jame's Fees blog, I just see as another way of generating stylized tiles (and a more use-friendly way of stylizing) similar to MapNik. Solves a different problem from what WMS does.

I also don't think WMS is just for GIS. The projection support may only be relevant for GIS, but the rest of it is useful for other things.

Imagine if you had like 20 different layers of interest - imagine how much effort that would be to create tiles for that for all the zoom levels you would need (when most of them will NEVER BE USED) - to publish that like in mapserver I just add another 5 lines of code. And of course we do have tilecaches, geocaches too which help out a bit and are best on actual use. Also at some point speed becomes less relevant with advancing CPU, GPUs technology. So thinking about speed is short-sighted and to say that just because 80% of your map need is tiles forgets about the 20% that makes your map more interesting and different from Joe Blow's down the street.

As Sean said WMS is more programmer friendly because it provides more flexibility and face it you aren't hard-coding those calls. My OpenLayers does that -- why do I give a damn what it looks like.

February 19, 2011 9:53 PM 

sophia said... 

Regina,
WMS (and by extension other OGC services) is a dead end not because of speed but because it does how the Internet has evolved. This Sean's point about with tile servers "you get cachability, scalability, all the goodness of URIs" — we both agree with Sean ;-).  

Tiles do not have to be pre-generated, they can and are typically generated dynamically. While WMS are typically used to generate tiles, there really isn't a reason to use WMS. 

The geospatial industry is moving towards deployment on cloud infrastructures along with the rest if the IT industry. This style of infrastructure favors tools that natively leverage the attributes of the internet over a protocol designed by committee.
February 20, 2011 9:43 PM 

Slar said...

Sophia,
I think it is an important discussion, but I believe reports of WMS's demise are greatly exaggerated. The web is filled with really bad specifications that have become ubiquitous - HTML, JavaScript, JPG, XML, etc. The reason they stick around is not because they are any good, but because they have reached that critical mass of adoption so there is interoperability.

I have no doubt that if you control the entire stack then a different approach will deliver better performance. Often in the federal government the entire stack can not (or should not) be controlled by one implementer and there is benefit from Server A serving stuff that Application B (or C or Q) can display. WMS certainly isn't perfect, but it does support this case reasonably well.

If you have an approach for efficiently serving up tiles that everyone from ESRI to the FOSS community to the Luciad/Gallium/other-trendy-company-du-jour will agree to support in their clients, I'd love to see it. What we're finding is that in most cases, WMS is delivering acceptable performance faster than other approaches are proving interoperable. As long as this is the case, WMS will remain relevant.

February 20, 2011 8:16 AM 

sophia said... 

Slar,
Since I'm on a roll, I'll venture another heresy. In practice, interoperability is a red herring. I have yet to see a production site that uses multiple web map servers that originate from different domains. The speed of a mapping client is determined by it's slowest link. If your mapping application includes a slow or frequently down WMS, chances are that people will complain or not use it. More often than not, you deploy the data yourself to ensure availability. 

Getting back the "interoperability is great" world, GeoWebCache (FOSS) now serves ArcGIS tile caches. So you can use all the great ArcGIS tools to create your map and use a FOSS solution to serve it up. When it comes down to interoperability, what is more interoperable than a URL and a png?  

At the risk of redundancy, Federal IT infrastructure is moving to the cloud: http://gcn.com/articles/2011/02/18/kundra-plan-25-percent-of-it-spending-on-cloud.aspx It doesn't make sense to deploy a load balanced WMS infrastructure when storage with an HTTP interface will scale for far less.
February 20, 2011 10:19 PM 

Anonymous said...

In practice the advantage of WMS is that the configuration is discoverable from a single URL from most GIS software. Usually with tile based services you need to configure the origin and resolutions yourself. I don't doubt that there are significant advantages using tiles. But it should be indistinguishable to GIS users.

February 21, 2011 6:07 AM  

sophia said... 

Anonymous,
In general, GetCapabilities document works well when you have less than 100 layers to serve with relatively few styles or SRS supported. I have had customers with this volume of data to serve, think federal agencies. The main problem is that the GetCapabilities document becomes too big to parse efficiently, or hold in memory, especially for web based clients. On the server side, generating this document dynamically can also take quite a bit of time and compute resources to generate. Certain implementations, such as ArcGIS, cache the GetCapabilities document and hold it in cache until ArcGIS is restarted. This works fine unless your use case requires dynamic additions to layers to the WMS. 

The response to handling discovery for WMS with lots of layers has been to use a catalog. The problem is that it adds another layer of complexity that a client must negotiate. EBRIM much?  

In contrast, tile servers are dead simple. Tile size can specified and resolutions are set both dejuré and defacto. There is no handshake, no processing of XML, you just get maps. Of course you have to know apriori what you are requesting, but in practice do you really know what you are getting when you select a layer in a WMS?
February 21, 2011 6:50 AM 

Anonymous said...

I been in GIS App development for over three years now and I did'nt have opportunity to work with WMS as they SUCK..You are right Sophia. I worked with all kind of geospatial developer and know tons of real world geo spatial application, among all those share of WMS is negligible. People who are crying about it are same people who cried for arcIMS but its gone too.
If anyone has doubt, just Google the Web App and see who is using WMS...

February 24, 2011 8:31 AM 

sophia said... 

I'm not sure how to find out who is using WMS in their applications, but there is quite a bit of data served through WMS. For example, there are 14K WMS layers listed at the FGDC monitoring site: http://registry.fgdc.gov/statuschecker/wmsResultsReport.php 

I'm sure that there are more catalogs of WMS services out there if you search for "GeoNetwork". I agree with you about adoption/use, despite the number of of WMS services and layers available, I have yet to see actual usage numbers. This is what I mean about the long tail of WMS.
February 24, 2011 2:15 PM  

mc.prins said...

INSPIRE viewservice dictates WMS on steroids (WMS will learn to speak all the European languages) and WMS has evolved so it now does tilecaches (WMS-C/WMTS) as well.
oh yeah, we also want to support 20 or so different SRS's

February 25, 2011 8:14 AM 

sophia said... 

mc.prins,
That's great but if the INSPIRE viewer (http://www.inspire-geoportal.eu/index.cfm/pageid/341) is any indication of performance, the 20-30 second load times is not very inspiring. 

I think the saying that a map service can be either be fast or interoperable, but not both is not true. It's a matter of picking the right technology and architecture.
February 25, 2011 9:32 AM  

Slar said...

With something like this, there are three categories of interoperability.
1. Adopted standards. Things like WMS give almost complete interoperability but sometimes you feel like you are trying to pound a round peg into a square hole.

2. Emerging standards. You get better performance, but you usually have to do some development to get there. Tile servers fit into this category - they are more or less standardized but most vendors haven't implemented support yet. If you have full control over the solution, it's fine. The FOSS community tries to inhabit this space at every opportunity.

3. Proprietary solutions. Customers will often sacrifice interoperability if they find a solution that is performant and requires minimal development.

The problem I'm having with the Army is that it moves at glacial speeds. If I suggest Door #2 the response is "Sounds good. We can field that by 2017. So...what would you like us to do until then? Unless you have a better plan, I'm just going to pick Door #3." Therefore I am often stuck promoting Door #1 and hoping that it isn't the weakest link in the system. (That's usually when ESRI pipes up and says "Don't worry, we'll be sure to break something so that we can justify our expensive maintenance agreement.") I don't see this changing any time soon so I don't see WMS going anywhere.

In the meantime, please stand up a WMS so that I can at least see your data, even if I'm ultimately going to want to acquire it and host it myself.

February 25, 2011 10:42 PM 

sophia said... 

Slar,
I concur with everything you have said. I've also been in the same situation selling WMS as a solution. We often would make a sale as a "special project" , i.e. it's an experiment. It was frustrating that it was difficult to get anything into production and customers would default to Option 3. 

However, I don't believe that WMS is the only way to make web maps discoverable. A simple README or metadata document of your choice at the root of a tile cache would suffice. The best part of this is that this can be indexed by crawlers instead of having to do the GetCapabilities two step dance to get an ISO19115 doc that may point to actual metadata buried in an ISO 19139 doc. 

If we really wanted to get fancy, metadata can be embedded in the tiles as EXIF for jpgs or XMP for pngs. 

Of course there is also the WMS-C or WMTS spec, but again you have to deal with a service interface that I feel is overly complicated.
February 26, 2011 9:19 AM 

mc.prins said...

The INSPIRE legislation has fairly strict requirements on performance (response time for 800x600 pixels, 8 bits==470Kb image is =< 5 sec.) and availability (99%). Just because some proof of concept viewer makes 20 or so 15KB requests for a layer (instead of one) doesn't make the services slow, it just demonstrates a poor use of the requirements.

February 26, 2011 5:29 AM 

sophia said... 

mc.prins, 

That's the problem with legislated requirements, people end up building systems that only meets the minimum. I'll be bold and predict that by 2015 INSPIRE requirements will be OBE as well, which is well before the 2019 end date for implementation.
sophia
February 26, 2011 9:43 AM 

Overcome By Events or Rearranging Deck Chairs

OBE, the polite acronym to signify that you have become irrelevant. Compare these two requests to a tile server and a Web Map Server:

Cloudmade/OSM tile server

http://b.tile.cloudmade.com/BC9A493B41014CAABB98F0471D759707/1/256/15/17599/10746.png

Web Map Server (from ESRI example)

http://hostname/deploy_name/com.esri.wms.Esrimap?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=Oceans,Countries,Cities&STYLES=&SRS=EPSG:4326&BBOX=-124,21,-66,49&WIDTH=600&HEIGHT=400&FORMAT=image/png

Both requests return a map, but which one would you rather use? The tile server URL can be generated programmatically. The WMS URL requires a GetCapabilities handshake, parsing an XML doc, and formulating a request. It's true that you can compose a map with a WMS, but are developers (or even users) interested in all that sausage making? Amazon now supports static website hosting in S3 storage; cheap storage and hosting all in one, boom. Tools such as TileMill ease the process of generating tiles without the clunkiness of Style Layer Descriptor (SLDs) used by WMS. 


Don't get me wrong, W*S style services will have a long tail, because we've spent a decade expounding it's virtues to the Federal government. However, it's time we recognize the WMS is OBE.

UPDATE:

 "we'll lie in the W*S beds we've made"  - Sean Gillies

UPDATE 2: 
Nicely (or better) formatted comments posted here.

Postgres/PostGIS with RAID 10 on Amazon EBS without Leaving the Browser

Since some one asked about installing Postgres on AWS without leaving the browser, I've updated the Postgres install script to Ubuntu 10.10 Maverick Meerkat and made the size and number of the volumes as command line arguments. The script below can be cut and pasted into the User Data textbox when launching the AMI as in the previous GeoServer install instructions.

You will need to place your private key and X.509 certificate where they can be downloaded. For the sake of simplicity, the example script retrieves the credentials  using wget protected by a username and password. In practice, I would use sftp or scp to transfer the credentials


UPDATE 1/14/11: The script from 1/13/11 contained errors and has been replaced. The current script has been tested against ami-cef405a7

#!/bin/bash -ex
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1

# download the install script and run it
cd /home/ubuntu

# grab your private key and X.509 cert
wget --user=user --password='myPassword' http://example.server.com/home/*.pem
sudo chmod 600 *pem 

# change this to your keypair and cert
export EC2_PRIVATE_KEY=~/myKey.pem
export EC2_CERT=~/myCert.pem

# get install script
wget http://dl.dropbox.com/u/6706687/postgres-ubuntu-ec2-install.sh 
chmod 755 postgres-ubuntu-ec2-install.sh

# run it, note that the install script now takes arguments for the number of volumes
#   and the size of each volume in gigbytes, args below create a 500GB RAID10 
sudo -u ubuntu ./postgres-ubuntu-ec2-install.sh 10 100

Install GeoServer on Amazon EC2 without leaving the browser

Learned a new trick from @waxpancake at ThinkUp, so here's how to install GeoServer on a Ubuntu Maverick 10.10 EC2 instance with out leaving the browser.

Launch an Ubuntu AMI, in this example I use ami-cef405a7, but check this page to find a suitable AMI from Canonical. 

I use a micro instance in this example, but you can use any size.

Copy the following script in the User Data textbox. The script downloads the install script from the earlier blog post and runs it.

#!/bin/bash -ex
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1

# download the install script and run it
cd /home/ubuntu
wget http://dl.dropbox.com/u/6706687/geoserver-ubuntu-ec2-install.sh 
chmod 755 geoserver-ubuntu-ec2-install.sh
./geoserver-ubuntu-ec2-install.sh

It should look like this:

You can tag your instance or leave it blank.

Use an existing keypair or create a new one.

Use and existing security group or create a new one. Note that port 80 and 22 need to be open for Apache and ssh respectively.

Launch the instance.

That's it! It may take a minute or three to get everything installed, configured, and running even if the console shows that the instance is running.