Give us THUNDERDOME!

The FOSS4G WMS Shootout is bikeshedding. I have the utmost respect to the WMS Shootout/Benchmarking participant teams, but the exercise is pointless. There should be a new and more relevant competition for web map servers. There should be Thunderdome.


First things first, this is lower case wms — web map server, not the deceased Web Map Service specification. Similar to the Programming M*therf*cker manifesto, we do one thing in Thunderdome: we serve maps.


Second, there is no such thing as a level playing field in the real world. Assuming all things being equal and spherical cows is fine for theory, but the world doesn't work that way. So the first rule is no limits on technology or architecture, if a vendor wants to spend a million dollars, no problem! The catch is that the architecture, tweaks, and costs have to be disclosed. To make this a little more realistic, the infrastructure has to be deployed on approved Amazon Machine Images (AMIs) and the billing has to be publicly available. So deploy one machine or deploy a cluster of a 100 machines, doesn't matter. We want to see the price/performance ratio.


Third, all participants must serve the same data set. You choose the backend: file, database, or NoSQL — your choice, doesn't matter. Again your architecture is fully documented, including indexes built, connection pooling, etc. We're after configuration and maintenance information.


Fourth, the prescribed set data and maps must be served in a format or protocol supported by OpenLayers. This means no plugins. WMS, tiles, GML, GeoJSON, ArcGIS, whatever works best. Have at it, just as long as it can be retrieved by OpenLayers. Why OpenLayers? It has the best support for a variety of formats and protocols of all the javascript mapping libraries.


Fifth, users will be able to download a tool similar in design to Anonymous' LOIC (Low Orbit Ion Canon) to fully exercise each system for a week. This geoLOIC, as it were, will send the appropriate requests to the targeted system. This tool will gather real world performance across a variety of platforms and network conditions, the results will be sent back to the organizers at the conclusion of the test.


Sixth, logs for each server will record down time, response time, and other metrics. If a geoLOIC shuts down a server, the team has to get it back up. Uptime is crucial.


All, participant data will be presented across a standard set of metrics such as uptime, total response time, response time at the server and cost. All documentation, logs, etc will be publicly available.


The goal of wms Thunderdome is develop a set of statistics that implementors can use to evaluate web map servers across a variety of axis whether it be price/performance, availability/cost, maintenance/performance, etc.


Now go listen to the creepy man's intro.