How-To: Install the Locker Project

The Locker Project collects your personal data into a single platform. You can read more about its core values and goals at the Locker Project blog.


This how-to covers installing on OSX. It assumes you have gcc, git, python, and setuptools

1. Install node.js

$ git clone --depth 1 https://github.com/joyent/node.git
$ cd node/
$ export JOBS=2
$ mkdir ~/local
$ ./configure --prefix=$HOME/local/node
$ make
$ make install
$ export PATH=$HOME/local/node/bin:$PATH

2. Install npm

$ curl http://npmjs.org/install.sh | sh

3. Install the locker

$ git clone https://github.com/quartzjer/Locker.git
$ cd Locker
$ npm install
$ python setupEnv.py

4. Start the locker and go to http://localhost:8042 in your browser

$ node lockerd.js

If you see "TypeError: Invalid argument to getAgent" you will need to change the following in node-http-proxy.js 

$ cd ./Locker/node-modules/http-proxy/lib

# using your favorite editor, change node-http-proxy.js to look like this:

function _getAgent (host, port, secure) {
//var agent = !secure ? http.getAgent(host, port) : https.getAgent({
// host: host,
// port: port
//});
var opts = {
host: host,
port: port
};
var agent = !secure ? http.getAgent(opts) : https.getAgent(opts);

agent.maxSockets = maxSockets;
return agent;
}

Weekend Hack: Adding an Auxiliary Input to VW Radio with Satellite and iPod Adapter

I've bought cars over the internet for the past decade without ever seeing then, so when I bought a 2009 VW Jetta TDI with an iPod adapter I thought, "that would be a nice option." However, VW's iPod interface is terrible. For example, albums and playlists are treated as CDs and even basic mp3 tags such as album or song names are displayed as Folder N or Track N. The iPod option also removes the auxiliary audio input. After trying to hack an auxiliary input on to the iPod adapter and failing because the the adapter needs an iPod attached to work, I decided to try a different route.

The car came with a Sirius satellite radio (pre XM acquisition) that I don't use for a number of reasons, but mainly because Sirius pissed me off with the way they handled pre-merger XM contracts. It occurred to me that I could use the satellite radio audio output as an auxiliary input.

The satellite radio box is mounted in the trunk. The speaker outputs from the radio are on a plug with a group of three purple wires and two orange striped wires.

I originally thought about running a line with 3.5mm mini stereo plug to the center console, but I decided to add a blue tooth receiver instead. This offers more flexibility and I can play music from multiple devices including Pandora on my Android phone.


Parts list:
Bluelink Bluetooth Music Receiver
USB Car Charger
A to A USB Cable


The Bluelink Bluetooth Music receiver is an inexpensive bluetooth receiver that can run on either 110 volts or through a USB cable. The Jetta has a convenient 12 volt socket in the trunk so the Bluelink can be powered with a USB car charger connected via a male A to A USB cable.

The Bluelink comes with a 3.5mm mini stereo to RCA cable which I used to make a new cable connecting the audio wires fron the satellite radio to the Bluelink input.

The wiring schematic from the satellite radio is:

• purple w/ blue stripe > ground (black)
• purple w/ white stripe > right channel (red)
• purple w/ green stripe > left channel (white)

After the wires were soldered, they were covered with heat shrink tubing and wire connectors were added. I've found that electronics in cars are constantly vibrating and using connectors improves the reliability of the installation.


The Bluelink was ziptied to the rear deck frame and the cable from the satellite radio and USB power cable were tucked away and ziptied as needed.

Install script for the OpenGeo Suite Community Edition on Ubuntu

OpenGeo released a packaged version of the OpenGeo Suite that can be installed through apt or yum. The install requires some familiarity with Linux, which maybe a barrier if you are coming from the Windows world. I've posted an install script for OpenGeo Suite Community Edition with the Apache2 proxy configured. The script is available on gist. 


The script does the following:

• update the list of repositories to include OpenGeo's repository
• install the OpenGeo Suite Community Edition
• install apache2
• configure apache2 to proxy the Suite web applications to port 80 

The script was tested on Ubuntu Desktop 10.10 Desktop and the Canonical Ubuntu 10.10 64-bit AMI ami-cef405a7.

How-To: Spatial and Ruby on OSX

Notes on getting started using Ruby and rgeo on OSX to manipulate spatial data. 


Setup:
1. Download and install ruby 1.92

$ curl ftp://ftp.ruby-lang.org//pub/ruby/1.9/ruby-1.9.2-p180.tar.gz -o ruby-1.9.2-p180.tar.gz 
$ tar xvfz ruby-1.9.2-p180.tar.gz
$ cd ruby-1.9.2-p180
$ ./configure
$ make
$ sudo make install

2. Download and install homebrew package manager

$ ruby -e "$(curl -fsSLk https://gist.github.com/raw/323731/install_homebrew.rb)"

3. Download and install geos and proj libraries. The ruby gem rgeo requires geos and proj. While it is possible to build rgeo against  the Kyng Chaos Frameworks, it's far easier to install these libraries using brew.

$ brew install geos
$ brew intall proj

4. Install rgeo. RGeo is a ruby gem based on the OGC Simple Features Specification.

$ sudo gem install rgeo

5. Install supporting rgeo gems.
Shapefile:

$ sudo gem install dbf
$ sudo gem install rgeo-shapefile

PostGIS:

$ gem install pg
$ gem install rgeo-activerecord
$ gem install activerecord-postgis-adapter

GeoJSON:

$ gem install rgeo-geojson

Overcome by Events: Chicago Tribune shows how to map

Clint Waggoner¹ (an old boss) once told me, "If you wait just a little bit longer, things tend to resolve themselves." After saying WMS is OBE and baiting people with the #WMSisdead hashtag on Twitter, I felt that I should prove my point by using the tools I mentioned and documenting the process to create custom tiles and display them on the web. 


Project deadlines, a long weekend at the beach, telecons, and a cruise to Cozumel :) resulted in putting it off. As usual Clint was right.


The Chicago Tribune published a five part series on how to make custom maps using PostGIS, TileMill, Mapnik and display it on Google maps. Their code is also available on github.



¹I have undying respect for Clint because he used to dynamically allocate memory by wiring magnetic-core memory as his program was running. He did this on a PDP11. In a sub. Next to a nuclear reactor.

and the shift away from W*S architecture continues

On the Safe blog, Paul Nalos writes about Iowa DOT's data publishing strategy:

Second – and this is where it gets really interesting – they presented an approach where multiple data sources are fused together in the visualization stage. The idea is that it is easier to query multiple data sources (e.g., using web services) in the visualization layer than to keep composite or centralized data stores in sync. Merging data this late in the game likely doesn’t make sense in all cases (e.g., if complex analysis or transformation is required), but the basic advantages are clear: users benefit from an integrated display of data in one system when they would previously have consulted two.

"An integrated display of data" sounds a lot like a tile cache.

 

OBE: Where we last left off ...

Blogger's default commenting system is pathetic and I can't be arsed to find a better solution. So here are the comments to Overcome by Events formatted for easier reading. My responses are indented.

Regina Obe said...

I think you are wrong that WMS is dead.
Aside from the fact you used my name in vein -- I'll let that piece slide.

Both Tile Servers and WMS have there place and for different reasons.

Tile Servers are great for standard stuff like a general overlay when you don't care about being able to feed in different projections, or don't care about other datasets. This is because they are easy to scale and are fast and face it are static (well as far as the viewer is concerned anyway).

They suck when you are talking about very custom layers that few people use or people need in different projections are are changing constantly (by constantly I mean by seconds and you are feeding a lot of it you can't just push with JSON/XML/take your pick ajax overlay).

This is where you need WMS. Both taste great and work great together. Just don't use the wrong tool for the job.

February 19, 2011 1:36 PM 

sophia said... 

I'm afraid WMS is dead, it's just lurching around like a zombie waiting for pickaxe to the brain. At least, WMS did have some life unlike the majority of OGC specs which were still births. 

Your line of argument is correct if we assume the point of WMS is Web based GIS. However, that isn't how web mapping is used in practice. Serving light weight vector data formats over static maps is the dominant pattern for web mapping. So if you want a custom map, why not use Tile Mill to generate a custom layer? Sure only a couple of projections are available, but in practice do the majority of users care? If your require custom projections why not use a desktop solution which has better cartographic support? 

As for dynamic maps, tile servers are evolving to use different algorithms to account for rapidly changing data. Frankly, having seen the amount of effort expended on the FOSS4G WMS Shootout, I think that a code sprint improving tile servers would be a better expenditure of time. 

Thanks for letting the acronym slide, overuse of acronyms is a nasty habit I picked up in DC and OGC meetings.
February 19, 2011 4:45 PM  

sgillies said...

In tiles you have what are essentially positional arguments (http://twitter.com/mnot/status/34854552700194816), which suck for programmers, but you get cachability, scalability, all the goodness of URIs. In WMS you have keyword arguments - more programmer friendly, but no consideration for caching, none of benefits of URIs. Trade-offs.

February 19, 2011 5:59 PM  

Regina Obe said...

Sophia,

I'm with Sean on this. I agree that the majority of web mapping is dishing up common variety stuff best handled with tile servers. TielMill I'm not familiar with which gathering from Jame's Fees blog, I just see as another way of generating stylized tiles (and a more use-friendly way of stylizing) similar to MapNik. Solves a different problem from what WMS does.

I also don't think WMS is just for GIS. The projection support may only be relevant for GIS, but the rest of it is useful for other things.

Imagine if you had like 20 different layers of interest - imagine how much effort that would be to create tiles for that for all the zoom levels you would need (when most of them will NEVER BE USED) - to publish that like in mapserver I just add another 5 lines of code. And of course we do have tilecaches, geocaches too which help out a bit and are best on actual use. Also at some point speed becomes less relevant with advancing CPU, GPUs technology. So thinking about speed is short-sighted and to say that just because 80% of your map need is tiles forgets about the 20% that makes your map more interesting and different from Joe Blow's down the street.

As Sean said WMS is more programmer friendly because it provides more flexibility and face it you aren't hard-coding those calls. My OpenLayers does that -- why do I give a damn what it looks like.

February 19, 2011 9:53 PM 

sophia said... 

Regina,
WMS (and by extension other OGC services) is a dead end not because of speed but because it does how the Internet has evolved. This Sean's point about with tile servers "you get cachability, scalability, all the goodness of URIs" — we both agree with Sean ;-).  

Tiles do not have to be pre-generated, they can and are typically generated dynamically. While WMS are typically used to generate tiles, there really isn't a reason to use WMS. 

The geospatial industry is moving towards deployment on cloud infrastructures along with the rest if the IT industry. This style of infrastructure favors tools that natively leverage the attributes of the internet over a protocol designed by committee.
February 20, 2011 9:43 PM 

Slar said...

Sophia,
I think it is an important discussion, but I believe reports of WMS's demise are greatly exaggerated. The web is filled with really bad specifications that have become ubiquitous - HTML, JavaScript, JPG, XML, etc. The reason they stick around is not because they are any good, but because they have reached that critical mass of adoption so there is interoperability.

I have no doubt that if you control the entire stack then a different approach will deliver better performance. Often in the federal government the entire stack can not (or should not) be controlled by one implementer and there is benefit from Server A serving stuff that Application B (or C or Q) can display. WMS certainly isn't perfect, but it does support this case reasonably well.

If you have an approach for efficiently serving up tiles that everyone from ESRI to the FOSS community to the Luciad/Gallium/other-trendy-company-du-jour will agree to support in their clients, I'd love to see it. What we're finding is that in most cases, WMS is delivering acceptable performance faster than other approaches are proving interoperable. As long as this is the case, WMS will remain relevant.

February 20, 2011 8:16 AM 

sophia said... 

Slar,
Since I'm on a roll, I'll venture another heresy. In practice, interoperability is a red herring. I have yet to see a production site that uses multiple web map servers that originate from different domains. The speed of a mapping client is determined by it's slowest link. If your mapping application includes a slow or frequently down WMS, chances are that people will complain or not use it. More often than not, you deploy the data yourself to ensure availability. 

Getting back the "interoperability is great" world, GeoWebCache (FOSS) now serves ArcGIS tile caches. So you can use all the great ArcGIS tools to create your map and use a FOSS solution to serve it up. When it comes down to interoperability, what is more interoperable than a URL and a png?  

At the risk of redundancy, Federal IT infrastructure is moving to the cloud: http://gcn.com/articles/2011/02/18/kundra-plan-25-percent-of-it-spending-on-cloud.aspx It doesn't make sense to deploy a load balanced WMS infrastructure when storage with an HTTP interface will scale for far less.
February 20, 2011 10:19 PM 

Anonymous said...

In practice the advantage of WMS is that the configuration is discoverable from a single URL from most GIS software. Usually with tile based services you need to configure the origin and resolutions yourself. I don't doubt that there are significant advantages using tiles. But it should be indistinguishable to GIS users.

February 21, 2011 6:07 AM  

sophia said... 

Anonymous,
In general, GetCapabilities document works well when you have less than 100 layers to serve with relatively few styles or SRS supported. I have had customers with this volume of data to serve, think federal agencies. The main problem is that the GetCapabilities document becomes too big to parse efficiently, or hold in memory, especially for web based clients. On the server side, generating this document dynamically can also take quite a bit of time and compute resources to generate. Certain implementations, such as ArcGIS, cache the GetCapabilities document and hold it in cache until ArcGIS is restarted. This works fine unless your use case requires dynamic additions to layers to the WMS. 

The response to handling discovery for WMS with lots of layers has been to use a catalog. The problem is that it adds another layer of complexity that a client must negotiate. EBRIM much?  

In contrast, tile servers are dead simple. Tile size can specified and resolutions are set both dejuré and defacto. There is no handshake, no processing of XML, you just get maps. Of course you have to know apriori what you are requesting, but in practice do you really know what you are getting when you select a layer in a WMS?
February 21, 2011 6:50 AM 

Anonymous said...

I been in GIS App development for over three years now and I did'nt have opportunity to work with WMS as they SUCK..You are right Sophia. I worked with all kind of geospatial developer and know tons of real world geo spatial application, among all those share of WMS is negligible. People who are crying about it are same people who cried for arcIMS but its gone too.
If anyone has doubt, just Google the Web App and see who is using WMS...

February 24, 2011 8:31 AM 

sophia said... 

I'm not sure how to find out who is using WMS in their applications, but there is quite a bit of data served through WMS. For example, there are 14K WMS layers listed at the FGDC monitoring site: http://registry.fgdc.gov/statuschecker/wmsResultsReport.php 

I'm sure that there are more catalogs of WMS services out there if you search for "GeoNetwork". I agree with you about adoption/use, despite the number of of WMS services and layers available, I have yet to see actual usage numbers. This is what I mean about the long tail of WMS.
February 24, 2011 2:15 PM  

mc.prins said...

INSPIRE viewservice dictates WMS on steroids (WMS will learn to speak all the European languages) and WMS has evolved so it now does tilecaches (WMS-C/WMTS) as well.
oh yeah, we also want to support 20 or so different SRS's

February 25, 2011 8:14 AM 

sophia said... 

mc.prins,
That's great but if the INSPIRE viewer (http://www.inspire-geoportal.eu/index.cfm/pageid/341) is any indication of performance, the 20-30 second load times is not very inspiring. 

I think the saying that a map service can be either be fast or interoperable, but not both is not true. It's a matter of picking the right technology and architecture.
February 25, 2011 9:32 AM  

Slar said...

With something like this, there are three categories of interoperability.
1. Adopted standards. Things like WMS give almost complete interoperability but sometimes you feel like you are trying to pound a round peg into a square hole.

2. Emerging standards. You get better performance, but you usually have to do some development to get there. Tile servers fit into this category - they are more or less standardized but most vendors haven't implemented support yet. If you have full control over the solution, it's fine. The FOSS community tries to inhabit this space at every opportunity.

3. Proprietary solutions. Customers will often sacrifice interoperability if they find a solution that is performant and requires minimal development.

The problem I'm having with the Army is that it moves at glacial speeds. If I suggest Door #2 the response is "Sounds good. We can field that by 2017. So...what would you like us to do until then? Unless you have a better plan, I'm just going to pick Door #3." Therefore I am often stuck promoting Door #1 and hoping that it isn't the weakest link in the system. (That's usually when ESRI pipes up and says "Don't worry, we'll be sure to break something so that we can justify our expensive maintenance agreement.") I don't see this changing any time soon so I don't see WMS going anywhere.

In the meantime, please stand up a WMS so that I can at least see your data, even if I'm ultimately going to want to acquire it and host it myself.

February 25, 2011 10:42 PM 

sophia said... 

Slar,
I concur with everything you have said. I've also been in the same situation selling WMS as a solution. We often would make a sale as a "special project" , i.e. it's an experiment. It was frustrating that it was difficult to get anything into production and customers would default to Option 3. 

However, I don't believe that WMS is the only way to make web maps discoverable. A simple README or metadata document of your choice at the root of a tile cache would suffice. The best part of this is that this can be indexed by crawlers instead of having to do the GetCapabilities two step dance to get an ISO19115 doc that may point to actual metadata buried in an ISO 19139 doc. 

If we really wanted to get fancy, metadata can be embedded in the tiles as EXIF for jpgs or XMP for pngs. 

Of course there is also the WMS-C or WMTS spec, but again you have to deal with a service interface that I feel is overly complicated.
February 26, 2011 9:19 AM 

mc.prins said...

The INSPIRE legislation has fairly strict requirements on performance (response time for 800x600 pixels, 8 bits==470Kb image is =< 5 sec.) and availability (99%). Just because some proof of concept viewer makes 20 or so 15KB requests for a layer (instead of one) doesn't make the services slow, it just demonstrates a poor use of the requirements.

February 26, 2011 5:29 AM 

sophia said... 

mc.prins, 

That's the problem with legislated requirements, people end up building systems that only meets the minimum. I'll be bold and predict that by 2015 INSPIRE requirements will be OBE as well, which is well before the 2019 end date for implementation.
sophia
February 26, 2011 9:43 AM 

Overcome By Events or Rearranging Deck Chairs

OBE, the polite acronym to signify that you have become irrelevant. Compare these two requests to a tile server and a Web Map Server:

Cloudmade/OSM tile server

http://b.tile.cloudmade.com/BC9A493B41014CAABB98F0471D759707/1/256/15/17599/10746.png

Web Map Server (from ESRI example)

http://hostname/deploy_name/com.esri.wms.Esrimap?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=Oceans,Countries,Cities&STYLES=&SRS=EPSG:4326&BBOX=-124,21,-66,49&WIDTH=600&HEIGHT=400&FORMAT=image/png

Both requests return a map, but which one would you rather use? The tile server URL can be generated programmatically. The WMS URL requires a GetCapabilities handshake, parsing an XML doc, and formulating a request. It's true that you can compose a map with a WMS, but are developers (or even users) interested in all that sausage making? Amazon now supports static website hosting in S3 storage; cheap storage and hosting all in one, boom. Tools such as TileMill ease the process of generating tiles without the clunkiness of Style Layer Descriptor (SLDs) used by WMS. 


Don't get me wrong, W*S style services will have a long tail, because we've spent a decade expounding it's virtues to the Federal government. However, it's time we recognize the WMS is OBE.

UPDATE:

 "we'll lie in the W*S beds we've made"  - Sean Gillies

UPDATE 2: 
Nicely (or better) formatted comments posted here.

Postgres/PostGIS with RAID 10 on Amazon EBS without Leaving the Browser

Since some one asked about installing Postgres on AWS without leaving the browser, I've updated the Postgres install script to Ubuntu 10.10 Maverick Meerkat and made the size and number of the volumes as command line arguments. The script below can be cut and pasted into the User Data textbox when launching the AMI as in the previous GeoServer install instructions.

You will need to place your private key and X.509 certificate where they can be downloaded. For the sake of simplicity, the example script retrieves the credentials  using wget protected by a username and password. In practice, I would use sftp or scp to transfer the credentials


UPDATE 1/14/11: The script from 1/13/11 contained errors and has been replaced. The current script has been tested against ami-cef405a7

#!/bin/bash -ex
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1

# download the install script and run it
cd /home/ubuntu

# grab your private key and X.509 cert
wget --user=user --password='myPassword' http://example.server.com/home/*.pem
sudo chmod 600 *pem 

# change this to your keypair and cert
export EC2_PRIVATE_KEY=~/myKey.pem
export EC2_CERT=~/myCert.pem

# get install script
wget http://dl.dropbox.com/u/6706687/postgres-ubuntu-ec2-install.sh 
chmod 755 postgres-ubuntu-ec2-install.sh

# run it, note that the install script now takes arguments for the number of volumes
#   and the size of each volume in gigbytes, args below create a 500GB RAID10 
sudo -u ubuntu ./postgres-ubuntu-ec2-install.sh 10 100

Install GeoServer on Amazon EC2 without leaving the browser

Learned a new trick from @waxpancake at ThinkUp, so here's how to install GeoServer on a Ubuntu Maverick 10.10 EC2 instance with out leaving the browser.

Launch an Ubuntu AMI, in this example I use ami-cef405a7, but check this page to find a suitable AMI from Canonical. 

I use a micro instance in this example, but you can use any size.

Copy the following script in the User Data textbox. The script downloads the install script from the earlier blog post and runs it.

#!/bin/bash -ex
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1

# download the install script and run it
cd /home/ubuntu
wget http://dl.dropbox.com/u/6706687/geoserver-ubuntu-ec2-install.sh 
chmod 755 geoserver-ubuntu-ec2-install.sh
./geoserver-ubuntu-ec2-install.sh

It should look like this:

You can tag your instance or leave it blank.

Use an existing keypair or create a new one.

Use and existing security group or create a new one. Note that port 80 and 22 need to be open for Apache and ssh respectively.

Launch the instance.

That's it! It may take a minute or three to get everything installed, configured, and running even if the console shows that the instance is running.