The map fades

At Wherecamp 2010, I twittered, "as we extend our senses with sensors the map fades." The idea that maps will eventually become obsolete and affectations for future descendants of steampunk aficionados has been difficult to let go. Mike Liebhold started the idea by asking the question, "is 1:1 scale mapping a reasonable idea" on the geowanking list. This touched off a thread that included David Asbury's response containing a link to quotes that touch on the various facets of the 1:1 map. Quotes from the comedian Steven Wright and writers Jorge Luis Borges and Lewis Carroll highlight the absurdity of such a map; and respondents to the question cited information overload would make such a map impractical.


The answer is simple, there is no map. It's gone. Moving freely through the world without resorting to abstractions of location places a large cognitive burden.  Studies of societies steeped in wayfinding have shown that they have developed acute abilities to distinguish environmental clues and extract meaningful information.  What if you no longer need years of experience to develop wayfinding cognitive abilities? 



Wikitude Drive - Test Drivers Wanted from Wikitude on Vimeo.

Locative Media: A New Augmented Reality Blog

Yep, just like we all need a hole in the head, I'm starting an augmented reality blog. The augmented reality stuff doesn't really fit; well nothing really fits in this curmudgeonly hodgepodge of technology, geospatial, and occasional cello posts.


I've been hoarding the locativemedia.org domain for years, so it's time to put it to good use.  With out further ado: Locatively

Where2.0 and WhereCamp 2010

Where2.0

After deciding to skip Where2.0 and go straight to WhereCamp for 2010, I have to eat my words from twitter describing Where2.0 as, "low signal to noise ratio, unless you count product launches as signal."  Since saying that, the Where2.0 program has proven me wrong and is probably one of the best conference schedules I've seen in a while. 

I'm really excited by the Where2.0 program and I'm having a tough time deciding which session to attend.  I'm happy that all the location based product launches occurred at SXSW, leaving Where2.0 more technically focused this year. Let the marriage of location and real time analytics begin.

That said, I'm also speaking at Where2.0, not once but 3 times.  I'm on two augmented reality panels and an Ignite talk.

Truly Open Augmented Reality

Convene and lead a panel on requirements and specifications for an open software stack for augmented reality, based on the assumption that AR is both a discrete medium, and it is the intersection of many media, including web, CAD, mapping, games, virtual worlds... Read more.

The Next Wave of AR: Exploring Social Augmented Experiences

This panel will discuss shared augmented realities, considering some of the essential possibilities and challenges inherent in this new class of social augmented experiences. The format is presentation and discussion of a small set of scenarios (defined in advance, with audience input) describing likely future forms of shared augmented realities at differing scales of social engagement. Read more.  

Tish Shute also has more detailed information on our panel session on her blog ugotade.com.

Making Money at a Non-Profit

Times are hard and your angel funder just told you that your non-profit open source org needs to feed itself. Great, but it also means change, in terms of process, organization, culture and people's expectations. The talk is about achieving a balance between profitability and the open source mission.

WhereCamp
Since its start in 2007, WhereCamp has taken off with events happening Portland, Denver, Montreal, London, Quebec, and Africa as well as one in the Bay Area following Where2.0.  Google is once again hosting WhereCamp for 2010 and over 240 people have signed up for the event.  There are mentions of kites, cameras, and computers on twitter and Scoble has favorited WhereCamp.  Tish Shute and I are planning a session on the ARWave project which uses the Wave Federation Protocol to deliver augmented reality experiences. There will be camping on the Google grounds promising late night hacking sessions and geo-shenanigans.

Open Source AR Toolkit Roundup (Part 2)

Since the last post on augmented reality toolkits in December, more open source AR projects have been announced.  Without further ado here's a new roundup of AR code to play with.

Locatory is based on the Gamaray open source release and developed at the Open University of the Netherlands.  The project is hosted on Google Code and can be downloaded here.

Mixare is another augmented reality engine for the Android platform. Mixare is a platform to build on and uses json as a data source, lots of good documentation at mixare.org and the code and documentation hosted on Google and available for download.  Mixare based apps are already available in the Android Market Place.

augmentthis! is a server and Android app that let's users upload KML and display them on their Android device.  The server portion is available on Google code. The app is available on the Android Market, and the augmenthis! upload  service is on Google app spot with some additional documentation. 



SLARToolKit brings the venerable ARToolkit code base to Microsoft's SilverLight technology. SLARToolkit can be downloaded from Microsofts open source project hosting site Codeplex.


SLARToolkit - Silverlight Augmented Reality 3D projection sample from Rene Schulte on Vimeo.

Symbian gets its own augmented reality engine with OpenMAR. Source and community for OpenMAR is at their website.

One of the more exciting demos making the rounds around the AR community is the Parrot AR Drone, a quadri-copter controlled by an iPhone.  AR Drone has an open API and SDKs for multiple plaforms available on their site.  



While the following are not complete augmented reality applications, service API and SDKs that support AR applications are also becoming more common place.

Kooaba for both the iPhone and Android platforms provides mobile image search. While not open source, Kooaba provides a free query API and a commercial data upload API. Integrating image search into AR applications is a first step away from static AR content into fully interactive applicaitons.

SNAP or Sensor Networking Application Platform is an Android toolkit developed by Ericsson Labs that provides a uniform API for accessing sensors on Android devices. Interestingly, they choose to use the Open Geospatial Consortium SensorML specification to encode sensor information, can you say "Internet of things?"

Augmented Reality Meetup Videos

Chris Grayson (@chrisgrayson) set up ARMeetup.org, which is a video hub for Augmented Reality Meetup Groups. The video for the February 17th Augmented Reality New York (#arny) meetup is up and the demo line-up follows:

• Ori Inbar of Ogmento and gamesalfresco.com  demos the best and worst AR apps for the iPhone (and his rig for showing apps live on a mobile)
• Mike Kelly of Hudson Union - demos 3D object movies controlled by an AR marker to integrate high resolution print ads into a AR environment
• Chris Grayson of GigantiCo presented on the use of virtual mirrors in retail ecommerce to enhance user experience at point of sale.

Video below and checkout the twitter stream for #arny

AR DevCamp NYC: Recap and Going Forward

AR DevCamp NYC was great. We had demos of PyGoWave and Goblin XNA; as well as, numerous discussions on AR, markers, GPS, and AR games.  We had a number of people join us via Skype including the author of the Spads and Fokkers.  Tish Shute put together a comprehensive report of AR DevCamp NYC (within 12 hours of the event). I went through twitter and pulled the links to media that attendees posted.

• @mattlabs recorded Ohan Oda's presentation of Goblin XNA.
• @wonderchook also recorded a vide of a Goblin XNA marble game
• @comogard live blogged AR DevCamp which includes a video by Sean White
• Noah Zerkin brought a Nomad Unit for show and tell
• A video of the PyGoWave demonstration by Patrick Schneider.

Thank You
The event would not have been possible without @TOPPLabs providing their penthouse and Ashley DeVries' assistance with the event planning.  My co-conspirators Tish Shute and Ori Inbar assisted with planning and getting the word out. Kate Chapman created the logo. AR DevCamp was the brainchild of Mike Liebhold and  Anselm Hook provided encouragement (and the wiki) to organizers of the other DevCamps.  Finally,  thanks to all the attendees who brought their energy, enthusiasm and ideas.


Moving Forward
AR DevCamp NYC is just the beginning.  Two projects presented at DevCamp, ARWave and The Big ARNY, are building on the inertia of the event.  ARWave is a project to provide and open, distributed, and universally accessible platform for augmented reality built on the Google Wave Federation Protocol.  The Big ARNY is an Augmented Reality game for New York, a follow up meeting is scheduled for January 15, 2010.  In addition, Ori Inbar has started an AR New York Meetup.  Augmented Reality is starting off with a bang in 2010.

Augmented Reality Resources for Software and Hardware

One aspect of AR DevCamp is building and hacking stuff.  Augmented Reality has been around for quite some time and there are many software projects and some hardware projects.  I'll  cover what I found after a couple of days of searching.  This isn't intended as a comprehensive list, rather its a sampler of what's available right now.


Marker based Augmented Reality
Augmented Reality has been around for some time; but until the relatively recent advent of smart phones with compasses and GPS, the major focus of open source AR software has been around markers that register images superimposed on the a camera view.  The canonical implementation is ARToolKit from Washington University which has spawned implementations in multiple programming languages. In addition to ARToolKit derivatives, other marker based AR software are Goblin and Goblin XNA which was developed using C# and .NET, Simplified Spatial Target Tracker (SSTT) Visualizer developed by Hartmut Seichter in C++, and Studierstube Tracker which is conceptually similar to ARToolKit but is a different code base designed for mobile phones. Here is the sampler list for Augmented Reality marker software.


Flash/Shockwave: 

• FLARToolKit: http://www.libspark.org/wiki/saqoosha/FLARToolKit/en
• FLART example: http://www.adobe.com/devnet/flash/articles/augmented_reality.html
• EZFlar: http://www.ezflar.com/home/show_home (create FLAR app on line), code available: http://github.com/tcha-tcho/EZFLAR
• DART: http://www.cc.gatech.edu/dart/index.htm

C family:

• ARToolKit: http://www.hitl.washington.edu/artoolkit/download/
• Studierstube Tracker: http://studierstube.icg.tu-graz.ac.at/handheld_ar/stbtracker.php
• Goblin: http://www1.cs.columbia.edu/~eaddy/goblin/
• Goblin XNA: http://www.codeplex.com/goblinxna
• SSTT Visualizer: http://technotecture.com/ssttviz-download

Java:

• InstAR (ARToolKit based) http://code.google.com/p/instar/
• jARToolKit (Java binding to ARToolKit): http://sourceforge.net/projects/jartoolkit/
• NyARToolkit for Java/c#/AndroidL  http://nyatla.jp/nyartoolkit/wiki/index.php?NyARToolkit%20for%20Java.en

Processing: 

• Bryan Chung: http://www.bryanchung.net/?p=227

Mobile Augmented Reality

Mobile Augmented Reality places information, markers, and 3d objects in the camera view of a mobile device based upon location derived from GPS or positioning information and direction, typically from compass and accelerometer.  Markers are typically not used in mobile augmented reality platforms such as the iPhone or Android based phones.

Android and iPhone:

• Layar: http://layar.com/api/
• Wikitude:  http://www.wikitude.org/category/developer

iPhone:

• iphonearkit: http://code.google.com/p/iphonearkit/  and http://github.com/zac/iphonearkit/
• locative (Anselm Hook): A bare bones locative media server and client for sharing  favorite places http://github.com/anselm/locative
• Anselm Hook: augmentia: start of an iPhone AR app leveraging sio2  and http://github.com/anselm/augmentia

Android:

• OpenGamaray: http://sourceforge.net/projects/opengamaray/

Hardware
Augmented Reality hardware projects abound ranging from hacking heads-up displays (HUD) to wearable computing.  Pranav Mistry recently announced that the sixthsense project will be open sourced, probably so we won't have to resort to this:

There are also a lot of cool human computer interaction (HCI) projects coming from assistive technology that can be used to expand the AR experience.  Alternative modes interaction include using eye tracking for input and sound to provide additional cues that improve the user experience.  Also listed are a couple just plain cool projects

User Input:

• eyewriter: http://www.eyewriter.org/
• MICOLE: http://micole.cs.uta.fi/software/ haptics

Sound:

• soundmaps: http://makingmaps.net/2008/03/25/making-maps-with-sound/
• isonic: http://www.cs.umd.edu/hcil/audiomap/
• auditory icons: http://www.billbuxton.com/AudioUI06icons.pdf

Cool Stuff:

• Spads and Fokkers: http://spadsandfokkers.sourceforge.net/
• Johnny Lee: http://johnnylee.net/projects/   from wii remote hacks, to low cost eeg for brain control, kinetic typography
• EEG as HCI interface? : http://openeeg.sourceforge.net/doc/index.html

This just scratches the surface of AR apps. Mashups between other software and public Augmented Reality APIs, such as Wikitude, are just starting.

Gamaray AR Browser Open Sourced

Development on Gamaray the AR browser stopped on September 30th.  Today, the Gamaray source code was released as open source and available for download.


In light of Layar wanting to charge content developers then changing their minds for the immediate future its nice to have alternatives.

Augmented Reality and Android: This is not the droid I'm looking for

Brother Cavil: In all your travels, have you ever seen a star go supernova?

Ellen Tigh: No.

Brother Cavil: No? Well, I have. I saw a star explode and send out the building blocks of the Universe. Other stars, other planets and eventually other life. A supernova! Creation itself! I was there. I wanted to see it and be part of the moment. And you know how I perceived one of the most glorious events in the universe? With these ridiculous gelatinous orbs in my skull! With eyes designed to perceive only a tiny fraction of the EM spectrum. With ears designed only to hear vibrations in the air.

Battlestar Galactica [No Exit]

My Blackberry died of a broken USB connector a few weeks back and I replaced it with a myTouch; or as it is more geekily known, the second Android phone (this distinction in branding is important and I'll get back to it later). As with any new toy/technology, I can't resist loading the latest geek ware in an effort to break it. August was the Augmented Reality (AR) news month with Bruce Sterling playing beaming father to the fledgling industry; so what better way to try out new hardware than loading up a bunch of apps that make full use of the platform?

The Android based phones seems to be platform du jour for AR because of hardware access to the GPS, accelerometer, compass, and 3G (for the majority of the time). Augmented reality apps for iPhones have been recently released with the advent of the iPhone 3gs, and since the iPhone apps market is estimated to be worth 2.4 billion USD, in comparison to Android's miniscule apps market, expect AR apps to migrate to the iPhone real soon now. Returning to the branding issue, Joe Lamantia raises the geek-to-chic issue that will impact the future development of AR platforms.

I loaded Layar, Wikitude, and Gamaray and used them for a couple of weeks across different environments ranging from the suburbs and urban areas in Texas, Maryland, and New York City, inside and outside buildings, and while stationary and moving (car, train, plane). Comparisons are not terribly useful since they are very similar in function. In general, Layar currently has the most data layers but searching seems limited; for example a search for "sushi" returned no results but "food" did return results, but not always relevant. Wikitude imprints the closest street address if you take a photo, but I don't know if it writes the location data to the exif. Gamaray allows you to create content (including the insertion of 3D objects) without a developer key à la KML.

There are annoyances related to hardware. Getting a position fix on the myTouch's GPS can take over 2 minutes, and sometimes a reboot is necessary to kick the GPS out of its locational funk. Apparently, the iPhone also suffers from this problem. Layar and Wikitude handle this by using the last known position and Gamaray just refuses to start with a current position fix. AR apps are frequently rendered useless when in inside a building or in a moving vehicle. New York City's buildings create an urban canyon effect that hinders reception of GPS signals. This is unfortunate because AR apps seem to be targeted at the built environment.

For me, the biggest shortcomings of the current crop of AR apps has been the user experience, the lack interesting content, and the use of only locational sensors. I'll address them individually within the context of Joe Lamantia's article detailing four common AR user interaction patterns:

• Heads Up Display - "information about the real objects are added to a fixed point of view, typically the focus of the user’s visual field."
• Tricorder - "add pieces of information to an existing real-world experience, representing them directly within the combined, augmented-reality, or mixed-reality experience."
• Holochess - "adds new and wholly virtual objects directly into the augmented experience, combining them with existing, real objects. The virtual items in Holochess interaction patterns often interact with one another—and sometimes with the real elements of the mixed-reality experience."
• X-ray vision - "simulates seeing beneath the surface of objects, people, or places, showing their internal structure or contents. AR experiences using the X-ray Vision pattern often use a combination of projection and rendering—frequently, a schematic or abstracted rendering—of the object of interest"

UX sucks

Layar, Wikitude and Gamaray use a hybrid of the the Heads Up Display and the the Tricorder. My main pet peeve is the use of the Heads Up Display interaction pattern which is represented as a radar like scope with a wedge representing the field of view. Don't get me wrong, I spent many happy hours playing Battle Zone, but when I have to physically rotate the device (and myself) to see objects to my side or behind me, a simple 2D map is a more practical interface.

The Tricorder pattern is equally annoying, since this involves holding up the phone in front of you while scanning the horizon for alien life forms. Doing this made me feel like an alien in Herald Square. While I don't have a problem being an alien, it also made me feel like one of those self-important dorks who wore a blue-tooth headset constantly when they first came out.

Oh the promise!

Gamaray differs from Layar and Wikitude in that it implements the Holochess pattern by allowing users to place 3D object in the field view. This where things start getting interesting because users can interact with the application and reality instead of passively viewing info bubbles. It's not a far jump to the locative art installations as described in William Gibson's book Spook Country. The Mannahatta Project reconstructs the changing ecology of Manhattan back to 1609, and a AR app could overlay the historical view over the current view. AR applications are a natural fit for these visualizations. Temporal overlays could also be used to overlay houses,tax parcels, and demographic data in areas hit by tornados or hurricanes for use by emergency workers and insurance adjusters.

Where's the content?

Granted that AR is in its infancy, but the currently available content detracts from the idea of AR as a useful tool. A number of apps use Wikipedia as a starting point for content, but the entries in Wikipedia are scale inappropriate and are spatially coarse to be of interest. Tobler's law and its corollaries lands with a heavy thunk. Wikitude, Layar, and Gamaray all support user created content but there doesn't appear to be standardization around any particular content model. Crowd sourcing geospatial content worked well for Google, but Google had the advantage of a single data schema in KML as well as a single API.

AR is a natural fit for built environments but geospatial data is rarely collected at the resolution of individual manmade structures and data collection at this scale often means 3D. CAD data does exist for many urban areas but it is frequently locked up in hard drives of various planning agencies and engineering companies; its difficult to imagine that any of that data will be readily available. An alternative is Sketchup and apps are starting emerge, but none on a mobile platform so far.

Not significantly advanced technology

Considering that 18 years ago I carried a GPS that was larger and heavier than my current notebook and 7 years ago my Garmin Etrex was bigger than my circa 2000 Motorola Startac cell phone, you would think that I would be pretty happy with a device that combines GPS, compass, cell phone, camera, internet terminal, and music player into a single package. Actually, I am happy that I own this technological wonder that can begin to implement AR, but where AR apps fail to be indistinguishable from magic is that they don't use all the sensors. Amidst all the trumpeting that the compass was the killer feature on Android, features such as bluetooth, wifi, and of course the camera seemed to have been forgotten.

Makers have been busily hacking at phone cameras to make scanners and there are accessories to turn the phone camera into a microscope. An application that makes use of image recognition to identify objects in the field of view instead of using proximity and bearing to the phone's location would be infinitely more magical the current applications. An AR app can conceivably take a picture and send it to an image recognition service and then provide additional information based on the picture just like the folks at astronomy.net. That would be magic indeed.