I talked with people who studied haptics for their thesis in university, and people who currently develop haptic devices for people with disabilities.  I talked with people who have no sense of direction, and people who do but know someone with no sense of direction.  I talked with people from all over the world.  I think that in general, it was just one of the most unique experiences I have had.  So, i want to extend another big thanks to Nokia and all the people behind the scenes that made this event happen!

Going forward, we plan to develop the Haptic Guide much further.  In fact, we have already begun on new prototypes.  For any who are interested, we will be tracking future progress on our new website: http://www.hapticguide.com

We hope that one day, you will see this product in stores alongside cell phones and GPS units.

Following on from Episode 3, this video checks in with the team just before they took off for the PUSH N900 Showcase in London. Find out about a few of their last minute hiccups and what they thought of the PUSH projct as a whole.

If you’ve missed the first three video updates from the Haptic Guide team then you can check the links below:
Haptic Guide, Episode 1
Haptic Guide, Episode 2
Haptic Guide, Episode 3

And if you want to check out all the team’s blog entries, head on over to their Team page.

The video follows the team as they test out their haptic belt prototype in a bid to change the way we use GPS.

If you’ve missed the first two video updates from the team then they’re linked below:
Haptic Guide, Episode 1
Haptic Guide team page for all of their updates.

Overall, I think Mike and I were able to accomplish the primary (and many secondary) goals that we set out to accomplish with this project.  Two working vibrational haptic belts and a phone-based, internet-connected application to allow turn-by-turn geo-coded photo finding without reliance on visual or auditory cues.  This belt and software combination has many potential uses, from acting as a tourist sightseeing platform without the distraction of constantly checking a screen, to a simplified geo-caching tool, to finding a lost child (or spouse) if they have an N900 with them, and more.  Our final videos should make it up online before too long and we look forward to giving a live demo.

I want to extend a huge thank you to Nokia for making this project possible.  It was a great learning experience and a lot of fun to do.

-Tim

This video follows the team as they undergo completing the first prototype of their wearable ‘touring platform’ that lets its users share photos and be guided the to thier locations without distraction from the sights around them.

If you missed the first installment from the Haptic Guide, then you can check it out here or head on over to their team page to check out all their latest blog updates.

If you wanna check out the teams in a bit more detail, head on over to the Teams page or you can watch each of their first progress videos which I’ve linked below for all you PUSH lovers.

Solderin Skaters, Episode 1
Haptic Guide, Episode 1
Light Hack Crew, Episode 1
Sketch Your World, Episode 1
KAPing With The N900, Episode 1

First, some background.  The original milestone in the software was to take your current location (in GPS coordinates) and the target location (from GPS coordinates stored in a jpeg image taken from the N900’s camera) and use some calculus to determine bearing and distance to the target.  This we achieved a week or so ago.  This method alone would work for a target that is fairly close to where you are currently.  But as the distance increases, it becomes harder to discern the direction as the bearing changes caused by moving are more subtle.  Instead, we wanted to take your starting point and your destination and break it down into any number of “legs” so that you have turn-by-turn directions to the target that are easier to follow.  The downside to this is the requirement of having Internet access available.  In Canada, the N900 doesn’t seem to work fully with any of the current cellular service providers, so we are are limited to step-by-step directions only where we can get Wifi internet service.

After some late-night research, I was able to find a way to take two GPS coordinates and generate turn-by-turn directions for them.  So, when you first load a photo, the N900 will now request directions and use those if they are available.  Otherwise, it will just use the start/end point as before.

A couple of bugs surfaced on the first attempt, but after a couple of minor adjustments to the code (like a larger “in range of target” radius) everything worked beautifully.  On the way home, I activated a picture of my place and was able to successfully follow the instructions all the way home, with it changing to the next waypoint just as I came to the correct intersection to turn at, almost exactly like you would expect from a professional GPS device.

The finish line is definitely in sight as Mike said.  I have just got to do a bit of clean up, add a couple more relatively minor UI/display features, some additional testing, and we will have a functional product ready for demo.

Well, time’s running out, and I’m almost done! AHAHAHAHAHAHA!  The prototype (ugly as it may be) was a huge learning exercise for us, (and my first step into leather working).  It’s amazing to me how much faster the Mark II is coming along, and how much better it looks.  We’ve made some design decisions we hope you’ll like, including using a reversible belt!  We can run happily with the feedback belt looking like a regular leather belt or flip it around so the world can see the motors and wires!  Now, in this day and age it may not be entirely appropriate to walk down the street with various wires and lights buzzing and blinking away on your belt…

We made the decision to relocate just about all the fancier electronics to the water-resistant cellphone carrier for better protection from the elements and I’m still impressed at how it all just tucks right into a very small space right beside the N900 itself.  The N900 remains entirely usable as a phone during navigation, which was important to us.  Tim’s made huge strides on the UI on the phone as well, I can’t believe he’s teaching himself python as he goes.

To make life easier for everyone, I wanted post up the complete code for how our bluetooth communication class is implemented.  That way, at least somewhere on the internet is an easy to find, complete (though likely not optimal) example of how to do bluetooth from the N900.  But I do apologize now, because this is probably going to be a very long post…  I have added in a lot of comments to help with readability, they should explain what is going on throughout.  If you have any questions, leave a comment and I’ll try to answer as soon as I can.

In order for this example to work on your N900 and recognize the bluetooth module, you will need to install python-bluez (in the extras-devel repository).  From an X Terminal, run these commands:

sudo gainroot
apt-get install python-bluez

Here are the python files:

BeltConnection.py:

import threading
from bluetooth import *

# Because the application is multi-threaded, this is keep it thread-safe
_bclock = threading.Lock()

# This class Manages all connection stuff
class BeltConnection:
 def __init__(self):
   # Variable declaration
   self.isConnected = False
   self.lastKnownHeading = 0

 def connect(self):
   # This is the prefix of the device name to match
   target_name = "FireFly-"
   target_address = None
   number_of_tries = 2

   # Loop a couple times in case the device/name isn't found the first time
   while target_address is None and number_of_tries > 0:
     number_of_tries = number_of_tries - 1

     # Discover bluetooth devices in the vicinity
     print "Searching for belt..."
     nearby_devices = discover_devices()

     # Loop through each device and check its name for a prefix match
     for btaddr in nearby_devices:
       btname = lookup_name(btaddr)
       print " -Found device: %s (%s)" % (btname, btaddr)
       if btname is not None and btname.startswith(target_name) == True:
         # Use the first device it finds that matches name
         print '    Found firefly.'
         target_address = btaddr
         break

   if target_address is not None:
     # Begin the actual connection device using BT network socket communication
     self.sock=BluetoothSocket( RFCOMM )
     try: self.sock.connect((target_address, 1))
     except:
       print "Unable to connect to belt"
       self.isConnected = False
     else:
       # if connection succeeded, create the multi-thread "receiver"
       # which will handle receiving data in the background automatically
       print "Connected to belt (%s)" % (target_address)
       self.isConnected = True
       self.receiver = __ReceiverThread__(self)
       self.receiver.start()
   else:
     print "Could not locate belt."
     self.isConnected = False

 def sendHeading(self, newheading):
   # If connected, send the heading byte (0-255) to the belt
   if self.isConnected == True:
     self.sock.send(newheading)

 def getCurrentHeading(self):
   heading = 0
   # Acquire a thread lock (for safety) and return the last received heading
   _bclock.acquire()
   try: heading = self.lastKnownHeading
   finally: _bclock.release()
   return heading

 def disconnect(self):
   # If connected, do the socket cleanup stuff
   if self.isConnected == True:
     self.sock.close()
     self.isConnected = False
     print "Disconnected from belt"

# This is a private class that uses threading to receive data from the belt
class __ReceiverThread__ (threading.Thread):
 def __init__(self, parent):
   self.parent = parent
   threading.Thread.__init__(self)

 def run(self):
   data = ""
   while self.parent.isConnected == True:
     # Wait indefinitely until a byte is received.  This is likely not the
     # best way to do this.  For production, it would be better to use the
     # 'select' module to get notification when data is ready.
     data = self.parent.sock.recv(1)

     # Verify received
     if data != "":
       # Convert the byte (0-255) to a heading (0-359) and update belt
       # manager with new heading within the thread lock
       heading = int(float(ord(data))/256*360)                
       _bclock.acquire()
       try: self.parent.lastKnownHeading = heading
       finally: _bclock.release()
     else:
       # If nothing was received, this usually means the socket closed from the
       # remote end, so notify that the belt is disconnected.
       print "Remote client disconnected"
       self.parent.disconnect()

HapticGuide.py:

# This is a fairly large file, so I've cut out a lot of the extra stuff
# that isn't neccessary for this example. Image loading/parsing, gps, etc.

import sys
import gtk
import gobject
import hildon
import threading

# My custom data manager module
from BeltConnection import *

# This is ESSENTIAL if you are using multi-threading in a GTK app.  There are
# cleaner ways to do the same using gobject, but I'm using the brute force for
# the sake of sanity
gtk.gdk.threads_init()

# The overall UI form class
class HapticGuideUI:
 # A thread lock, in case needed
 lock = threading.Lock()
 # A frequency of the loop call (in milliseconds)
 TIMEOUT = 200

 def on_window_destroy(self, widget, data=None):
   self.belt.disconnect()
   gtk.main_quit()

 def on_window_delete(self, widget, data=None):
   self.belt.disconnect()
   gtk.main_quit()
   return False

 def on_send_click(self, widget, data=None):
   # Parse the custom heading to a value value and send to the belt.
   newHeading = self.manualHeading.get_text()
   self.manualHeading.set_text('')
   try: self.targetHeading = int(newHeading)
   except: print "invalid heading.  must be 0-359."
   else:
     if self.targetHeading>=0 and self.targetHeading<=359:
       adjusted = int(float(self.targetHeading)/360*256)
       self.belt.sendHeading(chr(adjusted))
       print 'New heading: %d   adjusted: %d' % (self.targetHeading, adjusted)
     else:
       print 'invalid heading.  must be 0-359.'

 def on_connect_click(self, widget, data=None):
   if self.belt.isConnected == False:
     self.belt.connect()

 def loop(self):
   # Get latest heading update from belt manager and display update if different from prev
   heading = self.belt.getCurrentHeading()
   if heading != self.lastBeltHeading:
     self.lastBeltHeading = heading
     print 'Current heading: %d' % (heading)
     # Update last received heading on UI
     self.currentHeading.set_text('%d' %(heading))  
   return True

 def __init__(self):
   self.app = hildon.Program.get_instance()    
   gtk.set_application_name("Haptic Guide")

   # This is the brilliant GTK+ Glade builder that lets me keep the entire UI
   # code in a .glade (xml) file to be loaded dynamically.
   builder = gtk.Builder()
   builder.add_from_file("HapticGuide.glade") 

   # Get variables to each of the ui widgets we will need to work with
   self.window = builder.get_object("window1")
   self.manualHeading = builder.get_object("manual_heading")
   self.currentHeading = builder.get_object("lbl_current_heading")

   # Create the belt object to handle data communication
   self.belt = BeltConnection()

   # Initialize tracking variables
   self.lastBeltHeading = 0

   # Connect signals exposed in the glade file to local functions
   signals = { "on_window_destroy" : self.on_window_destroy, \
               "on_window_delete" : self.on_window_delete, \
               "on_connect_clicked" : self.on_connect_click, \
               "on_send_clicked" : self.on_send_click }
   builder.connect_signals(signals)

 def show(self):
   self.window.show_all()
   # Add a recurring timeout call so we can continuously update headings
   gobject.timeout_add(self.TIMEOUT, self.loop)

if __name__ == "__main__":
 # Show the UI window
 ui = HapticGuideUI()
 ui.show()

 # The threads_enter() and threads_leave() calls are ESSENTIAL for apps with
 # multi-threaded intereaction
 gtk.gdk.threads_enter()
 gtk.main()
 gtk.gdk.threads_leave()

HapticGuide.glade (as requested, older version but it should work):

<?xml version="1.0"?>
<interface>
 <requires lib="gtk+" version="2.14"/>
 <!-- interface-naming-policy project-wide -->
 <object id="window1">
 <property name="width_request">800</property>
 <property name="height_request">480</property>
 <signal name="destroy_event" handler="on_window_destroy"/>
 <signal name="delete_event" handler="on_window_delete"/>
 <child>
 <object id="notebook1">
 <property name="width_request">800</property>
 <property name="height_request">380</property>
 <property name="visible">True</property>
 <property name="can_focus">True</property>
 <property name="group_id">0</property>
 <child>
 <object id="panelTarget">
 <property name="visible">True</property>
 <child>
 <object id="btnLoadImage">
 <property name="label" translatable="yes">Load
Image</property>
 <property name="width_request">138</property>
 <property name="height_request">92</property>
 <property name="visible">True</property>
 <property name="can_focus">True</property>
 <property name="receives_default">True</property>
 <signal name="clicked" handler="on_load_clicked"/>
 </object>
 <packing>
 <property name="x">656</property>
 <property name="y">170</property>
 </packing>
 </child>
 <child>
 <object id="btn_connect">
 <property name="label" translatable="yes">Connect
To Belt</property>
 <property name="width_request">138</property>
 <property name="height_request">92</property>
 <property name="visible">True</property>
 <property name="can_focus">True</property>
 <property name="receives_default">True</property>
 <accessibility>
 <relation type="flows-to" target="manual_heading"/>
 </accessibility>
 <signal name="clicked" handler="on_connect_clicked"/>
 </object>
 <packing>
 <property name="x">656</property>
 <property name="y">58</property>
 </packing>
 </child>
 <child>
 <object id="aspectframe2">
 <property name="width_request">650</property>
 <property name="height_request">288</property>
 <property name="visible">True</property>
 <property name="label_xalign">0</property>
 <property name="shadow_type">none</property>
 <child>
 <object id="imgTargetPic">
 <property name="visible">True</property>
 <property name="stock">gtk-missing-image</property>
 </object>
 </child>
 </object>
 <packing>
 <property name="y">4</property>
 </packing>
 </child>
 <child>
 <object id="aspectframe1">
 <property name="width_request">100</property>
 <property name="height_request">80</property>
 <property name="visible">True</property>
 <property name="label_xalign">0</property>
 <property name="shadow_type">none</property>
 <child>
 <object id="image3">
 <property name="visible">True</property>
 <property name="stock">gtk-go-up</property>
 </object>
 </child>
 </object>
 </child>
 <child>
 <object id="lblTargetCoords">
 <property name="width_request">794</property>
 <property name="height_request">32</property>
 <property name="visible">True</property>
 <property name="label" translatable="yes">Target Location:      NOT SET</property>
 </object>
 <packing>
 <property name="y">313</property>
 </packing>
 </child>
 <child>
 <object id="lblCurrentCoords">
 <property name="width_request">794</property>
 <property name="height_request">32</property>
 <property name="visible">True</property>
 <property name="label" translatable="yes">Current Location:      UNKNOWN</property>
 </object>
 <packing>
 <property name="y">289</property>
 </packing>
 </child>
 </object>
 </child>
 <child type="tab">
 <object id="tabImage">
 <property name="visible">True</property>
 <property name="label" translatable="yes">Target</property>
 </object>
 <packing>
 <property name="tab_fill">False</property>
 </packing>
 </child>
 <child>
 <object id="image2">
 <property name="visible">True</property>
 <property name="pixbuf">map-sim.png</property>
 </object>
 <packing>
 <property name="position">1</property>
 </packing>
 </child>
 <child type="tab">
 <object id="tabMap">
 <property name="visible">True</property>
 <property name="label" translatable="yes">Map</property>
 </object>
 <packing>
 <property name="position">1</property>
 <property name="tab_fill">False</property>
 </packing>
 </child>
 <child>
 <object id="panelManual">
 <property name="visible">True</property>
 <property name="orientation">vertical</property>
 <child>
 <object id="lbl_current_heading">
 <property name="visible">True</property>
 <property name="label" translatable="yes">0</property>
 </object>
 <packing>
 <property name="position">0</property>
 </packing>
 </child>
 <child>
 <object id="hbox1">
 <property name="visible">True</property>
 <child>
 <object id="label4">
 <property name="visible">True</property>
 <property name="label" translatable="yes">New Heading (0-359): </property>
 </object>
 <packing>
 <property name="position">0</property>
 </packing>
 </child>
 <child>
 <object id="manual_heading">
 <property name="visible">True</property>
 <property name="can_focus">True</property>
 <property name="invisible_char">&#x25CF;</property>
 </object>
 <packing>
 <property name="position">1</property>
 </packing>
 </child>
 <child>
 <object id="btn_send">
 <property name="label" translatable="yes">send</property>
 <property name="visible">True</property>
 <property name="can_focus">True</property>
 <property name="receives_default">True</property>
 <accessibility>
 <relation type="flows-to" target="manual_heading"/>
 </accessibility>
 <signal name="clicked" handler="on_send_clicked"/>
 </object>
 <packing>
 <property name="position">2</property>
 </packing>
 </child>
 </object>
 <packing>
 <property name="position">1</property>
 </packing>
 </child>
 </object>
 <packing>
 <property name="position">2</property>
 </packing>
 </child>
 <child type="tab">
 <object id="tabManual">
 <property name="visible">True</property>
 <property name="label" translatable="yes">Manual Control</property>
 </object>
 <packing>
 <property name="position">2</property>
 <property name="tab_fill">False</property>
 </packing>
 </child>
 </object>
 </child>
 </object>
</interface>

I fired up the application and connected to the belt over bluetooth and Mike stood across the room with his back to me.  It took a minute or so to get used to it, but very soon Mike was able to accurately show me which direction I was telling him to go.  There were just a couple of small problems: 1) Human error – I had trouble remembering my own directions, so I would send a bearing in one direction and get mixed up which way I was telling him to go and 2) There is a minor sensory perception that was unexpected.  When the belt was facing the direction and I sent a bearing directly behind, the difference was not very noticeable.  To fix this, we will have to do a smooth transition from the initial motor to the new going through all the motors along the way.  This will feel more natural to the wearer and should resolve that problem.

Overall, we deemed this first test a major success and we learned a couple things in the process.  The software and hardware communication works well but more importantly, the belt appears to have a VERY quick learning time and starts to feel completely natural after only a couple minutes of wearing it.

A video of our first test will likely be posted somewhere in the near future.

-Tim