UR5 Robot

An experiment in puppeteering a UR5 robot through the browser

I was invited to participate at a residency at The Studio for Creative Inquiry at Carnegie Mellon University during the Spring of 2019, where I received a grant to assist with the development of Handsfree.js. What you’re seeing in the video above is me mirroring my head movements on a Universal Robot.

After attaching a rubber head to the end of the arm, I used computer vision to detect my head pose (yaw, pitch, roll) from my browsers webcam using just JavaScript! I then loaded a small Python server on the robot itself that took the pose values and matched the rubber head’s pose using inverse kinematics. I had so much fun making this!

Handsfree.js

A JavaScript library for interacting with pages through face gestures

As I was working on NextCheckbox.com in 2018, I befriended an individual who was recovering from a severe stroke at a homeless shelter. The stroke had left him unable to effectively communicate with friends and family as he couldn’t speak clearly on the phone nor could he type or text. This inspired me to begin working on a “Face Pointer”.

Using computer vision, I started a library that took in a user’s head pose through the browser’s webcam and used that to place a pointer on the screen. The user could then smile, raise an eyebrow, or use any other custom gesture to activate clicks. The project received numerous awards, including a grant from Google PAIR, a residency at The Studio for Creative Inquiry Carnegie Mellon, School of AI Fellowship, a grant from Glitch.com, and support through GoFundMe.

Later versions of the library included support for body pose estimation and hand gesture recognition:

NextCheckbox

A Vue.js app with a Firebase backend and custom print layouts.

I started working on NextCheckbox.com full time on February 2017 as a way to help the homeless population in Portland, OR connect with local resources. The problem was that despite having over 270+ organizations in downtown alone that helped homeless and low income individuals, there wasn’t one central place to look up the information. In fact, many of the organizations didn’t have a website of their own!

The project is currently in use by several shelters in downtown Portland.

Vue.js, Firebase, Cloud Functions

The Tech

Built on Vue.js with a custom Vuetify theme, NextCheckbox uses Firebase Firestore as the database. I built a custom Server Side Renderer with Cloud Functions to render SEO meta tags for each service. Additionally, the project comes with a custom print layout to help social workers create printouts.