PDA

View Full Version : iPod/iPhone - can the "away" camera be used as an optical mouse sensor?



IsaacKuo
2011-Nov-23, 10:30 PM
I'm wondering if the "away" facing camera can be used as an optical mouse sensor, thus turning the entire phone into a mouse.

I'd like to use this for a game interface, where you simply slide the phone around the tabletop in order to "scroll" your view and point at things with a central crosshair.

The way the normal interface works, your finger blocks whatever it is you're trying to point at, and it's imprecise compared to using a mouse. Also, the normal interface requires different actions for scrolling and pointing. Turning the phone into a big mouse combines scrolling and pointing into a single action (this is similar to how many computer games work in full screen mode).

Thanks!

PraedSt
2011-Nov-24, 06:22 PM
Have you got solutions for (1) lighting and (2) taking pictures quickly? If so, then you probably just need an app.

Ara Pacis
2011-Nov-24, 11:20 PM
IIRC, several mice use a pattern recognition type of code to detect movement from one instant to the next. I wonder if a smartphone camera has the ability to see something sharp at that short distance... and then there's lighting to worry about but you might use the LED if it has one, so maybe you'd want a cradle to give you some standoff distance. If you can write code or know someone who can, I don't see any physical reason why it couldn't work.

Moose
2011-Nov-25, 12:56 AM
In theory, yes, it's possible. It's an interesting idea, though not a trivial one to implement. Off the top of my head, here are at least some of the challenges you'd have to address:

1) A USB optical mouse, one suitable for gaming, polls 125 times per second. iPod cameras will generate in the neighborhood of 30 frames per second. Without doing something fancy so that your camera "records" at high speed using only part of its image sensor, your iPod mouse motion would feel choppy (even by 1985 standards... early serial mice polled at 40hz), imprecise, and uncomfortable for gaming.

2) Processing the image would also tie up a not-insignificant portion of the processor budget you might rather use for generating frames for the game itself.

3) Running the camera continuously would increase power consumption, though I don't know how much. Less than the screen, surely, but still non-zero. (And don't call you Shirley.)

[Edit to update:] None of this is entirely insurmountable. The hard part is written in my next post.

Moose
2011-Nov-25, 01:04 AM
I wonder if a smartphone camera has the ability to see something sharp at that short distance...

You wouldn't necessarily need sharp. It's not the image itself that you're tracking, but the motion of the features. It's probably enough to track the blur, just so it's consistent blur.

The hard part would be the coarse angular motion. It's a much larger lens, after all, and much less frequent polling. This (and the CPU time it would take to process this sans specialized hardware) would probably wind up being the show-stoppers.

Ara Pacis
2011-Nov-25, 01:55 AM
You wouldn't necessarily need sharp. It's not the image itself that you're tracking, but the motion of the features. It's probably enough to track the blur, just so it's consistent blur.

The hard part would be the coarse angular motion. It's a much larger lens, after all, and much less frequent polling. This (and the CPU time it would take to process this sans specialized hardware) would probably wind up being the show-stoppers.

The problem tracking the blur is that it may change faster and in a manner unrelated to motion since it is, ya know, blurred.

a1call
2011-Nov-25, 02:17 AM
The Iteleport app turns an iPhone to a touch-pad. So you could sit away (miles if you'd like) and push the pointer around and click. However there is a slight performance degrading compared to an actual touch-pad.

Moose
2011-Nov-25, 03:16 AM
The problem tracking the blur is that it may change faster and in a manner unrelated to motion since it is, ya know, blurred.

Maybe. I certainly can't claim to be an expert on blur or anything (other than being myopic, myself), but I'm pretty sure you could use the same algorithm optical mice use, more or less unchanged, if you kept your area small. If your area of processing is small and centered, the lens's curvature shouldn't affect the blur (from edge to edge) enough to matter.

a1call
2011-Nov-25, 03:35 AM
There is an app called "Heart Rate" which detects heart beats via color change on the tip of a finger, accurately!
So one method that should work would be to slide the iPhone/camera over a sheet with say blue gradient top-down and green gradient left-right. Then the color could be analyzed to yield mouse's x-y coordinates.
Just a brain storm.

IsaacKuo
2011-Nov-25, 06:02 AM
I've been thinking of using the "toward" facing camera instead. It could use the position of the user's eyes, on the assumption that the user's head will remain in roughly the same place. Or it could lock in on features in the ceiling if the user's face isn't in view.

But locking in on the user's eyes would be best. It offers alternative methods of use--keeping the phone in place while moving one's head, or holding the phone in your hands and tilting to pan/point.

Due to the limited frame rate, I don't think this would be suitable for a fast paced FPS, but it could be good enough for a turn based game or slower paced RTS.

IsaacKuo
2011-Nov-25, 09:07 AM
In any case, these solutions seem clunky and probably not very precise. I suppose it's not practical.

So essentially, I just wish that smart phones incorporated an optical mouse sensor on the "back". Oh well...

jokergirl
2011-Nov-25, 10:44 AM
It doesn't seem practical, no. Considering the wide variety of surfaces the "mouse" would have to work on, and the general slowness of getting anything out of the camera, it would surprise me if you got anything close to real-time there.
Why not use the gyro instead? I know it can't handle everything but it might give you a basic idea?

;)

IsaacKuo
2011-Nov-25, 10:50 AM
The gyro is absolutely useless for this. It can't sense translation motion.

PraedSt
2011-Nov-25, 11:37 AM
How often does the GPS signal refresh? :D

jokergirl
2011-Nov-25, 12:50 PM
GPS signals are 1Hz.

Why can't you use the gyro? Mouse movements would give very straight sideways acceleration. That's (delta)(delta)position. Should be enough with some number-crunching if the precision is enough.
How often is the accelerometer on the iPhone sampled?

;)

PS: Yes, I know I used gyro as a synonym for accelerometer again, and I know it's wrong. It's an accelerometer. Sorry.

PraedSt
2011-Nov-25, 02:05 PM
A measly 1Hz. Pfft.

Moose
2011-Nov-25, 02:08 PM
A measly 1Hz. Pfft.

That's fine if you have a half-km mousepad. And have some time on your hands.

Moose
2011-Nov-25, 02:09 PM
There is an app called "Heart Rate" which detects heart beats via color change on the tip of a finger, accurately!
So one method that should work would be to slide the iPhone/camera over a sheet with say blue gradient top-down and green gradient left-right. Then the color could be analyzed to yield mouse's x-y coordinates.
Just a brain storm.

Some early optical mice used a gridded mousepad. Come to think of it, their lenses were pretty big. Maybe the rotation issue wouldn't be all that bad.

JCoyote
2011-Nov-25, 05:07 PM
Most optical mice though use dedicated hardware and it's all stripped down to detect motion and the end result is all that is sent to the pc.

The processing overhead on this could be pretty high depending on what the smallest frame area the camera can deliver is. Also, the camera sensor may "charge" in a way that would be insurmountable for getting fast enough response.

And there is no reference beam. Optical mice tend to have LED's and lasers in them for a reason, to create a consistent visual of the surface. No such thing on the iPhone.

If you can create a mouse input for the iPhone, perhaps you would be better suited by creating a new phone case that incorporates a mouse into it? Just an idea.

IsaacKuo
2011-Nov-25, 05:27 PM
Why can't you use the gyro? Mouse movements would give very straight sideways acceleration. That's (delta)(delta)position. Should be enough with some number-crunching if the precision is enough.
How often is the accelerometer on the iPhone sampled?

PS: Yes, I know I used gyro as a synonym for accelerometer again, and I know it's wrong. It's an accelerometer. Sorry.

Poking around, I find that iPhones have both a gyro and an accelerometer. The gyro can't detect translation acceleration, but the accelerometer can. It seems that the capabilities are crude in practice.

PraedSt
2011-Nov-25, 07:00 PM
I found someone who's done it with a Nokia. Maybe you could pick up clues.
http://www.pyrofersprojects.com/blog/nokia-6230i-bluetooth-mouse-mod/

And a clip of it in action. Seems to work ok.
http://www.youtube.com/watch?v=yT1h_ITR0G0

IsaacKuo
2011-Nov-25, 11:31 PM
I found someone who's done it with a Nokia. Maybe you could pick up clues.
http://www.pyrofersprojects.com/blog/nokia-6230i-bluetooth-mouse-mod/

And a clip of it in action. Seems to work ok.
http://www.youtube.com/watch?v=yT1h_ITR0G0

Awesome! This is the sort of thing I was thinking of, and it does look impractical. Holding the phone above the surface is a non-starter, although maybe it would be okay for the upward staring camera (looking at the ceiling or player's face).

orionjim
2011-Nov-26, 01:36 AM
Apple is working on an optical LCD sensor panel.
http://www.getgek.com/tag/iphone-optical-lcd-sensor-panel

Jim

IsaacKuo
2011-Nov-26, 06:48 AM
Apple is working on an optical LCD sensor panel.
http://www.getgek.com/tag/iphone-optical-lcd-sensor-panel

This is the sort of patent which stifles innovation. Fortunately, you can't patent something which is a simple combination of two existing things (a smart phone and a mouse sensor).

Anyway, this thing isn't relevant to the game interface I'd like, because it requires the screen to be face-down.