I do hardware. I do software. I do computer vision. I built some software that ran on a cellphone used by LEO (law enforcement officers) to determine if the person they are quizzing is inebriated or impaired through controlled substances by examining the person's eyes and having them focus on images displayed on the phone screen. I've done eye tracking using fully custom solutions and also through a few of the off-the-shelf SDKs such as GazeSense from eyeware and a few other SDKs.
The problem is not the eye-tracking, it is reasonably easy to build robust systems that can do that easily enough, even with custom hardware under all sorts of lighting conditions. The hard part is the UX if you are trying to build something that isn't hampered by current UI paradigms.
Rapid typing and menus of custom actions with just eye movement, though fatiguing, shouldn't be hard to solve, and then render the output however you want; text, text to speech, commands issued to an machine, etc. Making a usable user interface to do anything else, that's where the rubber hits the road.
@pg, which software is your friend using? If it is anything like I've looked in to in the past, it's over-priced accessibility crap with a UI straight out of the 1990s.
Yes, UX is the key. The iPhone succeeded because they didn't just take macOS's mouse/keyboard UI and slap it under a touchscreen. They took the limitations and strengths of capacitive touch and designed a bespoke UX with new concepts everywhere.
Input modalities define platforms. Eye tracking is a new input modality and will define a new platform. It needs a whole new UX designed around its limitations and strengths. It needs a keyboard, it needs a browser, it needs copy and paste, it needs an app switcher, it needs a whole vocabulary of standard interactions and best practices. Apple has a good start in Vision Pro but they're not going to be the only ones doing UX for eye tracking. There's definitely room for other players with fresh ideas.
Another good example is the early "handwriting recognition" of Palm Pilot. They didn't try to increase the accuracy of recognizing all letters, they tried to hit the sweet spot between what they coudl reliably interpret and what users would learn/put up with.
I've got ALS (MND). Completely agree UX is the problem, gazing at a keyboard on a screen designed to stop multiple keys clogging (QWERTY) feels wrong.
Some ideas
- gesture based eye movements, maybe two sweeps on a nine by nine grid, which map directly to phonemes
- enormous 4k 75inch tv with thousands of words or ideograms or phrases
- "writing" with your eyes then doing line to text AI to clean up
- standardish keyboard with massive LLM prediction and clean UX for autocomplete/throwaway with branching options
Ideas are cheap so no clue if these work. Also Tobi split between cheap good non-hackable gaming eyetracking and medical products doesn't help. Finally, with ALS you want to communicate about different things and are more tired.
I worked on the Eye Control Keyboard that has shipped as part of Windows since version 10. You are right that having a QWERTY keyboard is wrong, in many ways.
The Windows keyboard does actually implement something similar to your first and third suggestion. You can spell words by fixating on the first letter of a word, glancing between the following letters and finally fixating on the last letter. Some people can do this to successfully and achieve good input speeds, however, it is a skill that takes some mastery.
For me the real problem comes from three places.
Firstly, having to spell out words either with some kind of keyboard or even with a Dasher-like system means word length matters, long words are harder to enter than short words. The amount of effort needed to express an idea should be proportional to how unusual that idea is, not how many letters are needed to express it; "Hello Dave, nice to see you, how are you today?" should be easier to write than "Eel shoes".
Secondly, in order to achieve some level of throughput, you need to accept that you're going to be living near an edge where typos are inevitable. On current systems, the mechanisms for making corrections are extremely disruptive to throughput, mostly involving repeatedly pressing a key to delete the last character, word or sentence.
Thirdly, similar to the second issue, revising finished text is also a fraught problem that is inadequately addressed and often requires repeated pressing of arrow keys and the like.
I am working on a solution that I believe addresses all of these issues. A solution that allows text to be created quickly using input methods slower than typing, be it eye tracking or switch access scanning. Similarly, a system that allows the same input methods to be used to review and revise text efficiently.
I will be looking to see if I can take advantage of Paul's interest in this area to help his friend and others.
Thank you as well for the encouragement. I'm currently head down working on getting my code into something that could ship as a minimum-viable-product on a variety of platforms.
> I built some software that ran on a cellphone used by LEO (law enforcement officers) to determine if the person they are quizzing is inebriated or impaired through controlled substances by examining the person's eyes and having them focus on images displayed on the phone screen.
Is this used in the field, and what level of testing and validation was required?
This is coming from a skeptical POV, but I’m genuinely curious to hear about the process. Historically, of course, law enforcement technology has a history of being anything but evidence based. It’s good that there’s finally progress away from pseudoscience like fiber analysis and smoke patterns.
But from your experience is there anything to stop LEOs from adopting new tools and technologies without adequate evaluation and standards, aside from the courts, which have such a poor track record in this space?
The problem is not the eye-tracking, it is reasonably easy to build robust systems that can do that easily enough, even with custom hardware under all sorts of lighting conditions. The hard part is the UX if you are trying to build something that isn't hampered by current UI paradigms.
Rapid typing and menus of custom actions with just eye movement, though fatiguing, shouldn't be hard to solve, and then render the output however you want; text, text to speech, commands issued to an machine, etc. Making a usable user interface to do anything else, that's where the rubber hits the road.
@pg, which software is your friend using? If it is anything like I've looked in to in the past, it's over-priced accessibility crap with a UI straight out of the 1990s.