This page is intended to describe completed research projects. It is under construction.
Automating repetitive tasks in web applications
(with Zachary Bray)
Many tasks in web applications are highly repetitive. The research field that tries to automate tedious tasks for users is called Programming by Example (PBE). In this paper we contribute to this field by introducing a new set of algorithms to learn users' intended tasks within a web browser. Our technique is based on using ensembles of decision trees, in which each individual decision tree represents a particular hypothesis about the steps the user is trying to perform. The system is implemented as a Firefox web browser add-on. For two difficult use cases we have demonstrated that this system quickly converges to the solution, even in the presence of high noise.
Research paper:
- Bray, Z. and Kristensson, P.O. 2010. Using ensembles of decision trees to automate repetitive tasks in web applications. In Proceedings of the 2nd ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS '10). ACM Press: 35-40.
Parakeet: Efficient Mobile Speech Recogniton
(with Keith Vertanen)
Speech recognition has undergone tremendous progress in the recent decades. However, an Achilles heel of speech recognition is the difficulty of correcting inevitable recognition errors. We developed Parakeet: a mobile continuous speech recognition system that enables users to correct recognition errors using a touch-screen interface. The system uses a word-confusion network to reveal the next best hypotheses in an efficient way to the user. There are also a number of other user experience enhancements in this system, for details see the publications below. In a formative study we found that novice users could enter text at a text entry rate of 13 wpm while simultaneously walking outdoors during a windy day. Ignoring rather long recognition delays, an expert user reached a text entry rate of 45 wpm for the same task. This indicates that mobile speech recognition holds tremendous potential as an efficient mobile text entry method, as long as users are provided with an efficient correction interface.
Press / media:
- Parakeet squawks a mighty solution to speech recognition accuracy. TMCnet. Brendan B. Read, September 3, 2010.
- Correct me if I'm wrong.... The Economist's Technology Quarterly. September 2, 2010.
- Correct me if I'm wrong... Better speech-recognition technology. The Economist's Technology Monitor (online). July 12, 2010.
Most important research papers:
- Vertanen, K. and Kristensson, P.O. 2010. Intelligently aiding human-guided correction of speech recognition. In Proceedings of the 24th AAAI Conference on Artificial Intelligence (AAAI '10). AAAI Press: 1698-1701.
- Vertanen, K. and Kristensson, P.O. 2009. Parakeet: a continuous speech recognition system for mobile touch-screen devices. In Proceedings of the 14th ACM International Conference on Intelligent User Interfaces (IUI '09). ACM Press: 237-246.
The efficacy of handwriting recognition
(with Leif Denby)
Research paper:
- Kristensson, P.O. and Denby, L.C. 2009. Text entry performance of state of the art unconstrained handwriting recognition: a longitudinal user study. In Proceedings of the 27th ACM Conference on Human Factors in Computing Systems (CHI '09). ACM Press: 567-570.
The efficacy of space time cube visualization
InfoTouch: Multitouch and Visualization
(with Olof Arnell, Anneli Björk, Nils Dahlbäck, Joakim Pennerup, Erik Prytz, Johan Wikman and Niclas Åström)
In this project we explored how a multitouch table could be used to encourage exploration in information visualization. Our application let users explore tagged images in Flickr by manipulating an interconnected tag cloud. Users could use both hands to push and pull tags together. When users pulled tags into a certain area the system showed the user a selection of photos with the corresponding tags. We discovered that users generally liked the system and that they used multitouch interaction about 10% of the time. This indicates that multitouch may be a useful complement to information visualization displays.
Research paper:
- Kristensson, P.O., Arnell, O., Björk, A., Dahlbäck, N., Pennerup, J., Prytz, E., Wikman, J. and Åström, N. 2008. InfoTouch: an explorative multi-touch information visualization interface for tagged photo collections. In Proceedings of the 5th Nordic Conference on Human-Computer Interaction (NordiCHI '08). ACM Press: 491-494.
Gesture keyboard Tech (SHARK/ShapeWriter)
(with Shumin Zhai)
This technology enables users to write quickly on their mobile phones by sliding or swiping the finger over a touch-sensitive on-screen keyboard. For instance, to write the word "the" the user pushes down the finger on the T key, slides to the H and E keys, and then lifts up the finger on the E key. This input paradigm recognizes the shape of this gesture using a pattern recognizer. During practice, the shapes of words builds up in users' muscle memory which enables users to quickly recall the shapes for words without looking much at the keyboard (similar to how you remember your ATM code). The basic research was first published in 2002 and 2003. In 2004 we released a free public release (called SHARK Shorthand) that was available for download on the IBM alphaWorks website and hailed as "text entry epiphany" by jkOnTheRun, a top ten tech blog. In 2007 we founded ShapeWriter, Inc. to commercialize the technology. We made several public releases of ShapeWriter software for iPhone, Android and Windows Mobile devices. In 2010 the company was acquired by Nuance Communications, Inc.
Tech start-up:
- ShapeWriter, Inc. I co-founded this company in 2007 and was its Director of Engineering in 2007-2010. Acquired by Nuance Communications, Inc. in 2010.
Press / media:
- More than 100 international press articles in newspaper and magazines such as BBC World News, Die Zeit, The New York Times and San Jose Mercury News. See this list of international press articles. Please note that some of the articles given in the link are about other research projects I have been involved in.
Most important research papers:
- Kristensson, P.O. and Zhai, S. 2007. Command strokes with and without preview: using pen gestures on keyboard for command selection. In Proceedings of the 25th ACM Conference on Human Factors in Computing Systems (CHI '07). ACM Press: 1137-1146.
- Kristensson, P.O. and Zhai, S. 2004. SHARK2: a large vocabulary shorthand writing system for pen-based computers. In Proceedings of the 17th Annual ACM Symposium on User Interface Software and Technology (UIST '04). ACM Press: 43-52.
- Zhai, S. and Kristensson, P.O. 2003. Shorthand writing on stylus keyboard. In Proceedings of the 21st ACM Conference on Human Factors in Computing Systems (CHI '03). ACM Press: 97-104.