Within the last twelve to eighteen months, I’ve crossed a threshold whereby the vast majority of my work is now aimed at touch-screen devices. I often have to sketch out feature specs, interaction designs and so forth, and I enjoy working on paper whenever I can. I quickly encountered a problem: touch-screen gestures are difficult to describe concisely. To solve this problem, I created a means of talking about such gestures symbolically; I call it Touch Notation.
There are three primary problems I face when talking about multi-touch gestures in documents:
- Elaborateness. To visually describe a gesture, it’s fairly common to see actual life-like drawings of fingers and hands; that’s far too much trouble to sketch, and takes up too much room when reproduced. It also doesn’t scale to small sizes.
- Verbosity. To instead textually describe a gesture, disproportionately many words are required: “double-tap-and-hold the object then drag to the right”. That’s tiring, and again takes up far too much room.
- Ambiguity. There’s no absolutely standard mapping of terminology to gestures, so we’re faced with deciding whether we really want “swipe”, “flick” or something else, and whether the term will be understood later.
Such situations are usually helped by creating a formal terminology or notation of some kind, and indeed we all find ourselves doing so automatically and organically in our own notes. My own notation has developed over the course of a few months, and has stabilised sufficiently to be worth sharing; I find it very useful. Touch Notation’s purpose is to help me talk about and design features and interactions for touch-screen (and usually multi-touch) software.
That’s not all it’s useful for, though – I also use it to display gestural input in a concise, reference-suitable way. This usage is analogous to lists of keyboard shortcuts on the desktop, which make use of standard modifier-key symbols like Shift, Alt, Ctrl etc.
Touch Notation lets me make gestural-input “cheat sheets” for touch-screen apps just as easily as I can make lists of keyboard shortcuts for desktop apps. The goal is to provide one possible way of talking precisely and accurately (I hesitate to quite say “scientifically”) about touch-screen interaction. I thought it might be of interest to you too, so I’ve decided to share it.
Design Priorities
When developing the notation, my priorities were:
- It should be (relatively) easy to sketch quickly, and definitely easier than textual descriptions or finger-drawings.
- It should be scaleable, so it can be reproduced at any size (particularly at small sizes).
- Each symbol should be unambiguous within the set.
- We should prefer using the simplest, strongest forms for the most common gestures.
- All symbols should fit within constant vertical space, and have similar vertical visual weight.
These are simple, utilitarian and common-sense constraints, with an emphasis on making something that’s usable rather than conceptually perfect (which is almost always the right approach for anything).
Fingers vs Gestures
Before I list the symbols, I want to briefly mention the number of fingers issue. I’ve seen plenty of sketches and icons where, for example, a two-finger swipe-right is represented as two right-pointing arrows, one above the other. That’s fine in itself, but it completely breaks scaleability at small sizes – particularly if you’re requiring a four-finger swipe for some reason.
I wanted to avoid that problem with Touch Notation, so my approach was to separate the number of fingers from the gesture. After all, a tap is conceptually the same “gesture” whether you’re tapping with one finger or with two. I specify the number of fingers first (which is optional for the most common case of only one finger), then the actual gesture (with a gap between the two sections).
There are a few alternate notations I’ve experimented with for showing the number of fingers:
- Number Only. This is the one I actually use on a day-to-day basis, because it’s as simple as writing/typing a number.
- Number and Symbol. As with Number Only, but followed by a “Finger” symbol (shown below in the full list of symbols). Since the Finger symbol is a sort of vertical lozenge shape, you’d likely want to distinguish this from (say) the numeral 1 by using colour or some such thing.
- Multiple Symbols. No number, and instead repeating the Finger symbol the appropriate number of times.
It really depends what you find most useful or pleasing. There’s actually less visual ambiguity from the Finger symbol than you might think, since (for example) confusing it for the numeral 1 doesn’t change its meaning in isolation, and in Number and Symbol style you’re unlikely to think it’s an additional numeral 1 after the actual number because it’s very uncommon indeed to require a double-digits number of fingers to perform a gesture!
Personally, I find the Number Only style to be the easiest to write, and least visually ambiguous at smaller sizes. I’ll be using it in all the examples below.
Touch Notation Symbols
Here’s the full set of symbols, and their meanings.
Tap
A standard tap (and release).
Tap and Hold
A tap and hold for a moment (without releasing). This gesture is sometimes also called a long tap.
I deliberated for a while about whether this symbol should actually be Tap, and the solid circle be Tap and Hold (the hollow area representing releasing the tap, in that case), but I decided against it since Tap is by far the most common gesture and I wanted it to have as strong and simple a form as possible.
Swipe
A swipe or flick across the screen, usually with no particular origin. The arrow is rotated to indicate the actual direction of the swipe.
Rotate
A rotation gesture, involving at least two fingers.
Pinch Out
An expansion/pinch-out gesture, involving at least two (and usually exactly two) fingers.
It’s worth noting that the Pinch Out/In symbols are horizontal reflections of the fairly-standard “Enter/Exit Full Screen” icons used on Mac OS X and elsewhere. The context of use will remove any ambiguity, and the symbols are too meaningful to pass up on that basis alone.
Pinch In
A pinch (inwards) gesture, involving at least two (and usually exactly two) fingers.
Tap and Drag
A tap and then drag, without releasing and without any explicit pause between tapping and dragging. This is distinct from a Swipe in that a Tap and Drag usually has a specific origin point: the object you’re dragging.
Tap, Hold and Drag
A tap, hold for a moment, and then drag (commonly involving a transition in object-state after the hold period has elapsed).
Finger
The symbol for a finger, which can optionally be used when specifying the number of fingers required for a gesture (as described above).
Examples
Here are a few examples of using Touch Notation to encode various gestures. When I’m creating formal documents using Touch Notation, I tend to use black for the number of fingers and blue for the entire gestural part, just to give a subtle sense of energy and motion to the gesture. You’ll find that the gesture instructions in tutorials in the iPad versions of the iWork apps do the same thing.
- Tap (with 1 finger)
- Double-tap with 3 fingers
- Swipe right with 2 fingers
I find that the best way to read these gestures is to first read the black number, prepare that number of fingers, then follow along with the blue symbols of the gesture: “with this number of fingers, do that“.
Quantity Modifiers
I sometimes want to specify that, say, the user will tap and hold for two seconds, or will rotate for 45 degrees. I use leading or trailing numbers to indicate these situations, with the position of the number depending on the actual affinity of the value to the symbol. There are a couple of examples below.
- Tap and Hold for 2 seconds, then drag
- Tap and Hold (for some default period), then drag for 2 seconds
Sequential and Split Input
By default, since we’re dealing with multi-touch gestures, two consecutive tap symbols (for example) indicate performing two taps quickly, i.e. a double-tap. Sometimes, however, I want to express that the user should tap, wait a moment, then tap again – as two sequential gestures. I use the comma for this, since it has a fairly standard meaning of indicating a brief pause.
- Double Tap
- Tap, then Tap again
In touch-screen gestures, there’s also the (not so well-known) concept of a split tap; this is where you’re performing one action with one hand, and then (simultaneously) perform an action with the other hand, as a single gesture.
This is commonly used in VoiceOver, where for example the user can roll a finger over the keyboard until they hear that they’re arrived at the correct key, then tap a finger of their other hand anywhere on screen to enter that key. Split taps in VoiceOver allow for much easier text-entry where precision is only important for one hand.
I think the general concept of split input is interesting and useful, so I included it in the notation: I use a forward-slash to indicate the split. I don’t consider this to create ambiguity, since I only ever encode one gesture in a notation sequence, so I wouldn’t be using a forward-slash to mean “or”, as it can in English.
- Tap and Hold, then Drag, and then Tap elsewhere with another finger
Download your own set of Touch Notation symbols
I find this notation useful, and you’re welcome to use it too. If you use it anywhere other than in-house, I’d appreciate attribution. I’m offering a Photoshop (.psd) file containing each symbol as a Shape Layer (with all symbols at their original relative sizes), a Photoshop Custom Shapes (.csh) file so you can use Touch Notation in any Photoshop document, and also the original Illustrator (.ai) file I created the symbols in.
You can download Touch Notation here.
License info is included with the download; it’s a Creative Commons Attribution-ShareAlike license.
Oliver Toscan also made an OmniGraffle Stencil of the Touch Notation symbols; you can download it here.
Final Thoughts
I’ve found Touch Notation to be really useful in my own work; I just keep a quick symbol-reference around (though I rarely need to refer to it now) for when I’m sketching, and I’ll occasionally make formal lists of gestures for features electronically using the templates. I find that it has just the right balance of simplicity and expressiveness that I actually do use it.
I hope you find a use for it in your workflow, or at the very least get some value from comparing it to your own notation style; I always welcome any feedback you may have. If you’re feeling grateful/generous, feel free to take a look at my Amazon Wishlist, or the Paypal Donate link on my Cocoa Source Code page.
Remember that you can always keep up to date with what I’m doing by following me (mattgemmell) on Twitter. Now, back to work.
Great stuff. Team is working on a touch interface as we speak and this will be introduced as an option for communication today. Thanks.
Excellent idea!
[...] This post was mentioned on Twitter by Matt Legend Gemmell, James Hall and others. James Hall said: Very Cool Sir, Thanks! > @mattgemmell: Touch Notation: http://mattgemmell.com/touch-notation [...]
Excellent idea, but you have some of it inside out.
The number of fingers must always match the number of circles shown. Put the number of taps inside the circles.
The amount of time is not needed and should not be specified, in the same way that a double-click is just a double-click.
Tap and hold should be a horizontal lozenge, using the concept that time goes left to right. Arrows should show the gesture,as you have, but should always be rooted in circles or lozenges, because those are the fingers.
Hi Steve,
“Excellent idea, but you have some of it inside out.”
Ahem. _In your opinion_, right?
“The number of fingers must always match the number of circles shown. Put the number of taps inside the circles.”
I think that’d be a verbose and visually unusual notation. The concept of splitting the number of fingers from the gesture works really well for conciseness. It’d also be unnecessary duplication to put the number of taps in each circle (you’re never going to ask the user to double-tap with finger 1 whilst triple-tapping with finger 2), and extremely unnecessarily space-consuming to have a gesture-symbol for each finger.
“The amount of time is not needed and should not be specified, in the same way that a double-click is just a double-click.”
On the contrary, it’s often useful to be able to specify a quantity modifier; the times are valid examples, and by no means the only usage.
“Tap and hold should be a horizontal lozenge, using the concept that time goes left to right.”
I considered that as an option, but I think the implied directionality of a lozenge is confusing. Gestures are more inherently spatial than temporal, which this notation reflects.
“Arrows should show the gesture,as you have, but should always be rooted in circles or lozenges, because those are the fingers.”
That would raise an ambiguity between a swipe and a drag, hence I avoided it. Try not to take the notation too (physically) literally. It really does work pretty well here for me.
If you write up your own version, by all means link it from here so I can take a look.
For some reason, my brain keeps wanting to associate number of circles with the number of fingers I need to use, which means that in the case of the three-finger double tap, I read that as three times double tap instead (where if this was the case, you would use the comma).
Not knowing exactly what the number is for or that there are commas for sequential gestures kind of makes it even more difficult to just “get” what the notation is supposed to be. I wonder if there’s a better way to represent that (or, if, we’re just doomed because this *is* a good idea and there’s not much to do about some gestures being fairly complicated to represent intuitively…)
I like where you’re going here. Explaining these symbols once would no doubt be easier than explaining the gestures over and over again.
My only real feedback is that it might make more sense if the pinch gestures had solid dots to indicate where the fingers go down, like the rotate gesture. This would make for a more consistent visual language, and I don’t think the weighting change would be a disadvantage.
Just wanted to highlight some other work in this area:
Kicker and Rachel Glaves’ drawings from December 2008: http://www.kickerstudio.com/blog/2008/12/touchscreen-stencils/
Gestureworks’ icons from November 2009: http://gestureworks.com/about/open-source-gesture-library/
Ron George’s gesturcons from January 2010: http://blog.rongeorge.com/design/gesturcons/
LukeW’s from April 2010: http://www.lukew.com/ff/entry.asp?1071
Very interesting read. Are these applicable on my girlfriend? Oh…
Sure you can, she likes it. And is actually mentioning that she wishes you would more often. Oh …
I read the top part of the article and then overroad your stylesheet to hide all the text. I thought it’d be fun to guess. Here is what I guess for the symbols:
3…
3 presses
fingers?
touch and hold
touch
swipe
rotate with two fingers
pinch out
pinch in
tap hold & swipe
tap swipe
finger..?
I gave up on the combinations. At this point I could tell that I had guessed some things wrong.
The biggest one for me was “tap and hold”. To me the inner section being blank looks like an unpressed button while the fully filled in looks like a depressed button. The three fingers were also confusing but with some context it makes sense.
I don’t know if this means anything to you but I enjoyed myself haha.
Very interesting and complete thinking about touch notation.
Here is the touch notation I choose for my app (third picture) : http://www.mancingdolecules.com/everyday-looper/screenshots/
Main difference is that I display 3 circles for a single tap with 3 fingers. Mostly because the app is multi-finger centric and not multi-tap centric, this way I spared the numbers.
Nice, well written article on a strange topic. I did some multitouch design myself (for a couple of years now but never on the iPad), and I would still consider anything beyond click, drag, double click and pinch/zoom/rotate (which is the same thing), to be utterly complex for the majority of users.
But maybe, I just watched too many user-tests with none Mac addicted participants. On the iPad and MacBooks, gestures introduced by the OS are probable learned by some.
Great Idea! I made a version for Omni Graffle based on your work. You can download it here…
http://www.lighthouseinmypocket.com/touchnotation.zip
Hope you like it.
[...] This post was mentioned on Twitter by Matt Legend Gemmell and Martin Dufort, Bjoern Knafla. Bjoern Knafla said: RT @mattgemmell: Touch Notation for OmniGraffle, contributed by a reader: http://tinyurl.com/39zhhv3 [...]
Excellent idea and contribution to the field. Thanks!
This is fantastic work, Matt. Will be really helpful to me to have a notation system when we start working on iPad projects at Balsamiq.
I created a Balsamiq Mockups version of the file as well that Mockups users can download for free.
http://mockupstogo.net/touch-notation
Links back to this page, as well as the CC license are included as well as attribution to you. Thanks for sharing this!
One thing I worry about is the layer of abstraction that gets added into systems like this. My personal preference is to show actual hands as I think it is more intuitive.
http://www.gesturecons.com
We should formalize such notations and used it as a “language” for developers and users.
Thanks for sharing the OmniGraffle template!
As with several others, the numbers immediately suggested times (# of times to tap) rather than # of fingers, and the dots/circles suggested fingertips (& the # thereof) rather than # of taps. Thus, I would suggest putting the dot(s) first, followed by the number: “solid-dot 3″ = single-finger triple-tap; “three circles” = three fingers holding.
It’d be great if you could upload CS4 compatible versions of the files included.
Otherwise, looking good. Thanks for sharing your thought process and the result.
Regards,
DS
Have you checked this? http://www.lukew.com/ff/entry.asp?1071
[...] Touch Notation » Matt Legend Gemmell – Nice simple notation for annotating touch screen designs with gesture information. [...]
[...] Touch Notation by Matt Legend Gemmell Matt Gemell создал простую и понятную нотацию для проектирования тач-интерфейсов. Она позволяет описывать сложные жесты без перегруженных служебных иллюстраций и легко масштабируется до небольших размеров. Прямая ссылка на скачивание — http://mattgemmell.com/files/touch_notation.zip (ZIP, 200 КБ). [...]
[...] Touch Notation » Matt Legend Gemmell (tags: symbols ui usability gestures interface ipad iphone multitouch notation touchscreen) [...]
[...] Template (.ai & .eps) Creative Commons Attribution-Share Alike 3.0 United States License”>Touch Notation (.ai) Creative Commons Attribution-ShareAlike license”>Nokia S60 Wireframing Stencils (.ai)Windows [...]