Lecture 21 Multi-Touch

In this short chapter, you will practice working with touch interaction by implementing support for multi-touch gestures, or the ability to detect two or more different “contacts” (fingers) independently. Multi-touch is actually a pretty awesome interaction mode; it’s very “sci-fi”.

Specifically, you will be implementing an simple animated app that graphically tracks the location of all 5 of your fingers.

The code for this tutorial can be found at https://github.com/info448/lab-multitouch. Note that the starter code builds on the example presented in Lecture 15.

The emulator doesn’t support multi-touch, so you will need to run this project on a physical device.

21.1 Identifying Fingers

Android lets you react to multiple touches by responding to ACTION_POINTER_DOWN events, which occur when a second “pointer” (finger) joins the gesture (after the ACTION_DOWN event).

  • The first finger starts the gesture with an ACTION_DOWN event. then subsequent fingers produce ACTION_POINTER_DOWN events.

  • Similarly, there are ACTION_POINTER_UP events for removing fingers, until the last finger is removed which causes the ACTION_UP event.

Practice: In MainActivity.java, add further cases to the onTouchEvent() callback and log out when subsequent fingers are placed and lifted.

Here the tricky part: each finger that is currently “down” can cause events independently. That is, if we move a finger, then an ACTION_MOVE event will occur. Similarly, if we remove a finger, then an ACTION_POINTER_UP event will occur. So the question is, how do we know which finger caused the event?

Underneath the hood, pointers are stored in a list (think: an ArrayList), so each has a pointer index (a number representing their index in that list). But these indices can change as you interact with the device. For example, lifting up a finger will could cause that pointer to be removed from the list, thus moving all of the other pointers up an index. In fact, the index is allowed to change between each event—while they often stay in order, there is no assurance that they will. The exact behavior of these indices is not specified or enforced by the framework, so we need to treat those values as unstable and cannot use them to “track” particular fingers.

However, each pointer that comes down is assigned a consistent pointer id number that we can refer to it by. This id will be associated with that finger for the length of time that contact is being made. In general, the first finger down will be id 0, and no matter what happens to the list order that pointer’s id will stay the same.

Practice: track pointer ids using the following procedure:

  1. When a Touch event occurs, determine the pointer index which caused the event. Do this by calling the getActionIndex() method on the MotionEvent object. Note that this only actually applies to POINTER_DOWN or POINTER_UP events, otherwise it will just return 0 (for the “main finger” of the event).

  2. Get the unique pointer id for whatever finger caused the event. Do this by calling the getPointerId(pointerIndex) method on the MotionEvent. This will give you the unique id associated with the event’s finger.

21.2 Drawing Touches

Once you know which pointer has gone up and down, you can respond to it by modifying the drawing displayed by the App. Add the falling functionality to the the DrawingSurfaceView:

  • Add an instance variable touches that is a HashMap mapping pointer ids (Integers) to Ball objects. This will track a single “ball” for each touch.

  • Add a method addTouch() that takes in a pointer id as well as the x,y coordinates of the touch point. This method should add a new Ball (at the given coordinates) to the touches map (with the given pointer id).

    • Because this method will need to work across threads in the DrawingSurfaceView, you should make sure the method is synchronized (specifying that keyword in the method signature).

    • Call this method on the drawing View from MainActivity when a new finger is put down—including the first finger! This may be from two different types of events.

      • Pass the pointer index as a parameter to the getX() and getY() methods to determine the coordinates of that particular pointer!

  • Add a method removeTouch() that takes in a pointer id, and removes the Ball that corresponds to that touch.

    • This method should also be synchronized.

    • Call this method on the drawing View from MainActivity when a finger is lifted—including the last finger!

  • Modify the render() method so that the View draws each of the Ball objects in the HashMap at their stored location. You can use gold paint for this. Recall that you can get an iterable sequence of the values for a HashMap using the .values() method.

This should cause your app to show a small ball underneath each finger that goes down, with the balls disappearing when a finger is lifted. Make sure each ball is big enough to see!

21.3 Moving Fingers

Now we just need to handle finger movements. With a MOVE action, the event doesn’t track which finger has moved—instead, the pointer index of that action is always going to be 0 (the main pointer), even if a different finger moved!

However, the event does include information about how many pointers are involved in it (that is: how many pointers are currently down): we can get that number with the MotionEvent#getPointerCount() method. We don’t know which pointer index each finger has, but we do know that they will be consecutive indices (because they are stored in a list). Moreover—as was the case previously—each pointer will have its own x and y coordinates, representing the current position of that pointer (this may or may not have “moved” from the previous event).

Thus we can just loop through all of the pointer indices and get the pointer id for each one. We can then specify that we want the corresponding Ball to update its position to match the “current” pointer position. Again, most of the Balls will not have moved, but we know at least one of them did and so we will just update everything to make sure it works!

  • Add a method moveTouch() to the drawing View that takes in a pointer id (e.g., Ball id), and the “latest” x,y coordinates for that Ball. Update the appropriate Ball’s position to reflect these latest coordinates (use .get() to access a HashMap value by its key).

    • This method should again be synchronized.

  • In MainActivity, when a MOVE event occurs, loop through all of the pointer indices in the event. Get the pointer id and x,y coordinates of each, and use those to call the moveTouch() method on the drawing View. You will be “moving” all of the balls, even if most are just moving to the same place they were.

And with that, you should have multi-touch tracking! Try adding and removing fingers in unique orders, moving them around, and make sure that the Balls follow the contacts.

  • Note that tracking individual ids in this way is more commonly used to make sure you’re ignoring extra multiple touches. See the docs or this developer blog post for details.

21.4 Other Multi-Touch Gestures

We can respond to common multi-touch gestures (like “pinch to scale”) by using another kind of GestureDetector called a ScaleGestureDetector. As before, we subclass the simple version (ScaleGestureDetector.SimpleOnScaleGestureListener), and fill in the onScale() method. You can get the “scale factor” from the gesture with the .getScaleFactor() method. As a further bonus exercise, you might try to use the gesture to scale the size of a single ball.