Ning Zhang's Blog

This post is a modification of MSDN articles Multitouch Input and Gesture Support for Windows Phone, as a quick overview and reference for myself on Silverlight touch support.

Overview

Below class diagram shows key classes for Silverlight touch support:

• Touch
  • Touch events are asynchronous.
  • Touch events are always fired via application level Touch.FrameReported event.
  • Since touch events are global, they are direct, and there is no need for touch capture.
  • Multitouch is supported.
• Manipulation
  • Manipulation events are exposed by UIElement. They are routed events.
  • Manipulation events are always fired, IsManipulationEnabled property isn’t supported.
  • Manipulation events are synchronous.
  • Only translation and expansion are supported, rotation is not.
  • Concurrent multiple manipulations are not supported.
  • Gesture isn’t supported directly. It is supported by XNA and Silverlight for Windows Phone Toolkit.
• Controls: on Windows Phone 7, ScrollViewer supports pan & flick, Button supports tab.

Silverlight 3

Platform Requirements

Multitouch requires an environment (device; platform and operating system; hosting application such as a browser) that can propagate the touch input to an individual application, such as your Silverlight-based application.

Windows 7 supports multitouch input at the operating system level. This is supported in part through a message (WM_TOUCH). Already at this level, the operating system provides a promotion of multitouch messages to mouse messages. The promotion is present so that multitouch users can use touch and gestures to substitute as mouse moves or mouse clicks. This is useful when interacting with applications that may not be touch-aware, and the application does all its spatial input processing through mouse events and messages. Windows 7 also coalesces the messages when appropriate, so that applications do not have to process an overflow of intermediate messages that all generate incremental events.

Internet Explorer version 8 as a browser host is also multitouch aware. Internet Explorer version 8 forwards platform multitouch messages to plug-ins such as Silverlight that are running within Internet Explorer, such that Silverlight applications can interact with multitouch input.

Multitouch Input for Silverlight (shown for IE8 host)

Multitouch is also supported for Silverlight in current versions of Firefox hosts running on Windows 7, and for out-of-browser applications running on Windows 7. However, multitouch input is not supported for applications running in full-screen mode.

Registering for Multitouch

As part of the broader platform architecture for multitouch, each application that wants to receive multitouch messages must register its HWND (Touch API for Windows 7 includes RegisterTouchWindow for this purpose). The Silverlight 4 run time takes care of this registration step, and registers Silverlight as a runtime and all applications that use Silverlight as the runtime. Therefore it is not generally necessary to interact directly with platform code to process multitouch input. However, the characteristics of multitouch interaction within Touch API for Windows 7 and Silverlight 4 multitouch are fairly specific:

• Silverlight 4 is registered for raw touch input, not gestures. If your requirements include gestures, you have to process touch input into gestures using your own application code, within the context of Silverlight. Alternatively, you may need a larger interoperation design so that you can include a separate HWND that is multitouch registered for gestures from the platform, and interoperates with a Silverlight content area.

• In general, Silverlight 4 promotes raw touch input to mouse events. (However, on a per-touchframe basis, you can disable promotion, as is described in upcoming sections of this topic.)

• Within a host, certain gestures might be promoted by the browser host to become events other than mousedown/mousemove/mouseup.

Promotion to Mouse Events

Mouse event promotion exists so that multitouch users can use touch and gestures to substitute as mouse moves or mouse clicks. Conceptually this is the default, because applications that predate or do not consider multitouch would not know what API to call to perform the promotion, and therefore the platform does the mouse promotion in most cases. Silverlight 4 perpetuates the general concept of mouse event promotion for much the same reasons. Any given existing Silverlight control might have handlers for mouse events, but not specifically for multitouch events. For example, a button would be expected to act as if clicked when the user used a multitouch device to interact with it.

Mouse event promotion does have the potential for event dualism in cases where there is deliberate handling of a multitouch frame and its touch points. Within the body of the handler for a Silverlight multitouch event, you can suspend mouse event promotion for the duration of the primary touch down. For example, if you wanted to create a touch-aware button that performed different actions based on the touch characteristics, you could suspend the promotion so that your button did not promote to the usual "click" behavior, and instead went to your discrete logic for how to handle the input. To do this, call SuspendMousePromotionUntilTouchUp as one of the first operations of your handler.

The exact nature of mouse promotion is not documented here, because it is a platform characteristic. Generally speaking, the mechanism is a message-to-message promotion.

Touch to Gesture

Silverlight processes multitouch messages at the level of the raw message, analogous to the platform WM_TOUCH, along with access to other APIs that can capture touchpoint characteristics at that raw level that are then exposed as Silverlight APIs. Silverlight does not natively process touch to gestures, use the platform capabilities to that effect, or process WM_GESTURE. (Processing WM_GESTURE would require a registration state with the platform that Silverlight 4 does not opt into.)

If you want to process multitouch using the gesture metaphor, your code must handle the touch events and use the API exposed under Silverlight 4 and process into gestures, with or without using the platform API for gestures. This is not trivial.

Touch API

One important difference between multitouch input and other input techniques supported in Silverlight (mouse, keyboard, stylus) is that you register for multitouch events on an application-wide basis, not by adding handlers to specific input elements (UIElement objects). This is consistent with the metaphor that Silverlight as a whole is the "application" registered with the platform.

• To assign a multitouch event handler, you assign a handler for the static event Touch.FrameReported. System.Windows.Input.Touch is a static service class that exists solely for this purpose with Touch.FrameReported as its only API.

• The handler you write for Touch.FrameReported is based on the TouchFrameEventHandler delegate.

• In a typical UI design, you might have areas of the UI that you intend to support specific multitouch actions within, and other areas where it would be better to use mouse promotion and not necessarily process the input as multitouch. To determine where the primary touch point is, you may have to evaluate the overall coordinate against the location of your multitouch-aware element, and its bounds. See GetPrimaryTouchPoint for more information and example code.

• As mentioned previously, you may want to suspend mouse event promotion as part of your TouchFrameEventHandler logic. To do this, call SuspendMousePromotionUntilTouchUp as one of the first operations.

• Touch messages as reported to Silverlight 4 are typically combined as frames, which start with a primary "down" touch point. Sometimes you are only interested in the first touch point and the first "up" But the frame may contain other touch points and "move" actions. To access the full collection of points in a frame in your handlers, call GetTouchPoints. For a given touch point, probably the most important information is its Position.

• Other APIs expose information that in the platform API would be found in the TOUCHINPUT structure. Examples of such API are:TouchFrameEventArgs.Timestamp; TouchDevice.DirectlyOver; TouchPoint.Size; TouchPoint.TouchDevice. Depending on your scenario, you might not always need this level of information.

Manipulation API in Silverlight Version 4

The Silverlight 4 UIElement class has several events related to manipulations, as well as event support classes such as ManipulationStartedEventArgs. However, these APIs and the related manipulation concepts do not have full support under Silverlight 4 at run time. To handle these events, your application must be targeting Silverlight for Windows Phone. The events exist in the Silverlight 4 assemblies in order to provide common designer support for both Silverlight 4 and Silverlight for Windows Phone.

Silverlight for Windows Phone 7

Gesture Support in the Silverlight Framework

For a full list of supported Windows Phone Silverlight gestures, see the UI Design and Interaction Guide.

Silverlight for Windows Phone allows you to process touch input by using manipulation events. By using these events, you can move and scale objects in response to touch and multi-touch input. The events are described in the following table.

Note:

Silverlight controls that are supported on Windows Phone are gesture-aware, and support gestures such as tap, pan, and flick. You can handle simple gestures such as tap, double-tap, and tap-and-hold by using mouse events. For more information, see the Input for Windows Phone topic.

Gesture Support in the XNA Framework

The XNA framework offers a robust touch gesture system for developing applications. Developers can leverage this built-in gesture system instead of implementing one from scratch. The following XNA gestures are supported in Windows Phone.

The following list provides useful topics and articles for implementing gesture support in an XNA application:

• Working with Touch Input (Windows Phone)

• Detecting Gestures on a Multi-touch Screen (Windows Phone)

• Windows Phone 7 Gestures Compared