*C Game Dev Mouse Raw Input Or Direct Input System
*Raw_input Python-->
At the lowest layer you have raw input events from the OS. SDL keyboard input, mouse input, joystick input, etc. You might have several platforms (SDL is a least-common-denominator lacking several input forms, for instance, which you might later care about).
This section describes how the system provides mouse input to your application and how the application receives and processes that input.In this sectionTopicDescriptionAbout Mouse InputThis topic discusses mouse input.Using Mouse InputThis section covers tasks associated with mouse input.Mouse Input ReferenceFunctionsNameDescription_TrackMouseEventPosts messages when the mouse pointer leaves a window or hovers over a window for a specified amount of time. This function calls TrackMouseEvent if it exists, otherwise it emulates it.DragDetectCaptures the mouse and tracks its movement until the user releases the left button, presses the ESC key, or moves the mouse outside the drag rectangle around the specified point. The width and height of the drag rectangle are specified by the SM_CXDRAG and SM_CYDRAG values returned by the GetSystemMetrics function.EnableMouseInPointerEnables the mouse to act as a pointing device. GetCaptureRetrieves a handle to the window (if any) that has captured the mouse. Only one window at a time can capture the mouse; this window receives mouse input whether or not the cursor is within its borders. GetDoubleClickTimeRetrieves the current double-click time for the mouse. A double-click is a series of two clicks of the mouse button, the second occurring within a specified time after the first. The double-click time is the maximum number of milliseconds that may occur between the first and second click of a double-click. GetMouseMovePointsExRetrieves a history of up to 64 previous coordinates of the mouse or pen.ReleaseCaptureReleases the mouse capture from a window in the current thread and restores normal mouse input processing. A window that has captured the mouse receives all mouse input, regardless of the position of the cursor, except when a mouse button is clicked while the cursor hot spot is in the window of another thread. SetCaptureSets the mouse capture to the specified window belonging to the current thread. SetCapture captures mouse input either when the mouse is over the capturing window, or when the mouse button was pressed while the mouse was over the capturing window and the button is still down. Only one window at a time can capture the mouse. If the mouse cursor is over a window created by another thread, the system will direct mouse input to the specified window only if a mouse button is down.SetDoubleClickTimeSets the double-click time for the mouse. A double-click is a series of two clicks of a mouse button, the second occurring within a specified time after the first. The double-click time is the maximum number of milliseconds that may occur between the first and second clicks of a double-click. SwapMouseButtonReverses or restores the meaning of the left and right mouse buttons. TrackMouseEventPosts messages when the mouse pointer leaves a window or hovers over a window for a specified amount of time.
The following function is obsolete.FunctionDescriptionmouse_eventSynthesizes mouse motion and button clicks.NotificationsNameDescriptionWM_CAPTURECHANGEDSent to the window that is losing the mouse capture.WM_LBUTTONDBLCLKPosted when the user double-clicks the left mouse button while the cursor is in the client area of a window. If the mouse is not captured, the message is posted to the window beneath the cursor. Otherwise, the message is posted to the window that has captured the mouse.WM_LBUTTONDOWNPosted when the user presses the left mouse button while the cursor is in the client area of a window. If the mouse is not captured, the message is posted to the window beneath the cursor. Otherwise, the message is posted to the window that has captured the mouse.WM_LBUTTONUPPosted when the user releases the left mouse button while the cursor is in the client area of a window. If the mouse is not captured, the message is posted to the window beneath the cursor. Otherwise, the message is posted to the window that has captured the mouse.WM_MBUTTONDBLCLKPosted when the user double-clicks the middle mouse button while the cursor is in the client area of a window. If the mouse is not captured, the message is posted to the window beneath the cursor. Otherwise, the message is posted to the window that has captured the mouse.WM_MBUTTONDOWNPosted when the user presses the middle mouse button while the cursor is in the client area of a window. If the mouse is not captured, the message is posted to the window beneath the cursor. Otherwise, the message is posted to the window that has captured the mouse.WM_MBUTTONUPPosted when the user releases the middle mouse button while the cursor is in the client area of a window. If the mouse is not captured, the message is posted to the window beneath the cursor. Otherwise, the message is posted to the window that has captured the mouse.WM_MOUSEACTIVATESent when the cursor is in an inactive window and the user presses a mouse button. The parent window receives this message only if the child window passes it to the DefWindowProc function.WM_MOUSEHOVERPosted to a window when the cursor hovers over the client area of the window for the period of time specified in a prior call to TrackMouseEvent.WM_MOUSEHWHEELSent to the focus window when the mouse's horizontal scroll wheel is tilted or rotated. TheDefWindowProcfunction propagates the message to the window's parent. There should be no internal forwarding of the message, because DefWindowProc propagates it up the parent chain until it finds a window that processes it.WM_MOUSELEAVEPosted to a window when the cursor leaves the client area of the window specified in a prior call to TrackMouseEvent.WM_MOUSEMOVEPosted to a window when the cursor moves. If the mouse is not captured, the message is posted to the window that contains the cursor. Otherwise, the message is posted to the window that has captured the mouse.WM_MOUSEWHEELSent to the focus window when the mouse wheel is rotated. The DefWindowProc function propagates the message to the window's parent. There should be no internal forwarding of the message, because DefWindowProc propagates it up the parent chain until it finds a window that processes it.WM_NCHITTESTSent to a window in order to determine what part of the window corresponds to a particular screen coordinate. This can happen, for example, when the cursor moves, when a mouse button is pressed or released, or in response to a call to a function such as WindowFromPoint. If the mouse is not captured, the message is sent to the window beneath the cursor. Otherwise, the message is sent to the window that has captured the mouse.WM_NCLBUTTONDBLCLKPosted when the user double-clicks the left mouse button while the cursor is within the nonclient area of a window. This message is posted to the window that contains the cursor. If a window has captured the mouse, this message is not posted.WM_NCLBUTTONDOWNPosted when the user presses the left mouse button while the cursor is within the nonclient area of a window. This message is posted to the window that contains the cursor. If a window has captured the mouse, this message is not posted.WM_NCLBUTTONUPPosted when the user releases the left mouse button while the cursor is within the nonclient area of a window. This message is posted to the window that contains the cursor. If a window has captured the mouse, this message is not posted.WM_NCMBUTTONDBLCLKPosted when the user double-clicks the middle mouse button while the cursor is within the nonclient area of a window. This message is posted to the window that contains the cursor. If a window has captured the mouse, this message is not posted.WM_NCMBUTTONDOWNPosted when the user presses the middle mouse button while the cursor is within the nonclient area of a window. This message is posted to the window that contains the cursor. If a window has captured the mouse, this message is not posted.WM_NCMBUTTONUPPosted when the user releases the middle mouse button while the cursor is within the nonclient area of a window. This message is posted to the window that contains the cursor. If a window has captured the mouse, this message is not posted.WM_NCMOUSEHOVERPosted to a window when the cursor hovers over the nonclient area of the window for the period of time specified in a prior call toTrackMouseEvent.WM_NCMOUSELEAVEPosted to a window when the cursor leaves the nonclient area of the window specified in a prior call toTrackMouseEvent.WM_NCMOUSEMOVEPosted to a window when the cursor is moved within the nonclient area of the window. This message is posted to the window that contains the cursor. If a window has captured the mouse, this message is not posted.WM_NCRBUTTONDBLCLKPosted when the user double-clicks the right mouse button while the cursor is within the nonclient area of a window. This message is posted to the window that contains the cursor. If a window has captured the mouse, this message is not posted.WM_NCRBUTTONDOWNPosted when the user presses the right mouse button while the cursor is within the nonclient area of a window. This message is posted to the window that contains the cursor. If a window has captured the mouse, this message is not posted.WM_NCRBUTTONUPPosted when the user releases the right mouse button while the cursor is within the nonclient area of a window. This message is posted to the window that contains the cursor. If a window has captured the mouse, this message is not posted.WM_NCXBUTTONDBLCLKPosted when the user double-clicks the first or second X button while the cursor is in the nonclient area of a window. This message is posted to the window that contains the cursor. If a window has captured the mouse, this message is not posted.WM_NCXBUTTONDOWNPosted when the user presses the first or second X button while the cursor is in the nonclient area of a window. This message is posted to the window that contains the cursor. If a window has captured the mouse, this message is not posted.WM_NCXBUTTONUPPosted when the user releases the first or second X button while the cursor is in the nonclient area of a window. This message is posted to the window that contains the cursor. If a window has captured the mouse, this message is not posted.WM_RBUTTONDBLCLKPosted when the user double-clicks the right mouse button while the cursor is in the client area of a window. If the mouse is not captured, the message is posted to the window beneath the cursor. Otherwise, the message is posted to the window that has captured the mouse.WM_RBUTTONDOWNPosted when the user presses the right mouse button while the cursor is in the client area of a window. If the mouse is not captured, the message is posted to the window beneath the cursor. Otherwise, the message is posted to the window that has captured the mouse.WM_RBUTTONUPPosted when the user releases the right mouse button while the cursor is in the client area of a window. If the mouse is not captured, the message is posted to the window beneath the cursor. Otherwise, the message is posted to the window that has captured the mouse.WM_XBUTTONDBLCLKPosted when the user double-clicks the first or second X button while the cursor is in the client area of a window. If the mouse is not captured, the message is posted to the window beneath the cursor. Otherwise, the message is posted to the window that has captured the mouse.WM_XBUTTONDOWNPosted when the user presses the first or second X button while the cursor is in the client area of a window. If the mouse is not captured, the message is posted to the window beneath the cursor. Otherwise, the message is posted to the window that has captured the mouse. WM_XBUTTONUPPosted when the user releases the first or second X button while the cursor is in the client area of a window. If the mouse is not captured, the message is posted to the window beneath the cursor. Otherwise, the message is posted to the window that has captured the mouse.StructuresNameDescriptionMOUSEMOVEPOINTContains information about the mouse's location in screen coordinates.TRACKMOUSEEVENTUsed by the TrackMouseEvent function to track when the mouse pointer leaves a window or hovers over a window for a specified amount of time.-->
Keyboards, mice, and game controllers can all be useful forms of input for Windows Mixed Reality devices. Bluetooth keyboards and mice are both supported on HoloLens, for use with debugging your app or as an alternate form of input. Windows Mixed Reality also supports immersive headsets attached to PCs - where mice, keyboards, and game controllers have historically been the norm.
To use keyboard input on HoloLens, pair a Bluetooth keyboard to your device or use virtual input via the Windows Device Portal. To use keyboard input while wearing a Windows Mixed Reality immersive headset, assign input focus to mixed reality by putting on the device or using the Windows Key + Y keyboard combination. Keep in mind that apps intended for HoloLens must provide functionality without these devices attached.
Note
The code snippets in this article currently demonstrate use of C++/CX rather than C++17-compliant C++/WinRT as used in the C++ holographic project template. The concepts are equivalent for a C++/WinRT project, though you will need to translate the code.Subscribe for CoreWindow input eventsKeyboard input
In the Windows Holographic app template, we include an event handler for keyboard input just like any other UWP app. Your app consumes keyboard input data the same way in Windows Mixed Reality.
From AppView.cpp:Virtual keyboard input
For immersive desktop headsets, you can also support virtual keyboards rendered by Windows over your immersive view. To support this, your app can implement CoreTextEditContext. This lets Windows understand the state of your own app-rendered text boxes, so the virtual keyboard can correctly contribute to the text there.
For more information on implementing CoreTextEditContext support, see the CoreTextEditContext sample.Mouse Input
You can also use mouse input, again via the UWP CoreWindow input event handlers. Here's how to modify the Windows Holographic app template to support mouse clicks in the same way as pressed gestures. After making this modification, a mouse click while wearing an immersive headset device will reposition the cube.
Note that UWP apps can also get raw XY data for the mouse by using the MouseDevice API.
Start by declaring a new OnPointerPressed handler in AppView.h:
In AppView.cpp, add this code to SetWindow:
Then put this definition for OnPointerPressed at the bottom of the file:
The event handler we just added is a pass-through to the template main class. Let's modify the main class to support this pass-through. Add this public method declaration to the header file:
You'll need this private member variable, as well:
Finally, we will update the main class with new logic to support mouse clicks. Start by adding this event handler. Make sure to update the class name:
Now, in the Update method, replace the existing logic for getting a pointer pose with this:
Recompile and redeploy. You should notice that the mouse click will now reposition the cube in your immersive headset - or HoloLens with bluetooth mouse attached.Game controller support
Game controllers can be a fun and convenient way of allowing the user to control an immersive Windows Mixed Reality experience.
The first step in adding support for game controllers to the Windows Holographic app template, is to add the following private member declarations to the header class for your main file:
Initialize gamepad events, and any gamepads that are currently attached, in the constructor for your main class:
Add these event handlers to your main class. Make sure to update the class name:
Finally, update the input logic to recognize changes in controller state. Here, we use the same m_pointerPressed variable discussed in the section above for adding mouse events. Add this to the Update method, just before where it checks for the SpatialPointerPose:
Don't forget to unregister the events when cleaning up the main class:C Game Dev Mouse Raw Input Or Direct Input System
Recompile, and redeploy. You can now attach, or pair, a game controller and use it to reposition the spinning cube.Important guidelines for keyboard and mouse input
There are some key differences in how this code can be used on Microsoft HoloLens – which is a device that relies primarily on natural user input – versus what is available on a Windows Mixed Reality-enabled PC.
*You can’t rely on keyboard or mouse input to be present. All of your app's functionality must work with gaze, gesture, and speech input.
*When a Bluetooth keyboard is attached, it can be helpful to enable keyboard input for any text that your app might ask for. This can be a great supplement for dictation, for example.
*When it comes to designing your app, don’t rely on (for example) WASD and mouse look controls for your game. HoloLens is designed for the user to walk around the room. In this case, the user controls the camera directly. An interface for driving the camera around the room with move/look controls won't provide the same experience.
*Keyboard input can be an excellent way to control the debugging aspects of your app or game engine, especially since the user will not be required to use the keyboard. Wiring it up is the same as you're used to, with CoreWindow event APIs. In this scenario, you might choose to implement a way to configure your app to route keyboard events to a 'debug input only' mode during your debug sessions.
*Bluetooth controllers work as well.Raw_input PythonSee also
