Mouse support

[itsdax]

Any way to include mouse on screen grab?

---

Edit from @BoboTiG:

• Linux support done with Linux: added mouse support (partially fixes #55) #232 in v8.0.0.
• Mac support to be contributed.
• Windows support done with Windows: capture cursor #272 in vXXX.

Upvote & Fund

• We're using Polar.sh so you can upvote and help fund this issue.
• We receive the funding once the issue is completed & confirmed by you.
• Thank you in advance for helping prioritize & fund our backlog.

[BoboTiG]

As of now, no.
But it may be an interesting feature to add.

[jamespreed]

As an interim solution, could you capture the mouse position on screen and add it to the .png file metadata on write out?

A good example of getting the mouse position is given in the 2nd answer here:
https://stackoverflow.com/questions/3698635/getting-cursor-position-in-python

[BoboTiG]

It could be a good start.

Perhaps adding CLI arguments like --with-cursor and --cursor-file=FILE.
Add adding keywords/attributes with_cursor=False and cursor_file='' to the MSS class.

I think it can be interesting to merge the cursor into the pixels directly and not only in the final PNG.

What do ou think @itsdax and @jamespreed ?

[itsdax]

Yes! That sounds like the right approach.

In Java, there's a method for getting the cursor's state (text hover, resize, move, etc)
java.awt.Cursor.getType()

If there's a similar one in python, it can be used to trigger different icons.

[jamespreed]

Integrating a cursor directly into the pixels would be great. You could have several choices: cursor, crosshair, or concentric circles.

[BoboTiG]

FI we can find some clues from the pyinput module.

[BoboTiG]

This is how we could do it for GNU/Linux:
https://github.com/MaartenBaert/ssr/blob/master/src/AV/Input/X11Input.cpp

[skjerns]

I've written a solution for Windows.

The script gets the current mouse cursor icon to bitmap and adds it to the screenshot. It's very fast, so no performance problems.

However, I'm not quite sure how to implement it. The dependencies to win32gui can be exchanged for call to cTypes for sure. Maybe someone else can add the code in a beautiful fashion?

import numpy as np
import win32gui, win32ui
import mss
from PIL import Image

def set_pixel(img, w, x, y, rgb=(0,0,0)):
    """
    Set a pixel in a, RGB byte array
    """
    pos = (x*w + y)*3
    if pos>=len(img):return img # avoid setting pixel outside of frame
    img[pos:pos+3] = rgb
    return img

def add_mouse(img, w):
    flags, hcursor, (cx,cy) = win32gui.GetCursorInfo()
    cursor = get_cursor(hcursor)
    cursor_mean = cursor.mean(-1)
    where = np.where(cursor_mean>0)
    for x, y in zip(where[0], where[1]):
        rgb = [x for x in cursor[x,y]]
        img = set_pixel(img, w, x+cy, y+cx, rgb=rgb)
    return img


def get_cursor(hcursor):
    info = win32gui.GetCursorInfo()
    hdc = win32ui.CreateDCFromHandle(win32gui.GetDC(0))
    hbmp = win32ui.CreateBitmap()
    hbmp.CreateCompatibleBitmap(hdc, 36, 36)
    hdc = hdc.CreateCompatibleDC()
    hdc.SelectObject(hbmp)
    hdc.DrawIcon((0,0), hcursor)
    
    bmpinfo = hbmp.GetInfo()
    bmpbytes = hbmp.GetBitmapBits()
    bmpstr = hbmp.GetBitmapBits(True)
    im = np.array(Image.frombuffer(
        'RGB',
         (bmpinfo['bmWidth'], bmpinfo['bmHeight']),
         bmpstr, 'raw', 'BGRX', 0, 1))
    
    win32gui.DestroyIcon(hcursor)    
    win32gui.DeleteObject(hbmp.GetHandle())
    hdc.DeleteDC()
    return im

with mss.mss() as sct:
    screen = sct.monitors[0]
    img = bytearray(sct.grab(screen).rgb)
    img_with_mouse = add_mouse(img, screen['width'])

[matanox]

Maybe the best reference for linux would be ffmpeg, where mouse capture is implemented as part of screen grabbing (taking a screenshot).

For just getting the mouse coordinates at the time of making a screenshot, using python-xlib:

from Xlib import display
pointer = display.Display().screen().root.query_pointer()
position = (pointer.root_x, pointer.root_y)

But then you have to draw a cursor yourself into the screen grab, and you do not know the actual cursor image used by the display at the particular moment. Cursor rendering is its own little kingdom in X11 ...

[matanox]

Here is how ffmpeg goes about it. It may seem however, that python-xlib does not expose the xfixes API as of now, nor does ffmpeg-python expose that ffmpeg function retrieving the pointer image.

[matanox]

It should be possible to use ctypes to access the XFixesGetCursorImage function from libXfixes.so to get back a XFixesCursorImage C structure like here, but in python.

See also the structure of the image struct here or in the original X source ...

[zorvios]

I found a solution but it depends on GTK2 for now and it is written in python2
using x.py and xlib.py from http://code.google.com/p/pyxlib-ctypes/
here is a working exemple https://github.com/zorvios/X11CursorImagePy2

[zorvios]

I've written a working module for linux, that get the cursor image using Ctypes :
https://github.com/zorvios/PyXCursor
is there anything we could do to add this feature ?

[thiagoribeirodamotta]

Just to add my two cents: When trying to display the mouse via the "grab mouse position on screen and paste mouse icon" method, how to go about scenarios where an application uses custom mouse icons? Would one need to search for each application and see if it uses a custom mouse? Some applications even have more mouse states (or different ones) than OSs does. I know ffmpeg is agnostic to that issue, but don't know how the library does it.

[amadeok]

if using PIL for processing a workaround could be the one here:
https://github.com/swharden/pyScreenCapture/blob/master/go.py

[BoboTiG]

v8.0.0 added Linux support. Thanks to @zorvios it introduces all bricks needed for upcoming Mac, and Windows, supports (like MSSBase._merge(screenshot, cursor)).

[basket-ball]

with_cursor=True is only supported in linux. not in windows. Can it be supported in windows? thx

[BoboTiG]

Windows support is ongoing with #272.

[tamnguyenvan]

@BoboTiG , I'd like to suggest a solution.

• Linux:

from Xlib import display
import numpy as np
from Xlib.ext import xfixes

d = display.Display()
if not d.has_extension('XFIXES'):
    logger.error('XFIXES extension not supported.')
    return

xfixes_version = d.xfixes_query_version()

root = d.screen().root

image = d.xfixes_get_cursor_image(root)
cursor_image = image.cursor_image
width, height = image.width, image.height

cursor_data = np.array(cursor_image, dtype=np.uint32).reshape(height, width)

bgra = np.zeros((height, width, 4), dtype=np.uint8)
bgra[..., 0] = cursor_data & 0xFF          # Blue
bgra[..., 1] = (cursor_data >> 8) & 0xFF   # Green
bgra[..., 2] = (cursor_data >> 16) & 0xFF  # Red
bgra[..., 3] = (cursor_data >> 24) & 0xFF  # Alpha

• Windows:

import win32gui
import win32con
cursor_info = win32gui.GetCursorInfo()
cursor_handle = cursor_info[1]

# Define a dictionary mapping cursor handles to their states
cursor_states = {
    win32gui.LoadCursor(0, win32con.IDC_ARROW): "arrow",
    win32gui.LoadCursor(0, win32con.IDC_IBEAM): "ibeam",
    win32gui.LoadCursor(0, win32con.IDC_WAIT): "wait",
    win32gui.LoadCursor(0, win32con.IDC_CROSS): "cross",
    win32gui.LoadCursor(0, win32con.IDC_UPARROW): "uparrow",
    win32gui.LoadCursor(0, win32con.IDC_SIZENWSE): "sizenwse",
    win32gui.LoadCursor(0, win32con.IDC_SIZENESW): "sizenesw",
    win32gui.LoadCursor(0, win32con.IDC_SIZEWE): "sizewe",
    win32gui.LoadCursor(0, win32con.IDC_SIZENS): "sizens",
    win32gui.LoadCursor(0, win32con.IDC_SIZEALL): "sizeall",
    win32gui.LoadCursor(0, win32con.IDC_NO): "no",
    win32gui.LoadCursor(0, win32con.IDC_HAND): "hand",
    win32gui.LoadCursor(0, win32con.IDC_APPSTARTING): "appstarting",
    win32gui.LoadCursor(0, win32con.IDC_HELP): "help",
}
state = cursor_states.get(cursor_handle, "arrow")

# We can now use the state to load the appropriate cursor image for rendering

• macOS:

import AppKit
from Cocoa import NSBitmapImageRep, NSPNGFileType
import io
from PIL import Image
import cv2
import numpy as np

# Get current cursor
cursor = AppKit.NSCursor.currentSystemCursor()
image = cursor.image()
size = image.size()
width, height = int(size.width), int(size.height)
bitmap_rep = NSBitmapImageRep.imageRepWithData_(image.TIFFRepresentation())

png_data = bitmap_rep.representationUsingType_properties_(NSPNGFileType, None)

buffer = io.BytesIO(png_data)
img_array = Image.open(buffer)
rgba = np.array(img_array)
bgra = cv2.cvtColor(rgba, cv2.COLOR_RGBA2BGRA)

I have implemented it in my project, and it has worked perfectly.
Refs: https://github.com/tamnguyenvan/screenvivid/blob/main/screenvivid/models/utils/cursor/loader.py