sleap.gui.widgets.video

Module for showing and manipulating skeleton instances within a video.

All interactions should go through QtVideoPlayer.

Example usage:
>>> my_video = Video(...)
>>> my_instance = Instance(...)
>>> vp = QtVideoPlayer(video=my_video)
>>> vp.addInstance(instance=my_instance, color=(r, g, b))
class sleap.gui.widgets.video.GraphicsView(state=None, player=None, *args, **kwargs)[source]

Custom QGraphicsView used by QtVideoPlayer.

This contains elements for display of video and event handlers for zoom and selection of instances in view.

Signals:
  • updatedViewer: Emitted after update to view (e.g., zoom).

    Used internally so we know when to update points for each instance.

  • updatedSelection: Emitted after the user has (un)selected an instance.

  • instanceDoubleClicked: Emitted after an instance is double-clicked.

    Passes the Instance that was double-clicked.

  • areaSelected: Emitted after user selects an area when in “area”

    click mode. Passes x0, y0, x1, y1 for selected box coordinates.

  • pointSelected: Emitted after user clicks a point (in “point” click

    mode.) Passes x, y coordinates of point.

  • leftMouseButtonPressed: Emitted by event handler.

  • rightMouseButtonPressed: Emitted by event handler.

  • leftMouseButtonReleased: Emitted by event handler.

  • rightMouseButtonReleased: Emitted by event handler.

  • leftMouseButtonDoubleClicked: Emitted by event handler.

  • rightMouseButtonDoubleClicked: Emitted by event handler.

property all_instances

Returns a list of all QtInstance objects in scene.

Order should match the order in which instances were added to scene.

clear()[source]

Clears the displayed frame from the scene.

clearZoom()[source]

Clear zoom stack. Doesn’t update display.

static getInstancesBoundingRect(instances: List[QtInstance], margin: float = 0.0) → PySide2.QtCore.QRectF[source]

Return a rectangle containing all instances.

Parameters
  • instances – List of QtInstance objects.

  • margin – Margin for padding the rectangle. Padding is applied equally on all sides.

Returns

The QRectF which contains all of the instances.

Notes

The returned rectangle will be null if the instance list is empty.

getSelectionIndex() → Optional[int][source]

Returns the index of the currently selected instance. If no instance selected, returns None.

getSelectionInstance() → Optional[sleap.instance.Instance][source]

Returns the currently selected instance. If no instance selected, returns None.

getTopInstanceAt(scenePos) → Optional[sleap.instance.Instance][source]

Returns topmost instance at position in scene.

hasImage() → bool[source]

Returns whether or not the scene contains an image pixmap.

property instances

Returns a list of instances.

Order should match the order in which instances were added to scene.

instancesBoundingRect(margin: float = 0) → PySide2.QtCore.QRectF[source]

Returns a rect which contains all displayed skeleton instances.

Parameters

margin – Margin for padding the rect.

Returns

The QRectF which contains the skeleton instances.

keyPressEvent(event)[source]

Custom event hander, disables default QGraphicsView behavior.

keyReleaseEvent(event)[source]

Custom event hander, disables default QGraphicsView behavior.

mouseDoubleClickEvent(event: PySide2.QtGui.QMouseEvent)[source]

Custom event handler, clears zoom.

mouseMoveEvent(self, event: PySide2.QtGui.QMouseEvent)[source]
mousePressEvent(event)[source]

Start mouse pan or zoom mode.

mouseReleaseEvent(event)[source]

Stop mouse pan or zoom mode (apply zoom if valid).

property predicted_instances

Returns a list of predicted instances.

Order should match the order in which instances were added to scene.

resizeEvent(event)[source]

Maintain current zoom on resize.

selectInstance(select: Union[sleap.instance.Instance, int])[source]

Select a particular instance in view.

Parameters

select – Either Instance or index of instance in view.

Returns

None

property selectable_instances

Returns a list of instances which user can select.

Order should match the order in which instances were added to scene.

setImage(image: Union[PySide2.QtGui.QImage, PySide2.QtGui.QPixmap, numpy.ndarray])[source]

Set the scene’s current image pixmap to the input QImage or QPixmap.

Parameters

image – The QPixmap or QImage to display.

Raises

RuntimeError – If the input image is not QImage or QPixmap

Returns

None.

updateViewer()[source]

Apply current zoom.

wheelEvent(event)[source]

Custom event handler. Zoom in/out based on scroll wheel change.

zoomToRect(zoom_rect: PySide2.QtCore.QRectF)[source]

Method to zoom scene to a given rectangle.

The rect can either be given relative to the current zoom (this is useful if it’s the rect drawn by user) or it can be given in absolute coordinates for displayed frame.

Parameters

zoom_rect – The QRectF to which we want to zoom.

class sleap.gui.widgets.video.LoadImageWorker(*args, **kwargs)[source]

Object to load video frames in background thread.

Requests to load a frame image are sent by calling the request method with the frame idx; the video attribute should already be set to the correct video.

These requests are added to a FILO queue polled by the doProcessing method, called whenever there’s time during the Qt event loop. (It’s also added to the event queue if it hasn’t been called for a while and we get a request, since the timer doesn’t seem to emit events if the user has been holding down the mouse for a while.)

The actual frame loading is wrapped with a mutex lock so that we only load a single frame at a time; this helps us not get a bunch of older frame requests running concurrently.

Once the frame loads, the QImage is sent via the result signal. (Qt handles the cross-thread communication if we use signals.)

class sleap.gui.widgets.video.QtEdge(parent: PySide2.QtWidgets.QGraphicsObject, player: sleap.gui.widgets.video.QtVideoPlayer, src: sleap.gui.widgets.video.QtNode, dst: sleap.gui.widgets.video.QtNode, show_non_visible: bool = True, *args, **kwargs)[source]

QGraphicsLineItem to handle display of edge between skeleton instance nodes.

Parameters
  • parentQGraphicsObject which will contain this item.

  • src – The QtNode source node for the edge.

  • dst – The QtNode destination node for the edge.

  • show_non_visible – Whether to show “non-visible” nodes/edges.

angle_to(node: sleap.gui.widgets.video.QtNode) → float[source]

Returns the angle from one edge node to the other.

Parameters

node – The node from which we’re measuring the angle.

Returns

Angle (in radians) to the other node.

connected_to(node: sleap.gui.widgets.video.QtNode)[source]

Return the other node along the edge.

Parameters

node – One of the edge’s nodes.

Returns

The other node (or None if edge doesn’t have node).

updateEdge(node: sleap.gui.widgets.video.QtNode)[source]

Updates the visual display of node.

Parameters

node – The node to update.

Returns

None.

class sleap.gui.widgets.video.QtInstance(instance: sleap.instance.Instance = None, player: Optional[sleap.gui.widgets.video.QtVideoPlayer] = None, markerRadius=4, show_non_visible=True, *args, **kwargs)[source]

QGraphicsObject for skeleton instances.

This object stores the data for one skeleton instance and handles the events to manipulate the skeleton within a video frame (i.e., moving, rotating, marking nodes).

It should be instantiated with an Instance and added to the relevant QGraphicsScene.

When instantiated, it creates QtNode, QtEdge, and QtNodeLabel items as children of itself.

Parameters
  • instance – The Instance to show.

  • markerRadius – Radius of nodes.

  • show_non_visible – Whether to show “non-visible” nodes/edges.

boundingRect()[source]

Method required Qt to determine bounding rect for item.

getPointsBoundingRect() → PySide2.QtCore.QRectF[source]

Returns a rect which contains all the nodes in the skeleton.

paint(painter, option, widget=None)[source]

Method required by Qt.

property selected

Whether instance is selected.

showEdges(show: bool)[source]

Draws/hides the edges for this skeleton instance.

Parameters

show – Show edges if True, hide them otherwise.

showLabels(show: bool)[source]

Draws/hides the labels for this skeleton instance.

Parameters

show – Show labels if True, hide them otherwise.

updateBox(*args, **kwargs)[source]

Updates the box drawn around a selected skeleton.

This updates both the box attribute stored and the visual box. The box attribute is used to determine whether a click should select this instance.

updatePoints(complete: bool = False, user_change: bool = False)[source]

Updates data and display for all points in skeleton.

This is called any time the skeleton is manipulated as a whole.

Parameters
  • complete – Whether to update all nodes by setting “completed” attribute.

  • user_change – Whether method is called because of change made by user.

Returns

None.

class sleap.gui.widgets.video.QtNode(parent: PySide2.QtWidgets.QGraphicsObject, player: sleap.gui.widgets.video.QtVideoPlayer, node: sleap.skeleton.Node, point: sleap.instance.Point, radius: float, predicted=False, show_non_visible=True, callbacks=None, *args, **kwargs)[source]

QGraphicsEllipseItem to handle display of skeleton instance node.

Parameters
  • parent – The QtInstance which will contain this item.

  • node – The Node corresponding to this visual node.

  • point – The Point where this node is located. Note that this is a mutable object so we’re able to directly access the very same Point object that’s defined outside our class.

  • radius – Radius of the visual node item.

  • predicted – Whether this point is predicted.

  • show_non_visible – Whether to show points where visible is False.

  • callbacks – List of functions to call after we update to the Point.

calls()[source]

Method to call all callbacks.

mouseDoubleClickEvent(event: PySide2.QtGui.QMouseEvent)[source]

Custom event handler to emit signal on event.

mouseMoveEvent(event)[source]

Custom event handler for mouse move.

mousePressEvent(event)[source]

Custom event handler for mouse press.

mouseReleaseEvent(event)[source]

Custom event handler for mouse release.

updatePoint(user_change: bool = False)[source]

Method to update data for node/edge when node position is manipulated.

Parameters

user_change – Whether this being called because of change by user.

wheelEvent(event)[source]

Custom event handler for mouse scroll wheel.

class sleap.gui.widgets.video.QtNodeLabel(node: sleap.skeleton.Node, parent: PySide2.QtWidgets.QGraphicsObject, predicted: bool = False, *args, **kwargs)[source]

QGraphicsTextItem to handle display of node text label.

Parameters
  • node – The QtNode to which this label is attached.

  • parent – The QtInstance which will contain this item.

  • predicted – Whether this is for a predicted point.

adjustPos(*args, **kwargs)[source]

Update the position of the label based on the position of the node.

Parameters

arbitrary arguments so we can connect to various signals. (Accepts) –

adjustStyle()[source]

Update visual display of the label and its node.

boundingRect()[source]

Method required by Qt.

mouseMoveEvent(event)[source]

Pass events along so that clicking label is like clicking node.

mousePressEvent(event)[source]

Pass events along so that clicking label is like clicking node.

mouseReleaseEvent(event)[source]

Pass events along so that clicking label is like clicking node.

paint(*args, **kwargs)[source]

Method required by Qt.

wheelEvent(event)[source]

Pass events along so that clicking label is like clicking node.

class sleap.gui.widgets.video.QtTextWithBackground(*args, **kwargs)[source]

Inherits methods/behavior of QGraphicsTextItem, but with background box.

Color of background box is light or dark depending on the text color.

boundingRect()[source]

Method required by Qt.

paint(painter, option, *args, **kwargs)[source]

Method required by Qt.

class sleap.gui.widgets.video.QtVideoPlayer(video: sleap.io.video.Video = None, color_manager=None, state=None, context=None, *args, **kwargs)[source]

Main QWidget for displaying video with skeleton instances.

Signals:
  • changedPlot: Emitted whenever the plot is redrawn

video

The Video to display

color_manager

A ColorManager object which determines which color to show the instances.

addInstance(instance, **kwargs)[source]

Add a skeleton instance to the video.

Parameters
  • instance – this can be either a QtInstance or an Instance

  • other named args are passed along if/when creating QtInstance. (Any) –

getVisibleRect()[source]

Returns QRectF with currently visible portion of frame image.

property instances

Returns list of all QtInstance objects in view.

keyPressEvent(event: PySide2.QtGui.QKeyEvent)[source]

Custom event handler, allows navigation and selection within view.

keyReleaseEvent(event: PySide2.QtGui.QKeyEvent)[source]

Custom event handler, tracks when user releases modifier (shift) key.

load_video(video: sleap.io.video.Video, plot=True)[source]

Load video into viewer.

Parameters
  • video – the Video to display

  • plot – If True, plot the video frame. Otherwise, just load the data.

onAreaSelection(callback: Callable)[source]

Starts mode for user to select area, callback called when finished.

Parameters

callback – The function called after user clicks point, should take x0, y0, x1, y1 as arguments.

Returns

None.

onPointSelection(callback: Callable)[source]

Starts mode for user to click point, callback called when finished.

Parameters

callback – The function called after user clicks point, should take x and y as arguments.

Returns

None.

onSequenceSelect(seq_len: int, on_success: Callable, on_each: Optional[Callable] = None, on_failure: Optional[Callable] = None)[source]

Collect a sequence of instances (through user selection).

When the sequence is complete, the on_success callback is called. After each selection in sequence, the on_each callback is called (if given). If the user cancels (by unselecting without new selection), the on_failure callback is called (if given).

Note

If successful, we call >>> on_success(list_of_instances)

Parameters
  • seq_len – Number of instances we want to collect in sequence.

  • on_success – Callback for when user has selected desired number of instances.

  • on_each – Callback after user selects each instance.

  • on_failure – Callback if user cancels process before selecting enough instances.

plot(*args)[source]

Do the actual plotting of the video frame.

property predicted_instances

Returns list of predicted QtInstance objects in view.

reset()[source]

Reset viewer by removing all video data.

property scene

Returns QGraphicsScene for viewer.

property selectable_instances

Returns list of selectable QtInstance objects in view.

setFitZoom(value)[source]

Zooms or unzooms current view to fit all instances.

showEdges(show)[source]

Show/hide node edges for all instances in viewer.

Parameters

show – Show if True, hide otherwise.

showLabels(show)[source]

Show/hide node labels for all instances in viewer.

Parameters

show – Show if True, hide otherwise.

zoomToFit()[source]

Zoom view to fit all instances.

sleap.gui.widgets.video.plot_instances(scene, frame_idx, labels, video=None, fixed=True)[source]

Demo function for plotting instances.

sleap.gui.widgets.video.video_demo(video=None, labels=None, standalone=False)[source]

Demo function for showing video.