sleap.gui.widgets.video¶
Module for showing and manipulating skeleton instances within a video.
All interactions should go through QtVideoPlayer.
- Example usage:
>>> my_video = Video(...) >>> my_instance = Instance(...)
>>> vp = QtVideoPlayer(video=my_video) >>> vp.addInstance(instance=my_instance, color=(r, g, b))
-
class
sleap.gui.widgets.video.
GraphicsView
(state=None, player=None, *args, **kwargs)[source]¶ Custom QGraphicsView used by QtVideoPlayer.
This contains elements for display of video and event handlers for zoom and selection of instances in view.
- Signals:
- updatedViewer: Emitted after update to view (e.g., zoom).
Used internally so we know when to update points for each instance.
updatedSelection: Emitted after the user has (un)selected an instance.
- instanceDoubleClicked: Emitted after an instance is double-clicked.
Passes the
Instance
that was double-clicked.
- areaSelected: Emitted after user selects an area when in “area”
click mode. Passes x0, y0, x1, y1 for selected box coordinates.
- pointSelected: Emitted after user clicks a point (in “point” click
mode.) Passes x, y coordinates of point.
leftMouseButtonPressed: Emitted by event handler.
rightMouseButtonPressed: Emitted by event handler.
leftMouseButtonReleased: Emitted by event handler.
rightMouseButtonReleased: Emitted by event handler.
leftMouseButtonDoubleClicked: Emitted by event handler.
rightMouseButtonDoubleClicked: Emitted by event handler.
-
property
all_instances
¶ Returns a list of all QtInstance objects in scene.
Order should match the order in which instances were added to scene.
-
static
getInstancesBoundingRect
(instances: List[QtInstance], margin: float = 0.0) → PySide2.QtCore.QRectF[source]¶ Return a rectangle containing all instances.
- Parameters
instances – List of QtInstance objects.
margin – Margin for padding the rectangle. Padding is applied equally on all sides.
- Returns
The QRectF which contains all of the instances.
Notes
The returned rectangle will be null if the instance list is empty.
-
getSelectionIndex
() → Optional[int][source]¶ Returns the index of the currently selected instance. If no instance selected, returns None.
-
getSelectionInstance
() → Optional[sleap.instance.Instance][source]¶ Returns the currently selected instance. If no instance selected, returns None.
-
getTopInstanceAt
(scenePos) → Optional[sleap.instance.Instance][source]¶ Returns topmost instance at position in scene.
-
property
instances
¶ Returns a list of instances.
Order should match the order in which instances were added to scene.
-
instancesBoundingRect
(margin: float = 0) → PySide2.QtCore.QRectF[source]¶ Returns a rect which contains all displayed skeleton instances.
- Parameters
margin – Margin for padding the rect.
- Returns
The QRectF which contains the skeleton instances.
-
property
predicted_instances
¶ Returns a list of predicted instances.
Order should match the order in which instances were added to scene.
-
selectInstance
(select: Union[sleap.instance.Instance, int])[source]¶ Select a particular instance in view.
- Parameters
select – Either Instance or index of instance in view.
- Returns
None
-
property
selectable_instances
¶ Returns a list of instances which user can select.
Order should match the order in which instances were added to scene.
-
setImage
(image: Union[PySide2.QtGui.QImage, PySide2.QtGui.QPixmap, numpy.ndarray])[source]¶ Set the scene’s current image pixmap to the input QImage or QPixmap.
- Parameters
image – The QPixmap or QImage to display.
- Raises
RuntimeError – If the input image is not QImage or QPixmap
- Returns
None.
-
zoomToRect
(zoom_rect: PySide2.QtCore.QRectF)[source]¶ Method to zoom scene to a given rectangle.
The rect can either be given relative to the current zoom (this is useful if it’s the rect drawn by user) or it can be given in absolute coordinates for displayed frame.
- Parameters
zoom_rect – The QRectF to which we want to zoom.
-
class
sleap.gui.widgets.video.
LoadImageWorker
(*args, **kwargs)[source]¶ Object to load video frames in background thread.
Requests to load a frame image are sent by calling the request method with the frame idx; the video attribute should already be set to the correct video.
These requests are added to a FILO queue polled by the doProcessing method, called whenever there’s time during the Qt event loop. (It’s also added to the event queue if it hasn’t been called for a while and we get a request, since the timer doesn’t seem to emit events if the user has been holding down the mouse for a while.)
The actual frame loading is wrapped with a mutex lock so that we only load a single frame at a time; this helps us not get a bunch of older frame requests running concurrently.
Once the frame loads, the QImage is sent via the result signal. (Qt handles the cross-thread communication if we use signals.)
-
class
sleap.gui.widgets.video.
QtEdge
(parent: PySide2.QtWidgets.QGraphicsObject, player: sleap.gui.widgets.video.QtVideoPlayer, src: sleap.gui.widgets.video.QtNode, dst: sleap.gui.widgets.video.QtNode, show_non_visible: bool = True, *args, **kwargs)[source]¶ QGraphicsLineItem to handle display of edge between skeleton instance nodes.
- Parameters
parent – QGraphicsObject which will contain this item.
src – The QtNode source node for the edge.
dst – The QtNode destination node for the edge.
show_non_visible – Whether to show “non-visible” nodes/edges.
-
angle_to
(node: sleap.gui.widgets.video.QtNode) → float[source]¶ Returns the angle from one edge node to the other.
- Parameters
node – The node from which we’re measuring the angle.
- Returns
Angle (in radians) to the other node.
-
connected_to
(node: sleap.gui.widgets.video.QtNode)[source]¶ Return the other node along the edge.
- Parameters
node – One of the edge’s nodes.
- Returns
The other node (or None if edge doesn’t have node).
-
updateEdge
(node: sleap.gui.widgets.video.QtNode)[source]¶ Updates the visual display of node.
- Parameters
node – The node to update.
- Returns
None.
-
class
sleap.gui.widgets.video.
QtInstance
(instance: sleap.instance.Instance = None, player: Optional[sleap.gui.widgets.video.QtVideoPlayer] = None, markerRadius=4, show_non_visible=True, *args, **kwargs)[source]¶ QGraphicsObject for skeleton instances.
This object stores the data for one skeleton instance and handles the events to manipulate the skeleton within a video frame (i.e., moving, rotating, marking nodes).
It should be instantiated with an Instance and added to the relevant QGraphicsScene.
When instantiated, it creates QtNode, QtEdge, and QtNodeLabel items as children of itself.
- Parameters
instance – The
Instance
to show.markerRadius – Radius of nodes.
show_non_visible – Whether to show “non-visible” nodes/edges.
-
getPointsBoundingRect
() → PySide2.QtCore.QRectF[source]¶ Returns a rect which contains all the nodes in the skeleton.
-
property
selected
¶ Whether instance is selected.
-
showEdges
(show: bool)[source]¶ Draws/hides the edges for this skeleton instance.
- Parameters
show – Show edges if True, hide them otherwise.
-
showLabels
(show: bool)[source]¶ Draws/hides the labels for this skeleton instance.
- Parameters
show – Show labels if True, hide them otherwise.
-
updateBox
(*args, **kwargs)[source]¶ Updates the box drawn around a selected skeleton.
This updates both the box attribute stored and the visual box. The box attribute is used to determine whether a click should select this instance.
-
updatePoints
(complete: bool = False, user_change: bool = False)[source]¶ Updates data and display for all points in skeleton.
This is called any time the skeleton is manipulated as a whole.
- Parameters
complete – Whether to update all nodes by setting “completed” attribute.
user_change – Whether method is called because of change made by user.
- Returns
None.
-
class
sleap.gui.widgets.video.
QtNode
(parent: PySide2.QtWidgets.QGraphicsObject, player: sleap.gui.widgets.video.QtVideoPlayer, node: sleap.skeleton.Node, point: sleap.instance.Point, radius: float, predicted=False, show_non_visible=True, callbacks=None, *args, **kwargs)[source]¶ QGraphicsEllipseItem to handle display of skeleton instance node.
- Parameters
parent – The QtInstance which will contain this item.
node – The
Node
corresponding to this visual node.point – The
Point
where this node is located. Note that this is a mutable object so we’re able to directly access the very same Point object that’s defined outside our class.radius – Radius of the visual node item.
predicted – Whether this point is predicted.
show_non_visible – Whether to show points where visible is False.
callbacks – List of functions to call after we update to the Point.
-
mouseDoubleClickEvent
(event: PySide2.QtGui.QMouseEvent)[source]¶ Custom event handler to emit signal on event.
-
class
sleap.gui.widgets.video.
QtNodeLabel
(node: sleap.skeleton.Node, parent: PySide2.QtWidgets.QGraphicsObject, predicted: bool = False, *args, **kwargs)[source]¶ QGraphicsTextItem to handle display of node text label.
- Parameters
node – The QtNode to which this label is attached.
parent – The QtInstance which will contain this item.
predicted – Whether this is for a predicted point.
-
class
sleap.gui.widgets.video.
QtTextWithBackground
(*args, **kwargs)[source]¶ Inherits methods/behavior of QGraphicsTextItem, but with background box.
Color of background box is light or dark depending on the text color.
-
class
sleap.gui.widgets.video.
QtVideoPlayer
(video: sleap.io.video.Video = None, color_manager=None, state=None, context=None, *args, **kwargs)[source]¶ Main QWidget for displaying video with skeleton instances.
- Signals:
changedPlot: Emitted whenever the plot is redrawn
-
video
¶ The
Video
to display
-
color_manager
¶ A
ColorManager
object which determines which color to show the instances.
-
addInstance
(instance, **kwargs)[source]¶ Add a skeleton instance to the video.
- Parameters
instance – this can be either a QtInstance or an Instance
other named args are passed along if/when creating QtInstance. (Any) –
-
property
instances
¶ Returns list of all QtInstance objects in view.
-
keyPressEvent
(event: PySide2.QtGui.QKeyEvent)[source]¶ Custom event handler, allows navigation and selection within view.
-
keyReleaseEvent
(event: PySide2.QtGui.QKeyEvent)[source]¶ Custom event handler, tracks when user releases modifier (shift) key.
-
load_video
(video: sleap.io.video.Video, plot=True)[source]¶ Load video into viewer.
- Parameters
video – the
Video
to displayplot – If True, plot the video frame. Otherwise, just load the data.
-
onAreaSelection
(callback: Callable)[source]¶ Starts mode for user to select area, callback called when finished.
- Parameters
callback – The function called after user clicks point, should take x0, y0, x1, y1 as arguments.
- Returns
None.
-
onPointSelection
(callback: Callable)[source]¶ Starts mode for user to click point, callback called when finished.
- Parameters
callback – The function called after user clicks point, should take x and y as arguments.
- Returns
None.
-
onSequenceSelect
(seq_len: int, on_success: Callable, on_each: Optional[Callable] = None, on_failure: Optional[Callable] = None)[source]¶ Collect a sequence of instances (through user selection).
When the sequence is complete, the on_success callback is called. After each selection in sequence, the on_each callback is called (if given). If the user cancels (by unselecting without new selection), the on_failure callback is called (if given).
Note
If successful, we call >>> on_success(list_of_instances)
- Parameters
seq_len – Number of instances we want to collect in sequence.
on_success – Callback for when user has selected desired number of instances.
on_each – Callback after user selects each instance.
on_failure – Callback if user cancels process before selecting enough instances.
-
property
predicted_instances
¶ Returns list of predicted QtInstance objects in view.
-
property
scene
¶ Returns QGraphicsScene for viewer.
-
property
selectable_instances
¶ Returns list of selectable QtInstance objects in view.
-
showEdges
(show)[source]¶ Show/hide node edges for all instances in viewer.
- Parameters
show – Show if True, hide otherwise.