sleap.instance#

Data structures for all labeled data contained with a SLEAP project.

The relationships between objects in this module:

class sleap.instance.Instance(skeleton: sleap.skeleton.Skeleton, track: sleap.instance.Track = None, from_predicted: Optional[PredictedInstance] = None, points: sleap.instance.PointArray = None, nodes: List = None, frame: Optional[LabeledFrame] = None)[source]#

This class represents a labeled instance.

Parameters

  • skeleton – The skeleton that this instance is associated with.

  • points – A dictionary where keys are skeleton node names and values are Point objects. Alternatively, a point array whose length and order matches skeleton.nodes.

  • track – An optional multi-frame object track associated with this instance. This allows individual animals/objects to be tracked across frames.

  • from_predicted – The predicted instance (if any) that this was copied from.

  • frame – A back reference to the LabeledFrame that this Instance belongs to. This field is set when instances are added to LabeledFrame objects.

property bounding_box: numpy.ndarray#

Return bounding box containing all points in [y1, x1, y2, x2] format.

property centroid: numpy.ndarray#

Return instance centroid as an array of (x, y) coordinates

Notes

This computes the centroid as the median of the visible points.

fill_missing(max_x: Optional[float] = None, max_y: Optional[float] = None)[source]#

Add points for skeleton nodes that are missing in the instance.

This is useful when modifying the skeleton so the nodes appears in the GUI.

Parameters

  • max_x – If specified, make sure points are not added outside of valid range.

  • max_y – If specified, make sure points are not added outside of valid range.

property frame_idx: Optional[int]#

Return the index of the labeled frame this instance is associated with.

classmethod from_numpy(points: numpy.ndarray, skeleton: sleap.skeleton.Skeleton, track: Optional[sleap.instance.Track] = None) → sleap.instance.Instance[source]#

Create an instance from a numpy array.

Parameters

  • points – A numpy array of shape (n_nodes, 2) and dtype float32 that contains the points in (x, y) coordinates of each node. Missing nodes should be represented as NaN.

  • skeleton – A sleap.Skeleton instance with n_nodes nodes to associate with the instance.

  • track – Optional sleap.Track object to associate with the instance.

Returns

A new Instance object.

Notes

This is an alias for Instance.from_pointsarray().

classmethod from_pointsarray(points: numpy.ndarray, skeleton: sleap.skeleton.Skeleton, track: Optional[sleap.instance.Track] = None) → sleap.instance.Instance[source]#

Create an instance from an array of points.

Parameters

  • points – A numpy array of shape (n_nodes, 2) and dtype float32 that contains the points in (x, y) coordinates of each node. Missing nodes should be represented as NaN.

  • skeleton – A sleap.Skeleton instance with n_nodes nodes to associate with the instance.

  • track – Optional sleap.Track object to associate with the instance.

Returns

A new Instance object.

get_points_array(copy: bool = True, invisible_as_nan: bool = False, full: bool = False) → Union[numpy.ndarray, numpy.recarray][source]#

Return the instance’s points in array form.

Parameters

  • copy – If True, the return a copy of the points array as an ndarray. If False, return a view of the underlying recarray.

  • invisible_as_nan – Should invisible points be marked as NaN. If copy is False, then invisible_as_nan is ignored since we don’t want to set invisible points to NaNs in original data.

  • full – If True, return all data for points. Otherwise, return just the x and y coordinates.

Returns

Either a recarray (if copy is False) or an ndarray (if copy True).

The order of the rows corresponds to the ordering of the skeleton nodes. Any skeleton node not defined will have NaNs present.

Columns in recarray are accessed by name, e.g., [“x”], [“y”].

Columns in ndarray are accessed by number. The order matches the order in Point.dtype or PredictedPoint.dtype.

matches(other: sleap.instance.Instance) → bool[source]#

Whether two instances match by value.

Checks the types, points, track, and frame index.

Parameters

other – The other Instance.

Returns

True if match, False otherwise.

property midpoint: numpy.ndarray#

Return the center of the bounding box of the instance points.

property n_visible_points: int#

Return the number of visible points in this instance.

property nodes: Tuple[sleap.skeleton.Node, ...]#

Return nodes that have been labelled for this instance.

property nodes_points: List[Tuple[sleap.skeleton.Node, sleap.instance.Point]]#

Return a list of (node, point) tuples for all labeled points.

numpy() → numpy.ndarray[source]#

Return the instance node coordinates as a numpy array.

Alias for points_array.

Returns

Array of shape (n_nodes, 2) of dtype float32 containing the coordinates of the instance’s nodes. Missing/not visible nodes will be replaced with NaN.

property points: Tuple[sleap.instance.Point, ...]#

Return a tuple of labelled points, in the order they were labelled.

property points_array: numpy.ndarray#

Return array of x and y coordinates for visible points.

Row in array corresponds to order of points in skeleton. Invisible points will be denoted by NaNs.

Returns

A numpy array of of shape (n_nodes, 2) point coordinates.

transform_points(transformation_matrix)[source]#

Apply affine transformation matrix to points in the instance.

Parameters

transformation_matrix – Affine transformation matrix as a numpy array of shape (3, 3).

property video: Optional[sleap.io.video.Video]#

Return the video of the labeled frame this instance is associated with.

class sleap.instance.LabeledFrame(video: sleap.io.video.Video, frame_idx, instances: Union[List[sleap.instance.Instance], List[sleap.instance.PredictedInstance]] = NOTHING)[source]#

Holds labeled data for a single frame of a video.

Parameters

  • video – The Video associated with this frame.

  • frame_idx – The index of frame in video.

  • instances – List of instances associated with the frame.

classmethod complex_frame_merge(base_frame: sleap.instance.LabeledFrame, new_frame: sleap.instance.LabeledFrame) → Tuple[List[sleap.instance.Instance], List[sleap.instance.Instance], List[sleap.instance.Instance]][source]#

Merge two frames, return conflicts if any.

A conflict occurs when * each frame has Instances which don’t perfectly match those

in the other frame, or

  • each frame has PredictedInstances which don’t perfectly match those in the other frame.

Parameters

  • base_frame – The LabeledFrame into which we want to merge.

  • new_frame – The LabeledFrame from which we want to merge.

Returns

  • list of instances that were merged

  • list of conflicting instances from base

  • list of conflicting instances from new

Return type

tuple of three items

classmethod complex_merge_between(base_labels: Labels, new_frames: List[LabeledFrame]) → Tuple[Dict[sleap.io.video.Video, Dict[int, List[sleap.instance.Instance]]], List[sleap.instance.Instance], List[sleap.instance.Instance]][source]#

Merge data from new frames into a Labels object.

Everything that can be merged cleanly is merged, any conflicts are returned.

Parameters

  • base_labels – The Labels into which we are merging.

  • new_frames – The list of LabeledFrame objects from which we are merging.

Returns

  • Dictionary, keys are Video, values are

    dictionary in which keys are frame index (int) and value is list of :class:`Instance`s

  • list of conflicting Instance objects from base

  • list of conflicting Instance objects from new frames

Return type

tuple of three items

find(track: Optional[Union[sleap.instance.Track, int]] = - 1, user: bool = False) → List[sleap.instance.Instance][source]#

Retrieve instances (if any) matching specifications.

Parameters

  • track – The Track to match. Note that None will only match instances where :attribute:`Instance.track` is None. If track is -1, then we’ll match any track.

  • user – Whether to only match user (non-predicted) instances.

Returns

List of instances.

property has_predicted_instances: bool#

Return whether the frame contains any predicted instances.

property has_tracked_instances: bool#

Return whether the frame contains any predicted instances with tracks.

property has_user_instances: bool#

Return whether the frame contains any user instances.

property image: numpy.ndarray#

Return the image for this frame of shape (height, width, channels).

index(value: sleap.instance.Instance) → int[source]#

Return index of given Instance.

insert(index: int, value: sleap.instance.Instance)[source]#

Add instance to frame.

Parameters

  • index – The index in list of frame instances where we should insert the new instance.

  • value – The instance to associate with frame.

Returns

None.

property instances: List[sleap.instance.Instance]#

Return list of all instances associated with this frame.

property instances_to_show: List[sleap.instance.Instance]#

Return a list of instances to show in GUI for this frame.

This list will not include any predicted instances for which there’s a corresponding regular instance.

Returns

List of instances to show in GUI.

static merge_frames(labeled_frames: List[sleap.instance.LabeledFrame], video: sleap.io.video.Video, remove_redundant=True) → List[sleap.instance.LabeledFrame][source]#

Return merged LabeledFrames for same video and frame index.

Parameters

  • labeled_frames – List of LabeledFrame objects to merge.

  • video – The Video for which to merge. This is specified so we don’t have to check all frames when we already know which video has new labeled frames.

  • remove_redundant – Whether to drop instances in the merged frames where there’s a perfect match.

Returns

The merged list of :class:`LabeledFrame`s.

property n_predicted_instances: int#

Return the number of predicted instances in the frame.

property n_tracked_instances: int#

Return the number of predicted instances with tracks in the frame.

property n_user_instances: int#

Return the number of user instances in the frame.

numpy() → numpy.ndarray[source]#

Return the instances as an array of shape (instances, nodes, 2).

plot(image: bool = True, scale: float = 1.0)[source]#

Plot the frame with all instances.

Parameters

  • image – If False, only the instances will be plotted without loading the original image.

  • scale – Relative scaling for the figure.

Notes

See sleap.nn.viz.plot_img and sleap.nn.viz.plot_instances for more plotting options.

plot_predicted(image: bool = True, scale: float = 1.0)[source]#

Plot the frame with all predicted instances.

Parameters

  • image – If False, only the instances will be plotted without loading the original image.

  • scale – Relative scaling for the figure.

Notes

See sleap.nn.viz.plot_img and sleap.nn.viz.plot_instances for more plotting options.

property predicted_instances: List[sleap.instance.PredictedInstance]#

Return list of predicted instances associated with frame.

remove_empty_instances()[source]#

Remove instances with no visible nodes from the labeled frame.

remove_untracked()[source]#

Removes any instances without a track assignment.

property tracked_instances: List[sleap.instance.PredictedInstance]#

Return list of predicted instances with tracks associated with frame.

property training_instances: List[sleap.instance.Instance]#

Return list of user instances with points for training.

property unused_predictions: List[sleap.instance.Instance]#

Return a list of “unused” PredictedInstance objects in frame.

This is all the PredictedInstance objects which do not have a corresponding Instance in the same track in frame.

property user_instances: List[sleap.instance.Instance]#

Return list of user instances associated with this frame.

class sleap.instance.Point(x: float = nan, y: float = nan, visible: bool = True, complete: bool = False)[source]#

A labelled point and any metadata associated with it.

Parameters

  • x – The horizontal pixel location of point within image frame.

  • y – The vertical pixel location of point within image frame.

  • visible – Whether point is visible in the labelled image or not.

  • complete – Has the point been verified by the user labeler.

dtype = dtype([('x', '<f8'), ('y', '<f8'), ('visible', '?'), ('complete', '?')])#
isnan() → bool[source]#

Are either of the coordinates a NaN value.

Returns

True if x or y is NaN, False otherwise.

numpy() → numpy.ndarray[source]#

Return the point as a numpy array.

class sleap.instance.PointArray(shape, buf=None, offset=0, strides=None, formats=None, names=None, titles=None, byteorder=None, aligned=False, order='C')[source]#

PointArray is a sub-class of numpy recarray which stores Point objects as records.

classmethod from_array(a: sleap.instance.PointArray) → sleap.instance.PointArray[source]#

Converts a PointArray (or child) to a new instance.

This will convert an object to the same type as itself, so a PredictedPointArray will result in the same.

Uses the default attribute values for new array.

Parameters

a – The array to convert.

Returns

A PointArray or PredictedPointArray with the same points as a.

classmethod make_default(size: int) → sleap.instance.PointArray[source]#

Construct a point array where points are all set to default.

The constructed PointArray will have specified size and each value in the array is assigned the default values for a Point`.

Parameters

size – The number of points to allocate.

Returns

A point array with all elements set to Point()

class sleap.instance.PredictedInstance(skeleton: sleap.skeleton.Skeleton, track: sleap.instance.Track = None, from_predicted: Optional[PredictedInstance] = None, points: sleap.instance.PointArray = None, nodes: List = None, frame: Optional[LabeledFrame] = None, score=0.0, tracking_score=0.0)[source]#

A predicted instance is an output of the inference procedure.

Parameters

  • score – The instance-level grouping prediction score.

  • tracking_score – The instance-level track matching score.

classmethod from_arrays(points: numpy.ndarray, point_confidences: numpy.ndarray, instance_score: float, skeleton: sleap.skeleton.Skeleton, track: Optional[sleap.instance.Track] = None) → sleap.instance.PredictedInstance[source]#

Create a predicted instance from data arrays.

Parameters

  • points – A numpy array of shape (n_nodes, 2) and dtype float32 that contains the points in (x, y) coordinates of each node. Missing nodes should be represented as NaN.

  • point_confidences – A numpy array of shape (n_nodes,) and dtype float32 that contains the confidence/score of the points.

  • instance_score – Scalar float representing the overall instance score, e.g., the PAF grouping score.

  • skeleton – A sleap.Skeleton instance with n_nodes nodes to associate with the predicted instance.

  • track – Optional sleap.Track to associate with the instance.

Returns

A new PredictedInstance.

classmethod from_instance(instance: sleap.instance.Instance, score: float) → sleap.instance.PredictedInstance[source]#

Create a PredictedInstance from an Instance.

The fields are copied in a shallow manner with the exception of points. For each point in the instance a PredictedPoint is created with score set to default value.

Parameters

  • instance – The Instance object to shallow copy data from.

  • score – The score for this instance.

Returns

A PredictedInstance for the given Instance.

classmethod from_numpy(points: numpy.ndarray, point_confidences: numpy.ndarray, instance_score: float, skeleton: sleap.skeleton.Skeleton, track: Optional[sleap.instance.Track] = None) → sleap.instance.PredictedInstance[source]#

Create a predicted instance from data arrays.

Parameters

  • points – A numpy array of shape (n_nodes, 2) and dtype float32 that contains the points in (x, y) coordinates of each node. Missing nodes should be represented as NaN.

  • point_confidences – A numpy array of shape (n_nodes,) and dtype float32 that contains the confidence/score of the points.

  • instance_score – Scalar float representing the overall instance score, e.g., the PAF grouping score.

  • skeleton – A sleap.Skeleton instance with n_nodes nodes to associate with the predicted instance.

  • track – Optional sleap.Track to associate with the instance.

Returns

A new PredictedInstance.

classmethod from_pointsarray(points: numpy.ndarray, point_confidences: numpy.ndarray, instance_score: float, skeleton: sleap.skeleton.Skeleton, track: Optional[sleap.instance.Track] = None) → sleap.instance.PredictedInstance[source]#

Create a predicted instance from data arrays.

Parameters

  • points – A numpy array of shape (n_nodes, 2) and dtype float32 that contains the points in (x, y) coordinates of each node. Missing nodes should be represented as NaN.

  • point_confidences – A numpy array of shape (n_nodes,) and dtype float32 that contains the confidence/score of the points.

  • instance_score – Scalar float representing the overall instance score, e.g., the PAF grouping score.

  • skeleton – A sleap.Skeleton instance with n_nodes nodes to associate with the predicted instance.

  • track – Optional sleap.Track to associate with the instance.

Returns

A new PredictedInstance.

property points_and_scores_array: numpy.ndarray#

Return the instance points and scores as an array.

This will be a (n_nodes, 3) array of (x, y, score) for each predicted point.

Rows in the array correspond to the order of points in skeleton. Invisible points will be represented as NaNs.

property scores: numpy.ndarray#

Return point scores for each predicted node.

class sleap.instance.PredictedPoint(x: float = nan, y: float = nan, visible: bool = True, complete: bool = False, score: float = 0.0)[source]#

A predicted point is an output of the inference procedure.

It has all the properties of a labeled point, plus a score.

Parameters

  • x – The horizontal pixel location of point within image frame.

  • y – The vertical pixel location of point within image frame.

  • visible – Whether point is visible in the labelled image or not.

  • complete – Has the point been verified by the user labeler.

  • score – The point-level prediction score.

dtype = dtype([('x', '<f8'), ('y', '<f8'), ('visible', '?'), ('complete', '?'), ('score', '<f8')])#
classmethod from_point(point: sleap.instance.Point, score: float = 0.0) → sleap.instance.PredictedPoint[source]#

Create a PredictedPoint from a Point

Parameters

  • point – The point to copy all data from.

  • score – The score for this predicted point.

Returns

A scored point based on the point passed in.

class sleap.instance.PredictedPointArray(shape, buf=None, offset=0, strides=None, formats=None, names=None, titles=None, byteorder=None, aligned=False, order='C')[source]#

PredictedPointArray is analogous to PointArray except for predicted points.

classmethod to_array(a: sleap.instance.PredictedPointArray) → sleap.instance.PointArray[source]#

Convert a PredictedPointArray to a normal PointArray.

Parameters

a – The array to convert.

Returns

The converted array.

class sleap.instance.Track(spawned_on=0, name='')[source]#

A track object is associated with a set of animal/object instances across multiple frames of video. This allows tracking of unique entities in the video over time and space.

Parameters

  • spawned_on – The video frame that this track was spawned on.

  • name – A name given to this track for identifying purposes.

matches(other: sleap.instance.Track)[source]#

Check if two tracks match by value.

Parameters

other – The other track to check

Returns

True if they match, False otherwise.

sleap.instance.make_instance_cattr() → cattr.converters.Converter[source]#

Create a cattr converter for Lists of Instances/PredictedInstances.

This is required because cattrs doesn’t automatically detect the class when the attributes of one class are a subset of another.

Returns

A cattr converter with hooks registered for structuring and unstructuring Instance and PredictedInstance objects.