sleap.instance

Data structures for all labeled data contained with a SLEAP project.

The relationships between objects in this module:

  • A LabeledFrame can contain zero or more Instance`s (and `PredictedInstance objects).

  • Instance objects (and PredictedInstance objects) have PointArray (or PredictedPointArray).

  • Instance (PredictedInstance) can be associated with a Track

  • A PointArray (or PredictedPointArray) contains zero or more Point objects (or PredictedPoint objectss), ideally as many as there are in the associated Skeleton although these can get out of sync if the skeleton is manipulated.

class sleap.instance.Instance(skeleton: sleap.skeleton.Skeleton, track: sleap.instance.Track = None, from_predicted: Optional[PredictedInstance] = None, points: sleap.instance.PointArray = None, nodes: List = None, frame: Optional[LabeledFrame] = None)[source]

The class Instance represents a labelled instance of a skeleton.

Parameters

  • skeleton – The skeleton that this instance is associated with.

  • points – A dictionary where keys are skeleton node names and values are Point objects. Alternatively, a point array whose length and order matches skeleton.nodes.

  • track – An optional multi-frame object track associated with this instance. This allows individual animals/objects to be tracked across frames.

  • from_predicted – The predicted instance (if any) that this was copied from.

  • frame – A back reference to the LabeledFrame that this Instance belongs to. This field is set when instances are added to LabeledFrame objects.

property bounding_box

Returns the instance’s containing bounding box in [y1, x1, y2, x2] format.

property centroid

Returns instance centroid as (x,y) numpy row vector.

property frame_idx

Get the index of the frame that this instance was found on.

This is a convenience method for Instance.frame.frame_idx.

Returns

The frame number this instance was found on, or None if the instance is not associated with frame.

classmethod from_pointsarray(points: numpy.ndarray, skeleton: sleap.skeleton.Skeleton, track: Optional[sleap.instance.Track] = None) → sleap.instance.Instance[source]

Create an instance from pointsarray.

Parameters

  • points – A numpy array of shape (n_nodes, 2) and dtype float32 that contains the points in (x, y) coordinates of each node. Missing nodes should be represented as NaNs.

  • skeleton – A sleap.Skeleton instance with n_nodes nodes to associate with the predicted instance.

Returns

A new Instance.

get_points_array(copy: bool = True, invisible_as_nan: bool = False, full: bool = False) → numpy.ndarray[source]

Return the instance’s points in array form.

Parameters

  • copy – If True, the return a copy of the points array as an ndarray. If False, return a view of the underlying recarray.

  • invisible_as_nan – Should invisible points be marked as NaN. If copy is False, then invisible_as_nan is ignored since we don’t want to set invisible points to NaNs in original data.

  • full – If True, return all data for points. Otherwise, return just the x and y coordinates.

Returns

Either a recarray (if copy is False) or an ndarray (if copy True).

The order of the rows corresponds to the ordering of the skeleton nodes. Any skeleton node not defined will have NaNs present.

Columns in recarray are accessed by name, e.g., [“x”], [“y”].

Columns in ndarray are accessed by number. The order matches the order in Point.dtype or PredictedPoint.dtype.

matches(other: sleap.instance.Instance) → bool[source]

Whether two instances match by value.

Checks the types, points, track, and frame index.

Parameters

other – The other Instance.

Returns

True if match, False otherwise.

property n_visible_points

Returns the count of points that are visible in this instance.

property nodes

The tuple of nodes that have been labelled for this instance.

property nodes_points

The list of (node, point) tuples for all labelled points.

numpy() → numpy.ndarray[source]

Return the instance node coordinates as a numpy array.

Alias for points_array.

Returns

Array of shape (n_nodes, 2) of dtype float32 containing the coordinates of the instance’s nodes. Missing/not visible nodes will be replaced with NaNs.

property points

The tuple of labelled points, in order they were labelled.

property points_array

Nx2 array of x and y for visible points.

Row in array corresponds to order of points in skeleton. Invisible points will have nans.

Returns

ndarray of visible point coordinates.

transform_points(transformation_matrix)[source]

Applies transformation matrix to points.

class sleap.instance.LabeledFrame(video: sleap.io.video.Video, frame_idx, instances: Union[List[sleap.instance.Instance], List[sleap.instance.PredictedInstance]] = NOTHING)[source]

Holds labeled data for a single frame of a video.

Parameters

  • video – The Video associated with this frame.

  • frame_idx – The index of frame in video.

classmethod complex_frame_merge(base_frame: sleap.instance.LabeledFrame, new_frame: sleap.instance.LabeledFrame) → Tuple[List[sleap.instance.Instance], List[sleap.instance.Instance], List[sleap.instance.Instance]][source]

Merge two frames, return conflicts if any.

A conflict occurs when * each frame has Instances which don’t perfectly match those

in the other frame, or

  • each frame has PredictedInstances which don’t perfectly match those in the other frame.

Parameters

  • base_frame – The LabeledFrame into which we want to merge.

  • new_frame – The LabeledFrame from which we want to merge.

Returns

  • list of instances that were merged

  • list of conflicting instances from base

  • list of conflicting instances from new

Return type

tuple of three items

classmethod complex_merge_between(base_labels: Labels, new_frames: List[LabeledFrame]) → Tuple[Dict[sleap.io.video.Video, Dict[int, List[sleap.instance.Instance]]], List[sleap.instance.Instance], List[sleap.instance.Instance]][source]

Merge data from new frames into a Labels object.

Everything that can be merged cleanly is merged, any conflicts are returned.

Parameters

  • base_labels – The Labels into which we are merging.

  • new_frames – The list of LabeledFrame objects from which we are merging.

Returns

  • Dictionary, keys are Video, values are

    dictionary in which keys are frame index (int) and value is list of :class:`Instance`s

  • list of conflicting Instance objects from base

  • list of conflicting Instance objects from new frames

Return type

tuple of three items

find(track: Optional[Union[sleap.instance.Track, int]] = - 1, user: bool = False) → List[sleap.instance.Instance][source]

Retrieves instances (if any) matching specifications.

Parameters

  • track – The Track to match. Note that None will only match instances where :attribute:`Instance.track` is None. If track is -1, then we’ll match any track.

  • user – Whether to only match user (non-predicted) instances.

Returns

List of instances.

property has_predicted_instances

Whether the frame contains any predicted instances.

property has_user_instances

Whether the frame contains any user instances.

property image

Return the image for this frame of shape (height, width, channels).

index(value: sleap.instance.Instance) → int[source]

Returns index of given Instance.

insert(index: int, value: sleap.instance.Instance)[source]

Adds instance to frame.

Parameters

  • index – The index in list of frame instances where we should insert the new instance.

  • value – The instance to associate with frame.

Returns

None.

property instances

Returns list of all instances associated with this frame.

property instances_to_show

Return a list of instances to show in GUI for this frame.

This list will not include any predicted instances for which there’s a corresponding regular instance.

Returns

List of instances to show in GUI.

static merge_frames(labeled_frames: List[LabeledFrame], video: sleap.io.video.Video, remove_redundant=True) → List[sleap.instance.LabeledFrame][source]

Merged LabeledFrames for same video and frame index.

Parameters

  • labeled_frames – List of LabeledFrame objects to merge.

  • video – The Video for which to merge. This is specified so we don’t have to check all frames when we already know which video has new labeled frames.

  • remove_redundant – Whether to drop instances in the merged frames where there’s a perfect match.

Returns

The merged list of :class:`LabeledFrame`s.

plot(image: bool = True)[source]

Plot the frame with all instances.

Parameters

image – If False, only the instances will be plotted without loading the original image.

Notes

See sleap.nn.viz.plot_img and sleap.nn.viz.plot_instances for more plotting options.

plot_predicted(image: bool = True)[source]

Plot the frame with all predicted instances.

Parameters

image – If False, only the instances will be plotted without loading the original image.

Notes

See sleap.nn.viz.plot_img and sleap.nn.viz.plot_instances for more plotting options.

property predicted_instances

Returns list of predicted instances associated with frame.

property training_instances

Returns list of user instances with points for training.

property unused_predictions

Returns list of “unused” PredictedInstance objects in frame.

This is all the PredictedInstance objects which do not have a corresponding Instance in the same track in frame.

property user_instances

Returns list of user instances associated with this frame.

class sleap.instance.Point(x: float = nan, y: float = nan, visible: bool = True, complete: bool = False)[source]

A labelled point and any metadata associated with it.

Parameters

  • x – The horizontal pixel location of point within image frame.

  • y – The vertical pixel location of point within image frame.

  • visible – Whether point is visible in the labelled image or not.

  • complete – Has the point been verified by the user labeler.

isnan() → bool[source]

Are either of the coordinates a NaN value.

Returns

True if x or y is NaN, False otherwise.

class sleap.instance.PointArray(shape, buf=None, offset=0, strides=None, formats=None, names=None, titles=None, byteorder=None, aligned=False, order='C')[source]

PointArray is a sub-class of numpy recarray which stores Point objects as records.

classmethod from_array(a: sleap.instance.PointArray) → sleap.instance.PointArray[source]

Converts a PointArray (or child) to a new instance.

This will convert an object to the same type as itself, so a PredictedPointArray will result in the same.

Uses the default attribute values for new array.

Parameters

a – The array to convert.

Returns

A PointArray or PredictedPointArray with the same points as a.

classmethod make_default(size: int) → sleap.instance.PointArray[source]

Construct a point array where points are all set to default.

The constructed PointArray will have specified size and each value in the array is assigned the default values for a Point`.

Parameters

size – The number of points to allocate.

Returns

A point array with all elements set to Point()

class sleap.instance.PredictedInstance(skeleton: sleap.skeleton.Skeleton, track: sleap.instance.Track = None, from_predicted: Optional[PredictedInstance] = None, points: sleap.instance.PointArray = None, nodes: List = None, frame: Optional[LabeledFrame] = None, score=0.0, tracking_score=0.0)[source]

A predicted instance is an output of the inference procedure.

Parameters

  • score – The instance-level grouping prediction score.

  • tracking_score – The instance-level track matching score.

classmethod from_arrays(points: numpy.ndarray, point_confidences: numpy.ndarray, instance_score: float, skeleton: sleap.skeleton.Skeleton, track: Optional[sleap.instance.Track] = None) → sleap.instance.PredictedInstance[source]

Create a predicted instance from data arrays.

Parameters

  • points – A numpy array of shape (n_nodes, 2) and dtype float32 that contains the points in (x, y) coordinates of each node. Missing nodes should be represented as NaNs.

  • point_confidences – A numpy array of shape (n_nodes,) and dtype float32 that contains the confidence/score of the points.

  • instance_score – Scalar float representing the overall instance score, e.g., the PAF grouping score.

  • skeleton – A sleap.Skeleton instance with n_nodes nodes to associate with the predicted instance.

Returns

A new PredictedInstance.

classmethod from_instance(instance: sleap.instance.Instance, score: float) → sleap.instance.PredictedInstance[source]

Create a PredictedInstance from an Instance.

The fields are copied in a shallow manner with the exception of points. For each point in the instance a PredictedPoint is created with score set to default value.

Parameters

  • instance – The Instance object to shallow copy data from.

  • score – The score for this instance.

Returns

A PredictedInstance for the given Instance.

property points_and_scores_array

(N, 3) array of (x, y, score) for predicted points.

Row in arrow corresponds to order of points in skeleton. Invisible points will have NaNs.

Returns

ndarray of visible point coordinates and scores.

class sleap.instance.PredictedPoint(x: float = nan, y: float = nan, visible: bool = True, complete: bool = False, score: float = 0.0)[source]

A predicted point is an output of the inference procedure.

It has all the properties of a labeled point, plus a score.

Parameters

  • x – The horizontal pixel location of point within image frame.

  • y – The vertical pixel location of point within image frame.

  • visible – Whether point is visible in the labelled image or not.

  • complete – Has the point been verified by the user labeler.

  • score – The point-level prediction score.

classmethod from_point(point: sleap.instance.Point, score: float = 0.0) → sleap.instance.PredictedPoint[source]

Create a PredictedPoint from a Point

Parameters

  • point – The point to copy all data from.

  • score – The score for this predicted point.

Returns

A scored point based on the point passed in.

class sleap.instance.PredictedPointArray(shape, buf=None, offset=0, strides=None, formats=None, names=None, titles=None, byteorder=None, aligned=False, order='C')[source]

PredictedPointArray is analogous to PointArray except for predicted points.

classmethod to_array(a: sleap.instance.PredictedPointArray) → sleap.instance.PointArray[source]

Convert a PredictedPointArray to a normal PointArray.

Parameters

a – The array to convert.

Returns

The converted array.

class sleap.instance.Track(spawned_on, name='')[source]

A track object is associated with a set of animal/object instances across multiple frames of video. This allows tracking of unique entities in the video over time and space.

Parameters

  • spawned_on – The video frame that this track was spawned on.

  • name – A name given to this track for identifying purposes.

matches(other: sleap.instance.Track)[source]

Check if two tracks match by value.

Parameters

other – The other track to check

Returns

True if they match, False otherwise.

sleap.instance.make_instance_cattr() → cattr.converters.Converter[source]

Create a cattr converter for Lists of Instances/PredictedInstances.

This is required because cattrs doesn’t automatically detect the class when the attributes of one class are a subset of another.

Returns

A cattr converter with hooks registered for structuring and

unstructuring Instance objects and :class:`PredictedInstance`s.