sleap.nn.tracker.components

Functions/classes used by multiple trackers.

Main types of functions:

1. Calculate pair-wise instance similarity; used for populating similarity/cost matrix.

2. Pick matches based on cost matrix.

3. Other clean-up (e.g., cull instances, connect track breaks).

class sleap.nn.tracker.components.FrameMatches(matches: List[Match], cost_matrix: ndarray, unmatched_instances: List[InstanceType] = NOTHING)

Calculates (and stores) matches for a frame.

This class encapsulates the logic to generate matches (using a custom matching function) from a cost matrix. One key feature is that it retains additional information, such as whether all the matches were first-choice (i.e., if each instance got the instance it would have matched to if there weren’t other instances).

Typically this will be created using the from_candidate_instances method which creates the cost matrix and then uses the matching function to find matches.

matches

the list of Match objects.

Type:

List[sleap.nn.tracker.components.Match]

cost_matrix

the cost matrix, shape is (number of untracked instances, number of candidate tracks).

Type:

numpy.ndarray

unmatched_instances

the instances for which we are finding matches.

Type:

List[sleap.nn.tracker.components.InstanceType]

classmethod from_candidate_instances(untracked_instances: List[InstanceType], candidate_instances: List[InstanceType], similarity_function: Callable, matching_function: Callable, robust_best_instance: float = 1.0)

Calculates (and stores) matches for a frame from candidate instance.

Parameters:

• untracked_instances – list of untracked instances in the frame.

• candidate_instances – list of instances use as match.

• similarity_function – a function that returns the similarity between two instances (untracked and candidate).

• matching_function – function used to find the best match from the cost matrix. See the classmethod from_cost_matrix.

• robust_best_instance (float) – if the value is between 0 and 1 (excluded), use a robust quantile similarity score for the track. If the value is 1, use the max similarity (non-robust). For selecting a robust score, 0.95 is a good value.

property has_only_first_choice_matches: bool

Whether all the matches were first-choice.

A match is a ‘first-choice’ for an instance if that instance would have matched to the same track even if there were no other instances.

class sleap.nn.tracker.components.Match(track: Track, instance: Instance, score: float | None = None, is_first_choice: bool = False)

Stores a match between a specific instance and specific track.

sleap.nn.tracker.components.centroid_distance(ref_instance: InstanceType, query_instance: InstanceType, cache: dict = {}) → float

Returns the negative distance between the centroids of two instances.

Uses cache dictionary (created with function so it persists between calls) since without cache this method is significantly slower than others.

sleap.nn.tracker.components.connect_single_track_breaks(frames: List[LabeledFrame], instance_count: int) → List[LabeledFrame]

Merges breaks in tracks by connecting single lost with single new track.

Parameters:

• frames – The list of LabeledFrame objects with predictions.

• instance_count – The maximum number of instances we want per frame.

Returns:

Updated list of frames, also modifies frames in place.

sleap.nn.tracker.components.cull_frame_instances(instances_list: List[InstanceType], instance_count: int, iou_threshold: float | None = None) → List[LabeledFrame]

Removes instances (for single frame) over instance per frame threshold.

Parameters:

• instances_list – The list of instances for a single frame.

• instance_count – The maximum number of instances we want per frame.

• iou_threshold – Intersection over Union (IOU) threshold to use when removing overlapping instances over target count; if None, then only use score to determine which instances to remove.

Returns:

Updated list of frames, also modifies frames in place.

sleap.nn.tracker.components.cull_instances(frames: List[LabeledFrame], instance_count: int, iou_threshold: float | None = None)

Removes instances from frames over instance per frame threshold.

Parameters:

• frames – The list of LabeledFrame objects with predictions.

• instance_count – The maximum number of instances we want per frame.

• iou_threshold – Intersection over Union (IOU) threshold to use when removing overlapping instances over target count; if None, then only use score to determine which instances to remove.

Returns:

None; modifies frames in place.

sleap.nn.tracker.components.factory_object_keypoint_similarity(keypoint_errors: List | int | float | None = None, score_weighting: bool = False, normalization_keypoints: str = 'all') → Callable

Factory for similarity function based on object keypoints.

Parameters:

• keypoint_errors – The standard error of the distance between the predicted keypoint and the true value, in pixels. If None or empty list, defaults to 1. If a scalar or singleton list, every keypoint has the same error. If a list, defines the error for each keypoint, the length should be equal to the number of keypoints in the skeleton.

• score_weighting – If True, use score of PredictedPoint to weigh keypoint_errors. If False, do not add a weight to keypoint_errors.

• normalization_keypoints – Determine how to normalize similarity score. One of [“all”, “ref”, “union”]. If “all”, similarity score is normalized by number of reference points. If “ref”, similarity score is normalized by number of visible reference points. If “union”, similarity score is normalized by number of points both visible in query and reference instance. Default is “all”.

Returns:

Callable that returns object keypoint similarity between two `Instance`s.

sleap.nn.tracker.components.first_choice_matching(cost_matrix: ndarray) → List[Tuple[int, int]]

Returns match indices where each row gets matched to best column.

The means that multiple rows might be matched to the same column.

sleap.nn.tracker.components.greedy_matching(cost_matrix: ndarray) → List[Tuple[int, int]]

Performs greedy bipartite matching.

sleap.nn.tracker.components.hungarian_matching(cost_matrix: ndarray) → List[Tuple[int, int]]

Wrapper for Hungarian matching algorithm in scipy.

sleap.nn.tracker.components.instance_iou(ref_instance: InstanceType, query_instance: InstanceType, cache: dict = {}) → float

Computes IOU between bounding boxes of instances.

sleap.nn.tracker.components.instance_similarity(ref_instance: InstanceType, query_instance: InstanceType) → float

Computes similarity between instances.

sleap.nn.tracker.components.nms_fast(boxes, scores, iou_threshold, target_count=None) → List[int]

https://www.pyimagesearch.com/2015/02/16/faster-non-maximum-suppression-python/

sleap.nn.tracker.components.normalized_instance_similarity(ref_instance: InstanceType, query_instance: InstanceType, img_hw: Tuple[int]) → float

Computes similarity between instances with normalized keypoints.