sleap.nn.tracking

Tracking tools for linking grouped instances over time.

class sleap.nn.tracking.BaseTracker

class sleap.nn.tracking.FlowCandidateMaker(min_points: int = 0, img_scale: float = 1.0, of_window_size: int = 21, of_max_levels: int = 3, save_shifted_instances: bool = False, shifted_instances: Dict[Tuple[int, int], List[sleap.nn.tracking.ShiftedInstance]] = NOTHING)

Class for producing optical flow shift matching candidates.

static flow_shift_instances(ref_instances: List[InstanceType], ref_img: numpy.ndarray, new_img: numpy.ndarray, min_shifted_points: int = 0, scale: float = 1.0, window_size: int = 21, max_levels: int = 3) → List[sleap.nn.tracking.ShiftedInstance]

Generates instances in a new frame by applying optical flow displacements.

Parameters

• ref_instances – Reference instances in the previous frame.

• ref_img – Previous frame image as a numpy array.

• new_img – New frame image as a numpy array.

• min_shifted_points – Minimum number of points that must be detected in the new frame in order to generate a new shifted instance.

• scale – Factor to scale the images by when computing optical flow. Decrease this to increase performance at the cost of finer accuracy. Sometimes decreasing the image scale can improve performance with fast movements.

• window_size – Optical flow window size to consider at each pyramid scale level.

• max_levels – Number of pyramid scale levels to consider. This is different from the scale parameter, which determines the initial image scaling.

Returns

A list of ShiftedInstances with the optical flow displacements applied to the reference instance points. Points that are not found will be represented as NaNs in the points array for each shifted instance.

Notes

This function relies on the Lucas-Kanade method for optical flow estimation.

class sleap.nn.tracking.FlowTracker(track_window: int = 5, cleaner: Optional[Callable] = None, target_instance_count: int = 0, pre_cull_function: Optional[Callable] = None, post_connect_single_breaks: bool = False, min_new_track_points: int = 0, track_matching_queue: Deque[sleap.nn.tracking.MatchedFrameInstances] = NOTHING, spawned_tracks: List[sleap.instance.Track] = NOTHING, save_tracked_instances: bool = False, tracked_instances: Dict[int, List[InstanceType]] = NOTHING, last_matches: Optional[sleap.nn.tracker.components.FrameMatches] = None, similarity_function: Callable = <function instance_similarity>, matching_function: Callable = <function greedy_matching>, candidate_maker: object = NOTHING)

A Tracker pre-configured to use optical flow shifted candidates.

class sleap.nn.tracking.KalmanTracker(init_tracker: Optional[sleap.nn.tracking.Tracker], init_set: sleap.nn.tracking.KalmanInitSet, kalman_tracker: sleap.nn.tracker.kalman.BareKalmanTracker, cull_function: Optional[Callable] = None, init_frame_count: int = 10, re_init_cooldown: int = 100, re_init_after: int = 20, init_done: bool = False, pre_tracked: bool = False, last_t: int = 0, last_init_t: int = 0)

Class for Kalman filter-based tracking pipeline.

Kalman filters need to be initialized with a certain number of already tracked instances.

Parameters

• init_tracker – The regular Tracker we can use to track data needed for initializing Kalman filters. If not specified, then you can use pre-tracked data (i.e., track assignments already set on instances) if pre_tracked is True.

• init_set – Object to keep track of tracked “init” data and determine when we have enough good data to initialize filters.

• kalman_tracker – The object which handles the actual Kalman filter-based tracking.

• cull_function – If given, this is called to cull instances before tracking.

• init_frame_count – The target number of instances/identities per frame.

• re_init_cooldown – Number of frames to wait after initializing filters before checking if we need to re-init (because they aren’t successfully matching tracks).

• re_init_after – If there’s a gap of this many frames since filters have matched tracks (and we’ve also waited for cooldown frames), start using the regular tracker so that we can re-initialize Kalman filters.

• init_done – Keeps track of whether we’re initialized the filters yet.

• pre_tracked – Whether to use init_tracker or tracks already set on instances.

• last_t – The last frame index we’ve tracked.

• last_init_t – The last frame index on which Kalman filters were initialized; used to checking cooldown period.

property is_valid

Do we have everything we need to run tracking?

classmethod make_tracker(init_tracker: Optional[sleap.nn.tracking.Tracker], node_indices: List[int], instance_count: int, instance_iou_threshold: float = 0.8, init_frame_count: int = 10)

Creates KalmanTracker object.

Parameters

• init_tracker – The Kalman filters need to be initialized with data that’s already been tracked. This is a regular Tracker which can be used to generate this tracked data (when needed).

• node_indices – Which nodes to track using Kalman filters; these should be nodes that are reliably present in the predictions.

• instance_count – The target number of instances to track per frame. A distinct Kalman filter is created/initialized to track each distinct identity. We’ll also use this to cull the number of predicted instances before trying to track.

• instance_iou_threshold – This is the IOU threshold so that we first cull instances which have high overlap.

• init_frame_count – How many frames of tracked data to use when initializing Kalman filters.

track(untracked_instances: List[InstanceType], img: Optional[numpy.ndarray] = None, t: int = None) → List[InstanceType]

Tracks individual frame, using Kalman filters if possible.

class sleap.nn.tracking.SimpleCandidateMaker(min_points: int = 0)

Class for producing list of matching candidates from prior frames.

class sleap.nn.tracking.SimpleTracker(track_window: int = 5, cleaner: Optional[Callable] = None, target_instance_count: int = 0, pre_cull_function: Optional[Callable] = None, post_connect_single_breaks: bool = False, min_new_track_points: int = 0, track_matching_queue: Deque[sleap.nn.tracking.MatchedFrameInstances] = NOTHING, spawned_tracks: List[sleap.instance.Track] = NOTHING, save_tracked_instances: bool = False, tracked_instances: Dict[int, List[InstanceType]] = NOTHING, last_matches: Optional[sleap.nn.tracker.components.FrameMatches] = None, similarity_function: Callable = <function instance_iou>, matching_function: Callable = <function hungarian_matching>, candidate_maker: object = NOTHING)

A Tracker pre-configured to use simple, non-image-based candidates.

class sleap.nn.tracking.TrackCleaner(instance_count: int, iou_threshold: Optional[float] = None)

Class for merging breaks in the predicted tracks.

Method: 1. You specify how many instances there should be in each frame. 2. The lowest scoring instances beyond this limit are deleting from each frame. 3. Going frame by frame, any time there’s exactly one missing track and exactly

one new track, we merge the new track into the missing track.

You should review the results to check for “swaps”. This can be done using the velocity threshold suggestion method.

instance_count

The maximum number of instances we want per frame.

iou_threshold

Intersection over Union (IOU) threshold to use when removing overlapping instances over target count; if None, then only use score to determine which instances to remove.

class sleap.nn.tracking.Tracker(track_window: int = 5, similarity_function: Optional[Callable] = <function instance_similarity>, matching_function: Callable = <function greedy_matching>, candidate_maker: object = NOTHING, cleaner: Optional[Callable] = None, target_instance_count: int = 0, pre_cull_function: Optional[Callable] = None, post_connect_single_breaks: bool = False, min_new_track_points: int = 0, track_matching_queue: Deque[sleap.nn.tracking.MatchedFrameInstances] = NOTHING, spawned_tracks: List[sleap.instance.Track] = NOTHING, save_tracked_instances: bool = False, tracked_instances: Dict[int, List[InstanceType]] = NOTHING, last_matches: Optional[sleap.nn.tracker.components.FrameMatches] = None)

Instance pose tracker.

Use by instantiated with the desired parameters and then calling the track method for each frame.

track_window

How many frames back to look for candidate instances to match instances in the current frame against.

similarity_function

A function that returns a numeric pairwise instance similarity value.

matching_function

A function that takes a matrix of pairwise similarities and determines the matches to use.

candidate_maker

A class instance with a get_candidates method which returns a list of Instances-like objects which we can match the predicted instances in a frame against.

cleaner

A class with a run method which attempts to clean tracks after the other tracking has run for all frames.

min_new_track_points

We won’t spawn a new track for an instance with fewer than this many points.

final_pass(frames: List[sleap.instance.LabeledFrame])

Called after tracking has run on all frames to do any post-processing.

track(untracked_instances: List[InstanceType], img: Optional[numpy.ndarray] = None, t: int = None) → List[InstanceType]

Performs a single step of tracking.

Parameters

• untracked_instances – List of instances to assign to tracks.

• img – Image data of the current frame for flow shifting.

• t – Current timestep. If not provided, increments from the internal queue.

Returns

A list of the instances that were tracked.

property unique_tracks_in_queue

Returns the unique tracks in the matching queue.

sleap.nn.tracking.run_tracker(frames: List[sleap.instance.LabeledFrame], tracker: sleap.nn.tracking.BaseTracker) → List[sleap.instance.LabeledFrame]

Run a tracker on a set of labeled frames.

Parameters

• frames – A list of labeled frames with instances.

• tracker – An initialized Tracker.

Returns

The input frames with the new tracks assigned. If the frames already had tracks, they will be cleared if the tracker has been re-initialized.