Inference

sleap.nn.inference

Class and command-line interface for running inference with trained models.

class sleap.nn.inference.Predictor(policies: Dict[str, object], tracker_takes_img: bool = False)

Encapsulates the inference pipeline.

policies

A dictionary with objects for different parts of the pipeline:

• “centroid”: Instance of region_proposal.CentroidPredictor

• “region”: Instance of region_proposal.RegionProposalExtractor

• “confmap”: Instance of peak_finding.ConfmapPeakFinder

• “paf”: Instance of paf_grouping.PAFGrouper

• “tracking”: Instance of tracking.Tracker

• “previous_predictions”: predicted.PredictedInstancePredictor

Note: the pipeline will be determined by which policies are given.

make_labeled_frames(instances_chunk: Dict[int, List[PredictedInstance]], frame_inds, sample_inds, video: Video) → List[sleap.instance.LabeledFrame]

Makes LabeledFrame objects for all predictions in chunk.

predict(video_filename: str, frames: Optional[List[int]] = None, video_kwargs: Optional[dict] = None) → List[sleap.instance.LabeledFrame]

Runs entire inference pipeline on frames from a video file.

predict_chunk(img_chunk, chunk_ind, chunk_size, frame_inds=None)

Runs the inference components of pipeline for a chunk.

track_chunk(predicted_instances_chunk, frame_inds, sample_inds, img_chunk)

Runs tracker for each frame in chunk.

track_next_sample(untracked_instances: List[PredictedInstance], t: int = None, img=None) → List[PredictedInstance]

Runs tracker for a single frame.

sleap.nn.inference.main()

CLI for running inference.

sleap.nn.tracking

class sleap.nn.tracking.FlowCandidateMaker(min_points: int = 0, img_scale: float = 1.0, of_window_size: int = 21, of_max_levels: int = 3, save_shifted_instances: bool = False, shifted_instances: Dict[Tuple[int, int], List[sleap.nn.tracking.ShiftedInstance]] = NOTHING)

Class for producing optical flow shift matching candidates.

static flow_shift_instances(ref_instances: List[InstanceType], ref_img: numpy.ndarray, new_img: numpy.ndarray, min_shifted_points: int = 0, scale: float = 1.0, window_size: int = 21, max_levels: int = 3) → List[sleap.nn.tracking.ShiftedInstance]

Generates instances in a new frame by applying optical flow displacements.

Parameters

• ref_instances – Reference instances in the previous frame.

• ref_img – Previous frame image as a numpy array.

• new_img – New frame image as a numpy array.

• min_shifted_points – Minimum number of points that must be detected in the new frame in order to generate a new shifted instance.

• scale – Factor to scale the images by when computing optical flow. Decrease this to increase performance at the cost of finer accuracy. Sometimes decreasing the image scale can improve performance with fast movements.

• window_size – Optical flow window size to consider at each pyramid scale level.

• max_levels – Number of pyramid scale levels to consider. This is different from the scale parameter, which determines the initial image scaling.

Returns

A list of ShiftedInstances with the optical flow displacements applied to the reference instance points. Points that are not found will be represented as NaNs in the points array for each shifted instance.

Notes

This function relies on the Lucas-Kanade method for optical flow estimation.

class sleap.nn.tracking.FlowTracker(track_window: int = 5, cleaner: Optional[Callable] = None, min_new_track_points: int = 0, track_matching_queue: Deque[sleap.nn.tracking.MatchedInstance] = NOTHING, spawned_tracks: List[sleap.instance.Track] = NOTHING, save_tracked_instances: bool = False, tracked_instances: Dict[int, List[InstanceType]] = NOTHING, similarity_function: Callable = <function instance_similarity>, matching_function: Callable = <function greedy_matching>, candidate_maker: object = NOTHING)

A Tracker pre-configured to use optical flow shifted candidates.

class sleap.nn.tracking.SimpleCandidateMaker(min_points: int = 0)

Class for producing list of matching candidates from prior frames.

class sleap.nn.tracking.SimpleTracker(track_window: int = 5, cleaner: Optional[Callable] = None, min_new_track_points: int = 0, track_matching_queue: Deque[sleap.nn.tracking.MatchedInstance] = NOTHING, spawned_tracks: List[sleap.instance.Track] = NOTHING, save_tracked_instances: bool = False, tracked_instances: Dict[int, List[InstanceType]] = NOTHING, similarity_function: Callable = <function instance_iou>, matching_function: Callable = <function hungarian_matching>, candidate_maker: object = NOTHING)

A Tracker pre-configured to use simple, non-image-based candidates.

class sleap.nn.tracking.TrackCleaner(instance_count: int)

Class for merging breaks in the predicted tracks.

Method: 1. You specify how many instances there should be in each frame. 2. The lowest scoring instances beyond this limit are deleting from each frame. 3. Going frame by frame, any time there’s exactly one missing track and exactly

one new track, we merge the new track into the missing track.

You should review the results to check for “swaps”. This can be done using the velocity threshold suggestion method.

instance_count

The maximum number of instances we want per frame.

run(frames: List[LabeledFrame])

Attempts to merge tracks for given frames.

Parameters

frames – The list of LabeldFrame objects with predictions.

Returns

None; modifies frames in place.

class sleap.nn.tracking.Tracker(track_window: int = 5, similarity_function: Callable = <function instance_similarity>, matching_function: Callable = <function greedy_matching>, candidate_maker: object = NOTHING, cleaner: Optional[Callable] = None, min_new_track_points: int = 0, track_matching_queue: Deque[sleap.nn.tracking.MatchedInstance] = NOTHING, spawned_tracks: List[sleap.instance.Track] = NOTHING, save_tracked_instances: bool = False, tracked_instances: Dict[int, List[InstanceType]] = NOTHING)

Instance pose tracker.

Use by instantiated with the desired parameters and then calling the track method for each frame.

track_window

How many frames back to look for candidate instances to match instances in the current frame against.

similarity_function

A function that returns a numeric pairwise instance similarity value.

matching_function

A function that takes a matrix of pairwise similarities and determines the matches to use.

candidate_maker

A class instance with a get_candidates method which returns a list of Instances-like objects which we can match the predicted instances in a frame against.

cleaner

A class with a run method which attempts to clean tracks after the other tracking has run for all frames.

min_new_track_points

We won’t spawn a new track for an instance with fewer than this many points.

final_pass(frames: List[sleap.instance.LabeledFrame])

Called after tracking has run on all chunks.

track(untracked_instances: List[InstanceType], img: Optional[numpy.ndarray] = None, t: int = None) → List[InstanceType]

Performs a single step of tracking.

Parameters

• untracked_instances – List of instances to assign to tracks.

• img – Image data of the current frame for flow shifting.

• t – Current timestep. If not provided, increments from the internal queue.

Returns

A list of the instances that were tracked.

property unique_tracks_in_queue

Returns the unique tracks in the matching queue.

sleap.nn.tracking.centroid_distance(ref_instance: InstanceType, query_instance: InstanceType, cache: dict = {}) → float

Returns the negative distance between the centroids of two instances.

Uses cache dictionary (created with function so it persists between calls) since without cache this method is significantly slower than others.

sleap.nn.tracking.greedy_matching(cost_matrix: numpy.ndarray) → List[Tuple[int, int]]

Performs greedy bipartite matching.

sleap.nn.tracking.hungarian_matching(cost_matrix: numpy.ndarray) → List[Tuple[int, int]]

Wrapper for Hungarian matching algorithm in scipy.

sleap.nn.tracking.instance_iou(ref_instance: InstanceType, query_instance: InstanceType, cache: dict = {}) → float

Computes IOU between bounding boxes of instances.

sleap.nn.tracking.instance_similarity(ref_instance: InstanceType, query_instance: InstanceType) → float

Computes similarity between instances.