sleap.nn.paf_grouping

This module provides a set of utilities for grouping peaks based on PAFs.

Part affinity fields (PAFs) are a representation used to resolve the peak grouping problem for multi-instance pose estimation [1].

They are a convenient way to represent directed graphs with support in image space. For each edge, a PAF can be represented by an image with two channels, corresponding to the x and y components of a unit vector pointing along the direction of the underlying directed graph formed by the connections of the landmarks belonging to an instance.

Given a pair of putatively connected landmarks, the agreement between the line segment that connects them and the PAF vectors found at the coordinates along the same line can be used as a measure of “connectedness”. These scores can then be used to guide the instance-wise grouping of landmarks.

This image space representation is particularly useful as it is amenable to neural network-based prediction from unlabeled images.

References

1

Zhe Cao, Tomas Simon, Shih-En Wei, Yaser Sheikh. Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields. In _CVPR_, 2017.

sleap.nn.paf_grouping.assign_connections_to_instances(connections: Dict[sleap.nn.paf_grouping.EdgeType, List[sleap.nn.paf_grouping.EdgeConnection]], min_instance_peaks: Union[int, float] = 0, n_nodes: int = None) → Dict[sleap.nn.paf_grouping.PeakID, int][source]

Assigns connected edges to instances via greedy graph partitioning.

Parameters

  • connections – A dict that maps EdgeType to a list of EdgeConnections found through connection scoring. This can be generated by the filter_connection_candidates function.

  • min_instance_peaks – If this is greater than 0, grouped instances with fewer assigned peaks than this threshold will be excluded. If a float in the range (0., 1.] is provided, this is interpreted as a fraction of the total number of nodes in the skeleton. If an integer is provided, this is the absolute minimum number of peaks.

  • n_nodes – Total node type count. Used to convert min_instance_peaks to an absolute number when a fraction is specified. If not provided, the node count is inferred from the unique node inds in connections.

Returns

A dict mapping PeakID to a unique instance ID specified as an integer.

A PeakID is a tuple of (node_type_ind, peak_ind), where the peak_ind is the index or identifier specified in a EdgeConnection as a src_peak_ind or dst_peak_ind.

Return type

instance_assignments

Note

Instance IDs are not necessarily consecutive since some instances may be filtered out during the partitioning or filtering.

This function expects connections from a single sample/frame!