sleap.nn.peak_finding

This module contains TensorFlow-based peak finding methods.

In general, the inputs to the functions provided here operate on confidence maps (sometimes referred to as heatmaps), which are image-based representations of the locations of landmark coordinates.

In these representations, landmark locations are encoded as probability that it is present each pixel. This is often represented by an unnormalized 2D Gaussian PDF centered at the true location and evaluated over the entire image grid.

Peak finding entails finding either the global or local maxima of these confidence maps.

sleap.nn.peak_finding.crop_centered_boxes(img: tensorflow.python.framework.ops.Tensor, peaks: tensorflow.python.framework.ops.Tensor, window_length: int) → tensorflow.python.framework.ops.Tensor[source]

Crops boxes centered around peaks.

Parameters

  • img – Tensor of shape (samples, height, width, channels).

  • peaks – Tensor of shape (n_peaks, 4) where subscripts of peak locations are specified in each row as [sample, row, col, channel].

  • window_length – Size (width and height) of windows to be cropped. This parameter will be rounded up to nearest odd number.

Returns

A tensor of shape (n_peaks, window_length, window_length, 1) corresponding to the box cropped around each peak.

sleap.nn.peak_finding.ensure_odd(x: tensorflow.python.framework.ops.Tensor) → tensorflow.python.framework.ops.Tensor[source]

Rounds numbers up to the nearest odd value.

sleap.nn.peak_finding.find_offsets_local_direction(centered_patches: tensorflow.python.framework.ops.Tensor, delta: float) → tensorflow.python.framework.ops.Tensor[source]

Computes subpixel offsets from the direction of the pixels around the peak.

This function finds the delta-offset from the center pixel of peak-centered patches by finding the direction of the gradient around each center.

Parameters

  • centered_patches – A rank-4 tensor of shape (samples, 3, 3, 1) corresponding to the centered crops around the grid-anchored peaks. For multi-channel images, stack the channels along the samples axis before calling this function.

  • delta – Scalar float that will scaled by the gradient direction.

Returns

offsets, a float32 tensor of shape (samples, 2) where the columns correspond to the offsets relative to the center pixel for the y and x directions respectively, i.e., for the i-th sample:

dy_i, dx_i = offsets[i]

Note

For symmetric patches, the offset will be 0.

Example

>>> find_offsets_local_direction(np.array(
...     [[0., 1., 0.],
...      [1., 3., 2.],
...      [0., 1., 0.]]).reshape(1, 3, 3, 1), 0.25)
<tf.Tensor: id=21250, shape=(1, 2), dtype=float64, numpy=array([[0.  , 0.25]])>
sleap.nn.peak_finding.make_gaussian_kernel(size: int, sigma: float) → tensorflow.python.framework.ops.Tensor[source]

Generates a square unnormalized 2D symmetric Gaussian kernel.

Parameters

  • size – Length of kernel. This should be an odd integer.

  • sigma – Standard deviation of the Gaussian specified as a scalar float.

Returns

kernel, a float32 tensor of shape (size, size) with values corresponding to the unnormalized probability density of a 2D Gaussian distribution with symmetric covariance along the x and y directions.

Note

The maximum value of this kernel will be 1.0. To normalize it, divide each element by (2 * np.pi * sigma ** 2).

sleap.nn.peak_finding.refine_peaks_local_direction(imgs: tensorflow.python.framework.ops.Tensor, peaks: tensorflow.python.framework.ops.Tensor, delta: float = 0.25) → tensorflow.python.framework.ops.Tensor[source]

Refines peaks by applying a fixed offset along the gradients around the peaks.

This function wraps other methods to refine peak coordinates by:

  1. Cropping 3 x 3 patches around each peak.

  2. Stacking patches along the samples axis.

  3. Computing the local gradient around each centered patch.

  4. Applying subpixel offsets to each peak.

This is a commonly used algorithm for subpixel peak refinement, described for pose estimation applications in [1].

Parameters

  • imgs – A float32 tensor of shape (samples, height, width, channels) in which the peaks were detected.

  • peaks – Tensor of shape (n_peaks, 4) where subscripts of peak locations are specified in each row as [sample, row, col, channel].

  • delta – Scalar float specifying the step to take along the local peak gradients.

Returns

refined_peaks, a float32 tensor of shape (n_peaks, 4) in the same format as the input peaks, but with offsets applied.

References

1

Alejandro Newell, Kaiyu Yang, and Jia Deng. Stacked Hourglass Networks for Human Pose Estimation. In _European conference on computer vision_, 2016.

sleap.nn.peak_finding.smooth_imgs(imgs: tensorflow.python.framework.ops.Tensor, kernel_size: int = 5, sigma: float = 1.0) → tensorflow.python.framework.ops.Tensor[source]

Smooths the input image by convolving it with a Gaussian kernel.

Parameters

  • imgs – A rank-4 tensor of shape (samples, height, width, channels).

  • kernel_size – An odd-valued scalar integer specifying the width and height of the Gaussian kernel.

  • sigma – Standard deviation of the Gaussian specified as a scalar float.

Returns

A tensor of the same shape as imgs after convolving with a Gaussian sample and channelwise.