Saliency Detection for Unconstrained Videos Using Superpixel-level Graph and Spatiotemporal Propagation

Z. Liu¹     J. Li¹     L. Ye¹     G. Sun¹     L. Shen¹

Image and Video Processing LAB,Shanghai University¹

 

Abstract

This paper proposes an effective spatiotemporal saliency model for unconstrained videos with complicated motion and complex scenes. First, superpixel-level motion and color histograms as well as global motion histogram are extracted as the features for saliency measurement. Then a superpixel-level graph with the addition of a virtual background node representing the global motion is constructed, and an iterative motion saliency measurement method which utilizes the shortest path algorithm on the graph is exploited to reasonably generate motion saliency maps. Temporal propagation of saliency in both forward and backward directions is performed by using efficient operations on inter-frame similarity matrices to obtain the integrated temporal saliency maps with the better coherence. Finally, spatial propagation of saliency both locally and globally is performed via the use of intra-frame similarity matrices to obtain the spatiotemporal saliency maps with the even better quality. Experimental results on two video datasets with various unconstrained videos demonstrate that the proposed model consistently outperforms the state-of-the-art spatiotemporal saliency models on saliency detection performance.

 

Proposed Model

Illustration of the proposed SGSP model. (a) Original video frames; (b) motion vector fields; (c) superpixel segmentation results; (d) superrpixel-level graph; (e) motion saliency map; (f) forward temporal saliency map; (g) backward temporal saliency map; (h) integrated temporal saliency map; (i) initial spatiotemporal saliency map; (j) final spatiotemporal saliency map.

Results

Examples of spatiotemporal saliency maps for some videos in SegTrackV2 (shown with an interval of 6, 10, and 5 frames, respectively, from top to bottom). (a) Video frames, (b) binary ground truths, and spatiotemporal saliency maps generated using (c) SR, (d) CE, (e) QFT, (f) MB, (g) DCMR, (h) SP, (i) SLT, (j) GD and (k) our SGSP model.

Examples of spatiotemporal saliency maps for some videos in UVSD (shown with an interval of 10, 15, and 40 frames, respectively, from top to bottom). (a) Video frames, (b) binary ground truths, and spatiotemporal saliency maps generated using (c) SR, (d) CE, (e) QFT, (f) MB, (g) DCMR, (h) SP, (i) SLT, (j) GD and (k) our SGSP model.

 

Quantitative Comparison

PR curves of final spatiotemporal saliency (FStS) maps generated using our SGSP model with different number of superpixels on (a) SegTrackV2 and (b) UVSD.

PR curves of different saliency maps generated using our SGSP model, i.e., MS, FTS, BTS, ITS, IStS and FStS, respectively, on (a) SegTrackV2 and (b) UVSD.

PR curves of different saliency models on (a) SegTrackV2 and (b) UVSD.

Comparison with PR curves of saliency maps generated using the belief propagation algorithm and the proposed temporal propagation method on (a) SegTrackV2 and (b) UVSD.

Citation

Z. Liu, J. Li, L. Ye, G. Sun, and L. Shen, "Saliency detection for unconstrained videos using superpixel-level graph and spatiotemporal propagation," IEEE Transactions on Circuits and Systems for Video Technology, Jul. 2016.
@ARTICLE{ Saliency detection for unconstrained videos using superpixel-level graph and spatiotemporal propagation_TCSVT2016,
author={ Zhi Liu, Haojun Li, Linwei Ye, Guanglin Sun, Liquan Shen},
journal={Circuits and Systems for Video Technology, IEEE Transactions on},
title={ Saliency detection for unconstrained videos using superpixel-level graph and spatiotemporal propagation },
year={2016},
doi={10.1109/TCSVT.2016.2595324},
month={Jul}
}.

 

Downloads

"Saliency detection for unconstrained videos using superpixel-level graph and spatiotemporal propagation" Z. Liu, J. Li, L. Ye, G. Sun, and L. Shen, IEEE Transactions on Circuits and Systems for Video Technology, doi.: 10.1109/TCSVT.2016.2595324, Jul. 2016.    [Paper]   [Code]