×

Real-time hand tracking using a mean shift embedded particle filter. (English) Zbl 1111.68648

Summary: Particle filtering and Mean Shift (MS) are two successful approaches to visual tracking. Both have their respective strengths and weaknesses. In this paper, we propose to integrate advantages of the two approaches for improved tracking. By incorporating the MS optimization into particle filtering to move particles to local peaks in the likelihood, the proposed Mean Shift Embedded Particle Filter (MSEPF) improves the sampling efficiency considerably. Our work is conducted in the context of developing a hand control interface for a robotic wheelchair. We realize real-time hand tracking in dynamic environments of the wheelchair using MSEPF. Extensive experimental results demonstrate that MSEPF outperforms the MS tracker and the conventional particle filter in hand tracking. Our approach produces reliable tracking while effectively handling rapid motion and distraction with roughly 85% fewer particles. We also present a simple method for dynamic gesture recognition. The hand control interface based on the proposed algorithms works well in dynamic environments of the wheelchair.

MSC:

68T10 Pattern recognition, speech recognition
PDFBibTeX XMLCite
Full Text: DOI

References:

[1] Y. Wu, Vision and learning for intelligent human-computer interaction, Ph.D. Thesis, University of Illinois at Urbana-Champaign, 2001.; Y. Wu, Vision and learning for intelligent human-computer interaction, Ph.D. Thesis, University of Illinois at Urbana-Champaign, 2001.
[2] Pavlovic, V.; Sharma, R.; Huang, T., Visual interpretation of hand gestures for human-computer interaction: a review, IEEE Trans. Pattern Anal. Mach. Intell., 19, 7, 677-695 (1997)
[3] McAllister, G.; McKenna, S.; Ricketts, I., Hand tracking for behaviour understanding, Image Vision Comput., 20, 12, 827-840 (2002)
[4] K. Oka, Y. Sato, H. Koike, Real-time tracking of multiple fingertips and gesture recognition for augmented desk interface systems, in: Proceedings of IEEE International Conference on Automated Face and Gesture Recognition (FG’02), 2002, pp. 411-416.; K. Oka, Y. Sato, H. Koike, Real-time tracking of multiple fingertips and gesture recognition for augmented desk interface systems, in: Proceedings of IEEE International Conference on Automated Face and Gesture Recognition (FG’02), 2002, pp. 411-416.
[5] I. Laptev, T. Lindeberg, Tracking of multi-state hand models using particle filtering and a hierarchy of multi-scale image features, in: Proceedings of IEEE Workshop on Scale-Space and Morphology, 2001.; I. Laptev, T. Lindeberg, Tracking of multi-state hand models using particle filtering and a hierarchy of multi-scale image features, in: Proceedings of IEEE Workshop on Scale-Space and Morphology, 2001. · Zbl 0991.68584
[6] H. Fei, I. Reid, Probabilistic tracking and recognition of non-rigid hand motion, in: Proceedings of IEEE International Workshop on Analysis and Modeling of Faces and Gestures (AMFG’03), 2003, pp. 60-67.; H. Fei, I. Reid, Probabilistic tracking and recognition of non-rigid hand motion, in: Proceedings of IEEE International Workshop on Analysis and Modeling of Faces and Gestures (AMFG’03), 2003, pp. 60-67.
[7] S. Lu, D. Metaxas, D. Samaras, J. Oliensis, Using multiple cues for hand tracking and model refinement, in: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR’03), 2003, pp. II: 443-450.; S. Lu, D. Metaxas, D. Samaras, J. Oliensis, Using multiple cues for hand tracking and model refinement, in: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR’03), 2003, pp. II: 443-450.
[8] H. Zhou, T. Huang, Tracking articulated hand motion with eigen dynamics analysis in: Proceedings of IEEE International Conference on Computer Vision (ICCV’03), 2003, pp. 1102-1109.; H. Zhou, T. Huang, Tracking articulated hand motion with eigen dynamics analysis in: Proceedings of IEEE International Conference on Computer Vision (ICCV’03), 2003, pp. 1102-1109.
[9] Tsap, L., Gesture-tracking in real time with dynamic regional range computation, Real Time Imaging, 8, 2, 115-126 (2002) · Zbl 1011.68778
[10] A. Shamaie, A. Sutherland, A dynamic model for real-time tracking of hands in bimanual movements, Gesture-Based Communication in Human-Computer Interaction (International Gesture Workshop, GW03).; A. Shamaie, A. Sutherland, A dynamic model for real-time tracking of hands in bimanual movements, Gesture-Based Communication in Human-Computer Interaction (International Gesture Workshop, GW03).
[11] E.B. Sudderth, M.I. Mandel, W.T. Freeman, A.S. Willsky, Visual hand tracking using nonparametric belief propagation, in: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition Workshop (CVPRW’04), 2004.; E.B. Sudderth, M.I. Mandel, W.T. Freeman, A.S. Willsky, Visual hand tracking using nonparametric belief propagation, in: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition Workshop (CVPRW’04), 2004.
[12] M. Kolsch, M. Turk, Fast 2d hand tracking with flocks of features and multi-cue integration, in: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition Workshop (CVPRW’04), 2004.; M. Kolsch, M. Turk, Fast 2d hand tracking with flocks of features and multi-cue integration, in: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition Workshop (CVPRW’04), 2004.
[13] Q. Yuan, S. Sclaroff, V. Athitsos, Automatic 2d hand tracking in video sequences in: Proceedings of IEEE Workshop on Applications of Computer Vision (WACV’05), 2005.; Q. Yuan, S. Sclaroff, V. Athitsos, Automatic 2d hand tracking in video sequences in: Proceedings of IEEE Workshop on Applications of Computer Vision (WACV’05), 2005.
[14] Inaguma, T.; Saji, H.; Nakatani, H., Hand motion tracking based on a constraint of three-dimensional continuity, J. Electron. Imaging, 14, 1, 013021 (2005)
[15] W.Y. Chang, C.S. Chen, Y.P. Hung, Appearance-guided particle filtering for articulated hand tracking, in: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR’05), 2005.; W.Y. Chang, C.S. Chen, Y.P. Hung, Appearance-guided particle filtering for articulated hand tracking, in: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR’05), 2005.
[16] Wu, S.; Hong, L., Hand tracking in a natural conversational environment by the interacting multiple model and probabilistic data association (imm-pda) algorithm, Pattern Recognition, 38, 2143-2158 (2005)
[17] Stenger, B.; Thayananthan, A.; Torr, P. H.S.; Cipolla, R., Model-based hand tracking using a hierarchical Bayesian filter, IEEE Trans. Pattern Anal. Mach. Intell., 28, 9, 1372-1384 (2006)
[18] Isard, M.; Blake, A., Condensation—conditional density propagation for visual tracking, Int. J. Comput. Vision, 29, 1, 5-28 (1998)
[19] Ng, C.; Ranganath, S., Real-time gesture recognition system and application, Image Vision Comput., 20, 13-14, 993-1007 (2002)
[20] M. Isard, A. Blake, ICONDENSATION: unifying low-level tracking in a stochastic framework, in: Proceedings of European Conference on Computer Vision (ECCV’98), vol. 1, Freiburg, Germany, 1998, pp. 893-908.; M. Isard, A. Blake, ICONDENSATION: unifying low-level tracking in a stochastic framework, in: Proceedings of European Conference on Computer Vision (ECCV’98), vol. 1, Freiburg, Germany, 1998, pp. 893-908.
[21] Yang, M.; Ahuja, N.; Tabb, M., Extraction of 2d motion trajectories and its application to hand gesture recognition, IEEE Trans. Pattern Anal. Mach. Intell., 24, 8, 1061-1074 (2002)
[22] Vision based hand gesture recognition systems. URL \(\langle;\) http://ls7-www.cs.uni-dortmund.de/research/gesture/vbgr-table.html \(\rangle;\); Vision based hand gesture recognition systems. URL \(\langle;\) http://ls7-www.cs.uni-dortmund.de/research/gesture/vbgr-table.html \(\rangle;\)
[23] M. Bray, E. Koller-Meier, L. Van Gool, Smart particle filtering for 3d hand tracking, in: Proceedings of IEEE International Conference on Automated Face and Gesture Recognition (FG’04), 2004, pp. 675-680.; M. Bray, E. Koller-Meier, L. Van Gool, Smart particle filtering for 3d hand tracking, in: Proceedings of IEEE International Conference on Automated Face and Gesture Recognition (FG’04), 2004, pp. 675-680.
[24] Arulampalam, M.; Maskell, S.; Gordon, N.; Clapp, T., A tutorial on particle filters for online nonlinear/non-Gaussian Bayesian tracking, IEEE Trans. Signal Process., 50, 2, 174-189 (2002)
[25] Y. Rui, Y. Chen, Better proposal distributions: object tracking using the unscented particle filter, in: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR’01), vol. 2, Hawaii, 2001, pp. 786-793.; Y. Rui, Y. Chen, Better proposal distributions: object tracking using the unscented particle filter, in: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR’01), vol. 2, Hawaii, 2001, pp. 786-793.
[26] J. MacCormick, M. Isard, Partitioned sampling, articulated objects, and interface-quality hand tracking, in: Proceedings of European Conference on Computer Vision (ECCV’00), 2000, pp. 3-19.; J. MacCormick, M. Isard, Partitioned sampling, articulated objects, and interface-quality hand tracking, in: Proceedings of European Conference on Computer Vision (ECCV’00), 2000, pp. 3-19.
[27] T. Cham, J.M. Rehg, A multiple hypothesis approach to figure tracking, in: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR’99), 1999, pp. 239-245.; T. Cham, J.M. Rehg, A multiple hypothesis approach to figure tracking, in: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR’99), 1999, pp. 239-245.
[28] J. Deutscher, A. Blake, I. Reid, Articulated body motion capture by annealled particle filtering, in: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR’00), 2000, pp. 126-133.; J. Deutscher, A. Blake, I. Reid, Articulated body motion capture by annealled particle filtering, in: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR’00), 2000, pp. 126-133.
[29] K. Choo, D.J. Fleet, People tracking using hybrid Monte Carlo filtering, in: Proceedings of IEEE International Conference on Computer Vision (ICCV’01), 2001, pp. 1068-1075.; K. Choo, D.J. Fleet, People tracking using hybrid Monte Carlo filtering, in: Proceedings of IEEE International Conference on Computer Vision (ICCV’01), 2001, pp. 1068-1075.
[30] C. Chang, R. Ansari, Kernel particle filter: iterative sampling for efficient visual tracking, in: Proceedings of IEEE International Conference on Image Processing (ICIP’03), 2003.; C. Chang, R. Ansari, Kernel particle filter: iterative sampling for efficient visual tracking, in: Proceedings of IEEE International Conference on Image Processing (ICIP’03), 2003.
[31] B. Han, D. Comaniciu, Y. Zhu, L. Davis, Incremental density approximation and kernel-based Bayesian filtering for object tracking, in: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR’04), Washington, DC, 2004.; B. Han, D. Comaniciu, Y. Zhu, L. Davis, Incremental density approximation and kernel-based Bayesian filtering for object tracking, in: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR’04), Washington, DC, 2004.
[32] D. Comaniciu, V. Ramesh, P. Meer, Real-time tracking of non-rigid objects using mean shift, in: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR’00), South Carolina, 2000, pp. 142-149.; D. Comaniciu, V. Ramesh, P. Meer, Real-time tracking of non-rigid objects using mean shift, in: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR’00), South Carolina, 2000, pp. 142-149.
[33] C. Shan, Y. Wei, T. Tan, F. Ojardias, Real time hand tracking by combining particle filtering and mean shift, in: Proceedings of IEEE International Conference on Automated Face and Gesture Recognition (FG’04), Seoul, Korea, 2004, pp. 669-674.; C. Shan, Y. Wei, T. Tan, F. Ojardias, Real time hand tracking by combining particle filtering and mean shift, in: Proceedings of IEEE International Conference on Automated Face and Gesture Recognition (FG’04), Seoul, Korea, 2004, pp. 669-674.
[34] P. Perez, C. Hue, J. Vermaak, M. Gangnet, Color-based probabilistic tracking, in: Proceedings of European Conference on Computer Vision (ECCV’02), 2002, pp. I: 661-675.; P. Perez, C. Hue, J. Vermaak, M. Gangnet, Color-based probabilistic tracking, in: Proceedings of European Conference on Computer Vision (ECCV’02), 2002, pp. I: 661-675. · Zbl 1034.68652
[35] Nummiaro, K.; Koller-Meier, E.; Van Gool, L., An adaptive color-based particle filter, Image Vision Comput., 21, 1, 99-110 (2003)
[36] G. Bradski, Computer vision face tracking for use in a perceptual user interface, Intel Technol. J. Q2.; G. Bradski, Computer vision face tracking for use in a perceptual user interface, Intel Technol. J. Q2.
[37] D. Comaniciu, V. Ramesh, Mean shift and optimal prediction for efficient object tracking, in: Proceedings of IEEE Conference on Image Processing (ICIP’00), Vancouver, Canada, 2000, pp. 70-73.; D. Comaniciu, V. Ramesh, Mean shift and optimal prediction for efficient object tracking, in: Proceedings of IEEE Conference on Image Processing (ICIP’00), Vancouver, Canada, 2000, pp. 70-73.
[38] C. Yang, R. Duraiswami, L. Davis, Efficient mean-shift tracking via a new similarity measure, in: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR’05), 2005.; C. Yang, R. Duraiswami, L. Davis, Efficient mean-shift tracking via a new similarity measure, in: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR’05), 2005.
[39] Blake, A.; Isard, M., Active Contours (1998), Springer: Springer Berlin
[40] M. Soriano, B. Martinkauppi, S. Huovinen, M. Laaksonen, Skin detection in video under changing illumination conditions, in: Proceedings of International Conference on Pattern Recognition (ICPR’00), 2000, pp. 839-842.; M. Soriano, B. Martinkauppi, S. Huovinen, M. Laaksonen, Skin detection in video under changing illumination conditions, in: Proceedings of International Conference on Pattern Recognition (ICPR’00), 2000, pp. 839-842.
[41] H.P. Graf, E. Cosatto, D. Gibbon, M. Kocheisen, E. Petajan, Multi-modal system for locating heads and faces, in: Proceedings of International Conference on Automatic Face and Gesture Recognition (FG’96), 1996, pp. 88-93.; H.P. Graf, E. Cosatto, D. Gibbon, M. Kocheisen, E. Petajan, Multi-modal system for locating heads and faces, in: Proceedings of International Conference on Automatic Face and Gesture Recognition (FG’96), 1996, pp. 88-93.
[42] Starner, T.; Weaver, J.; Pentland, A., Real-time american sign language recognition using desk and wearable computer based video, IEEE Trans. Pattern Anal. Mach. Intell., 20, 12, 1371-1375 (1998)
[43] M.J. Black, A.D. Jepson, Recognition temporal trajectories using the CONDENSATION algorithm, in: Proceedings of IEEE International Conference on Automated Face and Gesture Recognition (FG’98), Japan, 1998, pp. 16-21.; M.J. Black, A.D. Jepson, Recognition temporal trajectories using the CONDENSATION algorithm, in: Proceedings of IEEE International Conference on Automated Face and Gesture Recognition (FG’98), Japan, 1998, pp. 16-21.
[44] Psarrou, A.; Gong, S.; Walter, M., Recognition of human gestures and behaviour based on motion trajectories, Image Vision Comput., 20, 5-6, 349-358 (2002)
[45] Bobick, A.; Davis, J., The recognition of human movement using temporal templates, IEEE Trans. Pattern Anal. Mach. Intell., 23, 3, 257-267 (2001)
[46] Y. Boers, On the number of samples to be drawn in particle filtering, in: IEE Colloquium on Target Tracking: Algorithms and Applications, 1999, pp. 5/1-5/6.; Y. Boers, On the number of samples to be drawn in particle filtering, in: IEE Colloquium on Target Tracking: Algorithms and Applications, 1999, pp. 5/1-5/6.
[47] D. Koeller, R. Fratkina, Using learning for approximation in stochastic processes, in: Proceedings of International Conference on Machine Learning (ICML’98), 1998.; D. Koeller, R. Fratkina, Using learning for approximation in stochastic processes, in: Proceedings of International Conference on Machine Learning (ICML’98), 1998.
[48] D. Fox, W. Burgard, F. Dellaert, S. Thrun, Monte Carlo localization: efficient position estimation for mobile robots, in: Proceedings of National Conference on Artificial Intelligence (AAAI), 1999.; D. Fox, W. Burgard, F. Dellaert, S. Thrun, Monte Carlo localization: efficient position estimation for mobile robots, in: Proceedings of National Conference on Artificial Intelligence (AAAI), 1999. · Zbl 0971.68162
[49] D. Fox, Kld-sampling: adaptive particle filters, in: Advances in Neural Information Processing Systems (NIPS), 2001.; D. Fox, Kld-sampling: adaptive particle filters, in: Advances in Neural Information Processing Systems (NIPS), 2001.
[50] A. Soto, Self adaptive particle filter, in: Proceedings of International Joint Conference on Artificial Intelligence (IJCAI’05), 2005.; A. Soto, Self adaptive particle filter, in: Proceedings of International Joint Conference on Artificial Intelligence (IJCAI’05), 2005.
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.