×

Choosing the kernel parameters for support vector machines by the inter-cluster distance in the feature space. (English) Zbl 1162.68639

Summary: Determining the kernel and error penalty parameters for support vector machines (SVMs) is very problem-dependent in practice. A popular method to deciding the kernel parameters is the grid search method. In the training process, classifiers are trained with different kernel parameters, and only one of the classifiers is required for the testing process. This makes the training process time-consuming. In this paper we propose using the inter-cluster distances in the feature spaces to choose the kernel parameters. Calculating such distance costs much less computation time than training the corresponding SVM classifiers; thus the proper kernel parameters can be chosen much faster. Experiment results show that the inter-cluster distance can choose proper kernel parameters with which the testing accuracy of trained SVMs is competitive to the standard ones, and the training time can be significantly shortened.

MSC:

68T10 Pattern recognition, speech recognition
68T05 Learning and adaptive systems in artificial intelligence

Software:

LIBSVM; Torch
PDFBibTeX XMLCite
Full Text: DOI Link

References:

[1] Vapnik, V., Statistical Learning Theory (1998), Wiley: Wiley New York · Zbl 0935.62007
[2] Hsu, C.-W.; Lin, C.-J., A comparison of methods for multiclass support vector machines, IEEE Trans. Neural Networks, 13, 2, 415-425 (2002)
[3] Ancona, N.; Cicirelli, G.; Stella, E.; Distante, A., Object detection in images: run-time complexity and parameter selection of support vector machines, (16th International Conference on Pattern Recognition, vol. 2 (11-15 August, 2002)), 426-429
[4] Rojas, S. A.; Fernandez-Reyes, D., Adapting multiple kernel parameters for support vector machines using genetic algorithms, (The 2005 IEEE Congress on Evolutionary Computation, vol. 1 (02-05 September, 2005)), 626-631
[5] Liang, X.; Liu, F., Choosing multiple parameters for SVM based on genetic algorithm, (6th International Conference on Signal Processing, vol. 1 (26-30 August, 2002)), 117-119
[6] Liu, H.-J.; Wang, Y.-N.; Lu, X.-F., A method to choose kernel function and its parameters for support vector machines, (Proceedings of 2005 International Conference on Machine Learning and Cybernetics, vol. 7 (18-21 August, 2005)), 4277-4280
[7] Liu, S.; Jia, C.-Y.; Ma, H., A new weighted support vector machine with GA-based parameter selection, (Proceedings of 2005 International Conference on Machine Learning and Cybernetics, vol. 7 (18-21 August, 2005)), 4351-4355
[8] Quang, A. T.; Zhang, Q.-L.; Li, X., Evolving support vector machine parameters, (Proceedings of 2002 International Conference on Machine Learning and Cybernetics, vol. 1 (4-5 November, 2002)), 548-551
[9] Imbault, F.; Lebart, K., A stochastic optimization approach for parameter tuning of support vector machines, (Proceedings of 17th International Conference on Pattern Recognition, vol. 4 (23-26 August, 2004)), 597-600
[10] J.C. Platt, Sequential minimal optimization: a fast algorithm for training support vector machines, Microsoft Research, Technical Report MST-TR-98-14, 1998.; J.C. Platt, Sequential minimal optimization: a fast algorithm for training support vector machines, Microsoft Research, Technical Report MST-TR-98-14, 1998.
[11] Maruyama, K.-I.; Maruyama, M.; Miyao, H.; Nakano, Y., A method to make multiple hypotheses with high cumulative recognition rate using SVMs, Pattern Recognition, 37, 241-251 (2004) · Zbl 1059.68092
[12] Hush, D.; Scovel, C., Polynomial-time decomposition algorithms for support vector machines, Mach. Learn., 51, 51V71 (2003)
[13] Debnath, R.; Takahashi, H., An efficient method for tuning kernel parameter of the support vector machine, (IEEE International Symposium on Communications and Information Technology, vol. 2 (26-29 October, 2004)), 1023-1028
[14] Bi, L.-P.; Huang, H.; Zheng, Z.-Y.; Song, H.-T., New heuristic for determination Gaussian kernel’s parameter, (Proceedings of 2005 International Conference on Machine Learning and Cybernetics, vol. 7 (18-21 August, 2005)), 4299-4304
[15] Keerthi, S. S.; Lin, C.-J., Asymptotic behaviors of support vector machines with Gaussian kernel, Neural Comput., 15, 7, 1667-1689 (2003) · Zbl 1086.68569
[16] H.-T. Lin, C.-J. Lin, A study on sigmoid kernels for SVM and the training of non-PSD kernels by SMO-type methods, Technical Report, Department of Computer Science and Information Engineering, National Taiwan University.; H.-T. Lin, C.-J. Lin, A study on sigmoid kernels for SVM and the training of non-PSD kernels by SMO-type methods, Technical Report, Department of Computer Science and Information Engineering, National Taiwan University.
[17] C.-W. Hsu, C.-C. Chang, C.-J. Lin, A practical guide to support vector classification. [Online] Available from World Wide Web: \( \langle;\) http://www.csie.ntu.edu.tw/∼;cjlin/libsvm \(\rangle;\).; C.-W. Hsu, C.-C. Chang, C.-J. Lin, A practical guide to support vector classification. [Online] Available from World Wide Web: \( \langle;\) http://www.csie.ntu.edu.tw/∼;cjlin/libsvm \(\rangle;\).
[18] Bottou, L.; Cortes, C.; Denker, J.; Drucker, H.; Guyon, I.; Jackel, L.; LeCun, Y.; Muller, U.; Sackinger, E.; Simard, P.; Vapnik, V., Comparison of classifier methods: a case study in handwritten digit recognition, (Proceedings of International Conference on Pattern Recognition (1994)), 77-87
[19] Kreßel, U., Pairwise classification and support vector machines, (Schölkopf, B.; Burges, C. J.C.; Smola, A. J., Advances in Kernel Methods—Support Vector Learning (1999), MIT Press: MIT Press Cambridge, MA)
[20] Platt, J. C.; Cristianini, N.; Show-Tayler, J., Large margin DAG’s for multiclass classification, (Advances in Neural Information Processing Systems, vol. 12 (2000), MIT Press: MIT Press Cambridge, MA), 547-553
[21] F. Takahashi, S. Abe, Optimizing directed acyclic graph support vector machines, in: Proceedings of the Artificial Neural Networks in Pattern Recognition (ANNPR 2003), September 2003, pp. 166-170.; F. Takahashi, S. Abe, Optimizing directed acyclic graph support vector machines, in: Proceedings of the Artificial Neural Networks in Pattern Recognition (ANNPR 2003), September 2003, pp. 166-170.
[22] T. Phetkaew, B. Kijsirikul, W. Rivepiboon, Reordering adaptive directed acyclic graphs: an improved algorithm for multiclass support vector machines, in: Proceedings of the International Joint Conference on Neural Networks (IJCNN 2003), vol. 2, 2003, pp. 1605-1610.; T. Phetkaew, B. Kijsirikul, W. Rivepiboon, Reordering adaptive directed acyclic graphs: an improved algorithm for multiclass support vector machines, in: Proceedings of the International Joint Conference on Neural Networks (IJCNN 2003), vol. 2, 2003, pp. 1605-1610.
[23] K.-P. Wu, S.-D. Wang, Choosing the kernel parameters of support vector machines according to the inter-cluster distance, in: Proceedings of the International Joint Conference on Neural Networks (IJCNN 2006), 2006.; K.-P. Wu, S.-D. Wang, Choosing the kernel parameters of support vector machines according to the inter-cluster distance, in: Proceedings of the International Joint Conference on Neural Networks (IJCNN 2006), 2006.
[24] Bezdek, C.; Pal, N. R., Some new indexes of cluster validity, IEEE Trans. Syst. Man Cybern. Part B Cybern., 28, 3, 301-315 (1998)
[25] Platt, J. C., Fast training of support vector machines using sequential minimal optimization, (Schölkopf, B.; Burges, C. J.C.; Smola, A. J., Advances in Kernel Methods—Support Vector Learning (1999), MIT Press: MIT Press Cambridge, MA)
[26] D. Michie, D.J. Spiegelhalter, C.C. Taylor, Machine learning, neural and statistical classification, 1994. [Online] Available: \( \langle;\) ftp.ncc.up.pt/pub/statlog/\( \rangle;\).; D. Michie, D.J. Spiegelhalter, C.C. Taylor, Machine learning, neural and statistical classification, 1994. [Online] Available: \( \langle;\) ftp.ncc.up.pt/pub/statlog/\( \rangle;\). · Zbl 0827.68094
[27] C.-C. Chang, C.-J. Lin, LIBSVM: a library for support vector machines. [Online] Available from World Wide Web: \( \langle;\) http://www.csie.ntu.edu.tw/∼;cjlin/libsvm \(\rangle;\).; C.-C. Chang, C.-J. Lin, LIBSVM: a library for support vector machines. [Online] Available from World Wide Web: \( \langle;\) http://www.csie.ntu.edu.tw/∼;cjlin/libsvm \(\rangle;\).
[28] Dong, J. X.; Krzyzak, A.; Suen, C. Y., Fast SVM training algorithm with decomposition on very large datasets, IEEE Trans. Pattern Anal. Mach. Intell., 27, 4, 603-618 (2005)
[29] R. Collobert, S. Bengio, J. Mariethoz, Torch. [Online] Available from World Wide Web: \( \langle;\) http://www.torch.ch/\( \rangle;\).; R. Collobert, S. Bengio, J. Mariethoz, Torch. [Online] Available from World Wide Web: \( \langle;\) http://www.torch.ch/\( \rangle;\).
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.