MINISTRY OF EDUCATION AND TRAINING MINISTRY OF NATIONAL DEFENCE MILITARY TECHNICAL ACADEMY TRAN HUNG CUONG DC ALGORITHMS IN NONCONVEX QUADRATIC PROGRAMMING AND APPLICATIONS IN DATA CLUSTERING DOCTORAL DISSERTATION MATHEMATICS HANOI - 2021 MINISTRY OF EDUCATION AND TRAINING MINISTRY OF NATIONAL DEFENCE MILITARY TECHNICAL ACADEMY TRAN HUNG CUONG DC ALGORITHMS IN NONCONVEX QUADRATIC PROGRAMMING AND APPLICATIONS IN DATA CLUSTERING DOCTORAL DISSERTATION Major: Mathematical Foundations for Informatics Code: 46 01 10 RESEARCH SUPERVISIORS: Prof Dr.Sc Nguyen Dong Yen Prof Dr.Sc Pham The Long HANOI - 2021 Confirmation This dissertation was written on the basis of my research works carried out at the Military Technical Academy, under the guidance of Prof Nguyen Dong Yen and Prof Pham The Long All the results presented in this dissertation have got agreements of my coauthors to be used here February 25, 2021 The author Tran Hung Cuong i Acknowledgments I would like to express my deep gratitude to my advisor, Professor Nguyen Dong Yen and Professor Pham The Long, for their careful and effective guidance I would like to thank the board of directors of Military Technical Academy for providing me with pleasant working conditions I am grateful to the leaders of Hanoi University of Industry, the Faculty of Information Technology, and my colleagues, for granting me various financial supports and/or constant help during the three years of my PhD study I am sincerely grateful to Prof Jen-Chih Yao from Department of Applied Mathematics, National Sun Yat-sen University, Taiwan, and Prof ChingFeng Wen from Research Center for Nonlinear Analysis and Optimization, Kaohsiung Medical University, Taiwan, for granting several short-termed scholarships for my doctorate studies I would like to thank the following experts for their careful readings of this dissertation and for many useful suggestions which have helped me to improve the presentation: Prof Dang Quang A, Prof Pham Ky Anh, Prof Le Dung Muu, Assoc Prof Phan Thanh An, Assoc Prof Truong Xuan Duc Ha, Assoc Prof Luong Chi Mai, Assoc Prof Tran Nguyen Ngoc, Assoc Prof Nguyen Nang Tam, Assoc Prof Nguyen Quang Uy, Dr Duong Thi Viet An, Dr Bui Van Dinh, Dr Vu Van Dong, Dr Tran Nam Dung, Dr Phan Thi Hai Hong, Dr Nguyen Ngoc Luan, Dr Ngo Huu Phuc, Dr Le Xuan Thanh, Dr Le Quang Thuy, Dr Nguyen Thi Toan, Dr Ha Chi Trung, Dr Hoang Ngoc Tuan, Dr Nguyen Van Tuyen I am so much indebted to my family for their love, support and encouragement, not only in the present time, but also in the whole my life With love and gratitude, I dedicate this dissertation to them ii Contents Acknowledgments ii Table of Notations v Introduction vii Chapter Background Materials 1.1 Basic Definitions and Some Properties 1.2 DCA Schemes 1.3 General Convergence Theorem 1.4 Convergence Rates 11 1.5 Conclusions 13 Chapter Analysis of an Algorithm in Indefinite Quadratic Programming 14 2.1 Indefinite Quadratic Programs and DCAs 15 2.2 Convergence and Convergence Rate of the Algorithm 24 2.3 Asymptotical Stability of the Algorithm 30 2.4 Further Analysis 36 2.5 Conclusions 40 Chapter Qualitative Properties of the Minimum Sum-of-Squares Clustering Problem 41 3.1 Clustering Problems 41 3.2 Basic Properties of the MSSC Problem 44 3.3 The k-means Algorithm 49 iii 3.4 Characterizations of the Local Solutions 52 3.5 Stability Properties 59 3.6 Conclusions 65 Chapter Some Incremental Algorithms for the Clustering Problem 66 4.1 Incremental Clustering Algorithms 66 4.2 Ordin-Bagirov’s Clustering Algorithm 67 4.2.1 Basic constructions 68 4.2.2 Version of Ordin-Bagirov’s algorithm 71 4.2.3 Version of Ordin-Bagirov’s algorithm 73 4.2.4 The ε-neighborhoods technique 81 Incremental DC Clustering Algorithms 82 4.3 4.3.1 Bagirov’s DC Clustering Algorithm and Its Modification 82 4.3.2 The Third DC Clustering Algorithm 103 4.3.3 The Fourth DC Clustering Algorithm 105 4.4 Numerical Tests 107 4.5 Conclusions 111 General Conclusions 114 List of Author’s Related Papers 116 References 117 Index 125 iv Table of Notations N := {0, 1, 2, } (a, b) [a, b] x, y |x| x E AT pos Ω TC (x) NC (x) d(x, Ω) {xk } xk → x liminf αk the set of natural numbers empty set the set of real numbers the set of generalized real numbers n-dimensional Euclidean vector space set of m × n-real matrices set of x ∈ R with a < x < b set of x ∈ R with a ≤ x ≤ b canonical inner product absolute value of x ∈ R the Euclidean norm of a vector x the n × n unit matrix transposition of a matrix A convex cone generated by Ω tangent cone to C at x ∈ C normal cone to C at x ∈ C distance from x to Ω sequence of vectors xk converges to x in norm topology lower limit of a sequence {αk } of real numbers limsup αk upper limit of a sequence {αk } of real numbers ∅ R R := R ∪ {+∞, −∞} Rn Rm×n k→∞ k→∞ v χC ϕ : Rn → R dom ϕ ∂ ϕ(x) ϕ∗ : R n → R Γ0 (X) sol(P) loc(P) DC DCA PPA IQP KKT C∗ S MSSC KM indicator function of a set C extended-real-valued function effective domain of ϕ subdifferential of ϕ at x Fenchel conjugate function of ϕ the set of all lower semicontinuous, proper, convex functions on Rn the set of the solutions of problem (P) the set of the local solutions of problem (P) Difference-of-Convex functions DC algorithm proximal point algorithm indefinite quadratic programming Karush-Kuhn-Tucker the KKT point set of IQP the global solution set of IQP the minimum-sum-of-square clustering k-means algorithm vi Introduction 0.1 Literature Overview and Research Problems In this dissertation, we are concerned with several concrete topics in DC programming and data mining Here and in the sequel, the word “DC” stands for Difference of Convex functions Fundamental properties of DC functions and DC sets can be found in the book [94] of Professor Hoang Tuy, who made fundamental contributions to global optimization The whole Chapter of that book gives a deep analysis of DC optimization problems and their applications in design calculation, location, distance geometry, and clustering We refer to the books [37,46], the dissertation [36], and the references therein for methods of global optimization and numerous applications We will consider some algorithms for finding locally optimal solutions of optimization problems Thus, techniques of global optimization, like the branch and bound method and the cutting plane method, will not be applied herein Note that since global optimization algorithms are costly for many large-scale nonconvex optimization problems, local optimization algorithms play an important role in optimization theory and real world applications First, let us begin with some facts about DC programming As noted in [93], “DC programming and DC algorithms (DCA, for brevity) treat the problem of minimizing a function f = g − h, with g, h being lower semicontinuous, proper, convex functions on Rn , on the whole space Usually, g and h are called d.c components of f The DCA are constructed on the basis of the DC programming theory and the duality theory of J F Toland It was Pham Dinh Tao who suggested a general DCA theory, which has been developed intensively by him and Le Thi Hoai An, starting from their fundamental paper [77] published in Acta Mathematica Vietnamica in 1997.” The interested reader is referred to the comprehensive survey paper of Le Thi and Pham Dinh [55] on the thirty years (1985–2015) of the development vii of the DC programming and DCA, where as many as 343 research works have been commented and the following remarks have been given: “DC programming and DCA were the subject of several hundred articles in the high ranked scientific journals and the high-level international conferences, as well as various international research projects, and were the methodological basis of more than 50 PhD theses About 100 invited symposia/sessions dedicated to DC programming and DCA were presented in many international conferences The ever-growing number of works using DC programming and DCA proves their power and their key role in nonconvex programming/global optimization and many areas of applications.” DCA has been successfully applied to many large-scale DC optimization problems and proved to be more robust and efficient than related standard methods; see [55] The main applications of DC programming and DCA include: - Nonconvex optimization problems: The trust-region subproblems, indefinite quadratic programming problems, - Image analysis: Image analysis, signal and image restoration - Data mining and Machine learning: data clustering, robust support vector machines, learning with sparsity DCA has a tight connection with the proximal point algorithm (PPA, for brevity) One can apply PPA to solve monotone and pseudomonotone variational inequalities (see, e.g., [85] and [89] and the references therein) Since the necessary optimality conditions for an optimization problem can be written as a variational inequality, PPA is also a solution method for solving optimization problems In [69], PPA is applied to mixed variational inequalities by using DC decompositions of the cost function Linear convergence rate is achieved when the cost function is strongly convex In the nonconvex case, global algorithms are proposed to search a global solution Indefinite quadratic programming problems (IQPs for short) under linear constraints form an important class of optimization problems IQPs have various applications (see, e.g., [16, 29]) In general, the constraint set of an IQP can be unbounded Therefore, unlike the case of the trust-region subproblem (see, e.g., [58]), the boundedness of the iterative sequence generated by a DCA and a starting point for a given IQP require additional investigations viii Table 4.4: The summary table CPU time fbest Algorithm 4.2 vs Algorithm 4.1 37 Algorithm 4.5 vs Algorithm 4.4 14 48 Algorithm 4.5 vs Algorithm 4.6 17 32 Algorithm 4.2 vs KM 39 Algorithm 4.5 vs KM 45 Figure 4.1: The CPU time of the algorithms for the Wine data set 4.5 Conclusions We have presented the incremental DC clustering algorithm of Bagirov and proposed three modified versions Algorithms 4.4, 4.5, and 4.6 for this algorithm By constructing some concrete MSSC problems with small data sets, we have shown how these algorithms work Two convergence theorems and two theorems about the Q−linear convergence rate of the first modified version of Bagirov’s algorithm have been obtained by some delicate arguments Numerical tests of the above-mentioned algorithms on some real-world databases have shown the effectiveness of the proposed algorithms 111 Figure 4.2: The value of objective function of the algorithms for the Stock Wine data set Figure 4.3: The CPU time of the algorithms for the Stock Price data set 112 Figure 4.4: The value of objective function of the algorithms for the Stock Price data set 113 General Conclusions In this dissertation, we have applied DC programming and DCAs to analyze a solution algorithm for the indefinite quadratic programming problem (IQP problem) We have also used different tools from convex analysis, set-valued analysis, and optimization theory to study qualitative properties (solution existence, finiteness of the global solution set, and stability) of the minimum sum-of-squares clustering problem (MSSC problem) and develop some solution methods for this problem Our main results include: 1) The R-linear convergence of the Proximal DC decomposition algorithm (Algorithm B) and the asymptotic stability of that algorithm for the given IQP problem, as well as the analysis of the influence of the decomposition parameter on the rate of convergence of DCA sequences; 2) The solution existence theorem for the MSSC problem together with the necessary and sufficient conditions for a local solution of the problem, and three fundamental stability theorems for the MSSC problem when the data set is subject to change; 3) The analysis and development of the heuristic incremental algorithm of Ordin and Bagirov together with three modified versions of the DC incremental algorithms of Bagirov, including some theorems on the finite convergence and the Q−linear convergence, as well as numerical tests of the algorithms on several real-world databases In connection with the above results, we think that the following research topics deserve further investigations: - Qualitative properties of the clustering problems with L1 −distance and Euclidean distance; - Incremental algorithms for solving the clustering problems with L1 −distance 114 and Euclidean distance; - Booted DC algorithms (i.e., DCAs with a additional line search procedure at each iteration step; see [5]) to increase the computation speed; - Qualitative properties and solution methods for constrained clustering problems (see [14, 24, 73, 74] for the definition of constrained clustering problems and two basic solution methods) 115 List of Author’s Related Papers T H Cuong, Y Lim, N D Yen, Convergence of a solution algorithm in indefinite quadratic programming, Preprint (arXiv:1810.02044), submitted T H Cuong, J.-C Yao, N D Yen, Qualitative properties of the minimum sum-of-squares clustering problem, Optimization 69 (2020), No 9, 2131– 2154 (SCI-E; IF 1.206, Q1-Q2, H-index 37; MCQ of 2019: 0.75) T H Cuong, J.-C Yao, N D Yen, On some incremental algorithms for the minimum sum-of-squares clustering problem Part 1: Ordin and Bagirov’s incremental algorithm, Journal of Nonlinear and Convex Analysis 20 (2019), No 8, 1591–1608 (SCI-E; 0.710, Q2-Q3, H-index 18; MCQ of 2019: 0.56) T H Cuong, J.-C Yao, N D Yen, On some incremental algorithms for the minimum sum-of-squares clustering problem Part 2: Incremental DC algorithms, Journal of Nonlinear and Convex Analysis 21 (2020), No 5, 1109–1136 (SCI-E; 0.710, Q2-Q3, H-index 18; MCQ of 2019: 0.56) 116 References [1] C C Aggarwal, C K Reddy: Data Clustering Algorithms and Applications, Chapman & Hall/CRC Press, Boca Raton, Florida, 2014 [2] F B Akoa, Combining DC Algorithms (DCAs) and decomposition techniques for the training of nonpositive–semidefinite kernels, IEEE Trans Neur Networks 19 (2008), 1854–1872 [3] D Aloise, A Deshpande, P Hansen, P Popat, NP-hardness of Euclidean sum-of-squares clustering, Mach Learn 75 (2009), 245–248 [4] N T An, N M Nam, Convergence analysis of a proximal point algorithm for minimizing differences of functions, Optimization 66 (2017), 129– 147 [5] F J Arag´on Artacho, R M T Fleming, P T Vuong, Accelerating the DC algorithm for smooth functions, Math Program 169 (2018), 95–118 [6] A M Bagirov, Modified global k-means algorithm for minimum sum-ofsquares clustering problems, Pattern Recognit 41 (2008), 3192–3199 [7] A M Bagirov, An incremental DC algorithm for the minimum sum-ofsquares clustering, Iranian J Oper Res (2014), 1–14 [8] A M Bagirov, E Mohebi, An algorithm for clustering using L1 -norm based on hyperbolic smoothing technique, Comput Intell 32 (2016), 439– 457 [9] A M Bagirov, A M Rubinov, N V Soukhoroukova, J Yearwood, Unsupervised and supervised data classification via nonsmooth and global optimization, TOP 11 (2003), 1–93 [10] A M Bagirov, S Taher, A DC optimization algorithm for clustering problems with L1 −norm, Iranian J Oper Res (2017), 2–24 117 [11] A M Bagirov, J Ugon, Nonsmooth DC programming approach to clusterwise linear regression: optimality conditions and algorithms, Optim Methods Softw 33 (2018), 194–219 [12] A M Bagirov, J Ugon, D Webb, Fast modified global k-means algorithm for incremental cluster construction, Pattern Recognit 44 (2011), 866– 876 [13] A M Bagirov, J Yearwood, A new nonsmooth optimization algorithm for minimum sum-of-squares clustering problems, European J Oper Res 170 (2006), 578–596 [14] S Basu , I Davidson, K L Wagstaff, Constrained Clustering: Advances in Algorithms, Theory, and Applications, CRC Press, New York, 2009 [15] H H Bock, Clustering and neural networks In “Advances in Data Science and Classification”, Springer, Berlin (1998), pp 265–277 [16] I M Bomze, On standard quadratic optimization problems, J Global Optim 13 (1998), 369–387 [17] I M Bomze, G Danninger, A finite algorithm for solving general quadratic problems, J Global Optim (1994), 1–16 [18] M J Brusco, A repetitive branch-and-bound procedure for minimum within-cluster sum of squares partitioning, Psychometrika, 71 (2006), 347–363 [19] R Cambini, C Sodini, Decomposition methods for solving nonconvex quadratic programs via Branch and Bound, J Global Optim 33 (2005), 313–336 [20] F H Clarke, Optimization and Nonsmooth Analysis, Second edition, SIAM, Philadelphia, 1990 [21] G Cornu´ejols, J Pe˜ na, R Tă ută uncă u, Optimization Methods in Finance, Second edition, Cambridge University Press, Cambridge, 2018 [22] L R Costa, D Aloise, N Mladenovi´c, Less is more: basic variable neighborhood search heuristic for balanced minimum sum-of-squares clustering, Inform Sci 415/416 (2017), 247–253 [23] T F Cov˜oes, E R Hruschka, J Ghosh, A study of k-means-based algorithms for constrained clustering, Intelligent Data Analysis 17 (2013), 485–505 118 [24] I Davidson, S S Ravi, Clustering with constraints: Feasibility issues and the k-means algorithm, In: Proceedings of the 5th SIAM Data Mining Conference, 2005 [25] V F Dem’yanov, A M Rubinov, Constructive Nonsmooth Analysis, Peter Lang Verlag, Frankfurt am Main, 1995 [26] V F Dem’yanov, L V Vasil’ev, Nondifferentiable Optimization, Translated from the Russian by T Sasagawa, Optimization Software Inc., New York, 1985 [27] G Diehr, Evaluation of a branch and bound algorithm for clustering, SIAM J Sci Stat Comput (1985), 268–284 [28] O Du Merle, P Hansen, B Jaumard, N Mladenovi´c, An interior point algorithm for minimum sum of squares clustering, SIAM J Sci Comput 21 (2000), 1485–1505 [29] N I M Gould, Ph L Toint, A Quadratic Programming Page, http://www.numerical.rl.ac.uk/people/nimg/qp/qp.html [30] O K Gupta, Applications of quadratic programming, J Inf Optim Sci 16 (1995), 177–194 [31] N T V Hang, N D Yen, On the problem of minimizing a difference of polyhedral convex functions under linear constraints, J Optim Theory Appl 171 (2016), 617–642 [32] J Han, M Kamber, J Pei, Data Mining: Concepts and Techniques, Third edition, Morgan Kaufmann, New York, 2012 [33] P Hansen, E Ngai, B K Cheung, N Mladenovic, Analysis of global kmeans, an incremental heuristic for minimum sum-of-squares clustering, J Classif 22 (2005), 287–310 [34] P Hansen, N Mladenovi´c, Variable neighborhood decomposition search, J Heuristics (2001), 335–350 [35] P Hansen, N Mladenovi´c, J-means: a new heuristic for minimum sumof-squares clustering, Pattern Recognit (2001), 405–413 [36] P T Hoai, Some Nonconvex Optimization Problems: Algorithms and Applications, Ph.D Dissertation, Hanoi University of Science and Technology, Hanoi, 2019 119 [37] R Horst, H Tuy, Global Optimization, Deterministic Approaches, Second edition, Springer-Verlag, Berlin, 1993 [38] A D Ioffe, V M Tihomirov, Theory of Extremal Problems, NorthHolland Publishing Company, Amsterdam, 1979 [39] A K Jain, Data clustering: 50 years beyond K-means, Pattern Recognit Lett 31 (2010), 651–666 [40] N Jain, V Srivastava, Data mining techniques: A survey paper, Inter J Res Engineering Tech (2010), no 11, 116–119 [41] T.-C Jen, S.-J Wang, Image enhancement based on quadratic programming, Proceedings of the 15th IEEE International Conference on Image Processing, pp 3164–3167, 2008 [42] K Joki, A M Bagirov, N Karmitsa, M.M Măakelăa, S Taheri, Clusterwise support vector linear regression, European J Oper Res 287 (2020), 19–35 [43] M Kantardzic, Data Mining Concepts, Models, Methods, and Algorithms, Second edition, John Wiley & Sons, Hoboken, New Jersey, 2011 [44] N Karmitsa, A M Bagirov, S Taheri, New diagonal bundle method for clustering problems in large data sets, European J Oper Res 263 (2017), 367–379 [45] D Kinderlehrer, G Stampacchia, An Introduction to Variational Inequalities and Their Applications, Academic Press, Inc., New YorkLondon, 1980 [46] H Konno, P T Thach, H Tuy, Optimization on Low Rank Nonconvex Structures, Kluwer Academic Publishers, Dordrecht, 1997 [47] W L G Koontz, P M Narendra, K Fukunaga, A branch and bound clustering algorithm, IEEE Trans Comput 24 (1975), 908–915 [48] K M Kumar, A R M Reddy, An efficient k-means clustering filtering algorithm using density based initial cluster centers, Inform Sci 418/419 (2017), 286–301 [49] J Z C Lai, T.-J Huang, Fast global k-means clustering using cluster membership and inequality, Pattern Recognit 43 (2010), 731–737 120 [50] G M Lee, N N Tam, N D Yen, Quadratic Programming and Affine Variational Inequalities: A Qualitative Study, Springer–Verlag, New York, 2005 [51] H A Le Thi, M T Belghiti, T Pham Dinh, A new efficient algorithm based on DC programming and DCA for clustering, J Global Optim 37 (2007), 593–608 [52] H A Le Thi, M Le Hoai, T Pham Dinh, New and efficient DCA based algorithms for minimum sum-of-squares clustering, Pattern Recognition 47 (2014), 388–401 [53] H A Le Thi, , V N Huynh, T Pham Dinh, Convergence analysis of DCA with subanalytic data, J Optim Theory Appl 179 (2018), 103–126 [54] H A Le Thi, T Pham Dinh, The DC (difference of convex functions) programming and DCA revisited with DC models of real world nonconvex optimization problems, Ann Oper Res 133 (2005), 23–46 [55] H A Le Thi, T Pham Dinh, DC programming and DCA: thirty years of developments, Math Program 169 (2018), Ser B, 5–68 [56] M Lichman, UCI machine learning repository, University of California, Irvine, School of Information and Computer Sciences, 2013; http://archive.ics.uci.edu/ml [57] H A Le Thi, T Pham Dinh, N D Yen, Properties of two DC algorithms in quadratic programming, J Global Optim 49 (2011), 481–495 [58] H A Le Thi, T Pham Dinh, N D Yen, Behavior of DCA sequences for solving the trust-region subproblem, J Global Optim 53 (2012), 317–329 [59] W J Leong, B S Goh, Convergence and stability of line search methods for unconstrained optimization, Acta Appl Math 127 (2013), 155–167 [60] H A Le Thi, T Pham Dinh, Minimum sum-of-squares clustering by DC programming and DCA, ICIC 2009, LNAI 5755 (2009), 327–340 [61] A Likas, N Vlassis, J J Verbeek, The global k-means clustering algorithm, Pattern Recognit 36 (2003), 451–461 [62] F Liu, X Huang, J Yang, Indefinite kernel logistic regression, Preprint [arXiv:1707.01826v1], 2017 121 [63] F Liu, X Huang, C Peng, J Yang, N Kasabov, Robust kernel approximation for classification, Proceedings of the 24th International Conference “Neural Information Processing”, ICONIP 2017, Guangzhou, China, November 14–18, 2017, Proceedings, Part I, pp 289–296, 2017 [64] Z.-Q Luo, New error bounds and their applications to convergence analysis of iterative algorithms, Math Program 88 (2000), 341–355 [65] Z.-Q Luo, P Tseng, Error bound and convergence analysis of matrix splitting algorithms for the affine variational inequality problem, SIAM J Optim (1992), 43–54 [66] J MacQueen, Some methods for classification and analysis of multivariate observations, Proceedings of the 5th Berkeley Symposium on Mathematical Statistics and Probability, pp 281–297, 1967 [67] M Mahajan, P Nimbhorkar, K Varadarajan, The planar k-means problem is NP-hard, Theoret Comput Sci 442 (2012), 13–21 [68] B A McCarl, H Moskowitz, H Furtan, Quadratic programming applications, Omega (1977), 43–55 [69] L D Muu, T D Quoc, One step from DC optimization to DC mixed variational inequalities, Optimization 59 (2010), 63–76 [70] J Nocedal, S J Wright, Numerical Optimization, Springer-Verlag, New York, 1999 [71] B Ordin, A M Bagirov, A heuristic algorithm for solving the minimum sum-of-squares clustering problems, J Global Optim 61 (2015), 341–361 [72] P M Pardalos, S A Vavasis, Quadratic programming with one negative eigenvalue is NP-hard, J Global Optim (1991), 15–22 [73] D Pelleg, D Baras, K-Means with large and noisy constraint sets, In “Machine Learning: ECML 2007” (J N Kok et al., Eds.), Series “Lecture Notes in Artificial Intelligence” 4701, pp 674–682, 2007 [74] D Pelleg, D Baras, K-Means with large and noisy constraint sets, Technical Report H-0253, IBM, 2007 [75] J Peng, Y Xia, A cutting algorithm for the minimum sum-of-squared error clustering, Proceedings of the SIAM International Data Mining Conference, 2005 122 [76] T Pereira, D Aloise, B Daniel, J Brimberg, N Mladenovi´c, Review of basic local searches for solving the minimum sum-of-squares clustering problem Open problems in optimization and data analysis, Springer Optim Appl 141, pp 249–270, Springer, Cham, 2018 [77] T Pham Dinh, H A Le Thi, Convex analysis approach to d.c programming: theory, algorithms and applications, Acta Math Vietnam 22 (1997), 289–355 [78] T Pham Dinh, H A Le Thi, Solving a class of linearly constrained indefinite quadratic programming problems by d.c algorithms, J Global Optim 11 (1997), 253–285 [79] T Pham Dinh, H A Le Thi, A d.c optimization algorithm for solving the trust-region subproblem, SIAM J Optim (1998), 476–505 [80] T Pham Dinh, H A Le Thi, A branch and bound method via DC optimization algorithm and ellipsoidal techniques for box constrained nonconvex quadratic programming problems, J Global Optim 13 (1998), 171–206 [81] T Pham Dinh, H A Le Thi, DC (difference of convex functions) programming Theory, algorithms, applications: The state of the art, Proceedings of the First International Workshop on Global Constrained Optimization and Constraint Satisfaction (Cocos’02), Valbonne Sophia Antipolis, France, pp 2–4, 2002 [82] T Pham Dinh, H A Le Thi, F Akoa, Combining DCA (DC Algorithms) and interior point techniques for large-scale nonconvex quadratic programming, Optim Methods Softw 23 (2008), 609–629 [83] E Polak, Optimization Algorithms and Consistent Approximations, Springer-Verlag, New York, 1997 [84] R T Rockafellar, Convex Analysis, Princeton University Press, Princeton, 1970 [85] R T Rockafellar, Monotone operators and the proximal point algorithm, SIAM J Control Optim 14 (1976), 877–898 [86] S Z Selim, M A Ismail, K-means-type algorithms: A generalized convergence theorem and characterization of local optimality, IEEE Trans Pattern Anal Mach Intell (1984), 81–87 123 [87] H D Sherali, J Desai, A global optimization RLT-based approach for solving the hard clustering problem, J Global Optim 32 (2005), 281– 306 [88] J Stoer, R Burlisch, Introduction to Numerical Analysis, Third edition, Springer, New York, 2002 [89] N N Tam, J.-C Yao, N D Yen, Solution methods for pseudomonotone variational inequalities, J Optim Theory Appl 138 (2008), 253–273 [90] P Tseng, On linear convergence of iterative methods for the variational inequality problem, J Comput Appl Math 60 (1995), 237–252 [91] H N Tuan, Boundedness of a type of iterative sequences in twodimensional quadratic programming, J Optim Theory Appl 164 (2015), 234–245 [92] H N Tuan, Linear convergence of a type of DCA sequences in nonconvex quadratic programming, J Math Anal Appl 423 (2015), 1311–1319 [93] H N Tuan, DC Algorithms and Applications in Nonconvex Quadratic Programing, Ph.D Dissertation, Institute of Mathematics, Vietnam Academy of Science and Technology, Hanoi, 2015 [94] H Tuy, Convex Analysis and Global Optimization, Second edition, Springer, 2016 [95] H Tuy, A M Bagirov, A M Rubinov, Clustering via d.c optimization, In: “Advances in Convex Analysis and Global Optimization”, pp 221– 234, Kluwer Academic Publishers, Dordrecht, 2001 [96] R Wiebking, Selected applications of all-quadratic programming, OR Spektrum (1980), 243–249 [97] J Wu, Advances in k-means Clustering: A Data Mining Thinking, Springer-Verlag, Berlin-Heidelberg, 2012 [98] J Xie, S Jiang, W Xie, X Gao, An efficient global k-means clustering algorithm, J Comput (2011), 271–279 [99] H.-M Xu, H Xue, X.-H Chen, Y.-Y Wang, Solving indefinite kernel support vector machine with difference of convex functions programming, Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence (AAAI-17), Association for the Advancement of Artificial Intelligence, pp 2782–2788, 2017 124 [100] H Xue, Y Song, H.-M Xu, Multiple indefinite kernel learning for feature selection, Knowledge-Based Systems 191 (2020), Article 105272 (12 pages) [101] Y Ye, An extension of Karmarkar’s algorithm and the trust region method for quadratic programming, In “Progress in Mathematical Programming” (N Megiddo, Ed.), pp 49–63, Springer, New York, 1980 [102] Y Ye, On affine scaling algorithms for nonconvex quadratic programming, Math Program 56 (1992), 285–300 [103] Y Ye, Interior Point Algorithms: Theory and Analysis, Wiley, New York, 1997 125 ... Incremental DC Clustering Algorithms 82 4.3 4.3.1 Bagirov’s DC Clustering Algorithm and Its Modification 82 4.3.2 The Third DC Clustering Algorithm 103 4.3.3 The Fourth DC Clustering... concrete topics in DC programming and data mining Here and in the sequel, the word ? ?DC? ?? stands for Difference of Convex functions Fundamental properties of DC functions and DC sets can be found... world applications First, let us begin with some facts about DC programming As noted in [93], ? ?DC programming and DC algorithms (DCA, for brevity) treat the problem of minimizing a function f