ABSTRACT
Given a separation oracle for a convex set K ⊂ ℝ n that is contained in a box of radius R, the goal is to either compute a point in K or prove that K does not contain a ball of radius є. We propose a new cutting plane algorithm that uses an optimal O(n log(κ)) evaluations of the oracle and an additional O(n 2) time per evaluation, where κ = nR/є.
This improves upon Vaidya’s O( SO · n log(κ) + n ω+1 log(κ)) time algorithm [Vaidya, FOCS 1989a] in terms of polynomial dependence on n, where ω < 2.373 is the exponent of matrix multiplication and SO is the time for oracle evaluation. This improves upon Lee-Sidford-Wong’s O( SO · n log(κ) + n 3 log O(1) (κ)) time algorithm [Lee, Sidford and Wong, FOCS 2015] in terms of dependence on κ.
For many important applications in economics, κ = Ω(exp(n)) and this leads to a significant difference between log(κ) and (log(κ)). We also provide evidence that the n 2 time per evaluation cannot be improved and thus our running time is optimal.
A bottleneck of previous cutting plane methods is to compute leverage scores, a measure of the relative importance of past constraints. Our result is achieved by a novel multi-layered data structure for leverage score maintenance, which is a sophisticated combination of diverse techniques such as random projection, batched low-rank update, inverse maintenance, polynomial interpolation, and fast rectangular matrix multiplication. Interestingly, our method requires a combination of different fast rectangular matrix multiplication algorithms.
Our algorithm not only works for the classical convex optimization setting, but also generalizes to convex-concave games. We apply our algorithm to improve the runtimes of many interesting problems, e.g., Linear Arrow-Debreu Markets, Fisher Markets, and Walrasian equilibrium.
- [AD54] Kenneth J Arrow and Gerard Debreu. Existence of an equilibrium for a competitive economy. Econometrica: Journal of the Econometric Society, pages 265-290, 1954.Google ScholarCross Ref
- [AKM+17] Haim Avron, Michael Kapralov, Cameron Musco, Christopher Musco, Ameya Velingker, and Amir Zandieh. Random Fourier features for kernel ridge regression: Approximation bounds and statistical guarantees. In International Conference on Machine Learning (ICML). https://arxiv.org/ pdf/ 1804.09893.pdf, 2017.Google Scholar
- [AKM+19] Haim Avron, Michael Kapralov, Cameron Musco, Christopher Musco, Ameya Velingker, and Amir Zandieh. A universal sampling method for reconstructing signals with simple Fourier transforms. In 51st Annual ACM SIGACT Symposium on Theory of Computing (STOC). https://arxiv.Google Scholar
- org/pdf/ 1812.08723.pdf, 2019.Google Scholar
- [AM02] Lawrence M Ausubel and Paul R Milgrom. Ascending auctions with package bidding. Advances in Theoretical Economics, 1 ( 1 ), 2002.Google Scholar
- [AV95] David S Atkinson and Pravin M Vaidya. A cutting plane algorithm for convex programming that uses analytic centers. Mathematical Programming, 69 ( 1-3 ): 1-43, 1995.Google Scholar
- [BD76] Roger W Brockett and David Dobkin. On the number of multiplications required for matrix multiplication. SIAM Journal on Computing, 5 ( 4 ): 624-628, 1976.Google Scholar
- [BDX10] Benjamin Birnbaum, N Devanur, and Lin Xiao. New convex programs and distributed algorithms for fisher markets with linear and spending constraint utilities. Unpublished manuscript, 2010.Google Scholar
- [BLM89] Jean Bourgain, Joram Lindenstrauss, and V Milman. Approximation of zonoids by zonotopes. Acta mathematica, 162 ( 1 ): 73-141, 1989.Google Scholar
- [BLSS20] Jan van den Brand, Yin Tat Lee, Aaron Sidford, and Zhao Song. Solving tall dense linear programs in nearly linear time. In 52nd Annual ACM SIGACT Symposium on Theory of Computing (STOC). https://arxiv.org/ pdf/ 2002.02304.pdf, 2020.Google Scholar
- [BNS19] Jan van den Brand, Danupon Nanongkai, and Thatchaphol Saranurak.Google Scholar
- Dynamic matrix inverse: Improved algorithms and matching conditional lower bounds. In 60th Annual IEEE Symposium on Foundations of Computer Science (FOCS). https://arxiv.org/pdf/ 1905.05067.pdf, 2019.Google Scholar
- [Bra20] Jan van den Brand. A deterministic linear program solver in current matrix multiplication time. In ACM-SIAM Symposium on Discrete Algorithms (SODA). https://arxiv.org/pdf/ 1910.11957.pdf, 2020.Google Scholar
- [BV02] Dimitris Bertsimas and Santosh Vempala. Solving convex programs by random walks. In Proceedings of the thiry-fourth annual ACM symposium on Theory of computing (STOC), pages 109-115. ACM, 2002.Google ScholarDigital Library
- [BW14] Christos Boutsidis and David P Woodruf. Optimal cur matrix decompositions. In Proceedings of the 46th Annual ACM Symposium on Theory of Computing (STOC), pages 353-362. ACM, https://arxiv.org/pdf/1405.Google Scholar
- [CCLY19] Michael B Cohen, Ben Cousins, Yin Tat Lee, and Xin Yang. A nearoptimal algorithm for approximating the John ellipsoid. In Annual Conference on Learning Theory (COLT). https://arxiv.org/pdf/ 1905.11580.Google Scholar
- pdf, 2019.Google Scholar
- [CGLZ20] Matthias Christandl, François Le Gall, Vladimir Lysikov, and Jeroen Zuiddam. Barriers for rectangular matrix multiplication. In arXiv preprint. https://arxiv.org/ 2003.03019.pdf, 2020.Google Scholar
- [CLS19] Michael B Cohen, Yin Tat Lee, and Zhao Song. Solving linear programs in the current matrix multiplication time. In Proceedings of the 51st Annual ACM Symposium on Theory of Computing (STOC). https://arxiv.Google Scholar
- org/pdf/ 1810.07896.pdf, 2019.Google Scholar
- [CMM17] Michael B Cohen, Cameron Musco, and Christopher Musco. Input sparsity time low-rank approximation via ridge leverage score sampling.Google Scholar
- In Proceedings of the Twenty-Eighth Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), pages 1758-1777. SIAM, https://arxiv.org/ pdf/1511.07263.pdf, 2017.Google Scholar
- [Cop82] Don Coppersmith. Rapid multiplication of rectangular matrices. SIAM Journal on Computing, 11 ( 3 ): 467-471, 1982.Google Scholar
- [Cor89] Bernard Cornet. Linear exchange economies. Cahier Eco-Math, Université de Paris, 1, 1989.Google Scholar
- [CP15] Michael B. Cohen and Richard Peng. ℓp row sampling by lewis weights.Google Scholar
- In Proceedings of the Forty-Seventh Annual ACM on Symposium on Theory of Computing (STOC), pages 183-192. https://arxiv.org/pdf/1412.0588, 2015.Google Scholar
- [CW87] Don Coppersmith and Shmuel Winograd. Matrix multiplication via arithmetic progressions. In Proceedings of the nineteenth annual ACM symposium on Theory of computing (STOC), pages 1-6. ACM, 1987.Google ScholarDigital Library
- [CW13] Kenneth L. Clarkson and David P. Woodruf. Low rank approximation and regression in input sparsity time. In Symposium on Theory of Computing Conference (STOC), pages 81-90. https://arxiv.org/pdf/1207.Google Scholar
- [Dan47] George B Dantzig. Maximization of a linear function of variables subject to linear inequalities. Activity analysis of production and allocation, 13 : 339-347, 1947.Google Scholar
- [DGM16] Ran Duan, Jugal Garg, and Kurt Mehlhorn. An improved combinatorial polynomial algorithm for the linear arrow-debreu market. In Proceedings of the twenty-seventh annual ACM-SIAM symposium on Discrete algorithms (SODA), pages 90-106. SIAM, https://arxiv.org/pdf/1510.02694.Google Scholar
- pdf, 2016.Google Scholar
- [DGV16] Nikhil R Devanur, Jugal Garg, and László A Végh. A rational convex program for linear arrow-debreu markets. volume 5 ( 1 ), page 6. https: //arxiv.org/pdf/1307.8037.pdf, 2016.Google Scholar
- [DM15] Ran Duan and Kurt Mehlhorn. A combinatorial polynomial algorithm for the linear arrow-debreu market. Information and Computation, 243 : 112-132, 2015.Google Scholar
- [DMIMW12] Petros Drineas, Malik Magdon-Ismail, Michael W Mahoney, and David P Woodruf. Fast approximation of matrix coherence and statistical leverage. Journal of Machine Learning Research, 13 (Dec): 3475-3506, 2012.Google Scholar
- [DPSV08] Nikhil R Devanur, Christos H Papadimitriou, Amin Saberi, and Vijay V Vazirani. Market equilibrium via a primal-dual algorithm for a convex program. Journal of the ACM (JACM), 55 ( 5 ): 22, 2008.Google Scholar
- [DS13] Alexander Munro Davie and Andrew James Stothers. Improved bound for complexity of matrix multiplication. Proceedings of the Royal Society of Edinburgh Section A: Mathematics, 143 ( 2 ): 351-369, 2013.Google ScholarCross Ref
- [dVSV07] Sven de Vries, James Schummer, and Rakesh V Vohra. On ascending vickrey auctions for heterogeneous objects. Journal of Economic Theory, 132 ( 1 ): 95-118, 2007.Google Scholar
- [Eav75] B Curtis Eaves. A finite algorithm for the linear exchange model. Technical report, STANFORD UNIV CALIF SYSTEMS OPTIMIZATION LAB, 1975.Google Scholar
- [GLS12] Martin Grötschel, László Lovász, and Alexander Schrijver. Geometric algorithms and combinatorial optimization, volume 2. Springer Science & Business Media, 2012.Google Scholar
- [GU18] Francois Le Gall and Florent Urrutia. Improved rectangular matrix multiplication using powers of the coppersmith-winograd tensor. In Proceedings of the Twenty-Ninth Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), pages 1029-1046. SIAM, https://arxiv.org/pdf/ 1708.05622.pdf, 2018.Google ScholarDigital Library
- [GV19] Jugal Garg and László A Végh. A strongly polynomial algorithm for linear exchange markets. In Proceedings of the 51st Annual ACM SIGACT Symposium on Theory of Computing (STOC), pages 54-65. ACM, https: //arxiv.org/pdf/ 1809.06266.pdf, 2019.Google Scholar
- [HKNS15] Monika Henzinger, Sebastian Krinninger, Danupon Nanongkai, and Thatchaphol Saranurak. Unifying and strengthening hardness for dynamic problems via the online matrix-vector multiplication conjecture.Google Scholar
- In Proceedings of the forty-seventh annual ACM symposium on Theory of computing (STOC), pages 21-30. ACM, https://arxiv.org/pdf/1511.06773.Google Scholar
- pdf, 2015.Google Scholar
- [Jai07] Kamal Jain. A polynomial time algorithm for computing an arrowdebreu market equilibrium for linear utilities. SIAM Journal on Computing, 37 ( 1 ): 303-318, 2007.Google Scholar
- [JL84] William B Johnson and Joram Lindenstrauss. Extensions of lipschitz mappings into a hilbert space. Contemporary mathematics, 26 ( 189-206): 1, 1984.Google Scholar
- [JLSW20] Haotian Jiang, Yin Tat Lee, Zhao Song, and Sam Chiu-wai Wong. An improved cutting plane method for convex optimization, convex-concave games and its applications. In STOC. https://arxiv.org/pdf/ 2004.04250.Google Scholar
- pdf, 2020.Google Scholar
- [Kar84] Narendra Karmarkar. A new polynomial-time algorithm for linear programming. In Proceedings of the sixteenth annual ACM symposium on Theory of computing (STOC), pages 302-311. ACM, 1984.Google ScholarDigital Library
- [Kha80] Leonid G Khachiyan. Polynomial algorithms in linear programming.Google Scholar
- USSR Computational Mathematics and Mathematical Physics, 20 ( 1 ): 53-72, 1980.Google Scholar
- [KJC82] Alexander S Kelso Jr and Vincent P Crawford. Job matching, coalition formation, and gross substitutes. Econometrica: Journal of the Econometric Society, pages 1483-1504, 1982.Google ScholarCross Ref
- [KTE88] Leonid G Khachiyan, Sergei Pavlovich Tarasov, and I. I. Erlikh. The method of inscribed ellipsoids. In Soviet Math. Dokl, volume 37, pages 226-230, 1988.Google Scholar
- [KV06] Adam Tauman Kalai and Santosh Vempala. Simulated annealing for convex optimization. Mathematics of Operations Research, 31 ( 2 ): 253-266, 2006.Google Scholar
- [LDFU13] Yichao Lu, Paramveer Dhillon, Dean P Foster, and Lyle Ungar. Faster ridge regression via the subsampled randomized hadamard transform. In Advances in neural information processing systems (NIPS), pages 369-377, 2013.Google Scholar
- [Lee16] Yin Tat Lee. Faster algorithms for convex and combinatorial optimization.Google Scholar
- PhD thesis, Massachusetts Institute of Technology, 2016.Google Scholar
- [Lew78] D. Lewis. Finite dimensional subspaces of ℓp. Studia Mathematica, 63 ( 2 ): 207-212, 1978.Google Scholar
- [LG14] François Le Gall. Powers of tensors and fast matrix multiplication. In Proceedings of the 39th international symposium on symbolic and algebraic computation (ISSAC), pages 296-303. ACM, 2014.Google ScholarDigital Library
- [LS14] Yin Tat Lee and Aaron Sidford. Path finding methods for linear √ programming: Solving linear programs in O ( r ank ) iterations and faster algorithms for maximum flow. In 55th Annual IEEE Symposium on Foundations of Computer Science (FOCS), pages 424-433. https: //arxiv.org/pdf/1312.6677.pdf, https://arxiv.org/pdf/1312.6713.pdf, 2014.Google ScholarDigital Library
- [LS15] Yin Tat Lee and Aaron Sidford. Eficient inverse maintenance and faster algorithms for linear programming. In 56th Annual IEEE Symposium on Foundations of Computer Science (FOCS), pages 230-249. https://arxiv.Google Scholar
- org/pdf/1503.01752.pdf, 2015.Google Scholar
- [LSS+20] Jason Lee, Ruoqi Shen, Zhao Song, Mengdi Wang, and Zheng Yu. Leverage score sampling, neural tangent kernel, and kernel ridge regression.Google Scholar
- In Manuscript, 2020.Google Scholar
- [LSV18] Yin Tat Lee, Aaron Sidford, and Santosh S Vempala. Eficient convex optimization with membership oracles. In Annual Conference on Learning Theory (COLT). https://arxiv.org/pdf/1706.07357.pdf, 2018.Google Scholar
- [LSW15] Yin Tat Lee, Aaron Sidford, and Sam Chiu-wai Wong. A faster cutting plane method and its implications for combinatorial and convex optimization. In 56th Annual IEEE Symposium on Foundations of Computer Science (FOCS), pages 1049-1065. https://arxiv.org/pdf/1508.04874.pdf, 2015.Google ScholarDigital Library
- [LSZ19] Yin Tat Lee, Zhao Song, and Qiuyi Zhang. Solving empirical risk minimization in the current matrix multiplication time. In Annual Conference on Learning Theory (COLT). https://arxiv.org/pdf/ 1905.04447, 2019.Google Scholar
- [LV06] László Lovász and Santosh Vempala. Fast algorithms for logconcave functions: Sampling, rounding, integration and optimization. In 47th Annual IEEE Symposium on Foundations of Computer Science (FOCS), pages 57-68. IEEE, 2006.Google ScholarDigital Library
- [LW17] Renato Paes Leme and Sam Chiu-wai Wong. Computing walrasian equilibria: fast algorithms and structural properties. In Proceedings of the Twenty-Eighth Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), pages 632-651. SIAM, https://arxiv.org/pdf/1511.04032.pdf, 2017.Google ScholarCross Ref
- [Mad13] Aleksander Madry. Navigating central path with electrical flows: From lfows to matchings, and back. In 54th Annual IEEE Symposium on Foundations of Computer Science (FOCS), pages 253-262. https://arxiv.Google Scholar
- org/pdf/1307.2205.pdf, 2013.Google Scholar
- [Mad16] Aleksander Madry. Computing maximum flow with augmenting electrical flows. In 2016 IEEE 57th Annual Symposium on Foundations of Computer Science (FOCS), pages 593-602. IEEE, https://arxiv.org/pdf/ 1608.06016.pdf, 2016.Google ScholarCross Ref
- [Nem94] Arkadi Nemirovski. Eficient methods in convex programming. Lecture notes, 1994.Google Scholar
- [NN89] YE Nesterov and AS Nemirovskii. Self-concordant functions and polynomial time methods in convex programming. preprint, central economic & mathematical institute, ussr acad. Sci. Moscow, USSR, 1989.Google Scholar
- [NN13] Jelani Nelson and Huy L Nguyên. OSNAP: Faster numerical linear algebra algorithms via sparser subspace embeddings. In 54th Annual IEEE Symposium on Foundations of Computer Science (FOCS), pages 117-126. IEEE, https://arxiv.org/pdf/1211.1002, 2013.Google ScholarDigital Library
- [NP83] EI Nenakov and ME Primak. One algorithm for finding solutions of the arrow-debreu model. Kibernetica, 3 : 127-128, 1983.Google Scholar
- [NY83] Arkadii Semenovich Nemirovsky and David Borisovich Yudin. Problem complexity and method eficiency in optimization. 1983.Google Scholar
- [Par99] David C Parkes. ibundle: An eficient ascending price bundle auction.Google Scholar
- [Pro96] V Yu Protasov. Algorithms for approximate calculation of the minimum of a convex function from its values. Mathematical Notes, 59 ( 1 ): 69-74, 1996.Google Scholar
- [PSW17] Eric Price, Zhao Song, and David P. Woodruf. Fast regression with an ℓ∞ guarantee. In International Colloquium on Automata, Languages, and Programming (ICALP), 2017.Google Scholar
- [PU02] David C Parkes and Lyle H Ungar. An ascending-price generalized vickrey auction. 2002.Google Scholar
- [RSW16] Ilya Razenshteyn, Zhao Song, and David P Woodruf. Weighted low rank approximations with provable guarantees. In Proceedings of the 48th Annual Symposium on the Theory of Computing (STOC), 2016.Google ScholarDigital Library
- [San04] Piotr Sankowski. Dynamic transitive closure via dynamic matrix inverse.Google Scholar
- In 45th Annual IEEE Symposium on Foundations of Computer Science (FOCS), pages 509-517. IEEE, 2004.Google Scholar
- [Sch18] Aaron Schild. An almost-linear time algorithm for uniform random spanning tree generation. In Proceedings of the 50th Annual ACM SIGACT Symposium on Theory of Computing (STOC), pages 214-227. ACM, https: //arxiv.org/pdf/1711.06455.pdf, 2018.Google ScholarDigital Library
- [Sho77] Naum Z Shor. Cut-of method with space extension in convex programming problems. Cybernetics and systems analysis, 13 ( 1 ): 94-96, 1977.Google Scholar
- [Sid15] Aaron Daniel Sidford. Iterative methods, combinatorial optimization, and linear programming beyond the universal barrier. PhD thesis, Massachusetts Institute of Technology, 2015.Google Scholar
- [Son19] Zhao Song. Matrix Theory : Optimization, Concentration and Algorithms.Google Scholar
- PhD thesis, The University of Texas at Austin, 2019.Google Scholar
- [SS11] Daniel A Spielman and Nikhil Srivastava. Graph sparsification by effective resistances. In SIAM Journal on Computing, volume 40 ( 6 ), pages 1913-1926. https://arxiv.org/pdf/0803.0929.pdf, 2011.Google Scholar
- [SWZ17] Zhao Song, David P Woodruf, and Peilin Zhong. Low rank approximation with entrywise ℓ1-norm error. In Proceedings of the 49th Annual Symposium on the Theory of Computing (STOC). ACM, https://arxiv.org/pdf/1611.00898, 2017.Google ScholarDigital Library
- [SWZ19] Zhao Song, David P Woodruf, and Peilin Zhong. Relative error tensor low rank approximation. In ACM-SIAM Symposium on Discrete Algorithms (SODA). https://arxiv.org/pdf/1704.08246, 2019.Google ScholarCross Ref
- [Vai87] Pravin M Vaidya. An algorithm for linear programming which requires o(((m + n)n2 + (m + n)1.5n) l ) arithmetic operations. In 28th Annual IEEE Symposium on Foundations of Computer Science (FOCS), 1987.Google Scholar
- [Vai89a] Pravin M Vaidya. A new algorithm for minimizing convex functions over convex sets. In 30th Annual IEEE Symposium on Foundations of Computer Science (FOCS), pages 338-343, 1989.Google ScholarDigital Library
- [Vai89b] Pravin M Vaidya. Speeding-up linear programming using fast matrix multiplication. In 30th Annual Symposium on Foundations of Computer Science (FOCS), pages 332-337. IEEE, 1989.Google ScholarDigital Library
- [Vaz10] Vijay V Vazirani. Spending constraint utilities with applications to the adwords market. Mathematics of Operations Research, 35 ( 2 ): 458-478, 2010.Google Scholar
- [Vég16] László A Végh. A strongly polynomial algorithm for a class of minimumcost flow problems with separable convex objectives. volume 45 ( 5 ), pages 1729-1761. https://arxiv.org/pdf/1110.4882.pdf, 2016.Google Scholar
- [Wan16] Zi Wang. A New Strongly Polynomial Algorithm for Computing Fisher Market Equilibria with Spending Constraint Utilities. PhD thesis, New York University, 2016.Google Scholar
- [Wil12] Virginia Vassilevska Williams. Multiplying matrices faster than coppersmith-winograd. In Proceedings of the forty-fourth annual ACM symposium on Theory of computing (STOC), pages 887-898. ACM, 2012.Google ScholarDigital Library
- [Ye08] Yinyu Ye. A path to the arrow-debreu competitive market equilibrium.Google Scholar
- Mathematical Programming, 111 ( 1-2 ): 315-348, 2008.Google Scholar
- [YN76] David B Yudin and Arkadii S Nemirovski. Evaluation of the information complexity of mathematical programming problems. Ekonomika i Matematicheskie Metody, 12 : 128-142, 1976.Google Scholar
Index Terms
- An improved cutting plane method for convex optimization, convex-concave games, and its applications
Recommendations
Convex hulls of spheres and convex hulls of disjoint convex polytopes
Given a set @S of spheres in E^d, with d>=3 and d odd, having a constant number of m distinct radii @r"1,@r"2,...,@r"m, we show that the worst-case combinatorial complexity of the convex hull of @S is @Q(@__ __"1"=<"i"<>"j"=<"mn"in"j^@__ __^d^2^@__ __), ...
Towards a Practical Volumetric Cutting Plane Method for Convex Programming
We consider the volumetric cutting plane method for finding a point in a convex set ${\cal C}\subset\Re^n$ that is characterized by a separation oracle. We prove polynomiality of the algorithm with each added cut placed directly through the current ...
Convex hulls of spheres and convex hulls of convex polytopes lying on parallel hyperplanes
SoCG '11: Proceedings of the twenty-seventh annual symposium on Computational geometryGiven a set Σ of spheres in Ed, with d≥3 and d odd, having a fixed number of m distinct radii ρ1,ρ2,...,ρm, we show that the worst-case combinatorial complexity of the convex hull CHd(Σ) of Σ is Θ(Σ{1≤i≠j≤m}ninj⌊ d/2 ⌋), where ni is the number of ...
Comments