×

Biomolecular electrostatics using a fast multipole BEM on up to 512 GPUs and a billion unknowns. (English) Zbl 1259.78044

Summary: We present teraflop-scale calculations of biomolecular electrostatics enabled by the combination of algorithmic and hardware acceleration. The algorithmic acceleration is achieved with the fast multipole method (FMM) in conjunction with a boundary element method (BEM) formulation of the continuum electrostatic model, as well as the BIBEE approximation to BEM. The hardware acceleration is achieved through graphics processors, GPUs. We demonstrate the power of our algorithms and software for the calculation of the electrostatic interactions between biological molecules in solution. The applications demonstrated include the electrostatics of protein-drug binding and several multi-million atom systems consisting of hundreds to thousands of copies of lysozyme molecules. The parallel scalability of the software was studied in a cluster at the Nagasaki advanced computing center, using 128 nodes, each with four GPUs. Delicate tuning has resulted in strong scaling with parallel efficiency of 0.8 for 256 and 0.5 for 512 GPUs. The largest application run, with over 20 million atoms and one billion unknowns, required only one minute on 512 GPUs. We are currently adapting our BEM software to solve the linearized Poisson-Boltzmann equation for dilute ionic solutions, and it is also designed to be flexible enough to be extended for a variety of integral equation problems, ranging from Poisson problems to Helmholtz problems in electromagnetics and acoustics to high Reynolds number flows.

MSC:

78M16 Multipole methods applied to problems in optics and electromagnetic theory
78M15 Boundary element methods applied to problems in optics and electromagnetic theory
78-04 Software, source code, etc. for problems pertaining to optics and electromagnetic theory
82D60 Statistical mechanics of polymers
PDFBibTeX XMLCite
Full Text: DOI arXiv

References:

[1] Alexov, E. G.; Gunner, M. R., Incorporating protein conformational flexibility into the calculation of pH-dependent protein properties, Biophys. J., 72, 5, 2075-2093 (1997)
[2] Altman, M. D.; Bardhan, J. P.; Tidor, B.; White, J. K., FFTSVD: A fast multiscale boundary-element method solver suitable for BioMEMS and biomolecule simulation, IEEE Trans. Comput. Aided Des., 25, 274-284 (2006)
[3] Altman, M. D.; Bardhan, J. P.; White, J. K.; Tidor, B., Accurate solution of multi-region continuum electrostatic problems using the linearized Poisson-Boltzmann equation and curved boundary elements, J. Comput. Chem., 30, 132-153 (2009)
[4] Anandakrishnan, R.; Scogland, T. R.W.; Fenley, A. T.; Gordon, J. C.; Feng, W.-C.; Onufriev, A. V., Accelerating electrostatic surface potential calculation with multi-scale approximation on graphics processing units, J. Mol. Graph. Model., 28, 8, 904-910 (2010)
[5] Anderson, M.; Beattie, J.; Breault, G.; Breed, J.; Byth, K.; Culshaw, J.; Ellston, R.; Green, S.; Minshull, C.; Norman, R.; Pauptit, R.; Stanway, J.; Thomas, A.; Jewsbury, P., Imidazo[1,2-a]pyridines: A potent and selective class of cyclin dependent kinase inhibitors identified through structure-based hybridization, Bioorg. Med. Chem. Lett., 13, 3021 (2003)
[6] Atkinson, K. E., The Numerical Solution of Integral Equations of the Second Kind (1997), Cambridge University Press · Zbl 0155.47404
[7] Baker, N. A.; Sept, D.; Holst, M. J.; McCammon, J. A., Electrostatics of nanosystems: Application to microtubules and the ribosome, Proc. Natl. Acad. Sci. USA, 98, 10037-10041 (2001)
[8] Bardhan, J. P., Interpreting the Coulomb-field approximation for generalized-Born electrostatics using boundary-integral equation theory, J. Chem. Phys., 129, 144105 (2008)
[9] Bardhan, J. P., Numerical solution of boundary-integral equations for molecular electrostatics, J. Chem. Phys., 130, 094102 (2009)
[10] Bardhan, J. P.; Altman, M. D.; White, J. K.; Tidor, B., Numerical integration techniques for curved-panel discretizations of molecule-solvent interfaces, J. Chem. Phys., 127, 014701 (2007)
[11] Bardhan, J. P.; Eisenberg, R. S.; Gillespie, D., Discretization of the induced-charge boundary integral equation, Phys. Rev. E, 80, 011906 (2009)
[12] Bardhan, J. P.; Knepley, M. G.; Anitescu, M., Bounding the electrostatic free energies associated with linear continuum models of molecular solvation, J. Chem. Phys., 130, 10, 104108 (2009)
[13] Berman, H. M.; Westbrook, J.; Feng, Z.; Gilliland, G.; Bhat, T. N.; Weissig, H.; Shindyalov, I. N.; Bourne, P. E., The protein data bank, Nucl. Acids Res., 28, 1, 235-242 (2000)
[14] Bharadwaj, R.; Windemuth, A.; Sridharan, S.; Honig, B.; Nicholls, A., The fast multipole boundary element method for molecular electrostatics: An optimal approach for large systems, J. Comput. Chem., 16, 7, 898-913 (1995)
[15] Bordner, A. J.; Huber, G. A., Boundary element solution of the linear Poisson-Boltzmann equation and a multipole method for the rapid calculation of forces on macromolecules in solution, J. Comput. Chem., 24, 3, 353-367 (2003)
[16] Borgis, D.; Lévy, N.; Marchi, M., Computing the electrostatic free-energy of complex molecules: The variational Coulomb field approximation, J. Chem. Phys., 119, 6, 3516-3528 (2003)
[17] Brooks, B. R.; Bruccoleri, R. E.; Olafson, B. D.; States, D. J.; Swaminathan, S.; Karplus, M., CHARMM: A program for macromolecular energy, minimization, and dynamics calculations, J. Comput. Chem., 4, 187-217 (1983)
[18] Bruccoleri, R. E.; Novotny, J.; Davis, M. E.; Sharp, K. A., Finite difference Poisson-Boltzmann electrostatic calculations: Increased accuracy achieved by harmonic dielectric smoothing and charge antialiasing, J. Comput. Chem., 18, 2, 268-276 (1997)
[19] Caravella, J. A.; Carbeck, Jeffrey D.; Duffy, D. C.; Whitesides, G. M.; Tidor, B., Long-range electrostatic contributions to protein-ligand binding estimated using protein charge ladders, affinity capillary electrophoresis, and continuum electrostatic theory, J. Am. Chem. Soc., 121, 4340-4347 (1999)
[20] Carrascal, N.; Green, D. F., Energetic decomposition with the generalized-Born and Poisson-Boltzmann solvent models: Lessons from association of G-protein components, J. Phys. Chem. B, 114, 15, 5096-5116 (2010), PMID: 20355699
[21] Chang, R. C.; Asthagiri, D.; Lenhoff, A. M., Measured and calculated effects of mutations in bacteriophage T4 lysozyme on interactions in solution, Proteins, 41, 123-132 (2000)
[22] Cheng, H.-L.; Shi, X., Quality mesh generation for molecular skin surfaces using restricted union of balls, Comput. Geom., 42, 3, 196-206 (2009) · Zbl 1158.65014
[23] Chipman, D. M., Charge penetration in dielectric models of solvation, J. Chem. Phys., 106, 10194-10206 (1997)
[24] Connolly, M. L., Analytical molecular surface calculation, J. Appl. Cryst., 16, 548-558 (1983)
[25] Cruz, F. A.; Knepley, M. G.; Barba, L. A., PetFMM—a dynamically load-balancing parallel fast multipole library, Internat. J. Numer. Methods Engrg., 85, 4, 403-428 (2010) · Zbl 1217.70005
[26] Darden, T.; York, D.; Pedersen, L., Particle mesh Ewald: An \(N \log N\) method for Ewald sums in large systems, J. Chem. Phys., 93, 10089-10092 (1993)
[27] Dijkstra, W.; Mattheij, R. M.M. (2006), The condition number of the BEM-matrix arising from Laplaceʼs equation, Technical report, OAI Repository of the Technische Universiteit Eindhoven (TU/e) (Netherlands) · Zbl 1244.76039
[28] Dyson, H. J.; Wright, P. E., Intrinsically unstructured proteins and their functions, Nat. Rev. Mol. Cell. Biol., 6, 197-208 (2005)
[29] Elcock, A. H., Molecular simulations of diffusion and association in multimacromolecular systems, Meth. Enzymol., 383, 166-198 (2004)
[30] Ellis, R. J., Macromolecular crowding: An important but neglected aspect of the intracellular environment, Curr. Opin. Chem. Biol., 11, 1, 114-119 (2001)
[31] Fan, H.; Mark, A. E.; Zhu, J.; Honig, B., Comparative study of generalized Born models: Protein dynamics, Proc. Natl. Acad. Sci. USA, 102, 19, 6760-6764 (2005)
[32] Fenley, A. T.; Gordon, J. C.; Onufriev, A., An analytical approach to computing biomolecular electrostatic potential. I. Derivation and analysis, J. Chem. Phys., 129, 7, 075101 (2008)
[33] Gabdoulline, R. R.; Wade, R. C., Simulation of the diffusional association of barnase and barstar, Biophys. J., 72, 1917-1929 (1997)
[34] Geng, W. H.; Yu, S. N.; Wei, G. W., Treatment of charge singularities in implicit solvent models, J. Chem. Phys., 127, 114106 (2007)
[35] Ghosh, A.; Rapp, C. S.; Friesner, R. A., Generalized Born model based on a surface integral formulation, J. Phys. Chem. B, 102, 10983-10990 (1998)
[36] Gilson, M. K.; McCammon, J. A.; Madura, J. D., Molecular-dynamics simulation with a continuum electrostatic representation of the solvent, J. Comput. Chem., 16, 9, 1081-1095 (1995)
[37] Gilson, M. K.; Rashin, A.; Fine, R.; Honig, B., On the calculation of electrostatic interactions in proteins, J. Mol. Biol., 184, 503-516 (1985)
[38] Green, D. F.; Tidor, B., Design of improved protein inhibitors of HIV-1 cell entry: Optimization of electrostatic interactions at the binding interface, Proteins, 60, 644-657 (2005)
[39] Greengard, L.; Rokhlin, V., A fast algorithm for particle simulations, J. Comput. Phys., 73, 2, 325-348 (1987) · Zbl 0629.65005
[40] Hamada, T.; Narumi, T.; Yokota, R.; Yasuoka, K.; Nitadori, K.; Taiji, M., 42 TFlops hierarchical \(N\)-body simulations on GPUs with applications in both astrophysics and turbulence, (SCʼ09: Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis (2009), ACM: ACM New York, NY), 1-12
[41] Hardy, D. J.; Stone, J. E.; Schulten, K., Multilevel summation of electrostatic potentials using graphics processing units, Parallel Comput., 35, 164-177 (2009)
[42] Hess, J. L.; Smith, A. M.O., Calculation of non-lifting potential flow about arbitrary three-dimensional bodies, J. Ship Res., 8, 2, 22-44 (1962)
[43] Humphrey, W.; Dalke, A.; Schulten, K., VMD: Visual molecular dynamics, J. Mol. Graphics, 14, 1, 33-38 (1996)
[44] Jorgensen, W. L.; Ulmschneider, J. P.; Tirado-Rives, J., Free energies of hydration from a generalized Born model and an all-atom force field, J. Phys. Chem. B, 108, 16264-16270 (2004)
[45] Juffer, A. H.; Botta, E. F.F.; van Keulen, B. A.M.; van der Ploeg, A.; Berendsen, H. J.C., The electric potential of a macromolecule in a solvent: A fundamental approach, J. Comput. Phys., 97, 1, 144-171 (1991) · Zbl 0743.65094
[46] Kuo, S. S.; Altman, M. D.; Bardhan, J. P.; Tidor, B.; White, J. K., Fast methods for simulation of biomolecule electrostatics, (Proceedings of the 2002 IEEE/ACM International Conference on Computer-Aided Design, ICCAD ʼ02 (2002), ACM: ACM New York, NY, USA), 466-473
[47] I. Lashuk, A. Chandramowlishwaran, H. Langston, T. Nguyen, R. Sampath, A. Shringarpure, R. Vuduc, L. Ying, D. Zorin, G. Biros, A massively parallel adaptive fast-multipole method on heterogeneous architectures, in: Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis, SC ʼ09, Portland, Oregon, November 2009, pp. 1-12.; I. Lashuk, A. Chandramowlishwaran, H. Langston, T. Nguyen, R. Sampath, A. Shringarpure, R. Vuduc, L. Ying, D. Zorin, G. Biros, A massively parallel adaptive fast-multipole method on heterogeneous architectures, in: Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis, SC ʼ09, Portland, Oregon, November 2009, pp. 1-12.
[48] Liang, J.; Subramaniam, S., Computation of molecular electrostatics with boundary element methods, Biophys. J., 73, 4, 1830-1841 (1997)
[49] Liu, H.-Y.; Grinter, S. Z.; Zou, X., Multiscale generalized Born modeling of ligand binding energies for virtual database screening, J. Phys. Chem. B, 113, 35, 11793-11799 (2009), PMID: 19678651
[50] Lu, B.; Cheng, X.; Huang, J.; McCammon, J. A., Order \(N\) algorithm for computation of electrostatic interactions in biomolecular systems, Proc. Natl. Acad. Sci. USA, 103, 51, 19314-19319 (2006)
[51] Lu, B.; McCammon, J. A., Improved boundary element methods for Poisson-Boltzmann electrostatic potential and force calculations, J. Chem. Theory Comput., 3, 1134-1142 (2007)
[52] Lu, B.; Zhang, D.; McCammon, J. A., Computation of electrostatic forces between solvated molecules determined by the Poisson-Boltzmann equation using a boundary element method, J. Chem. Phys., 122, 21, 214102 (2005)
[53] Massova, I.; Kollman, P. A., Combined molecular mechanical and continuum solvent approach MM-PBSA/GBSA to predict ligand binding, Perspect. Drug Discov., 18, 113-135 (2000)
[54] McGuffee, S. R.; Elcock, A. H., Atomistically detailed simulations of concentrated protein solutions: The effects of salt, pH, point mutations, and protein concentration in simulations of 1000-molecule systems, J. Am. Chem. Soc., 128, 12098-12110 (2006)
[55] Miertus, S.; Scrocco, E.; Tomasi, J., Electrostatic interactions of a solute with a continuum — a direct utilization of ab initio molecular potentials for the prevision of solvent effects, Chem. Phys., 55, 1, 117-129 (1981)
[56] Minton, A. P., Implications of macromolecular crowding for protein assembly, Curr. Opin. Struc. Biol., 10, 34-39 (2000)
[57] Nabors, K.; White, J., FastCap: A multipole accelerated 3-D capacitance extraction program, IEEE Trans. Comput. Aided Des., 10, 11, 1447-1459 (1991)
[58] Neal, B. L.; Lenhoff, A. M., Excluded volume contribution to the osmotic second virial coefficient for proteins, AIChE J., 41, 1010-1014 (1995)
[59] Newman, J. N., Distribution of sources and normal dipoles over a quadrilateral panel, J. Engrg. Math., 20, 2, 113-126 (1986)
[60] NVIDIA Corp., CUDA Programming Guide. Version 2.2.1, May 2009.; NVIDIA Corp., CUDA Programming Guide. Version 2.2.1, May 2009.
[61] Phillips, J. C.; Braun, R.; Wang, W.; Gumbart, J.; Tajkhorshid, E.; Villa, E.; Chipot, C.; Skeel, R. D.; Kale, L.; Schulten, K., Scalable molecular dynamics with NAMD, J. Comput. Chem., 26, 1781-1802 (2005)
[62] Phillips, J. R.; White, J. K., A precorrected-FFT method for electrostatic analysis of complicated 3-D structures, IEEE Trans. Comput. Aided Des., 16, 10, 1059-1072 (1997)
[63] Qiu, D.; Shenkin, P. S.; Hollinger, F. P.; Still, W. C., The GB/SA continuum model for solvation. A fast analytical method for the calculation of approximate Born radii, J. Phys. Chem. A, 101, 16, 3005-3014 (1997)
[64] Rahimian, A.; Lashuk, I.; Veerapaneni, S.; Chandramowlishwaran, A.; Malhotra, D.; Moon, L.; Sampath, R.; Shringarpure, A.; Vetter, J.; Vuduc, R.; Zorin, D.; Biros, G., Petascale direct numerical simulation of blood flow on 200k cores and heterogeneous architectures, (Proceedings of the 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis, SC ʼ10 (2010), IEEE Computer Society: IEEE Computer Society Washington, DC, USA), 1-11
[65] Richards, F. M., Areas, volumes, packing and protein structure, Annu. Rev. Biophys. Bioeng., 6, 151-176 (1977)
[66] Rokhlin, V., Rapid solution of integral equation of classical potential theory, J. Comput. Phys., 60, 187-207 (1983) · Zbl 0629.65122
[67] Romanov, A. N.; Jabin, S. N.; Martynov, Y. B.; Sulimov, A. V.; Grigoriev, F. V.; Sulimov, V. B., Surface generalized Born method: A simple, fast, and precise implicit solvent model beyond the Coulomb approximation, J. Phys. Chem. A, 108, 43, 9323-9327 (2004)
[68] Rush, S.; Turner, A. H.; Cherin, A. H., Computer solution for time-invariant electric fields, J. Appl. Phys., 37, 6, 2211-2217 (1966)
[69] Saad, Y.; Schultz, M., GMRES: A generalized minimal residual algorithm for solving nonsymmetric linear systems, SIAM J. Sci. Stat. Comput., 7, 856-869 (1986) · Zbl 0599.65018
[70] Sanner, M. F. (1996), (last checked Feb. 4, 2010)
[71] Schutz, C. N.; Warshel, A., What are the dielectric constants of proteins and how to validate electrostatic models?, Proteins, 44, 400-417 (2001)
[72] Sharp, K. A.; Honig, B., Calculating total electrostatic energies with the nonlinear Poisson-Boltzmann equation, J. Phys. Chem., 94, 19, 7684-7692 (1990)
[73] Sharp, K. A.; Honig, B., Electrostatic interactions in macromolecules: Theory and applications, Annu. Rev. Biophys. Bioeng., 19, 301-332 (June 1990)
[74] Shaw, P. B., Theory of the Poisson Greenʼs-function for discontinuous dielectric media with an application to protein biophysics, Phys. Rev. A, 32, 4, 2476-2487 (1985)
[75] Sitkoff, D.; Sharp, K. A.; Honig, B., Accurate calculation of hydration free energies using macroscopic solvent models, J. Phys. Chem., 98, 7, 1978-1988 (1994)
[76] Skeel, R. D.; Tezcan, I.; Hardy, D. J., Multiple grid methods for classical molecular dynamics, J. Comput. Chem., 23, 6, 673-684 (2002)
[77] Spector, S.; Wang, M. H.; Carp, S. A.; Robblee, J.; Hendsch, Z. S.; Fairman, R.; Tidor, B.; Raleigh, D. P., Rational modification of protein stability by the mutation of charged surface residues, Biochemistry, 39, 872-879 (2000)
[78] Still, W. C.; Tempczyk, A.; Hawley, R. C.; Hendrickson, T. F., Semianalytical treatment of solvation for molecular mechanics and dynamics, J. Am. Chem. Soc., 112, 16, 6127-6129 (1990)
[79] Stork, M.; Tavan, P., Electrostatics of proteins in dielectric solvent continua. I. Newtonʼs third law marries qE forces, J. Chem. Phys., 126, 16, 165105 (2007)
[80] Takahashi, T.; Hamada, T., GPU-accelerated boundary element method for Helmholtzʼ equation in three dimensions, Internat. J. Numer. Methods Engrg., 80, 10, 1295-1321 (2009) · Zbl 1183.76829
[81] Vizcarra, C. L.; Mayo, S. L., Electrostatics in computational protein design, Curr. Opin. Chem. Biol., 9, 6, 622-626 (2005)
[82] Warren, M. S.; Salmon, J. K., A parallel hashed oct-tree \(N\)-body algorithm, (Proceedings of the 1993 ACM/IEEE Conference on Supercomputing (1993), ACM: ACM New York), 12-21
[83] Warshel, A.; Sharma, P. K.; Kato, M.; Parson, W. W., Modeling electrostatic effects in proteins, Biochim. Biophys. Acta, 1764, 1647-1676 (2006)
[84] Warwicker, J.; Watson, H. C., Calculation of the electric potential in the active site cleft due to alpha-helix dipoles, J. Mol. Biol., 157, 671-679 (1982)
[85] Ying, L.; Biros, G.; Zorin, D., A kernel-independent adaptive fast multipole algorithm in two and three dimensions, J. Comput. Phys., 196, 2, 591-626 (2004) · Zbl 1053.65095
[86] Yokota, R.; Sheel, T. K.; Obi, S., Calculation of isotropic turbulence using a pure Lagrangian vortex method, J. Comput. Phys., 226, 2, 1589-1606 (2007) · Zbl 1121.76046
[87] Yoon, B. J.; Lenhoff, A. M., A boundary element method for molecular electrostatics with electrolyte effects, J. Comput. Chem., 11, 9, 1080-1086 (1990)
[88] You, T. J.; Bashford, D., Conformation and hydrogen ion titration of proteins: A continuum electrostatic model with conformational flexibility, Biophys. J., 69, 5, 1721-1733 (1995)
[89] Zhang, Y.; Xu, G.; Bajaj, C., Quality meshing of implicit solvation models of biomolecular structures, Comput. Aided Geom. Design, 23, 6, 510-530 (2006) · Zbl 1098.92034
[90] Zhou, H. X., Boundary-element solution of macromolecular electrostatics—interaction energy between 2 proteins, Biophys. J., 65, 955-963 (1993)
[91] Zimmerman, S. B.; Minton, A. P., Macromolecular crowding: Biochemical, biophysical, and physiological consequences, Annu. Rev. Biophys. Biomol. Struct., 22, 1, 27-65 (1993)
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.