Materials Project computation and database infrastructure
1 like540 views
The document describes the Materials Project computation infrastructure, which uses the Atomate framework to automatically run density functional theory simulations on over 85,000 materials in a high-throughput manner, with the results stored in a MongoDB database for users to explore and analyze in order to accelerate materials innovation. The Materials Project infrastructure aims to make it easy for researchers to generate large amounts of computational data on materials properties through standardized and scalable workflows.
![4
Many data sets are available!
M. De Jong et
al. Sci. Data,
2015, 2,
150009.
]
M. De Jong et
al. Sci. Data,
2015, 2,
150009.](https://image.slidesharecdn.com/20181115presentationdei-181115164217/85/Materials-Project-computation-and-database-infrastructure-4-320.jpg)
![1. As described previously: for each data type (a
“material”, “task”, “workflow”, etc.) decide on a
set of fields that describe each instance of that
data type. In MongoDB, these fields can easily
be changed or added to later if needed.
2. Try to create a single collection and document
format that can handle any kind of materials
data!
– example 1: “PIF” file format from Citrine[1]
– example 2: MPContribs from Materials Project[2]
34
Two approaches to store data in MongoDB
[1] J. O’Mara, B. Meredig, K. Michel, Materials Data
Infrastructure : A Case Study of the Citrination Platform to
Examine Data Import , Storage , and Access, Jom. (2016).
[2] P. Huck, D. Gunter, S. Cholia, D. Winston, A.T. N’Diaye, K. Persson, User
applications driven by the community contribution framework MPContribs
in the Materials Project, Concurr. Comput. Pract. Exp. 22 (2015)](https://image.slidesharecdn.com/20181115presentationdei-181115164217/85/Materials-Project-computation-and-database-infrastructure-34-320.jpg)
More Related Content
Similar to Materials Project computation and database infrastructure (20)
More from Anubhav Jain (20)
Recently uploaded (20)
Materials Project computation and database infrastructure
- 1. Materials Project computation and database infrastructure Anubhav Jain Energy Technologies Area Lawrence Berkeley National Laboratory Berkeley, CA Presentation given to Delaware Energy Institute, 2018 Slides (already) posted to https://hackingmaterials.lbl.gov
- 2. Outline 2 ① Introduction to the Materials Project ② Materials Project computation infrastructure ③ Database considerations
- 3. The Materials Project database • Online resource of density functional theory simulation data for ~85,000 inorganic materials • Includes band structures, elastic tensors, piezoelectric tensors, battery properties and more • >60,000 registered users • Free • www.materialsproject.org 3 Jain et al. Commentary: The Materials Project: A materials genome approach to accelerating materials innovation. APL Mater. 1, 11002 (2013).
- 4. 4 Many data sets are available! M. De Jong et al. Sci. Data, 2015, 2, 150009. ] M. De Jong et al. Sci. Data, 2015, 2, 150009.
- 5. 5 As well as “apps” for exploring the data
- 6. Outline 6 ① Introduction to the Materials Project ② Materials Project computation infrastructure ③ Database considerations
- 7. A “black-box” view of performing a calculation 7 “something” Results! researcher What is the GGA-PBE elastic tensor of GaAs?
- 8. Unfortunately, the inside of the “black box” is usually tedious and “low-level” 8 lots of tedious, low-level work… Results! researcher What is the GGA-PBE elastic tensor of GaAs? Input file flags SLURM format how to fix ZPOTRF? q set up the structure coordinates q write input files, double-check all the flags q copy to supercomputer q submit job to queue q deal with supercomputer headaches q monitor job q fix error jobs, resubmit to queue, wait again q repeat process for subsequent calculations in workflow q parse output files to obtain results q copy and organize results, e.g., into Excel
- 9. What would be a better way? 9 “something” Results! researcher What is the GGA-PBE elastic tensor of GaAs?
- 10. What would be a better way? 10 Results! researcher What is the GGA-PBE elastic tensor of GaAs? Workflows to run q band structure q surface energies ü elastic tensor q Raman spectrum q QH thermal expansion
- 11. Ideally the method should scale to millions of calculations 11 Results! researcher Start with all binary oxides, replace O->S, run several different properties Workflows to run ü band structure ü surface energies ü elastic tensor q Raman spectrum q QH thermal expansion q spin-orbit coupling
- 12. Atomate tries make it easy, automatic, and flexible to generate data with existing simulation packages 12 Results! researcher Run many different properties of many different materials!
- 13. Atomate contains a library of simulation procedures 13 VASP-based • band structure • spin-orbit coupling • hybrid functional calcs • elastic tensor • piezoelectric tensor • Raman spectra • NEB • GIBBS method • QH thermal expansion • AIMD • ferroelectric • surface adsorption • work functions Other • BoltzTraP • FEFF method • LAMMPS MD Mathew, K. et al Atomate: A high-level interface to generate, execute, and analyze computational materials science workflows, Comput. Mater. Sci. 139 (2017) 140–152.
- 14. Each simulation procedure translates high-level instructions into a series of low-level tasks 14 quickly and automatically translate PI-style (minimal) specifications into well-defined FireWorks workflows What is the GGA-PBE elastic tensor of GaAs? M. De Jong, W. Chen, T. Angsten, A. Jain, R. Notestine, A. Gamst, et al., Charting the complete elastic properties of inorganic crystalline compounds, Sci. Data. 2 (2015).
- 15. Atomate thus encodes and standardizes knowledge about running various kinds of simulations from domain experts 15 K. Mathew J. Montoya S. Dwaraknath A. Faghaninia All past and present knowledge, from everyone in the group, everyone previously in the group, and our collaborators, about how to run calculations M. Aykol S.P. Ong B. Bocklund T. Smidt H. Tang I.H. Chu M. Horton J. Dagdalen B. Wood Z.K. Liu J. Neaton K. Persson A. Jain +
- 16. 16 Full operation diagram job 1 job 2 job 3 job 4 structure workflow database of all workflows automatically submit + executeoutput files + database
- 17. 17 Full operation diagram job 1 job 2 job 3 job 4 structure workflow database of all workflows automatically submit + executeoutput files + database
- 18. • Pymatgen can retrieve crystal structures from the Materials Project database (MPRester class) • It can also manipulate crystal structures – substitutions – supercell creation – order-disorder (shown at right) – interstitial finding – surface / slab generation • A visual interface to many of the tools are in Materials Project’s “Crystal Toolkit” app 18 Crystal structure generation via pymatgen Example: Order-disorder resolve partial or mixed occupancies into a fully ordered crystal structure (e.g., mixed oxide-fluoride site into separate oxygen/fluorine)
- 19. 19 Full operation diagram job 1 job 2 job 3 job 4 structure workflow database of all workflows automatically submit + executeoutput files + database
- 20. 20 Atomate’s main goal – convert structures to workflows Workflows consist of a series of jobs (“FireWorks”), each with multiple tasks. Atomate jobs typically (i) run a calculation and (ii) store the results in a database
- 21. 21 Full operation diagram job 1 job 2 job 3 job 4 structure workflow database of all workflows automatically submit + executeoutput files + database
- 22. FireWorks allows you to write your workflow once and execute (almost) anywhere 22 • Execute workflows locally or at a supercomputing center • Queue systems supported – PBS – SGE – SLURM – IBM LoadLeveler – NEWT (a REST-based API at NERSC) – Cobalt (Argonne LCF)
- 23. Dashboard with status of all jobs 23
- 24. • Job provenance and automatic metadata storage • Detect and rerun failures • “Dynamic” workflows that change behavior based on results • Customize job priorities • Much more… 24 Other features
- 25. 25 Full operation diagram job 1 job 2 job 3 job 4 structure workflow database of all workflows automatically submit + executeoutput files + database
- 26. Atomate – builders framework 26 “Builders” start with base collections in a database and create higher-level collections that summarize information or add metadata
- 27. 27 The atomate database makes it easy to perform various analyses with pymatgen atomate output database(s) phase diagrams Pourbaix diagrams diffusivity via MDband structure analysis
- 28. 28 Many research groups have run tens of thousands of materials science workflows with atomate also used by: • Persson research group, UC Berkeley • Ong research group, UC San Diego • Neaton research group, UC Berkeley • Liu research group, Penn State • Groups not developing on atomate! • e.g., see “Thermal expansion of quaternary nitride coatings” by Tasnadi et al. atomate now powers the Materials Project and will be used to run hundreds of thousands of simulations in the next year (www.materialsproject.org)
- 29. Outline 29 ① Introduction to the Materials Project ② Materials Project computation infrastructure ③ Database considerations
- 30. 30 About a decade ago, we were using a SQL infrastructure Main problems we ran into: • Too static – every time we wanted to store a new kind of data, the DB master needed to “design and update” the database schema • Too difficult for newcomers – constructing queries (joins, etc.). We actually designed a system to help people make queries, which is common
- 31. 31 Since then, we have switched to MongoDB – a “noSQL” database Major advantages • Very dynamic – easy to add new data types without interfering with old data types or redesigning everything. No central “database master” needed • Easy for newcomers – easy syntax, no complex “joins”, easy to visualize results • Easy object-relational mapping – built our pymatgen code so that any objects (e.g., band structures, crystal structures, etc.) could be exported to a database or imported from a database easily
- 32. 32 How we store computed data Data is stored in “collections”. Each collection is a set of documents that can be queried. Each document consists of nested key- value pairs (“dictionaries”) or arrays. e.g. one can search for: {“tags”: “phosphides”} to retrieve all documents tagged with “phosphide”
- 33. 33 Each collection has a set of standard keys Data is stored in “collections”. Each collection is a set of documents that can be queried. materials collection – each document represents a material, with keys like “formula” and “band_gap” tasks collection – each document represents a DFT calculation, with keys like “dir_name” and “input.parameters” workflows collection – each document represents a calculation workflow, with keys like “nodes” and “links” Typically, each document within a collection will be of a uniform format, but this not a hard requirement in MongoDB.
- 34. 1. As described previously: for each data type (a “material”, “task”, “workflow”, etc.) decide on a set of fields that describe each instance of that data type. In MongoDB, these fields can easily be changed or added to later if needed. 2. Try to create a single collection and document format that can handle any kind of materials data! – example 1: “PIF” file format from Citrine[1] – example 2: MPContribs from Materials Project[2] 34 Two approaches to store data in MongoDB [1] J. O’Mara, B. Meredig, K. Michel, Materials Data Infrastructure : A Case Study of the Citrination Platform to Examine Data Import , Storage , and Access, Jom. (2016). [2] P. Huck, D. Gunter, S. Cholia, D. Winston, A.T. N’Diaye, K. Persson, User applications driven by the community contribution framework MPContribs in the Materials Project, Concurr. Comput. Pract. Exp. 22 (2015)
- 35. 35 MPContribs and MPFile – storing / querying any kind of materials data into Materials Project
- 36. 36 MPContribs portal (currently in beta testing, access provided by request)
- 37. Funding: DOE-BES Materials Science Division, Computing: NERSC 37 Who to talk to next! The current “Guardians of the MP infrastructure” Slides (already) posted to https://hackingmaterials.lbl.gov