Spark access control on Amazon EMR with AWS Lake Formation
- 1. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Spark access control on Amazon EMR with AWS Lake Formation Anoop Johnson A N T 2 2 3 - R Principal Software Engineer Amazon Web Services
- 2. What you will learn Background Authentication Query execution Demo
- 3. Lake Formation: Secure once, access in many ways Amazon Athena Amazon Redshift Amazon EMR AWS Glue Amazon S3 Data catalog Permissions Lake Formation Admin
- 4. Control data access with grant and revoke
- 5. Permissions on tables and columns, not Amazon Simple Storage Service (Amazon S3)
- 6. View and audit data access
- 7. Why integrate Amazon EMR with Lake Formation? • Fine-grained, column-level access to databases and tables • Allows shared multi-tenant clusters to securely access data • Uses the AWS Glue Data Catalog as the metadata store • Federated single sign-on from your enterprise identity system • Active Directory (AD FS), Auth0, Okta, and many others • Uses Security Assertion Markup Language (SAML) 2.0
- 9. Amazon EMR authentication Amazon EMR cluster User Used for impersonation Lake Formation
- 10. Query execution under the hood Amazon EMR worker node 1 32 4 Amazon S3 5 6 Lake Formation
- 11. Amazon EMR: Supported applications • AWS Glue Data Catalog • Identity providers with support for SAML • Applications • Spark SQL • Amazon EMR Notebooks and Zeppelin with Livy
- 12. Integrating Amazon EMR with Lake Formation • Establish trust relationship between your corporate IdP and AWS • Configure IAM roles for Lake Formation • Configure Amazon EMR security features • Launch an Amazon EMR Lake Formation-enabled cluster
- 13. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.