Autoscaling Spark on AWS EC2 - 11th Spark London meetupRafal Kwasny
This document discusses autoscaling Spark clusters on AWS for efficiency and cost-effectiveness. It presents a typical AWS architecture with Spark running on EC2 and data stored in S3. It describes how autoscaling works to dynamically adjust the number of EC2 instances based on demand metrics to match resource usage. The spark-cloud tool is introduced to simplify managing Spark clusters on AWS with features like building AMIs, starting and shutting down clusters, and using spot instances for lower costs compared to on-demand pricing. Autoscaling helps remove the need to manually scale clusters up and down.
Introduction of OpsStack--Integrated Operation Platform developed by ChinaNetCloud
Operations Evolution Underway
Customer Challenges
OpsStack Killer Features
ChinaNetCloud talk on Operations-as-a-Service at AWS China Summit 2015 in Shanghai. Covers OaaS cases, technology, and AWS.
Given December 17th, 2015, in Shanghai, in Chinese by Yinan Gu.
This document provides guidance on troubleshooting issues with EC2 instances and Elastic Load Balancers (ELB) on AWS. It begins by recommending monitoring the AWS service health dashboard and CloudWatch metrics. Potential causes and resolutions are outlined for common problems with EC2 instance launching, health, networking, and EBS volumes. For ELBs, error messages, response metrics, health checks, and other potential problems are covered. The document concludes by listing information needed for support cases and additional resources.
This document discusses DevOps practices on AWS. It defines DevOps as a culture or practice to promote collaboration between development and operations teams. It then discusses why DevOps is needed to enable faster and more reliable software releases. It outlines various AWS services that can be used as part of a DevOps implementation, including CloudFormation, CodeDeploy, CloudWatch, Elastic Beanstalk, and OpsWorks. It provides examples of how these services can be used to provision infrastructure, deploy applications, and monitor systems as part of a DevOps workflow on AWS.
This document provides tips and best practices for using AWS Elastic Load Balancers (ELBs). It covers topics like load testing ELBs, using SSL with ELBs, CNAME records, balancing traffic both within and across availability zones, L4 load balancing support, internal ELBs, ELB logging, stickiness, blue/green deployments using ELBs, connection draining, using the ELB CLI for continuous integration/continuous delivery, auto scaling with ELB metrics, using CloudFront in front of ELBs, and some limitations around microservices support. The overall message is that ELBs are generally easy to use but have some limitations, so it's important to understand how to configure them properly
2017/06/21, AWS User Group Taiwan
https://rickhw.github.io/2017/06/21/AWS/Stategies-System-Monitor_and_CloudWatch/
The document discusses AWS services for continuous integration, delivery, and deployment based on AWS. It describes how CodeCommit can be used for source code management, CodePipeline for continuous delivery, and CodeDeploy for continuous deployment. It also discusses how Elastic Beanstalk can be used to deploy and manage applications on AWS.
This document discusses AWS services for automating infrastructure deployment and management across multiple environments in the cloud. It covers topics like infrastructure as code using templates, continuous integration/delivery, reuse across environments, testing with blue/green deployments, and integrating configuration management tools. CodeCommit is presented for source control, CodePipeline for release management, and CodeDeploy for automated deployments. Infrastructure is modeled after software development using templates, APIs, and desired state declarations.
Autoscaling Spark on AWS EC2 - 11th Spark London meetupRafal Kwasny
This document discusses autoscaling Spark clusters on AWS for efficiency and cost-effectiveness. It presents a typical AWS architecture with Spark running on EC2 and data stored in S3. It describes how autoscaling works to dynamically adjust the number of EC2 instances based on demand metrics to match resource usage. The spark-cloud tool is introduced to simplify managing Spark clusters on AWS with features like building AMIs, starting and shutting down clusters, and using spot instances for lower costs compared to on-demand pricing. Autoscaling helps remove the need to manually scale clusters up and down.
Introduction of OpsStack--Integrated Operation Platform developed by ChinaNetCloud
Operations Evolution Underway
Customer Challenges
OpsStack Killer Features
ChinaNetCloud talk on Operations-as-a-Service at AWS China Summit 2015 in Shanghai. Covers OaaS cases, technology, and AWS.
Given December 17th, 2015, in Shanghai, in Chinese by Yinan Gu.
This document provides guidance on troubleshooting issues with EC2 instances and Elastic Load Balancers (ELB) on AWS. It begins by recommending monitoring the AWS service health dashboard and CloudWatch metrics. Potential causes and resolutions are outlined for common problems with EC2 instance launching, health, networking, and EBS volumes. For ELBs, error messages, response metrics, health checks, and other potential problems are covered. The document concludes by listing information needed for support cases and additional resources.
This document discusses DevOps practices on AWS. It defines DevOps as a culture or practice to promote collaboration between development and operations teams. It then discusses why DevOps is needed to enable faster and more reliable software releases. It outlines various AWS services that can be used as part of a DevOps implementation, including CloudFormation, CodeDeploy, CloudWatch, Elastic Beanstalk, and OpsWorks. It provides examples of how these services can be used to provision infrastructure, deploy applications, and monitor systems as part of a DevOps workflow on AWS.
This document provides tips and best practices for using AWS Elastic Load Balancers (ELBs). It covers topics like load testing ELBs, using SSL with ELBs, CNAME records, balancing traffic both within and across availability zones, L4 load balancing support, internal ELBs, ELB logging, stickiness, blue/green deployments using ELBs, connection draining, using the ELB CLI for continuous integration/continuous delivery, auto scaling with ELB metrics, using CloudFront in front of ELBs, and some limitations around microservices support. The overall message is that ELBs are generally easy to use but have some limitations, so it's important to understand how to configure them properly
2017/06/21, AWS User Group Taiwan
https://rickhw.github.io/2017/06/21/AWS/Stategies-System-Monitor_and_CloudWatch/
The document discusses AWS services for continuous integration, delivery, and deployment based on AWS. It describes how CodeCommit can be used for source code management, CodePipeline for continuous delivery, and CodeDeploy for continuous deployment. It also discusses how Elastic Beanstalk can be used to deploy and manage applications on AWS.
This document discusses AWS services for automating infrastructure deployment and management across multiple environments in the cloud. It covers topics like infrastructure as code using templates, continuous integration/delivery, reuse across environments, testing with blue/green deployments, and integrating configuration management tools. CodeCommit is presented for source control, CodePipeline for release management, and CodeDeploy for automated deployments. Infrastructure is modeled after software development using templates, APIs, and desired state declarations.
8. Unauthenticated
API处理流程
Mobile apps AWS Lambda lambdaHandler
Register
Login
API Gateway
Authenticated
Mobile apps AWS Lambda lambdaHandler
ListPosts
GetPosts
API Gateway
GetComments
Invoke with
caller credentials
Authorized by IAM
22. The API definition
• POST
• 接收用户的username, password
• 加密密码并在DynamoDB中创建用户资料
• 生成基于JWT的密匙
• 返回用户和密匙
/register
• POST
• 接收用户的username, password
• 比对DynamoDB验证用户登录
• 成功后利用JWT加密并生成Token
• 返回登录成功后的JWT Token
/login
23. 自定义认证 - Lambda
Client
Lambda Auth
function
API Gateway
OAuth token
Policy is
evaluated
Policy is
cached
Endpoints on
Amazon EC2
Any other publicly
accessible endpoint
AWS Lambda
functions
403
AWS KMSJWT Provider
24. IAM Policy是所有AWS服务的权限工具
Mobile apps AWS Lambda lambdaHandler
API Gateway
Invoke with
caller credentials
Service calls are
authorized using
the IAM role
更多介绍
http://amzn.to/1YkxcjR
DynamoDB