© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.

SAS301

Multi-tenant meets ML: Building

ML-based SaaS environments
Michael Beardsley Anthony McClure
Principal Solutions Architect Sr. Solutions Architect
AWS SaaS Factory AWS SaaS Factory
AWS AWS

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agenda

SaaS and machine learning

Security concerns

Cost optimization

Data residency

Sample architectures

Open Q&A

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Software-as-a-Service (SaaS) is a
business and software delivery model
that enables organizations to offer
their solution in a low-friction,
service-centric approach.
- AWS SaaS Factory

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
SaaS value proposition

Benefits for customers Benefits for software providers

Simplified IT Time to value/ Agility and rapid Cost

management productivity innovation optimization

Lower switching Pay for what Operational Recurring

costs you use efficiencies revenue

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The use and development of computer
systems that are able to learn and adapt
without following explicit instructions,
by using algorithms and statistical models
to analyze and draw inferences from
patterns in data.
- Definition of machine learning, Oxford English Dictionary

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Machine learning

Model creation and ML ops Inference

• Data collection • Model access
• Data preparation • Results generation and storage
• Feature engineering
• Model training
• Model evaluation
• Model optimization
• Model deployment and management

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Machine learning and SaaS considerations

Security Cost optimization Data residency

Model creation and ML ops

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Security

Model creation and ML ops Inference

• Tenant isolation • Tenant isolation
• Access to training data • Deployed model access
• Model creation pipeline(s)
• Deployment

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Cost and other optimizations

Model creation and ML ops Inference

• The noisy neighbor problem • Scaling
• Pool vs. silo vs. bridge • Performance
• Training compute resources • Noisy neighbor
• Cost allocation
• Model deployment and updates
• Auditing and tracking
• AWS Marketplace

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Data residency and concerns

Model creation and ML ops Inference

• Where does the training data live? • Isolation of model inference results
• Cross account considerations? • Inputs
• Data quality • Parameters

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Sample architectures

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Silo, pool, and bridge inference

AWS Cloud

Amazon API Gateway AWS Lambda Amazon SageMaker Model

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Multi-tenant silo inference

Tenant1InferencePolicy Tenant1LambdaExecutionRole

Amazon AWS Lambda Single-model endpoint tenant1

tenant1 API Gateway Name: “tenant1” model

Tenant2InferencePolicy Tenant2LambdaExecutionRole

Amazon AWS Lambda Single-model endpoint tenant2

tenant2
API Gateway Name: “tenant2” model
© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Multi-tenant pool inference

JWT
TenantID: ”tenant1”
tenant1 model
tenant1

API Gateway AWS Lambda Multi-model

endpoint

JWT
TenantID: ”tenant2”

tenant2 tenant2 model

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Multi-tenant pool inference

JWT
TenantID: ”tenant1”
tenant1 model
tenant1
TenantID

Temporary
API Gateway security Multi-model
InvokeEndpointRole credential endpoint

JWT
TenantID: ”tenant2” AWS Lambda

tenant2 tenant2 model

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
InvokeEndpointPolicy example
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": "sagemaker:InvokeEndpoint",
"Resource": "*",
"Condition": {
"StringEquals": {
"sagemaker:TargetModel":"${aws:PrincipalTag/TenantID}.[Link]”
}
}
}
]
}
© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Multi-tenant model creation pipeline
4
Amazon SageMaker Pipeline

1 Pipeline 5
3 parameters Tenant 1 pipeline execution
tenant 1

Tenant 1
tenant1 training data
2 Rule SageMaker SageMaker SageMaker Tenant 1 6
tenant 1 data processing model training Model evaluation model artifacts
Amazon S3
Amazon S3
bucket
bucket

Amazon Pipeline
EventBridge Tenant 2 pipeline execution SageMaker
parameters
tenant 2 model registry

SageMaker SageMaker SageMaker Tenant 2

Tenant 2 Rule
tenant2 data processing model training Model evaluation model artifacts
training data tenant 2
Amazon S3 Amazon S3
bucket bucket

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Open Q&A

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
More SaaS sessions
Breakout sessions
▪ SAS305 – SaaS architecture patterns: From concept to implementation
▪ SAS405 – SaaS microservices deep dive: Simplifying multi-tenant development
▪ SAS306 – SaaS migration: Inside a real-world multi-tenant transformation
▪ SAS302 – Supporting extensibility in SaaS environments
▪ PEX310 – Optimizing your multi-tenant SaaS architecture

Workshops
▪ SAS403 - SaaS microservices deep dive: Multi-tenancy meets microservices
▪ SAS402 – Serverless meets SaaS: Inside a real-world serverless SaaS solution
▪ SAS401 – Amazon EKS SaaS: Building a working multi-tenant environment

Business session
▪ PEX209 – Building your SaaS journey on AWS

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
More SaaS sessions
Chalk talks
▪ SAS307 – DevOps and SaaS: Applying automation in multi-tenant environments
▪ SAS303 – SaaS anywhere: Building SaaS solutions that runs in hybrid models
▪ SAS301 – Multi-tenant meets ML: Building ML-based SaaS environments
▪ SAS304 – Solving the SaaS compliance puzzle
▪ PEX313 – The SaaS control plane: The heart of SaaS growth
▪ ARC403 – Amazon EKS SaaS deep dive: Inside a multi-tenant EKS solution
▪ ARC323 – Designing a multi-tenant SaaS tiering and throttling strategy
▪ SVS315 – Building multi-tenant applications with AWS Lambda and AWS Fargate

Builder session
▪ ARC327 – How to optimize cost in your multi-tenant architecture

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Additional resources

1 2 3

Subscribe to Explore the Discover resources

AWS SaaS insights SaaS on AWS hub for builders
Get monthly emails with bite- Check out the SaaS on AWS page Access our curated list of SaaS
sizes advice and the latest for more resources and insights reference solutions, demos, tech
updates events, and more

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Thank you!
Michael Beardsley Anthony McClure
mibeard@[Link] antmcl@[Link]

Please complete the session

survey in the mobile app

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.