Building

Scalable Web Architectures

Aaron Bannert
[email protected] / [email protected]

QuickTime™ and a
http://www.codemass.com/~aaron/presentations/ TIFF (Uncompressed)
are needed to see this decomp
picture
apachecon2005/ac2005scalablewebarch.ppt
Goal

 How do we build a massive web system

out of commodity parts and open source?
Agenda

1. LAMP Overview

2. LAMP Features

3. Performance

4. Surviving your first Slashdotting

a. Growing above a single box

b. Avoiding Bottlenecks
LAMP Overview
Architecture
LAMP

Linux
Apache
MySQL
PHP (Perl?)
The Big Picture
External Caching Tier
External Caching Tier

 What is this?
 Squid

 Apache’s mod_proxy
 Commercial HTTP Accelerator
External Caching Tier

 What does it do?

 Caches outbound HTTP objects
 Images, CSS, XML, HTML, etc…

 Flushes Connections
 Useful for modem users, frees up web tier

 Denial of Service Defense

External Caching Tier

 Hardware Requirements
 Lots of Memory
 Fast Network
 Moderate to little CPU
 Moderate Disk Capacity
 Room for cache, logs, etc… (disks are cheap)
 One slow disk is OK

 Two Cheapies > One Expensive

External Caching Tier

 Other Questions
 What to cache?
 How much to cache?
 Where to cache (internal vs. external)?
Web Serving Tier
Web Serving Tier

 What is this?
 Apache

 thttpd

 Tux Web Server

 IIS

 Netscape
Web Serving Tier

 What does it do?

 HTTP, HTTPS
 Serves Static Content from disk
 Generates Dynamic Content
 CGI/PHP/Python/mod_perl/etc…
 Dispatches requests to the App Server Tier
 Tomcat, Weblogic, Websphere, JRun, etc…
Web Serving Tier

 Hardware Requirements
 Lots and lots of Memory
 Memory is main bottleneck in web serving
 Memory determines max number of users
 Fast Network
 CPU depends on usage
 Dynamic content needs CPU
 Static file serving requires very little CPU
 Cheap slow disk, enough to hold your content
Web Serving Tier

 Choices
 How much dynamic content?
 When to offload dynamic processing?
 When to offload database operations?
 When to add more web servers?
Application Server Tier
Application Server Tier
 What does it do?
 Dynamic Page Processing
 JSP
 Servlets

 Standalone mod_perl/PHP/Python engines

 Internal Services
 Eg. Search, Shopping Cart, Credit Card Processing
Application Server Tier
• How does it work?
1. Web Tier generates the request using
 HTTP (aka “REST”, sortof)
 RPC/Corba
 Java RMI
 XMLRPC/Soap
 (or something homebrewed)
2. App Server processes request and responds
Application Server Tier

 Caveats
 Decoupling of services is GOOD
 Manage Complexity using well-defined APIs
 Don’t decouple for scaling, change your algorithms!
 Remote Calling overhead can be expensive
 Marshaling of data
 Sockets, net latency, throughput constraints…
 XML, Soap, XMLRPC, yuck (don’t scale well)
 Better to use Java’s RMI, good old RPC or even Corba
Application Server Tier

 More Caveats
 Remote Calling introduces new failure scenarios

 Classic Distributed Problems

• How to detect remote failures?
 How long to wait until deciding it’s failed?
• How to react to remote failures?
 What do we do when all app servers have failed?
Application Server Tier

 Hardware Requirements
 Lots and Lots and Lots of Memory
 App Servers are very memory hungry
 Java was hungry to being with
 Consider going to 64bit for larger memory-space
 Disk depends on application, typically minimal needed
 FAST CPU required, and lots of them
 (This will be an expensive machine.)
Database Tier
Database Tier
 Available DB Products
 Free/Open Source DBs
 PostgreSQL  MySQL
 GNU DBM
 SQLite
 Ingres
 mSQL
 SQLite
 Berkeley DB

 Commercial
 Oracle
 MS SQL
 IBM DB2
 Sybase
 SleepyCat
Database Tier
 What does it do?
 Data Storage and Retrieval
 Data Aggregation and Computation

 Sorting

 Filtering

 ACID properties
 (Atomic, Consistent, Isolated, Durable)
Database Tier
 Choices
 How much logic to place inside the DB?
 Use Connection Pooling?

 Data Partitioning?
 Spreading a dataset across multiple logical database
“slices” in order to achieve better performance.
Database Tier
 Hardware Requirements
 Entirely dependent upon application.
 Likely to be your most expensive machine(s).

 Tons of Memory
 Spindles galore
 RAID is useful (in software or hardware)
 Reliability usually trumps Speed
• RAID levels 0, 5, 1+0, and 5+0 are useful
 CPU also important
 Dual power supplies
 Dual Network
Internal Cache Tier
Internal Cache Tier
 What is this?
 Object Cache
 What Applications?
 Memcache

 Local Lookup Tables

 BDB, GDBM, SQL-based
 Application-local
Caching (eg. LRU tables)
 Homebrew Caching (disk or memory)
Internal Cache Tier
 What does it do?
 Caches objects closer to the
Application or Web Tiers
 Tuned for your application

 Very Fast Access

 Scales Horizontally
Internal Cache Tier
 Hardware Requirements
 Lots of Memory
 Note that 32bit processes are typically limited to 2GB
of RAM
 Little
or no disk
 Moderate to low CPU

 Fast Network
Misc. Services (DNS, Mail, etc…)
Misc. Services (DNS, Mail, etc…)

 Why mention these?

 Every LAMP system has them
 Crucial but often overlooked
 Source of hidden problems
Misc. Services: DNS

 Important Points
 Always have an offsite NS slave
 Always have an onsite NS slave
 Minimize network latency
 Don’t use NAT, load balancers, etc…
Misc. Services: Time Synchronization

 Synchronize the clocks on your systems!

 Hints:
 Use NTPDATE at boot time to set clock
 Use NTPD to stay in synch
 Don’t ever change the clock on a running
system!
Misc. Services: Monitoring

 System Health Monitoring

 Nagios

 Big Brother
 Orcalator

 Ganglia

 Fault Notification
The Glue

•Routers
•Switches
•Firewalls
•Load Balancers
Routers and Switches
 Expensive
 Complex
 Crucial Piece of the System

 Hints
 Use GigE if you can
 Jumbo Frames are GOOD
 VLans to manage complexity
 LACP (802.3ad) for failover/redundancy
Load Balancers
 What services to balance?
 HTTP Caches and Servers, App Servers, DB
Slaves
 What NOT to balance?
 DNS
 LDAP
 NIS
 Memcache
 Spread
 Anything with it’s own built-in balancing
Message Busses
 What is out there?
 Spread
 JMS
 MQSeries
 Tibco Rendezvous

 What does it do?

 Various forms of distributed message delivery.
 Guaranteed Delivery, Broadcasting, etc…
 Useful for heterogeneous distributed systems
What about the OS?
Operating System Selection
Lots of OS choices
 Linux
 FreeBSD
 NetBSD
 OpenBSD
 OpenSolaris
 Commercial Unix
What’s Important?
 Maintainability
 Upgrade Path
 Security Updates
 Bug Fixes

 Usability
 Do your engineers like it?
 Cost
 Hardware Requirements
 (you don’t need a commercial Unix anymore)
Features to look for

 Multi-processor Support
 64bit Capable
 Mature Thread Support
 Vibrant User Community
 Support for your devices
 The Age of LAMP
What does LAMP provide?
Scalability

 Grows in small steps

 Stays up when it counts

 Can grow with your traffic

 Room for the future

Reliability

 High Quality of Service

 Minimal Downtime
 Stability
 Redundancy
 Resilience
Low Cost

 Little or no software licensing costs

 Minimal hardware requirements

 Abundance of talent

 Reduced maintenance costs

Flexible

 Modular Components
 Public APIs
 Open Architecture
 Vendor Neutral
 Many options at all levels
Extendable
 Free/Open Source Licensing
 Right to Use
 Right to Inspect

 Right to Improve

 Plugins
 Some Free
 Some Commercial

 Can always customize

Free as in Beer?

Price
Speed
Quality

Pick any two.

Performance
What is Performance?

 For LAMP?

 Improving the User Experience

Architecture affects user experience?

 It affects it in two ways

 Speed
Fast Page Loads (Latency)
 Availability
Uptime
Problem: Concurrency

 Concurrency causes

slowdowns

 Latency suffers

 Pages load more slowly

Solution: Design for Concurrency

 Build parallel systems

 Eliminate bottlenecks

 Aim for horizontal scalability

Now for some real-world examples…

Surviving your first
Slashdotting
Strategies for Scalability
What is a “Slashdotting”?
 Massive traffic spike (1000x normal)
 High bandwidth needed
 VERY high concurrency needed
 Site inaccessible to some users
 If your system crashes, nobody gets in
Approach

1. Keep the system up, no crashing

 Some users are better than none

2. Let as many users in as possible

Strategies

1. Load Balancers (of course)

2. Static File Servers

3. External Caching
Load Balancers

 Hardware vs. Software

 Software is complex to set up, but cheaper
 Hardware is expensive, but dedicated
 IMHO: Use SW at first, graduate to HW
Static File Servers: Zero-copy

 Separate Static from Dynamic

 Scale them independently
 Later, dedicate static content serers
 Modern web servers are very good at serving static
content such as
• HTML
• CSS
• Images
• Zip/GZ/Tar files
External Caching
 Reduces internal load
 Scales horizontally
 Obeys HTTP-defined rules for caching
 Your app defines the caching headers
 Behaves no differently than other proxy servers
or your browser, only it’s dedicated

 Hint: Use mod_expires to override

Outgrowing your First Server
Strategies for Growth
Design for Horizontal Scalability

 Manage Complexity

 Design Stateless Systems (hard)

 Identify Bottlenecks (hard)

 Predict Growth

 Commodity Parts
Manage Complexity
 Decouple internal services
 Servicesscale independently
 Independent maintenance

 Well-defined APIs
 Facilitates
service decoupling
 Scales your engineering efforts
What is a Stateless System?

 Each connection hits a new server

 Server remembers nothing

 Benefits?
 Allows Better Caching
 Scales Horizontally
Designing Stateless Systems
 Decouple session state from web server
 Store session state in a DB
 Careful: may just move bottleneck to DB tier
 Use a distributed internal cache
 Memcached
 Reduces pressure on session database
Example: Scaling your User DB
 Assume you have a user-centric system
 Eg. User identity info, subscriptions, etc…

1. Group data by user

2. Distribute users across multiple DBs
3. Write a directory to track user->DB location
4. Cache user data to reduce DB round trips

Disadvantage: difficult to compare two users

Identify Bottlenecks
 Monitor your system performance
 Use tools for this, there are many
 Store and plot historical data
 Used to identify abnormalities
 Check out rrdtool
 Use system tools to troubleshoot
 vmstat, iostat, sar, top, strace, etc…
Predict Growth
 Use performance metrics
 Hits/sec

 Concurrent connections
 System load

 Total number of users

 Database table rows

 Database index size (on disk/memory)

…
Machine-sized Solutions
 Design for last year’s hardware
 (it’s cheaper)
 Leaves room for your software to grow
 Hardware will get faster
 And your systems will get busier
 Use Commodity Parts

 Standardize Hardware

 Use Commodity Software

(Open Source!)

 Avoid Fads
THE END
Thank You