Drupal Backend Performance and Scalability

1. Drupal Backend Performance and ScalabilityAshok and Christefano

2. August 7, 2010About me (BTMash)Systems programmer.

3. Work at California Institute of the Arts.

4. Working with Drupal since 2006 (4.6.x)

5. Primarily help with patches and upgrades for contrib modules on d.o.

6. Abuse

7. Favorites

8. Flag Content

9. Fivestar

10. Nice Menus

11. Simplenews

12. Userpoints

13. Userpoints_badges

14. User_badge

15. And more!

16. Strong interest in server optimizationAbout me (Christefano)LADrupal Manager

17. Camp Organizer

18. Drupal Core Contributor

19. Person of the Year

20. D.O and G.D.O webmaster teams.ResourcesKhalid Baheyeldin

21. http://2bits.com

22. Drupal.org infrastructure group

23. Peter Zaitsev

24. http://www.mysqlperformanceblog.com/about/GoalsDefine your objectives and goals first

25. Do you want faster response to the end user per page?

26. Do you want to handle more page views?

27. Do you want to minimize downtime?

28. Each is different, though they may be related

29. Sometimes, there are some ‘low hanging fruit’ that are easy to see, that provide noticeable improvement with little effort.

30. Gets hard to achieve more performance (more efforts, low return)

31. More infrastructure (splitting servers, multiple web/db servers, etc)

32. Patching Drupal

33. Revisions to Architecture (CCK, Views)DiagnosisProper diagnosis is essential before proposing and implementing a solution or you are running blind.

34. Must be based on proper data.

35. Analysis of data collected.

36. Can lead towards a few possible paths of optimization.Points of optimizationIntroductionTools to measure and diagnose issuesSpeed Optimizations

37. On to the Server…Much more complex

38. Measurement and monitoring tools

39. Hardware

40. The LAMP Stack

41. Linux, Apache, MySQL, PHP

42. Drupal

43. Database queries

44. Modules

45. CachingValidationAvoid the ‘wild goose chase’

46. Validate the results on a test server

47. Replicate the site on a development server (or local machine with the current data)

48. Backup and Migrate module help with that

49. Recreate the site

50. Gather a time difference ratio between the test and production server (production server is likely to be much faster)

51. Give a rough idea on any improvements made.

52. Measure again and see if relative times remain the same.The LAMP StackCurrently most commonly used stack for hosting Drupal and similar applications

53. Linux

54. Apache

55. MySQL

56. PHP

57. Most of this part of the presentation applies to *BSD as well.HardwarePhysical server matters

58. Dedicated

59. VPS

60. In the Cloud (Yes, it does work!)

61. Multiple cores are the norm

62. 8 > 4 > 2 > 1

63. Lots of RAM (caching the file system and database as much as possible)

64. Multiple disks

65. Not really applicable to shared hostingMultiple ServersOne database server, multiple web servers

66. Can use DNS round robin for load balancing

67. Or proper load balancers

68. Can use a reverse proxy (much like Drupal.org)

69. Do it only if you have the budget

70. Complexity is expensive (running cost, maintenance time)

71. Tuning a system can avoid (or delay) a splitTesting toolsab/ab2 (Apache Benchmark)

72. Ab –c 50 –n 10000

73. Do 50 concurrent requests for up to 10000 requests

74. Average response time per second

75. How many requests handled per second

76. Can test with authenticated sessions

77. Siege

78. http://www.joedog.org/index/siege-home

79. Similar in functionality to ab

80. Can test POST functionality

81. Cannot natively use on windows

82. JMeter

83. http://jakarta.apache.org/jmeter/

84. Desktop application written in java (can natively use on windows)Hardware Monitoring toolsTop

85. Classic Unix/Linux program

86. Real time monitoring (What is the system doing right now??)

87. Load average

88. CPU utilization

89. Memory usage

90. List of processes, sorted, with CPU memory

91. Can change order of sorting, time interval, grouping, along with many other things.

92. Htop

93. http://htop.sourceforge.net/

94. Similar to top but for multiple cores

95. Faster

96. Colours, stylesHardware Monitoring toolsatop

97. http://www.atcomputing.nl/Tools/atop/

98. Different format and information

99. Shows network statistics

100. Runs a collection daemon in the background

101. Vmstat

102. Report memory statistics

103. Can display in increments (vmstat 5 means it will show snapshots every 5 seconds)

104. netstat

105. Shows active network connections

106. netstat –anp

107. netstat –anp | grep ESTGraphical Monitoring toolsCacti

108. http://www.cacti.net/

109. Available as a package on Ubuntu, Debian (any other *nix / *bsd?)

110. Easy to understand graphs

111. Displays history over day, week, month, year

112. Graphs available to display statistics for CPU, memory, network, Apache, MySQL

113. Many others written by others available online (can find one for memcached, as an example)

114. Munin

115. http://munin.projects.linpro.no/

116. Very similar to Cacti

117. Can also create your own monitoring scripts

118. Nagios

119. Haven’t used it yet

120. Supposed to be very powerful (alerts by email, sms, etc)

121. Drupal module which integrates with toolLinux / BSDUse a proven stable distribution (Debian Stable, Ubuntu LTS, RHEL, CentOS)

122. Use recent versions

123. Use whatever your distribution your staff has expertise with (most important!)

124. Try to avoid bloat

125. Don’t install PostGreSQL if you are only using MySQL, no desktop server, java, etc

126. Balance compiling own programs versus using packages from distro

127. Compiling provides full control (and may let you get more out of) on the program.

128. Can be a pain to upgrade.

129. Using packages only let you use whatever the latest version may be (APC was not available for RHEL until middle of last year)

130. Upgrades are relatively painless.ApacheMost popular, supported, and feature rich

131. Fairly stable

132. Can also be enabled with too many unnecessary modules

133. Disable unnecessary modules (mod_proxy, cgi_module, etc may not be necessary on server if not hosting sites which take advantage of it)

134. Smaller process = more users can access site

135. ‘apachectl –M’ displays all modules currently enabled on site.

136. Monitoring Tool – apachetop

137. http://www.webta.org/projects/apachetop/

138. Reads and analyses Apache’s access log

139. Show all / recent hits

140. Requests / s, KB / s, KB / request

141. Good to detect crawlers

142. Run using ‘apachetop –f /path/to/site/access.log’Apache OptimizationsMaxClients (prevent swapping / thrashing)

143. Too low – cannot server enough clients

144. Too high – you run out of memory, and you start swapping (server dies and cannot server ANY clients)

145. MaxRequestsPerChild

146. Tune it to terminate the process faster and free up memory.

147. KeepAlive

148. Keep it on.

149. mod_gzip/deflate

150. Compress css, js, etcAlternativeslighttpd

151. http://www.lighttpd.net/

152. Popular with Ruby on Rails

153. Lighter than Apache (can go as low as 1MB)

154. Known to have leaks

155. Nginx

156. http://nginx.net/

157. More stable than lighty

158. Easier to setup than lighty

159. Drupalcampla is running on it…

160. Both run PHP as only as FastCGI

161. Separate processesAlternatives (cont’d)Varnish

162. HTTP accelerator

163. Set it up as a reverse proxy to send the call to apache if it cannot (or should not) serve something itself

164. Serve anonymous page requests and static files

165. Not particularly difficult to set up.

166. Does require some level of tuning.

167. Used on Drupal.org

168. Will be used on ladrupal.org

169. Squid

170. Similar to Varnish

171. Precursor to Varnish (so go with the above)MySQLMost popular database for Drupal

172. Not necessarily the best database from technology standpoint (default ending lacks ACID, transactions, concurrency) but still does a good job.

173. Easy to set up

174. Various pluggable engines (InnoDB, Archive, etc)MySQL Monitoringmtop / mytop

175. Like top, but for MySQL

176. Real time monitoring (no history)

177. Shows slow queries and locks

178. If you have neither –

179. SHOW FULL PROCESS LIST – mysqladmin processlist

180. Mysqlreport

181. http://hackmysql.com/mysqlreport

182. Reports on server – no recommendations (documentation on site explains a lot, however)

183. MySQL DB Tuner

184. http://wiki.mysqltuner.com/MySQLTuner

185. Another one at http://www.day32.com/MySQL/

186. Provides fairly decent recommendations (and warnings) on MySQL settingsMySQL MonitoringSlow Query Log

187. Can be enabled in your my.cnf file

188. Lists queries more than N seconds

189. Can also list queries which do not have indexes (if you choose)

190. Very useful to identify potential bottlenecks

191. Helped me identify that the COUNT(*) query used through the search module was cause for site search crawl

192. Removed count with a base query which cut the search query and render to less than half the time

193. Best way to interpret all of it

194. mysql_slow_log_parser_script(http://www.mysqlperformanceblog.com/files/utils/mysql_slow_log_parser)MySQL EnginesMyISAM

195. Fast reads

196. Less overhead

197. Poor concurrency (table level locking)

198. InnoDB

199. Transactional

200. Slower in some cases (SELECT COUNT(…))

201. Lots of settings to analyze and change

202. Better concurrency (good for tables that are constantly hit, such as sessions, watchdog, etc)

203. New Engines in development

204. Falcon (promising ACID)

205. SolidDB

206. Concerns over the many new variants of MySQL

207. MariaDB, Percona (forked InnoDB), Google Patchsets, Mysql 5.5MySQL TuningQuery Cache, Key Buffer

208. VERY Important!

209. Also important

210. Table Cache (caches your tables)

211. sort_buffer_size, read_buffer_size

212. tmp_table_size (required to be the same or greater than query cache)

213. InnoDB

214. Innodb_buffer_pool_size (most important parameter)

215. Set this value to be 50 – 80% of total memory you wish to allocate towards MySQL

216. Loads of settings!

217. http://www.mysqlperformanceblog.com/ is a great resource to learn lots about tuning MySQLMySQL ReplicationUsed on Drupal.org

218. INSERT/DELETE/UPDATE queries go to master

219. SELECT queries go to slave(s)

220. Provide noticeable improvement

221. Usage on http://www.zimmertwins.com dramatically sped up the site

222. D.O has also seen noticeable improvements

223. Patch for 6.x seen at http://drupal.org/node/147160

224. Beware of complexity (connection between master/slave goes wrong, can bring down site)

225. Did extensive tuning of Zimmer Twins DB server

226. Noticeable improvement despite lack of querying slave server

227. Removed need for slave server.PHPUse a recent, stable release

228. Drupal 7 will require 5.2.x, as do a few current 6.x contributed modules

229. Install an Op-code cacher / accelerator

230. eAccelerator

231. APC

232. Xcache

233. Zend optimizer (commercial)

234. Very useful in bringing down memory usage for a siteOp-code CachingBenefits

235. Lowered memory usage

236. Significant decrease in CPU utilization

237. Can be biggest impact on a site

238. Usage on http://www.zimmertwins.com lowered memory usage per process from 20 MB down to less than 4 MB

239. Usage on http://calarts.edu lowered memory usage per process down from 35 MB down to 7 MB

240. Cached page usage was even lower (2 MB or less)

241. 2bits.com has written comparisons between the 3 cachers at http://tinyurl.com/5uo874

242. Drawbacks

243. May crash

244. May require restarts after updating code

245. Using the capistrano script by the TxT folk (who use xcache) may be useful.Op-code CachingOp-code caching will not work in all circumstances

246. Network connections

247. Sorting arrays

248. Database queries

249. All of the above

250. Some culprit modules

251. Tagadelic

252. Node access modules

253. Admin menu

254. Forum

255. TrackerRunning PHPmod_php

256. Standard php module used by Apache

257. No state retained between requests

258. Few issues

259. Well tested and supported

260. May not need to use anything else

261. Can be as low as 10 – 12 MB

262. Can easily be over 100 MB (depending on drupal modules installed, apache modules enabled on server, etc)

263. mod_cgi

264. Old way of running php

265. Don’t use it now (inefficient, each request forks process)

266. Fast CGI

267. Much faster than cgi

268. Can be used with nginx, lighttpd, apache

269. Apache + fcgi tested to be more stable with lower memory usage than mod_phpDrupalMainly database intensive

270. Can be a CPU hog

271. Can be memory intensive

272. Some modules are known to be slow

273. Site may not be affected by bottleneck.

274. Maintenance tips

275. Disable unnecessary modules

276. Make sure cron runs regularlyDrupal ToolsDevel

277. http://drupal.org/project/devel

278. Total page execution

279. Query execution time

280. Query log

281. Memory utilization

282. Can be combined with stress testing

283. Log pages which use most queries

284. Log pages which take most time to generate

285. Trace

286. http://drupal.org/project/trace

287. Use for debugging

288. Traces output, hook invocations, warnings

289. Can filter by query typeModule calls over networkDoes your module do interesting things over the network?

290. Email users (og)

291. Call web widgets / APIs (digg, youtube, flickr)

292. Cache data as much as possible (from web widgets/apis)

293. Use helper modules to aid with reducing bog

294. job_queue module

295. Queue Mail module

296. SMTP mail module

297. WeatherDrupal page cachingPage Caching

298. Primarily for anonymous users

299. Doesn’t necessarily affect authenticated users

300. May expire too soon

301. Set minimum cache expiry time

302. Aggressive cache is not compatible with all modules (statistics from core) but can provide better performance.

303. Other types of caching are not necessarily turned off

304. Filter

305. Menu

306. Blocks (6.x)

307. Variables

308. FormsUseful Drupal caching modulesBoost

309. http://drupal.org/project/boost

310. Creates HTML for pages and stores it in files

311. Requires changes to .htaccess files and symlinks (admin section of module now shows what to write into htaccess file)

312. Usable on shared hosts as well as VPS and dedicated servers

313. Enhances ability to handle traffic spikes (from anonymous users)

314. Can also use module to display site while in maintenance mode (only uncached pages and authenticated users will get maintenance mode page)

315. Development Seed has case study on using boost for Anti Poverty event site that notes performance gain by nearly 1000 times (http://tinyurl.com/l7ktfy)

316. Cache Router

317. FS Fastpath

318. Memcached

319. Authenticated User CachePluggable cachingUse $conf variable in settings.php

320. $conf[‘cache_inc’] = ‘./path/to/caching_mechanism.inc’;

321. Allows you to have a custom caching module

322. Can even use this to completely disable caching (DO NOT USE ON A PRODUCTION SITE)MemcachedDistributed object caching in memory

323. Written by danga for livejournal

324. Lives in memory

325. Can span multiple servers (

326. Drupal install uses pluggable caching mechanism

327. Should be seamless for Drupal 6.x

328. Drupal 5.x requires a patch + schema changes

329. Can lower db queries in ½

330. Has own requirements (apache needs to restarted, more instances = more complexity)Changing the search mechanismDrupal Core Search is good…?

331. But can quickly get bad with more and more content

332. DB Tuning can only go so far

333. With nearly 1M nodes, search on zimmertwins.ca would take over 30 seconds.

334. Try another engine!

335. ApacheSOLR

336. Very fast

337. Pain to configure

338. Requires server on which Java can be installed

339. Available as a drupal module

340. Acquia offers service to use their setup (wholly takes load off your server)

341. LuceneAPI

342. Not as fast as ApacheSolr, but easy to install and can work with that before moving on to Solr

343. Google CSE module might also be a good fitOther optionsUsing an optimized distribution

344. Pressflow

345. http://fourkitchens.com/pressflow-makes-drupal-scale

346. Supports database replication

347. Only supports MySQL

348. Cleanly supports reverse proxies such as Varnish

349. Optimized for PHP5

350. Mercury

351. Build based on Pressflow

352. Is an Amazon Machine Image to be used as instance of Amazon EC2

353. Install image contains installation of Solr, Varnish, Memcached

354. Tuned Apache settings

355. Performance is very promising.Other options (cont’d)Patching Drupal

356. Other name for it is ‘hack core’

357. Not recommended (unless you know what you are doing)

358. Sometimes it is necessary

359. Hack core from a safe distance if possible

360. Copy core module into sites/site_name_directory/modules and apply changes

361. Use pluggable caching to have your own version of caching

362. Create schema changes in a separate module (so you can remove them cleanly in the future)

363. Create own module and override original mechanism if process is called through form builder (change the submission or validation to point to yours)

364. Have an easy way to track changes you have madeOther options (cont’d)Case 1:

365. User login on zimmertwins.ca was painfully slow (5+ seconds per user)

366. Gist of problem: DB not using index on username due to lower()

367. Original issue at http://drupal.org/node/83738 (now ongoing in a new issue)

368. ‘Bug’ has been around since 2006

369. Solution: Modified patch on d.o for site

370. Copied user module into sites/all/modules to use override code

371. user login time now less than 0.1 seconds

372. As a note, d.o has also been patched (login has required matching case for past few months)

373. Case 2:

374. Comments do not have index on user id: problem with abuse module

375. Solution: Created index on user id as an update

376. Added update to module that requires this schema change (my own module)

377. Loading for comment by user no longer an issue.Module DevelopersTake advantage of caching

378. Use memory wisely

379. Unset the variable if you do not have a need for it down the line)

380. Save variable to memory for future use so processing is not done multiple times

381. Take Advantage of AHAH functionality

382. Fewer queries

383. Not reloading the page

384. Saving bandwidth

385. Learn to use jQuery (falls in line with above)(Possibly) Related Camp PresentationsDrupal Performance: Flash Storage as a Game Changer

386. Christoph Weber

387. Using Flash Storage to power your database for one.

388. Install Drupal 7 with Ubuntu 9.10 on Amazon EC2

389. HarnishGoradia (thatlaguy)

390. Building on SOLR Search

391. John FialaThank you!Questions? Answers? Tips from you all? Discuss!

Drupal Backend Performance and Scalability

Recommended

More Related Content

What's hot (20)

Similar to Drupal Backend Performance and Scalability (20)

More from Ashok Modi (7)

Recently uploaded (20)

Drupal Backend Performance and Scalability

Editor's Notes