![Submit a Question: http://tinyurl.com/gae-dd09
Datastore 3 / 3
• GQL:
Google Query Language
• If it’s not SQL, it must be good!
SELECT [* | __key__] FROM <kind>
[WHERE <condition> [AND <condition> ...]]
[ORDER BY <property> [ASC | DESC] [, <property> [ASC | DESC] ...]]
[LIMIT [<offset>,]<count>]
[OFFSET <offset>]
<condition> := <property> {< | <= | > | >= | = | != } <value>
<condition> := <property> IN <list>
<condition> := ANCESTOR IS <entity or key>](https://image.slidesharecdn.com/appenginedevdaysdc20091026-091030140054-phpapp01/85/App-Engine-Dev-Days-DC-20091026-15-320.jpg)
More Related Content
What's hot (19)
Similar to App Engine Dev Days DC 20091026 (20)
Recently uploaded (20)
App Engine Dev Days DC 20091026
- 1. Google App Engine Jonathan Blocksom Stack Overflow DevDays DC, 10/26/09
- 2. Submit a Question: http://tinyurl.com/gae-dd09 Attention Conservation Notice • Jonathan Blocksom [email protected] @jblocksom Software Engineer Google Public Sector Projects Team
- 3. Submit a Question: http://tinyurl.com/gae-dd09 We’re lonely Open positions at Google in DC
- 4. Submit a Question: http://tinyurl.com/gae-dd09 App Engine! • Launched April 2008 • Paid Use Feb 2009 • Java support May 2009 • Latest Release (1.2.7) October 2009 • Incoming Email • Previous Release (1.2.5) September 2009 • XMPP Support
- 5. Submit a Question: http://tinyurl.com/gae-dd09 Your Scalable Website? • Brad Fitzpatrick describes LiveJournal in ’07: net. LiveJournal Backend: 2007 Today (Roughly.) BIG-IP perlbal (httpd/proxy) Global Database bigip1 mod_perl bigip2 proxy1 master_a master_b web1 proxy2 web2 proxy3 Memcached web3 slave1 slave2 ... slave5 djabberd proxy4 mc1 web4 djabberd proxy5 ... mc2 User DB Cluster 1 djabberd webN mc3 uc1a uc1b mc4 User DB Cluster 2 ... uc2a uc2b gearmand Mogile Storage Nodes gearmand1 mcN User DB Cluster 3 sto1 sto2 gearmandN uc3a uc3b Mogile Trackers ... sto8 tracker1 tracker3 User DB Cluster N ucNa ucNb MogileFS Database “workers” gearwrkN Job Queues (xN) mog_a mog_b theschwkN jqNa jqNb slave1 slaveN http://danga.com/words/ 3
- 6. Submit a Question: http://tinyurl.com/gae-dd09 App Engine Scalable Website Request App Engine Response
- 7. Submit a Question: http://tinyurl.com/gae-dd09 What’s Scalable? • Google Moderator on App Engineused in March 2009 for White House Town Hall meeting
- 8. Submit a Question: http://tinyurl.com/gae-dd09 Scalable
- 9. Submit a Question: http://tinyurl.com/gae-dd09 Google Moderator team didn’t... • Provision a server • Install and configure an OS • Install and configure a web server • Install and configure a web framework • Install and configure a database • Install and configure a load balancer • Create user accounts • Crash in the middle of any of the above
- 10. Submit a Question: http://tinyurl.com/gae-dd09 App Engine Architecture
- 11. Submit a Question: http://tinyurl.com/gae-dd09 Static Content • Served by Google Static Content Servers • Very Efficient
- 12. Submit a Question: http://tinyurl.com/gae-dd09 Dynamic Content • Java xor Python • Sandboxed • No file writes • No sockets • No spawning threads/processes • Smart about caching • Fairly standard CGI Framework • Measured and Instrumented
- 13. Submit a Question: http://tinyurl.com/gae-dd09 Datastore 1 / 3 • Based on BigTable, transitioning to MegaStore • ACID Compliant • 2008 Talk on Implementation: http://sites.google.com/site/io/under-the- covers-of-the-google-app-engine-datastore
- 14. Submit a Question: http://tinyurl.com/gae-dd09 Datastore 2 / 3 • Typical types: Int, String, DateTime, GeoPt, PhoneNumber, PostalAddress, Email, Link, Blob...
- 15. Submit a Question: http://tinyurl.com/gae-dd09 Datastore 3 / 3 • GQL: Google Query Language • If it’s not SQL, it must be good! SELECT [* | __key__] FROM <kind> [WHERE <condition> [AND <condition> ...]] [ORDER BY <property> [ASC | DESC] [, <property> [ASC | DESC] ...]] [LIMIT [<offset>,]<count>] [OFFSET <offset>] <condition> := <property> {< | <= | > | >= | = | != } <value> <condition> := <property> IN <list> <condition> := ANCESTOR IS <entity or key>
- 16. Submit a Question: http://tinyurl.com/gae-dd09 Memcache • Arbitrary key/value cache • Similar API to memcached • get, set • get_multi, set_multi for speed • Namespace Support • 1MB limit per entry • Free Quota 8.6M/day, 48K/min (800/sec)
- 17. Submit a Question: http://tinyurl.com/gae-dd09 Users & Authentication • @gmail.com addresses • Apps for Domain • Admin Privileges
- 18. Submit a Question: http://tinyurl.com/gae-dd09 URL Fetch • Load URL • Asynchronous support • HTTP / HTTPS • Max 10 second response (default 5) • Max 1MB data
- 19. Submit a Question: http://tinyurl.com/gae-dd09 Even More... • Scheduled Tasks • Task Queue • Email • Image Manipulation cron jobs • XMPP email • ... XMPP
- 20. Submit a Question: http://tinyurl.com/gae-dd09 Web Hooks Pattern • Incoming data / operations to perform are URLs with POST data • Handled same as web request Scheduled Tasks Task Queue tasks Incoming Email XMPP Messages
- 21. Submit a Question: http://tinyurl.com/gae-dd09 Templates? • Treetop Wants to know
- 22. Submit a Question: http://tinyurl.com/gae-dd09 Templates! • Django Templates built in Variable substitution, function calls, loops, etc • Can add your own Python temlpate lib
- 23. Submit a Question: http://tinyurl.com/gae-dd09 Let’s Review!
- 24. Submit a Question: http://tinyurl.com/gae-dd09 Developing w/ App Engine • Local test server • Write your code, html, etc on local machine • Send code to App Engine • Monitor in App Engine Admin Console
- 25. Submit a Question: http://tinyurl.com/gae-dd09 Dashboard Demo • http://appengine.google.com/
- 26. Submit a Question: http://tinyurl.com/gae-dd09 What have we done? • Served static web pages • Served dynamic web pages • Stored data (cached & persistent) • Ran a cron Job • Authenticated users • We could have... • Scheduled a task, sent or received email, loaded a URL, modified an image, rocked out with XMPP
- 27. Submit a Question: http://tinyurl.com/gae-dd09 Quotas • 34 global quotas in Dashboard • Individual quotas: request size time amount of data • Exceptions thrown when hit • Free and Paid
- 28. Submit a Question: http://tinyurl.com/gae-dd09 Quota Rule of Thumb • Less than 1000 of everything • The Zen of Quotas: Meet the quotas and you will be scalable
- 29. Submit a Question: http://tinyurl.com/gae-dd09 All For Good • Volunteering Opportunity Search Engine • Started March 2009 in cooperation with White House, CNCS, Craigslist, Google... • Released June 2009 • Open source, released under Apache License http://code.google.com/p/allforgood/ • Front end at http://allforgood.org API powers search results on serve.gov
- 30. Submit a Question: http://tinyurl.com/gae-dd09 Tech Overview • Partners provide listings in XML or Google Spreadsheets • AARP, United Way, AmeriCorp, HandsOnNetwork, United Way • AFG indexes feeds, stores outside GAE • Initially Google Base • Now SOLR on private server • GAE front end • API for external apps, partners • Integration with Google Maps, FriendConnect, Facebook
- 31. Submit a Question: http://tinyurl.com/gae-dd09 Tech Overview User or API Search Queries Backend Volunteering Opportunities (Base or App Engine s SOLR) lt su Volunteering Opportunities Re memcache Listings ... Volunteering Opportunities Crawler Static Content Google Code SVN
- 32. Submit a Question: http://tinyurl.com/gae-dd09 Why A Different Back End? • No full text search in App Engine Datastore
- 33. Submit a Question: http://tinyurl.com/gae-dd09 On The Promiscuous Nature of GAE • GAE plays well with others • Do full text search with something else! • Serve large files with S3! • Serve videos with YouTube!
- 34. Submit a Question: http://tinyurl.com/gae-dd09 Challenges with AFG • Parsing questionable data • Integration w/ Social Networks • Google Base is an Odd Duck • I Was New And Had No Clue What Was Going • App Engine Outages
- 35. Submit a Question: http://tinyurl.com/gae-dd09 Q &A
- 36. Submit a Question: http://tinyurl.com/gae-dd09 More Information • App Engine Documentation • Google I/O 2008 & 2009 • 2008 for intro, 2009 for deep dives / Java • http://code.google.com/events/io/2009/ sessions.html#appengine “App Engine Nitty-Gritty” “Spark Plug to Drive Train” • Google Group • Samples http://code.google.com/p/google-app-engine-samples/