Highway to heaven - XConf Manchester 2015
Download as PPTX, PDF1 like1,288 views
The document discusses the transition from monolithic to microservices architecture at Autoscout24, focusing on organizational structure, cultural shifts, and technical principles essential for successful implementation. Key strategies include building autonomous teams, emphasizing data-driven decision-making, and ensuring operational responsibility remains with the development teams. The document also highlights challenges faced during the transformation and the importance of continuous delivery, infrastructure as code, and prioritizing security and data privacy.
![Attributions
Blue sky, white-gray clouds by nature protector Natubico, www.vivism.info [CC BY-SA 3.0]
http://commons.wikimedia.org/wiki/File%3ABlue_sky%2C_white-gray_clouds.JPG
A Danish Perspective by NASA [Public domain] http://commons.wikimedia.org/wiki/File%3AA_Danish_Perspective.jpg
http://commons.wikimedia.org/wiki/File%3ANASAComputerRoom7090.NARA.jpg
GREG
EINRAD
Amazon16 by Neil Palmer/CIAT [CC BY-SA 2.0] https://www.flickr.com/photos/ciat/5641594952
BERGSTEIGER
Barber in Cameroon by James Emery from Douglasville, United States (Daddy Joe_1355) [CC BY 2.0]
http://commons.wikimedia.org/wiki/File%3ABarber_in_Cameroon.jpg
Wide objectives by Kivela (Own work) [Public domain]
href="http://commons.wikimedia.org/wiki/File%3AWide_objectives.jpg
Transformer Fire Barrier by GerryS1 (Own work) [CC BY-SA 3.0 or GFDL]
http://commons.wikimedia.org/wiki/File%3ATransformer_Fire_Barrier.jpg](https://image.slidesharecdn.com/wruzti6saop9eywnzzsg-signature-dd1b38b5f14ba02a6b4dc613652c4ebf19a2bb30f06cfc3d131fc5835d909d68-poli-150710092113-lva1-app6892/85/Highway-to-heaven-XConf-Manchester-2015-27-320.jpg)
![Attributions (cont)
Alonso Renault Pitstop Chinese GP 2008 by Bert van Dijk (Pitstop F1 ING Renault) [CC BY-SA 2.0]
http://commons.wikimedia.org/wiki/File%3AAlonso_Renault_Pitstop_Chinese_GP_2008.jpg
Principle of Panchasheel by Prakash Adhikary (Own work) [CC BY 3.0]
http://commons.wikimedia.org/wiki/File%3APrinciple_of_Panchasheel.JPG
Traffic Jam by Doo Ho Kim [CC BY-SA 2.0] https://www.flickr.com/photos/titicat/3049591547
Pellets by The original uploader was Richard Mayer at German Wikipedia [GFDL or CC-BY-SA-3.0]
http://commons.wikimedia.org/wiki/File%3APellets.jpg
Pipes and Valves by Uwe Hermann [CC BY-SA 2.0] https://www.flickr.com/photos/73628542@N00/6272975359
Size variation in Coccinella undecimpunctata (2127991716) by Gilles San Martin from Namur, Belgium [CC BY-SA 2.0]
http://commons.wikimedia.org/wiki/File%3ASize_variation_in_Coccinella_undecimpunctata_(2127991716).jpg
Mille crêpe by Laitr Keiows (Own work) [CC BY-SA 3.0 or GFDL]
http://commons.wikimedia.org/wiki/File%3AMille_cr%C3%AApe.jpg
Country Energy power line replacement 01 by Bidgee (Own work) [CC BY-SA 3.0]
http://commons.wikimedia.org/wiki/File%3ACountry_Energy_power_line_replacement_01.jpg](https://image.slidesharecdn.com/wruzti6saop9eywnzzsg-signature-dd1b38b5f14ba02a6b4dc613652c4ebf19a2bb30f06cfc3d131fc5835d909d68-poli-150710092113-lva1-app6892/85/Highway-to-heaven-XConf-Manchester-2015-28-320.jpg)
![Attributions (cont)
Sharing Sucks (4536747557) by eyeliam from Portland, United States [CC BY 2.0]
http://commons.wikimedia.org/wiki/File%3ASharing_Sucks_(4536747557).jpg
7Line 9184 (8263568241) by Metropolitan Transportation Authority of the State of New York (7Line_9184 Uploaded by
tm) [CC BY 2.0] http://commons.wikimedia.org/wiki/File%3A7Line_9184_(8263568241).jpg
England rugby team 1905 by Russell & Sons (The Graphic) [Public domain or Public domain]
http://commons.wikimedia.org/wiki/File%3AEngland_rugby_team_1905.jpg
Wandergeselle by Sigismund von Dobschütz [CC BY-SA 3.0]
http://commons.wikimedia.org/wiki/File%3AWandergeselle_02.JPG
Faber-Rechenschieber 5304 by User:Karl Gruber (Own work) [CC BY-SA 4.0]
http://commons.wikimedia.org/wiki/File%3AFaber-Rechenschieber_5304.JPG
Wheel clamps Texas by Richard Anderson from Denton, United States (Boots.) [CC BY-SA 2.0]
http://commons.wikimedia.org/wiki/File%3AWheel_clamps_Texas.jpg
GuadalupeNOLA15Oct07Thanks by Infrogmation of New Orleans (Photo by Infrogmation) [GFDL or CC BY-SA 3.0]
http://commons.wikimedia.org/wiki/File%3AGuadalupeNOLA15Oct07Thanks.jpg
AtariBasic by Calin99 (Own work) [GPL] http://commons.wikimedia.org/wiki/File%3AAtariBasic.png
Spare wheel by Brian Snelson [CC BY 2.0] https://commons.wikimedia.org/wiki/File:Spare_wheel_-_Flickr_-_exfordy.jpg](https://image.slidesharecdn.com/wruzti6saop9eywnzzsg-signature-dd1b38b5f14ba02a6b4dc613652c4ebf19a2bb30f06cfc3d131fc5835d909d68-poli-150710092113-lva1-app6892/85/Highway-to-heaven-XConf-Manchester-2015-29-320.jpg)
Highway to heaven - XConf Manchester 2015
- 1. Highway to heaven Building microservices in the cloud XConf Manchester, July 2015
- 2. Agenda Meet AutoScout24 Shifting gears How we build our services How we organize ourselves
- 4. New CEO CC
- 5. Do you attract talent? CC
- 6. CC
- 8. Monolith to Microservices Data center to AWS .NET / Windows to JVM / Linux Devs + Ops to DevOps culture Involve product people Five challenges WW
- 10. Change the wheels while driving WW
- 11. Strategic Goals Goals of the business side Architectural Principles High-Level Principles Design and Delivery Principles Tactical measures Reduce Time to Market Speed, Fast Feedback Cost Transparency Collect metrics to allow decisions cost vs. value. Support Data-Driven Decisions Listen to users and validate hypothesis. Provide as many relevant metrics & data as possible. You build it, you run it Responsibility to run and maintain a product stays with the building team. Fast feedback from live and customers helps us to continuously improve. Organized around Business Capabilities Build teams around products not projects. Follow the domain and respect bounded contexts. Inverse Conway Maneuver Shared Nothing Avoid shared infrastructure and tight coupling as much as possible. Don’t create the next monolith. Macro and Micro Architecture Clear separation. Autonomous micro services within the rules and constraints of the macro architecture. AWS First Favor AWS service over managed service, over self-hosted OSS, over self-rolled solutions. Data-Driven/ Metric-Driven Collect metrics from processes and applications. Analyze, alert and act on them. Eliminate Accidental Complexity Strive to keep it simple. Focus on essential complexity. No silver bullet. Event Sourcing and Publishing Keep history of state changes and publish application events. Autonomous Teams Make fast local decisions. Be responsible. Know your boundaries. Share findings. Continuous Delivery Deliver changes reliable, often and fast. Infrastructure As Code Automate everything: Reproducible, traceable and tested. Immutable servers over snowflake servers. DevOps Culture Developers and Ops work together in collaborative teams as engineers. No silos. Be Bold Go into production early. Value monitoring over tests. Recover and learn. Optimize for MTTR not MTBF. Security & Data Privacy Security must be first class citizen and everybody’s concern. Keep data-privacy in mind. CC
- 12. Principles Organized around business capabilities You build it, you run it Be bold Macro and micro architecture Event sourcing and publishing CWCW
- 13. Event Sourcing and data pumps One way data highway Event Sourcing - store history of all changes CC
- 14. SQS + S3 Kinesis + S3 Kinesis + DynamoDB SQS + DynamoDB Proxy + DynamoDB DynamoDB Evolution WW
- 15. Classified Events - Overview CC
- 16. Classified Events - Boundaries CC
- 17. DynamoDB as integration database? Two kinds of coupling Payload and connectivity Payload is DynamoDB agnostic DynamoDB as technical service contract WW
- 18. DynamoDB as integration database? Two kinds of coupling Payload and connectivity Payload is DynamoDB agnostic DynamoDB as technical service contract Event Sourcing rocks WW
- 19. From documents to events Refactoring toward deeper insight From CRUD to sync Offline first Writes are expensive USD 20 over USD 500 CC
- 20. Shared infrastructure Shared nothing Availability over shared nothing Convenience offerings No side effects Fast local decisions over committees CC
- 21. Shared nothing? How many environments? Use over re-use Re-use only after hardening How to share Copy n’paste, OSS, library Pull instead of push WW
- 22. How to build autonomous teams Do not fall back into old behaviours Beware of Mandelbrot teams Pager duty so that you run it Part-time ops not working Not all T-shapes are the same Wolf WW
- 23. Infrastructure guild Agree on things to do Share learnings Delegate implementation to teams Empty backlog should be normal How about infrastructure product teams? Mind the Shirky Principle CC
- 24. Focus sliders Product over platform platform product time WW
- 25. Focus sliders (cont.) Cash stack meets shiny new stack One company Lights on in cash stack Feature freeze Where to build new features? Ease of integration helps business people CC
- 27. Attributions Blue sky, white-gray clouds by nature protector Natubico, www.vivism.info [CC BY-SA 3.0] http://commons.wikimedia.org/wiki/File%3ABlue_sky%2C_white-gray_clouds.JPG A Danish Perspective by NASA [Public domain] http://commons.wikimedia.org/wiki/File%3AA_Danish_Perspective.jpg http://commons.wikimedia.org/wiki/File%3ANASAComputerRoom7090.NARA.jpg GREG EINRAD Amazon16 by Neil Palmer/CIAT [CC BY-SA 2.0] https://www.flickr.com/photos/ciat/5641594952 BERGSTEIGER Barber in Cameroon by James Emery from Douglasville, United States (Daddy Joe_1355) [CC BY 2.0] http://commons.wikimedia.org/wiki/File%3ABarber_in_Cameroon.jpg Wide objectives by Kivela (Own work) [Public domain] href="http://commons.wikimedia.org/wiki/File%3AWide_objectives.jpg Transformer Fire Barrier by GerryS1 (Own work) [CC BY-SA 3.0 or GFDL] http://commons.wikimedia.org/wiki/File%3ATransformer_Fire_Barrier.jpg
- 28. Attributions (cont) Alonso Renault Pitstop Chinese GP 2008 by Bert van Dijk (Pitstop F1 ING Renault) [CC BY-SA 2.0] http://commons.wikimedia.org/wiki/File%3AAlonso_Renault_Pitstop_Chinese_GP_2008.jpg Principle of Panchasheel by Prakash Adhikary (Own work) [CC BY 3.0] http://commons.wikimedia.org/wiki/File%3APrinciple_of_Panchasheel.JPG Traffic Jam by Doo Ho Kim [CC BY-SA 2.0] https://www.flickr.com/photos/titicat/3049591547 Pellets by The original uploader was Richard Mayer at German Wikipedia [GFDL or CC-BY-SA-3.0] http://commons.wikimedia.org/wiki/File%3APellets.jpg Pipes and Valves by Uwe Hermann [CC BY-SA 2.0] https://www.flickr.com/photos/73628542@N00/6272975359 Size variation in Coccinella undecimpunctata (2127991716) by Gilles San Martin from Namur, Belgium [CC BY-SA 2.0] http://commons.wikimedia.org/wiki/File%3ASize_variation_in_Coccinella_undecimpunctata_(2127991716).jpg Mille crêpe by Laitr Keiows (Own work) [CC BY-SA 3.0 or GFDL] http://commons.wikimedia.org/wiki/File%3AMille_cr%C3%AApe.jpg Country Energy power line replacement 01 by Bidgee (Own work) [CC BY-SA 3.0] http://commons.wikimedia.org/wiki/File%3ACountry_Energy_power_line_replacement_01.jpg
- 29. Attributions (cont) Sharing Sucks (4536747557) by eyeliam from Portland, United States [CC BY 2.0] http://commons.wikimedia.org/wiki/File%3ASharing_Sucks_(4536747557).jpg 7Line 9184 (8263568241) by Metropolitan Transportation Authority of the State of New York (7Line_9184 Uploaded by tm) [CC BY 2.0] http://commons.wikimedia.org/wiki/File%3A7Line_9184_(8263568241).jpg England rugby team 1905 by Russell & Sons (The Graphic) [Public domain or Public domain] http://commons.wikimedia.org/wiki/File%3AEngland_rugby_team_1905.jpg Wandergeselle by Sigismund von Dobschütz [CC BY-SA 3.0] http://commons.wikimedia.org/wiki/File%3AWandergeselle_02.JPG Faber-Rechenschieber 5304 by User:Karl Gruber (Own work) [CC BY-SA 4.0] http://commons.wikimedia.org/wiki/File%3AFaber-Rechenschieber_5304.JPG Wheel clamps Texas by Richard Anderson from Denton, United States (Boots.) [CC BY-SA 2.0] http://commons.wikimedia.org/wiki/File%3AWheel_clamps_Texas.jpg GuadalupeNOLA15Oct07Thanks by Infrogmation of New Orleans (Photo by Infrogmation) [GFDL or CC BY-SA 3.0] http://commons.wikimedia.org/wiki/File%3AGuadalupeNOLA15Oct07Thanks.jpg AtariBasic by Calin99 (Own work) [GPL] http://commons.wikimedia.org/wiki/File%3AAtariBasic.png Spare wheel by Brian Snelson [CC BY 2.0] https://commons.wikimedia.org/wiki/File:Spare_wheel_-_Flickr_-_exfordy.jpg
Editor's Notes
- #3: Meetup AutoScout24 Shifting gears How we build our services Event Sourcing DynamoDB as Atom Feed From documents to events Evolving architecture PriceEstimation Shared nothing? Infrastructure CSV Log processing How we organize ourselves Autonomous teams Infrastructure guild Focus sliders
- #4: Printed ads in paper analogy. Bold parts are in progress.
- #5: Scout24 was sold. New CEO Greg Ellis. Are you ready for the future? 21st century internet company?
- #6: We could hire good .NET devs, but mostly from banks, insurances companies No internet background; we need to teach Talents that could help us where sparse
- #7: We started think about our ecosystem and the flywheel that drives it. .NET small, slow Linux/JVM larger, faster
- #8: “Fly at the speed of fear” - Disruptive Japanese dragon: Flying beast Rollercoaster: Sixflags Magic Mountain Started Nov. 2014 with one team, now at 4 teams.
- #9: .NET/Windows -> Scala/Linux Own Data centers -> AWS Monolith + Swimlanes -> Microservices Dev + Ops -> Engineers Product: Include business, find core and add some polish -> Shiny new cut
- #12: This is for reference only. Make your own. We use this to guide discussions. Poster hangs in every room. The items on the right support items on the left.
- #13: Organized around Business Capabilities Build teams around products not projects. Follow the domain and respect bounded contexts. Inverse Conway Maneuver You build it, you run it Responsibility to run and maintain a product stays with the building team. Fast feedback from live and customers helps us to continuously improve. Be Bold Go into production early. Value semantic monitoring over tests. Recover and learn. Optimize for MTTR not MTBF. Macro and Micro Architecture Clear separation. Autonomous micro services within the rules and constraints of the macro architecture. Event Sourcing and Publishing Keep history of state changes and publish application events. -> Next story
- #14: Design decision: No queries against DC. Data needs to be pushed into AWS Ability to replay all events from beginning of time. Events = All changes to a classified
- #15: Just a summary. Skip: SQS + S3 Inspired by ImmobilienScout24 Kinesis + S3 Feedback from AWS Kinesis + DynomaDB Queryable, Storage costs not prohibitive (7x) SQS + DynamoDB Queue better than LATEST or TRIM_HORIZON Proxy + DynamoDB Changes are collected via queue table in Oracle DynamoDB DynamoDB is global service
- #16: Push vs. Pull Looks like a Integration DB
- #17: No Integration DB Only the owning service writes. Bounded contexts respected. DynamoDB as public API of service. Think publishing events into queue More consumers anticipated.
- #18: Skip: Cloud native learning curve, but with fast iterations. GSI necessary to make ordered scan. Later DynamoDB streams for changes. Treat DynamoDB as public facing contract and not as integration database. Only the owning service writes. Back of the napkin cost calculation for design decisions: 430 USD per month ofter one year. Full scan in 2 ½ h: 4 USD
- #20: DynamoDB is no document store, think about your data model and use cases. API to integrate with existing local storage implementation lead to think about syncing changes. Rediscovered offline first -> change/events Add/Remove/List -> Sync -> Almost Event Sourcing Do the right thing API followed evolution. Don’t be afraid of breaking changes. Don’t stick to legacy design. Timestamp vs SequenceID Writes: DynamoDB has an asymmetric provisioning model. Read to write costs 1:40 with eventual consistent reads Best write minimization: Write only changes = events
- #21: Shared Infrastucture vs. Shared Nothing Meeting avoidance movement Minimize coordination necessary Availability over Shared nothing -> Monitoring, Logging, required, Macro, enables troubleshooting and correlation across service boundaries, We share infrastructure via AWS platform GoCD optional, convenience offering, but how to e.g. improve cycle time. No side effects. Dashing, global shared state. -> Solution own dashing server Fast local decisions over committees When to tune?
- #22: What to share, classified service und taxonomy. CSV as a service How many environments: Two environments. Integration on prod. Be bold. Fakes on dev. CDCs as glue Use over re-use, re-use only after hardening Common helpers: shared library vs. copy & paste vs. OSS Size matters: copy & paste ok for 10 classes When to start OSS: on demand or speculatively? Logging as OSS? Shared lookup data as CSV via S3, not as a web service Provisioning of CSV is part of the service.
- #23: How to build autonomous teams Do not fall back into old behaviours Beware of Mandelbrot teams - dev team and ops team within the devops team Part-time ops not working Some devs do not like ops (and the other way round) Not all Ts are shaped the same. Pager duty enforces you build it you run it behaviour. How not to ignore the pager, no broken window syndrom On call rotation Name and shame
- #24: Why infrastructure guild? Agree on things to do Share learnings How we work: Work is done in the teams Focus work on infrastructure needed by team. (Macro decision: Shared infrastructure: Logging, Monitoring, Security) Teams are responsible - Subsidiarity How to handle long running/blocked stories? Keep delegating and resist temptation to take over Don’t treat infrastructure story as neglected child Empty backlog should be normal Hurdle to create new tasks. We want as few shared infrastructure as possible. Shirky: Institutions will try to preserve the problem to which they are the solution. Creates it own tasks. Risk of stories without value or prioritisation. “I have found an old sticky on the floor, let's implement that for two weeks!”
- #25: Focus Shift, PO missing Focus slider - AWS 20% product + 80% platform - first 2 months 50% product + 50% platform - ideally within next 3 months Current state: 70% product + 30 % platform Vision: 80% product + 20% platform War story: Beware spontaneous fluctuations in PO presence. (Compare priceestimation)
- #26: Focus slider - Business Don’t split into shiny new stack and legacy. Legacy is not used as a word anymore. Lights on in Cash stack - AWS Feature freeze: Where to build new features? Ease of integration between new and old stack helps business people to switch fluently between both worlds. Where to draw the line - Focus on technology change - Minimum viable integration Time to market vs. fast re-platforming (N+1 systems)