www.immobilienscout24.de




Continuous Evolution
Iterating to a Continuous Delivery Platform

Berlin | 25.04.2012 | Ingmar Krusch, DevOps believer
Seite 2   | Things we learned while scaling DevOps | Ingmar Krusch, André von Deetzen
Continuous Delivery
   at ImmobilienScout24



Seite 3 | Continuous Evolution | @IngmarKrusch
When we started




    running at capacity in
        ops and above



Seite 4 | Continuous Evolution | @IngmarKrusch
historically grown
               infrastructure
                 and code


Seite 5 | Continuous Evolution | @IngmarKrusch
Got root?
                         No!
                     Wait, what?!


Seite 6 | Continuous Evolution | @IngmarKrusch
we knew we had to
       fundamentally change



Seite 7 | Continuous Evolution | @IngmarKrusch
even without everyone
         pushing us



Seite 8 | Continuous Evolution | @IngmarKrusch
ground changing
                 beneath us



Seite 9 | Continuous Evolution | @IngmarKrusch
old and complicated
      build- and deployment
           infrastructure

Seite 10 | Continuous Evolution | @IngmarKrusch
Our dilemma




             we were devided
             amongst ourselves



Seite 11 | Continuous Evolution | @IngmarKrusch
taking
           evolutionary steps



Seite 12 | Continuous Evolution | @IngmarKrusch
Old world
                    Iteration zero



Seite 13 | Continuous Evolution | @IngmarKrusch
old world

   some                                           build &
   config                                         release




    config                                        47.0.1
    share                          deploy         48.2.1
                                    tool



                                 devapp01

Seite 14 | Continuous Evolution | @IngmarKrusch
Iteration one




Seite 15 | Continuous Evolution | @IngmarKrusch
old world -> first iterration

   some                                           build &
   config                                         release




    config                                        47.0.1
    share                          deploy         48.2.1
                                    tool



                                 devapp01

Seite 16 | Continuous Evolution | @IngmarKrusch
1st iteration – RPM tech demo
                                                                 build &
                                                                 release

 svn ci
                            config
                            viewer                               47.0.1
                                                                 48.2.1



  config
                                                  global       converter
   SVN
                                                  YUM            script

                                                     yum
  post
                                                     install
 commit
  hook
                                           devapp01

Seite 17 | Continuous Evolution | @IngmarKrusch
Config RPMs intermezzo

                         post
  config                                           is24-conf-devapp01
                        commit
   SVN
                         hook
models the                   generates one
landscape                    RPM                      application
                             for each host



                                                        tomcat



 Seite 18 | Continuous Evolution | @IngmarKrusch
Iteration two




Seite 19 | Continuous Evolution | @IngmarKrusch
2nd iteration – RPM POC
                                                                 build &
                                                                 release

 svn ci
                            config
                            viewer                               47.0.1
                                                                 48.2.1



  config
                                                  global       converter
   SVN
                                                  YUM            script

                                                     yum
  post
                                                     install
 commit
  hook
                                           devapp01

Seite 20 | Continuous Evolution | @IngmarKrusch
Taking a RPMized app live


 internet



    LB
                                        Tomcat / solr

                        http
Frontend


 Backend                                     DB


  Seite 21
    | Continuous Evolution | @IngmarKrusch
RPMize the maschines
     the solr app runs on

                                           devapp01

Seite 22 | Continuous Evolution | @IngmarKrusch
2nd iteration – RPM POC
svn ci
                                                             TeamCity
              config
              viewer



config
                        global
 SVN
                         YUM




 post
commit
 hook
               Cron:
                sudo yum upgrade -y


devsol01                 tuvsol01                 bersol01   hamsol01

Seite 23 | Continuous Evolution | @IngmarKrusch
Iteration three




Seite 24 | Continuous Evolution | @IngmarKrusch
3rd iteration: Combining two POCs

svn ci
                                                             TeamCity
              config
              viewer



config
                        global
 SVN
                         YUM




 post
commit
 hook
               Cron:
                sudo yum upgrade -y


devsol01                 tuvsol01                 bersol01   hamsol01

Seite 25 | Continuous Evolution | @IngmarKrusch
Map and reduce YADT 
                                                                       build &
 some                                                                  release
                                           UDT
 config

                                                          artefact 1
                config 1                                               47.0.1
 config                                                                48.2.1
 share
                                         YADT
                                                                tomcat / solr
                      start & stop                                service
                                                  rsync
                      services

                                    devapp01
Seite 26 | Continuous Evolution | @IngmarKrusch
First complete solution
                                                      TeamCity
svn ci

              config
              viewer



config
                        global
 SVN                                              tomcat / solr
                         YUM
                                                    service
                                                                     YADT
 post
commit
 hook

                                            UDT
                                                                        start
                                                                        stop
                                                                        status

devsol01                 tuvsol01                 bersol01        hamsol01

Seite 27 | Continuous Evolution | @IngmarKrusch
integrating with our
          datacenter partner



Seite 28 | Continuous Evolution | @IngmarKrusch
http://code.google.com/p/yadt




            YADT controls the
             whole datacenter



Seite 29 | Continuous Evolution | @IngmarKrusch
http://code.google.com/p/yadt




       Open Sourcing YADT
             was a great idea!



Seite 30 | Continuous Evolution | @IngmarKrusch
Splitting up legacy
        applications is HARD!



Seite 31 | Continuous Evolution | @IngmarKrusch
Iteration four




Seite 32 | Continuous Evolution | @IngmarKrusch
Iteration four – Package Promotion



          The maven release
            process is crap!
                                                  TeamCity


                       global
                        YUM




Seite 33 | Continuous Evolution | @IngmarKrusch
TeamCity

Package Promotion Pipeline



  DEV                                     TUV                  global
                  promote                           promote
  YUM                                    YUM                    YUM




                                                              bersol01
devsol01                             tuvsol01             hamsol01

Seite 34 | Continuous Evolution | @IngmarKrusch
Untold stories (so little time :)




                                 Nexus
           as YUM Repository



Seite 35 | Continuous Evolution | @IngmarKrusch
But it‘s Open Source




        code.google.com/p/
   nexus-yum-plugin


Seite 36 | Continuous Evolution | @IngmarKrusch
the journey continues…




Seite 37 | Continuous Evolution | @IngmarKrusch
dedicated vs. shared
            infrastructure



Seite 38 | Continuous Evolution | @IngmarKrusch
integrate better with
           the linux distro



Seite 39 | Continuous Evolution | @IngmarKrusch
Finally




       changing our culture




Seite 40 | Continuous Evolution | @IngmarKrusch
www.immobilienscout24.de




Let‘s discuss!!!
Contact:                   Google+: http://bit.ly/ingmar
ImmobilienScout24
Andreasstraße 10           Twitter: @IngmarKrusch
10243 Berlin
                           URL:   www.immobilienscout24.de

Continuous evolution - iterating to a continuous delivery platform

  • 1.
    www.immobilienscout24.de Continuous Evolution Iterating toa Continuous Delivery Platform Berlin | 25.04.2012 | Ingmar Krusch, DevOps believer
  • 2.
    Seite 2 | Things we learned while scaling DevOps | Ingmar Krusch, André von Deetzen
  • 3.
    Continuous Delivery at ImmobilienScout24 Seite 3 | Continuous Evolution | @IngmarKrusch
  • 4.
    When we started running at capacity in ops and above Seite 4 | Continuous Evolution | @IngmarKrusch
  • 5.
    historically grown infrastructure and code Seite 5 | Continuous Evolution | @IngmarKrusch
  • 6.
    Got root? No! Wait, what?! Seite 6 | Continuous Evolution | @IngmarKrusch
  • 7.
    we knew wehad to fundamentally change Seite 7 | Continuous Evolution | @IngmarKrusch
  • 8.
    even without everyone pushing us Seite 8 | Continuous Evolution | @IngmarKrusch
  • 9.
    ground changing beneath us Seite 9 | Continuous Evolution | @IngmarKrusch
  • 10.
    old and complicated build- and deployment infrastructure Seite 10 | Continuous Evolution | @IngmarKrusch
  • 11.
    Our dilemma we were devided amongst ourselves Seite 11 | Continuous Evolution | @IngmarKrusch
  • 12.
    taking evolutionary steps Seite 12 | Continuous Evolution | @IngmarKrusch
  • 13.
    Old world Iteration zero Seite 13 | Continuous Evolution | @IngmarKrusch
  • 14.
    old world some build & config release config 47.0.1 share deploy 48.2.1 tool devapp01 Seite 14 | Continuous Evolution | @IngmarKrusch
  • 15.
    Iteration one Seite 15| Continuous Evolution | @IngmarKrusch
  • 16.
    old world ->first iterration some build & config release config 47.0.1 share deploy 48.2.1 tool devapp01 Seite 16 | Continuous Evolution | @IngmarKrusch
  • 17.
    1st iteration –RPM tech demo build & release svn ci config viewer 47.0.1 48.2.1 config global converter SVN YUM script yum post install commit hook devapp01 Seite 17 | Continuous Evolution | @IngmarKrusch
  • 18.
    Config RPMs intermezzo post config is24-conf-devapp01 commit SVN hook models the generates one landscape RPM application for each host tomcat Seite 18 | Continuous Evolution | @IngmarKrusch
  • 19.
    Iteration two Seite 19| Continuous Evolution | @IngmarKrusch
  • 20.
    2nd iteration –RPM POC build & release svn ci config viewer 47.0.1 48.2.1 config global converter SVN YUM script yum post install commit hook devapp01 Seite 20 | Continuous Evolution | @IngmarKrusch
  • 21.
    Taking a RPMizedapp live internet LB Tomcat / solr http Frontend Backend DB Seite 21 | Continuous Evolution | @IngmarKrusch
  • 22.
    RPMize the maschines the solr app runs on devapp01 Seite 22 | Continuous Evolution | @IngmarKrusch
  • 23.
    2nd iteration –RPM POC svn ci TeamCity config viewer config global SVN YUM post commit hook Cron: sudo yum upgrade -y devsol01 tuvsol01 bersol01 hamsol01 Seite 23 | Continuous Evolution | @IngmarKrusch
  • 24.
    Iteration three Seite 24| Continuous Evolution | @IngmarKrusch
  • 25.
    3rd iteration: Combiningtwo POCs svn ci TeamCity config viewer config global SVN YUM post commit hook Cron: sudo yum upgrade -y devsol01 tuvsol01 bersol01 hamsol01 Seite 25 | Continuous Evolution | @IngmarKrusch
  • 26.
    Map and reduceYADT  build & some release UDT config artefact 1 config 1 47.0.1 config 48.2.1 share YADT tomcat / solr start & stop service rsync services devapp01 Seite 26 | Continuous Evolution | @IngmarKrusch
  • 27.
    First complete solution TeamCity svn ci config viewer config global SVN tomcat / solr YUM service YADT post commit hook UDT start stop status devsol01 tuvsol01 bersol01 hamsol01 Seite 27 | Continuous Evolution | @IngmarKrusch
  • 28.
    integrating with our datacenter partner Seite 28 | Continuous Evolution | @IngmarKrusch
  • 29.
    http://code.google.com/p/yadt YADT controls the whole datacenter Seite 29 | Continuous Evolution | @IngmarKrusch
  • 30.
    http://code.google.com/p/yadt Open Sourcing YADT was a great idea! Seite 30 | Continuous Evolution | @IngmarKrusch
  • 31.
    Splitting up legacy applications is HARD! Seite 31 | Continuous Evolution | @IngmarKrusch
  • 32.
    Iteration four Seite 32| Continuous Evolution | @IngmarKrusch
  • 33.
    Iteration four –Package Promotion The maven release process is crap! TeamCity global YUM Seite 33 | Continuous Evolution | @IngmarKrusch
  • 34.
    TeamCity Package Promotion Pipeline DEV TUV global promote promote YUM YUM YUM bersol01 devsol01 tuvsol01 hamsol01 Seite 34 | Continuous Evolution | @IngmarKrusch
  • 35.
    Untold stories (solittle time :) Nexus as YUM Repository Seite 35 | Continuous Evolution | @IngmarKrusch
  • 36.
    But it‘s OpenSource code.google.com/p/ nexus-yum-plugin Seite 36 | Continuous Evolution | @IngmarKrusch
  • 37.
    the journey continues… Seite37 | Continuous Evolution | @IngmarKrusch
  • 38.
    dedicated vs. shared infrastructure Seite 38 | Continuous Evolution | @IngmarKrusch
  • 39.
    integrate better with the linux distro Seite 39 | Continuous Evolution | @IngmarKrusch
  • 40.
    Finally changing our culture Seite 40 | Continuous Evolution | @IngmarKrusch
  • 41.
    www.immobilienscout24.de Let‘s discuss!!! Contact: Google+: http://bit.ly/ingmar ImmobilienScout24 Andreasstraße 10 Twitter: @IngmarKrusch 10243 Berlin URL: www.immobilienscout24.de