淺談 Live patching technology
5 likes•2,288 views
Live patching technology allows updating the Linux kernel without downtime. Ksplice was an early live patching solution released in 2009 but was limited and had licensing issues. kGraft and Kpatch were later developed by SUSE and Red Hat respectively as open source live patching solutions. Both use object code comparison and replacement at runtime, but kGraft can patch without stopping processes while Kpatch uses stop_machine to ensure safe replacement. Live patching is useful for critical bugs but has limitations around data structure and common function changes.
淺談 Live patching technology
- 1. Live Patching Technology Linux Kernel 4.0 SZ LIN [email protected] 1
- 2. Time for Kernel 4.0 2
- 3. Time for Kernel 4.0 3
- 4. 4 Live Patching 0 down time.
- 5. Why you need Patch? 1 Bug Fixes 2 3 5 New Feature Code Refactoring
- 7. 7
- 8. Live Patching Technology Oracle - 2009 Ksplice SUSE - 2014 kGraft RedHat - 2014 Kpatch 8
- 11. 11
- 14. 14
- 15. 15 Ksplice behavior Original Kernel Patched Kernel
- 16. 16 Ksplice behavior • foo method is updated • Find foo in running kernel • Find a safe time to insert jmp
- 17. 17 Ksplice behavior Limitation of Ksplice 1. Stop services < 1ms 2. Cannot patch common structures of kernel 3. Cannot patch common function (schedule, hrtimer)
- 18. Ksplice behavior Building an update Applying an update Actions 18 Match pre code to running kernel 1. Discover symbol values 2. Safety check Call stop_machine 1. Perform “safe time” check 2. Insert jmp instruction Find what has been changed Build pre and post source code to get object code Compare to find the list of changed functions Tell kernel to use new object code Ksplice.ko Load new object code Ksplice-new.ko
- 19. 19 kGraft
- 20. kGraft Target on… not even for short time periods unlike other technologies Doesn't require stopping the kernel, ever 1. kGraft patch can be built from C source directly, without the need for object code manipulation 2. Object-code based automated patch generation is provided as an alternative Allows code review on kGraft patch sources Small amount of code thanks to leveraging other Linux technologies, no complex instruction decoders or such kGraft is lean 20
- 21. How does kGraft work? 1 A kGraft patch is a .ko kernel module in a KMP RPM 2 The .ko is inserted into the kernel using 'insmod' at RPM install or update time 3 kGraft replaces whole functions in the kernel even while those functions may be executed 4 An updated kGraft RPM/module can replace an existing patch 21
- 22. kGraft Limitations 1 kGraft is designed for fixing critical bugs and thus primarily for simple changes 2 Changes in kernel data structure layout require special care and depending on the size of the change, the change may not be possible to do without rebooting at all – same as with other live patching tech 3 kGraft depends on a stable build environment and thus best suited for Linux distributions, their customers or anyone who builds their own kernels, rather than 3 rd party support companies 22
- 23. 23 kGraft 1 INT3/IPI-NMI self-modifying code 2 RCU-like update mechanism 3 mcount-based NOP space allocation 4 standard kernel module loading/linking mechanisms
- 24. kGraft Object Code Comparison 2 kernel compilations objcopy nm readelf O/S tools 24 With & without patch Compiler flags 1. ffunction-sections 2. fdata-sections
- 28. kGraft Behavior (4/7) 28 RCU-like replacement
- 29. kGraft Behavior (5/7) 29 RCU-like replacement
- 30. kGraft Behavior (6/7) 30 RCU-like replacement
- 33. 33 Kpatch
- 34. Kpatch What’s Kpatch? This applies a binary patch to kernel on-line Patching is done without shutdown Kpatch is a LIVE patching function for kernel Security and stability fixes Not for major kernel update Possible to fail patching with big patch 1. Constantly used system calls 2. Data structures Only for a small and critical issues 34
- 35. Kpatch Overview 35 Kpatch build: Build a binary patch module Kpatch.ko: The kernel module of Kpatch
- 36. Kpatch How to Patch 36 Kpatch uses Ftrace to patch 1. Hook the target function entry with registers 2. Change regs->ip to new function (change the flow)
- 37. Kpatch Conflict of Old and New Functions 37 Kpatch ensures the old functions are not executed when patching “Active Safeness Check” Do stack dump to check the target functions are not executed, for each thread.
- 38. Kpatch Active Safeness Check With Stop_machine 38 Kpatch uses stop_machine to check stacks
- 39. Kpatch Active Safeness Check With Stop_machine 39
- 40. Kpatch Active Safeness Check With Stop_machine 40
- 41. Kpatch Active Safeness Check With Stop_machine 41
- 42. Kpatch Stop_machine: Pros and Cons Pros Stop_machine stops all processes a while It is critical for control/network appliances In virtual environment, this takes longer time We need to wait all VCPUs are scheduled on the host machine Cons 42 Safe, simple and easy to review, Good for the 1st version Stop_machine-free kpatch is in discussion stage push current stop_machine-based kpatch to upstream
- 43. 1 Human safety analysis required! 2 Not a general purpose upgrade tool 3 ~80% of all CVE patches currently supported 1. Data structure changes, edge cases 2. Goal: 99% 4 stop_machine() latency: 1ms – 40ms 43 Kpatch Limitations Currently x86_64 only5
- 45. 45 Conclusion
- 46. 46 Conclusion 1 kGraft/ Kpatch RCU/ stop_machine 2 Implemented for x86 only as a reference architecture powerpc, s390 and arm is already in the works 3 Only for a small and critical issues Not for major kernel update 4 Two groups got together Combine kpatch and kGraft in the Linux Kernel 4.0 .
- 47. 47 Thank you
- 48. 48 References Linux Kernel https://www.kernel.org/ LWN https://lwn.net/ LKML.ORG - the Linux Kernel Mailing List Archive https://lkml.org/ Kgraft official site https://www.suse.com/promo/kgraft.html Kpatch official site http://rhelblog.redhat.com/2014/02/26/kpatch/ https://github.com/dynup/kpatch Kpslice – Oracle official site http://www.ksplice.com/ Kpatch Without Stop Machine Masami Hiramatsu
- 49. 49 References kpatch - Have your security and eat it too! Josh Poimboeuf Reboot adieu! Online Linux kernel patching Udo Seidel Oracle Ksplice for Oracle Linux Ksplice-quickstart Ksplice+ : Rebootless kernel updates in a distributed system Sanjay Kulhari KSPLICE: ZERO DOWNTIME UPDATES FOR ORACLE LINUX ORACLE DATA SHEET Note to Module Vendors With Respect to kGraft SUSE SolidDriver Program kGraft - Live patching of the Linux kernel Vojtěch Pavlik Ksplice: Automatic Rebootless Kernel Updates Jeff Arnold and M. Frans Kaashoek