萬事萬物皆是 LOG - 系統架構也來點科普
1 like1,103 views
The document discusses how logs (LOG) can be used as the basis for many software systems and applications. It describes logs as sequentially recorded events that provide an immutable record of what happened over time. It then gives examples of how databases, version control systems, distributed systems, messaging, and many other areas are based on logs and benefit from their append-only, immutable nature. Logs allow sharing state, reaching consensus, and building deterministic, scalable, and maintainable systems.
萬事萬物皆是 LOG - 系統架構也來點科普
- 1. 萬事萬物皆是 LOG Poga @ COSCUP 2016.08.20
- 2. LOG If you want maintainable, understandable software system
- 3. 什麼是 LOG
- 4. LOG 的性質 ● 按照時間出現 ● 出現過的就不會改變（append-only） ● 目的：記錄（某個時間）發生了（某件事） ○ 常常用來作為 debug 推理的依據
- 5. Things based on LOG ● Database ● Version Control ● Distributed System and Distributed Consensus ● Synchronization ● Replication ● Messaging ● UI Framework ● ...And almost EVERYTHING
- 7. POGA Software Developer, g0v.tw, KKTIX(前)
- 8. 資料庫 ID Username Balance 231 poga 100 7212 moon 12314 834 et 927 ... ... ...Database
- 9. ● 大量讀寫資料，長期儲存 ● 確保資料正確性（relation, validation, schema, ACID） ● 支援多種高效查詢（SQL, index） 如果你從沒用過資料庫...
- 10. 寫入資料時當機？
- 11. 如何避免寫入資料時當機導致資料消失？ 寫入： ID Username Balance 231 poga 100 ... 2016.08.20 13:40:05.0012621 Write {ID:231, Username:poga, Balance: 100} ... 真正寫入： Validation, Update Index… etc.
- 12. ... 2016.08.20 13:40:05.0012621 Write {ID:231, Username:poga, Balance: 100} ... 時間 發生了什麼事
- 13. WRITE-AHEAD LOG
- 14. LOG ● Fast and simple RECAP
- 15. 資料庫 ID Username Balance 231 poga 100 7212 moon 12314 834 et 927 ... ... ...Database
- 16. ● 單一 Database 擴充性有其極限 ● 多台 Database，分散負載 ○ 需要確保多台 Database 之間資料同步 Replication
- 17. 確保多台資料庫間資料同步？
- 18. 寫入一台資料庫 寫入： ID Username Balance 231 poga 100 ... 2016.08.20 13:40:05.0012621 Write {ID:231, Username:poga, Balance: 100} ... 真正寫入： Validation, Update Index… etc.
- 19. 一次寫入多台資料庫 寫入： ID Username Balance 231 poga 100 ... 2016.08.20 13:40:05.0012621 Write {ID:231, Username:poga, Balance: 100} ... 真正寫入： Validation, Update Index… etc. ... 2016.08.20 13:40:05.0012621 Write {ID:231, Username:poga, Balance: 100} ... 真正寫入： Validation, Update Index… etc. DB1 DB2
- 20. LOG-SHIPPING
- 21. LOG ● Fast and simple ● Can be used to share “current state” RECAP
- 22. 微服務 Microservice
- 25. Microservice ● 將邏輯按照 Domain 切割，有 獨立的儲存與運算資源 ● 容易擴充，團隊間分責明確 ● 門檻較高：不易維護，需要進階 系統維護能力
- 26. 微服務間怎麼互相溝通？
- 27. Event Stream
- 28. EVENT STREAM = LOG
- 29. LOG ● Fast and simple ● Can be used to share “current state” ● LOG are easy to parse/understand, Universal Interface RECAP
- 30. 前端架構 Flux/Redux
- 31. ● Many, many states ● Constantly changing state ○ User Interaction, Animation, Network ● Hard to reproduce bug and debug UI State
- 32. 前端介面狀態多變，如何維護？
- 33. Flux/Redux ● Unidirectional Data Flow ○ Append only ○ Handle action one by one, reasonable ● Immutable State ○ Produce new state based on previous state and action ● Everything else can be stateless(pure)
- 35. LOG ● Fast and simple ● Can be used to share “current state” ● Text are easy to parse/understand, Universal Interface ● Append-only, easy to reason, everything else can be stateless RECAP
- 36. 區塊鏈 Blockchain
- 37. ● 2016.08.20 13:31:23, A 給 B 100 bitcoin ● State: A 跟 B 身上剩下多少 bitcoin ● Double Spend: ○ A 把 bitcoin 交給 B 後，不能再把同樣的 bitcoin 交給 C Transaction
- 39. ● 如果 ○ 所有參與者都知道/可以驗證所有人的交易記錄 ○ 既有的交易紀錄無法被竄改 ○ 便沒有偽造空間 ● Blockchain ○ 將所有的交易記錄用 block 封裝，每個 block 替之前的交易記錄加密驗證 ○ 想要偽造一筆交易記錄 = 必須偽造整個 blockchain 上的交易記錄 ○ 所有人都基於相同的 blockchain 驗證交易 Distributed Consensus
- 40. LOG AGREEMENT = CONSENSUS Raft: A Consensus Algorithm for Replicated Logs Apache Mesos - The Mesos Replicated Log (PAXOS)
- 41. LOG ● Fast and simple ● Can be used to share “current state” ● Text are easy to parse/understand, Universal Interface ● Append-only, Immutable, easy to reason, everything else can be stateless ● Replicate log = consensus RECAP
- 43. ● > PB ● Data coming from everywhere ○ User Input ○ Business Data ○ Tracking ○ External Data ● ETL(Extract-Transform-Load) ● Analysis, Machine Learning… etc BIG Data
- 44. 如何處理雜亂、無序、大量的資料？
- 45. Data Pipeline
- 46. APPEND ONLY LOG = HIGH SCALABILITY
- 47. LOG ● Fast and simple ● Can be used to share “current state” ● Text are easy to parse/understand, Universal Interface ● Append-only, Immutable, easy to reason, everything else can be stateless ● Replicate log = consensus ● High Scalability RECAP
- 50. Make your system deterministic ● 容易理解 ● 容易DEBUG ● 容易同步 ● 容易儲存 ● 容易擴展
- 51. LOG = DETERMINISM = 穩定的系統
- 53. Thanks!