Downloaded 50 times
Uploaded byAlex Lau
openSUSE storage workshop 2016
This document provides an overview of openSUSE Cloud Storage Workshop presented by AvengerMoJo in November 2016. It covers introductory topics on traditional and cloud storage, key components of Ceph including MON, OSD, MDS, and CRUSH map. It also discusses features like thin provisioning, cache tiering, erasure coding, self management/repair. Development topics covered include Ceph source code, use of Salt for configuration and deployment, and SUSE's software lifecycle process.
More Related Content
Similar to openSUSE storage workshop 2016
openSUSE storage workshop 2016
- 1. openSUSE Cloud StorageWorkshop AvengerMoJo ( Alex Lau [email protected] ) Nov, 2016
- 2.
- 3.
- 4.
- 5. Storage Size Bits andBytes > Bytes (B) > Kilobyte (KB) > Megabyte (MB) > Gigabyte (GB) > Terabyte (TB) > Petabyte (PB) > Exabyte (EB) > 8 Bits > 8,192 Bits > 8,388,608 Bits > 8,589,934,592 Bits > 8,796,093,022,208 Bits > 9,007,199,254,740,992 Bits > 9,223,372,036,854,775,808 Bits
- 6. Hard Driver Terms >Capacity ( Size ) > Cylinders, Sectors and Tracks > Revolution per Minute ( Speed ) > Transfer Rate ( e.g. SATA III ) > Access Time ( Seek time + Latency )
- 7. RAID > Redundant Arrayof Independent Disks – 2 or more disks put together to act as 1
- 8. NAS and SAN >Network Attached Storage > TCP/IP > NFS/SMB > Serve Files > Storage Area Network > Fiber Channel > ISCSI > Serve Block ( LUN )
- 9. Storage Trend > DataSize and Capacity – Multimedia Contents – Big Demo binary, Detail Graphic / Photos, Audio and Video etc. > Data Functional need – Different Business requirement – More Data driven process – More application with data – More ecommerce > Data Backup for a longer period – Legislation and Compliance – Business analysis
- 10. Storage Usage Tier 0 UltraHigh Performance Tier 1 High-value, OLTP, Revenue Generating Tier 2 Backup/Recovery, Reference Data, Bulk Data Tier 3 Object, Archive, Compliance Archive, Long-term Retention 1-3% 15-20% 20-25% 50-60%
- 11. Storage Pricing JBOD Storage Mid-range Array Mid-range NAS High-end DiskArray SUSE Enterprise Storage Fully Featured NAS Device Entry-level Disk Array Dell EMC, Hitachi, HP, IBM NetApp, Pura Storage, Nexsan Promise, Synology, QNAP, Infortrend, ProWare, SansDigitial
- 12. CLOUD STORAGE INTRO SoftwareDefine Storage
- 13. Who is doingcloud storage?
- 14. Who is doingSoftware Define Storage
- 15. Completeness of Vision Leaders Visionaries Challengers Niche AbilitytoExecute Gartner’sReport http://www.theregister.co.uk/201 6/10/21/gartners_not_scoffing_ at_scofs_and_objects/ > SUSE has an aggressive pricing for deployment with commodity hardware > SES make both ceph and openstack enterprise ready
- 16. Software Define StorageDefinition From http://www.snia.org/sds > Virtualized storage with a service management interface, includes pools of storage with data service characteristics > Automation – Simplified management that reduces the cost of maintaining the storage infrastructure > Standard Interfaces – APIs for the management, provisioning and maintenance of storage devices and services > Virtualized Data Path – Block, File and/or Object interfaces that support applications written to these interfaces > Scalability – Seamless ability to scale the storage infrastructure without disruption to the specified availability or performance > Transparency – The ability for storage consumers to monitor and manage their own storage consumption against available resources and costs
- 17. SDS characters SUSE’s Cephbenefit point of view > High Extensibility: – Distributed over multiple nodes in cluster > High Availability: – No single point of failure > High Flexibility: – API, Block Device and Cloud Supported Architecture > Pure Software Define Architecture > Self Monitoring and Self Repairing
- 18. DevOps with SDS >Collaboration between – Development – Operations – QA ( Testing ) > SDS should enable DevOps to use a variety of data management tools to communicate their storage http://www.snia.org/sds
- 19. Why using ceph? >Thin Provisioning > Cache Tiering > Erasure Coding > Self Manage and Self Repair with continuous monitoring > High ROI compare to traditional Storage Solution Vendor
- 20. Thin Provisioning Traditional StorageProvision SDS Thin Provisioning Data Allocated Data Allocated Volume A Volume B Data Data Available Storage Volume A Volume B
- 21. Cache Tiers Writing QuicklyApplication like: • e.g. Video Recording • e.g. Lots of IoT Data Reading Quickly Application like: • e.g. Video Streaming • e.g. Big Data analysis Write Tier Hot Pool Normal Tier Cold Pool Read Tier Hot Pool SUSE ceph Storage Cluster Normal Tier Cold Pool
- 22. Control Costs Erasure Coding CopyCopy Copy Replication Pool SES CEPH CLUSTSER Control Costs Erasure Coded Pool SES CEPH CLUSTSER Data Data Data Data Parity Parity Multiple Copy of stored data • 300% cost of data size • Low Latency, Faster Recovery Single Copy with Parity • 150% cost of data size • Data/Parity ratio trade of CPU
- 23. Self Manage andSelf Repair > CRUSH map – Controlled Replication Under Scalable Hashing – Controlled, Scalable, Decentralized Placement of Replicated Data •Hash •Num of PG Object •Cluster state •Rule CRUSH •Peer OSD •Local Disk OSD
- 24. WHAT IS CEPH? Differentcomponents
- 25. Basic Ceph Cluster >Interface – Object Store – Block – File > MON – Cluster map > OSD – Data storage > MDS – cephfs LIBRADOS OSD MON OSD OSD MON MON MDS MDS MDS RADOS Object Store Block Store File Store Interface CEPHFS RDB RADOSG W
- 26. Ceph Monitor > PaxosRole – Proposers – Acceptors – Learners – Leader Mon OSD MAP MON MAP PG MAP CRUSH MAP Paxos Service Paxos LevelDB K/V K/V K/V Log …
- 27. ObjectStore Daemon > Lowlevel IO operation > FileJournal normally finished first before FileStore write to disk > DBObjectMap provide KeyValue omap for copy on write function File Store OSD OSDOSD OSD PG PG PG PG … Object Store File Store FileJournal DBObjectMap
- 28. FileStore Backend > OSDManage its own consistency of data > All write operation are transactional on top of existing filesystem – XFS, Btrfs, ext4 > ACID ( Atomicity, Consistency, Isolation, Durability ) operations to protect data write File Store OSD Disk Disk Disk BtrfsXFS ext4 File Store OSD File Store OSD OSD MON OSD OSD MON MON RADOS
- 29. cephfs MeatData Server >MDS store data at RADOS – Directories, Files ownership, access mode etc > POSIX compatible > Don’t Server File > Only Required for share filesystem > High Availability and Scalable OSD MON OSD OSD MON MON MDS MDS MDS RADOS CephFS Client META DataDataData
- 30. CRUSH map > Devices: –Devices consist of any object storage device–i.e., the storage drive corresponding to a ceph-osd daemon. You should have a device for each OSD daemon in your Ceph configuration file. > Bucket Types: – Bucket types define the types of buckets used in your CRUSH hierarchy. Buckets consist of a hierarchical aggregation of storage locations (e.g., rows, racks, chassis, hosts, etc.) and their assigned weights. > Bucket Instances: – Once you define bucket types, you must declare bucket instances for your hosts, and any other failure domain partitioning you choose. > Rules: – Rules consist of the manner of selecting buckets.
- 31. Kraken / SUSEKey Features > Client from multiple OS and hardware including ARM > Multi Path iSCSI support > Cloud Ready and S3 Supported > Data encryption over physical disk > Cephfs support > Bluestore support > Ceph-manager > openATTIC
- 32. ARM64 Server > Cephalready been tested with the following Gigabyte Cavium system > Gigabyte H270-H70 Cavium - 48 Core * 8 : 384 Cores - 32G * 32: 1T Memory - 256G * 16: 4T SSD - 40GbE * 8 Network
- 33. iSCSI Architecture Technical Background Protocol: ‒Block storage access over TCP/IP ‒ Initiators the client that access the iscsi target over tcp/ip ‒ Targets, the server that provide access to a local block SCSI and iSCSI: ‒ iSCSI encapsulated commands and responses ‒ TCP package of iscsi is representing SCSI command Remote access: ‒ iSCSI Initiators able to access a remote block like local disk ‒ Attach and format with XFS, brtfs etc. ‒ Booting directly from a iscsi target is supported
- 34. Public Network OSD1 OSD2OSD3 OSD4 Cluster Network iSCSI Gateway RBD Module iSCSI Gateway RBD Module iSCSI Initiator RBD image
- 35. BlueFS META DataDataData RocksDB Allocator Block BlockBlock BlueStore Backend > Rocksdb – Object metadata – Ceph key/value data > Block Device – Directly data object > Reduce Journal Write operation by half BlueStore
- 36. Ceph object gateway >RESTful gateway to ceph storage cluster – S3 Compatible – Swift Compatible LIBRADOS OSD MON OSD OSD MON MON RADOS RADOSGW RADOSGW S3 API Swift API
- 37. CephFS > POSIX compatible >MDS provide metadata information > Kernel cephfs module and FUSE cephfs module available > Advance features that is still require lots of testing – Directory Fragmentation – Inline Data – Snapshots – Multiple filesystems in a cluster libcephfs librados OSD MON OSD OSD MON MON MDS MDS MDS RADOS FUSE cephfsKernel cephfs.ko
- 38. openATTIC Architecture High LevelOverview Django Linux OS Tools openATTIC SYSTEMD RESTful API PostgreSQL DBUS Shell librados/li brbd Web UI REST Client HTTP NoDB
- 39. HARDWARE What is theminimal setup?
- 40. Ceph Cluster ina VM Requirement > At least 3 VM > 3 MON > 3 OSD – At least 15GB per osd – Host device better be on SSD VM OSD MON >15G VM OSD MON >15G VM OSD MON >15G
- 41. Minimal Production recommendation >OSD Storage Node ‒ 2GB RAM per OSD ‒ 1.5GHz CPU core per OSD ‒ 10GEb public and backend ‒ 4GB RAM for cache tier > MON Monitor Node ‒ 3 Mons minimal ‒ 2GB RAM per node ‒ SSD System OS ‒ Mon and OSD should not be virtualized ‒ Bonding 10GEb
- 42. For developer Dual 1GNetwork 6T = 220$ 220 * 3 = 660$ 512G = 150$ OSD1 OSD2 OSD3 OSD4 MON1 300$ 6T = 220$ 220 * 3 = 660$ 512G = 150$ 6T = 220$ 220 * 3 = 660$ 512G = 150$ OSD5 OSD6 OSD7 OSD8 MON2 300$ OSD9 OSD10 OSD11 OSD12 MON3 300$
- 43. HTPC AMD (A8-5545M) Formfactor: – 29.9 mm x 107.6 mm x 114.4mm CPU: – AMD A8-5545M ( Clock up 2.7GHz / 4M 4Core) RAM: – 8G DDR-3-1600 KingStone ( Up to 16G SO-DIMM ) Storage: – mS200 120G/m-SATA/read:550M, write: 520M Lan: – Gigabit LAN (RealTek RTL8111G) Connectivity: – USB3.0 * 4 Price: – $6980 (NTD)
- 44. Enclosure Form factor: – 215(D)x 126(w) x 166(H) mm Storage: – Support all brand of 3.5" SATA I / II / III hard disk drive 4 x 8TB = 32TB Connectivity: – USB 3.0 or eSATA Interface Price: – $3000 (NTD)
- 45. How to createmultiple price point? 1000$ = 1000G 2000MB rw 4 PCIe = 4000$ = 8000MB rw 4T Storage 400,000 IOPS 4$ per G 250$ = 1000G, 500MB rw 16 Driver = 4000$ = 8000MB rw 16T Storage 100,000 IOPS 1$ per G 250$ = 8000G 150MB rw 16 Driver = 4000$ = 2400MB rw 128T Storage 2000 IOPS 0.1$ per G
- 46. ARM64 hardware compareto Public Cloud price R120-T30 - 5700$ * 7 - 48 Core * 7 : 336 Cores - 8 * 16G * 7 : 896G Memory - 1T * 2 * 7 : 14T SSD - 8T * 6 * 7 : 336T HDD - 40GbE * 7 - 10GbE * 14 > EC 5+2 is about 250T > 2500 Customer 100GB > 2$ Storage = 5000$ > 8 Months = 40000$
- 47. CEPH DEVELOPMENT Source, andSalt in action
- 48. SUSE software lifecycle Upstream Repo openSUSE BuildService Internal Build Service QA and Test process Product •Tumbleweed •SLE->Leap > Upstream – Factory and Tumbleweed > SLE – Patch Upstream – Leap
- 49. Ceph Repo > Upstream –https://github.com/ceph/ceph > SUSE Upstream – https://github.com/SUSE/ceph > Open Build Service – https://build.opensuse.org/pac kage/show/filesystems:ceph:U nstable > Kraken Release – https://build.opensuse.org/proj ect/show/filesystems:ceph:kra ken
- 50. Tumbleweed Zypper Repo >Kraken – http://download.opensuse.org/repositori es/filesystems:/ceph:/kraken/openSUS E_Tumbleweed/ > Salt and Deepsea – http://download.opensuse.org/repositori es/home:/swiftgist/openSUSE_Tumble weed/ – http://download.opensuse.org/repositori es/filesystems:/ceph/openSUSE_Tumbl eweed/ > Tumbleweed OS – http://download.opensuse.org/tumblew eed/repo/oss/suse/ > Carbon + Diamond – http://download.opensuse.org/repositori es/systemsmanagement:/calamari/ope nSUSE_Tumbleweed
- 51. Salt files collectionfor ceph DeepSea > https://github.com/SUSE/DeepSea > A collection of Salt files to manage multiple Ceph clusters with a single salt master > The intended flow for the orchestration runners and related salt states – ceph.stage.0 or salt-run state.orch ceph.stage.prep – ceph.stage.1 or salt-run state.orch ceph.stage.discovery – Create /srv/pillar/ceph/proposals/policy.cfg – ceph.stage.2 or salt-run state.orch ceph.stage.configure – ceph.stage.3 or salt-run state.orch ceph.stage.deploy – ceph.stage.4 or salt-run state.orch ceph.stage.services
- 52. Salt enable ceph Existingcapability Sesceph ‒ Python API library that help deploy and manage ceph ‒ Already upstream in to salt available in next release ‒ https://github.com/oms4suse/sesceph Python-ceph-cfg ‒ Python salt module that use sesceph to deploy ‒ https://github.com/oms4suse/python-ceph-cfg
- 53. Why Salt? Existing capability Productsetup ‒ SUSE OpenStack cloud, SUSE manager and SUSE Enterprise Storage all come with salt enable Parallel execution ‒ E.g. Compare to ceph-deploy to prepare OSD > Customize Python module ‒ Continuous development on python api easy to manage > Flexible Configuration ‒ Default Jinja2 + YAML ( stateconf ) ‒ Pydsl if you like python directly, json, pyobject, etc
- 54. Quick salt deploymentexample > Git repo for fast deploy and benchmark https://github.com/AvengerMoJo/Ceph-Saltstack > Demo recording https://asciinema.org/a/81531 1) Salt setup 2) Git clone and copy module to salt _modules 3) Saltutil.sync_all push to all minion nodes 4) ntp_update all nodes 5) Create new mons, and create keys 6) Clean disk partitions and prepare OSD 7) Update crushmap
- 55.
- 56. ceph-deploy > ssh nopassword id need to pass over to all cluster nodes > echo nodes ceph user has sudo for root permission > ceph-deploy new <node1> <node2> <node3> – Create all the new MON > ceph.conf file will be created at the current directory for you to build your cluster configuration > Each cluster node should have identical ceph.conf file
- 57. OSD Prepare andActivate > ceph-deploy osd prepare <node1>:</dev/sda5>:</var/lib/ceph/osd/journal/osd-0> > ceph-deploy osd activate <node1>:</dev/sda5>
- 58. Cluster Status > cephstatus > ceph osd stat > ceph osd dump > ceph osd tree > ceph mon stat > ceph mon dump > ceph quorum_status > ceph osd lspools
- 59. Pool Management > cephosd lspools > ceph osd pool create <pool-name> <pg-num> <pgp- num> <pool-type> <crush-ruleset-name> > ceph osd pool delete <pool-name> <pool-name> --yes- i-really-really-mean-it > ceph osd pool set <pool-name> <key> <value>
- 60. CRUSH Map Management >ceph osd getcrushmap -o crushmap.out > crushtool -d crushmap.out -o decom_crushmap.txt > cp decom_crushmap.txt update_decom_crushmap.txt > crushtool -c update_decom_crushmap.txt -o update_crushmap.out > ceph osd setcrushmap -i update_crushmap.out > crushtool --test -i update_crushmap.out --show-choose-tries --rule 2 --num-rep=2 > crushtool --test -i update_crushmap.out --show-utilization --num- rep=2 ceph osd crush show-tunables
- 61. RBD Management > rbd--pool ssd create --size 10000 ssd_block – Create a 1G rbd in ssd pool > rbd map ssd/ssd_block ( in client ) – It should show up in /dev/rbd/<pool-name>/<block-name> > Then you can use it like a block device
- 62. Demo usage > Itcould be QEMU/KVM rbd client for VM > It could be also be NFS/CIFS server ( but you need to consider how to support HA over that )
- 63. WHAT NEXT? Email me[email protected] Let me know what you want to hear next
Editor's Notes
- #12 11
- #23 As previously mentioned SUSE Enterprise storage is a highly scalable and highly available storage solution. A SUSE Enterprise Storage Cluster is build using commodity server and disk drive components. Giving you freedom of choice to choose the hardware and significantly reduce capital cost by eliminating the need to purchase more expensive proprietary storage systems. Still your current investment is protected because different types and speeds of drives can be deployed dependent on your requirements. This could include flash drives for very high performance or high capacity hard disk drives for bulk storage. [click]
- #34 33
- #39 38
- #54 53