Implement a non-volatile SecondaryCache

[anand1976]

RocksDB added support for a SecondaryCache in #8271, #8191 and #8312. The SecondaryCache is an additional tier of caching below the block cache. It can be used to provide a non-volatile cache tier on local flash or NVM/SCM that can complement the DRAM block cache. More details about the design of SecondaryCache can be found here.

We are looking for a community contribution of SecondaryCache implementations, which would make this feature usable by the broader RocksDB userbase.

[kriti-sc]

Would like to pick this up.

[anand1976]

@kriti-sc That sounds great! Thanks for volunteering. Let me know what you have in mind. I'll be happy to discuss and help out in any way.

[kriti-sc]

If I have understood correctly, we are implementing a persistent memory cache. This cache will be controlled by the existing RocksDB APIs [reference PR].

I found this library by Intel to program persistent memory [reference]. It provides an abstraction over complexities such as error handling, data consistency & durability.

@anand1976 Please let me know your thoughts.

[zhichao-cao]

If I have understood correctly, we are implementing a persistent memory cache. This cache will be controlled by the existing RocksDB APIs [reference PR].

Actually, it is not only for persistent memory cache. The secondary cache can based on HDD, SSD or NVM or even remote DRAM.

[anand1976]

Thanks for your interest. I think long term a secondary cache based on persistent memory would be interesting. The hardware is not widely available or used yet, so it's still early days. In the near term, there's a need for a ssd/flash based secondary cache, for use cases where the database is on some kind of remote or cloud based storage and is accessed by a server with some direct attached flash. Let me know if you'd be interested in implementing it. Thanks,, Anand

…

On Tue, Aug 3, 2021, 3:29 PM Kriti Kathuria ***@***.***> wrote: If I have understood correctly, we are implementing a persistent memory cache. This cache will be controlled by the existing RocksDB APIs [reference PR <https://github.com/facebook/rocksdb/pull/8113/files>]. I found this library <https://pmem.io/vmemcache/manpages/master/vmemcache.3.html> by Intel to program persistent memory [reference <https://pmem.io/pmdk/>]. It provides an abstraction over complexities such as error handling, data consistency & durability. @anand1976 <https://github.com/anand1976> Please let me know your thoughts. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#8347 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AIAWX6TURW2CPSATK7QMRJTT3BUTLANCNFSM45XCMLJA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email> .

[zaorangyang]

Hi, All. Does this feature need only one community contributor? I am also glad to make contribution to this feature. :)

[mrambacher]

Hi, All. Does this feature need only one community contributor? I am also glad to make contribution to this feature. :)

The SecondaryCache is configurable and there can be multiple independent or related implementations. For example, there could be one based on PMEM and another based on SSD/Flash, and yet another stored in the cloud.

So my suggestion is to coordinate with anyone else who may be implementing a cache so that the same "SecondaryCache" is not implemented twice, but there is nothing that prevents multiple independent implementations!

[kriti-sc]

@anand1976 That aligns with what I have been thinking. I am interested in implementing this and will start with SSDs.

Thanks for your interest. I think long term a secondary cache based on persistent memory would be interesting. The hardware is not widely available or used yet, so it's still early days. In the near term, there's a need for a ssd/flash based secondary cache, for use cases where the database is on some kind of remote or cloud based storage and is accessed by a server with some direct attached flash. Let me know if you'd be interested in implementing it. Thanks,, Anand
…
On Tue, Aug 3, 2021, 3:29 PM Kriti Kathuria @.***> wrote: If I have understood correctly, we are implementing a persistent memory cache. This cache will be controlled by the existing RocksDB APIs [reference PR https://github.com/facebook/rocksdb/pull/8113/files]. I found this library https://pmem.io/vmemcache/manpages/master/vmemcache.3.html by Intel to program persistent memory [reference https://pmem.io/pmdk/]. It provides an abstraction over complexities such as error handling, data consistency & durability. @anand1976 https://github.com/anand1976 Please let me know your thoughts. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#8347 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIAWX6TURW2CPSATK7QMRJTT3BUTLANCNFSM45XCMLJA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

[kennthhz-zz]

I am interested in contributing. Our scenario (implementing NoSql using RocksDB as storage engine) is to reduce DRAM size and cost by placing larger part of block cache in Intel Optane PMEM or just regular NVME m2 (block based address pattern). So Block cache can be viewed as having hot (DRAM) and warm (PMEM) and cold (NVME m2) layer with customizable caching eviction strategy. Our read/write latency SLA is in the ms. So increase read latency from <10 ns to hundreds of ns or even at microsecond level shouldn't impact our latency SLA, but can save costs.

[chenyou-intel]

Hi, We are the Intel RPC Optane team. Now, we're building a secondary cache plugin implementation (along with other fs layer plugins) based on Optane Persistent Memory hardware and will be open-sourced soon. Hope we can contribute to the RocksDB community!

[chenyou-intel]

Hi, We are the Intel RPC Optane team. Now, we're building a secondary cache plugin implementation (along with other fs layer plugins) based on Optane Persistent Memory hardware and will be open-sourced soon. Hope we can contribute to the RocksDB community!

We have open sourced our persistent memory based secondary cache here !

[anand1976]

@chenyou-intel Thanks for the contribution! I'll take a look in the next week or so.

[chenyou-intel]

@chenyou-intel Thanks for the contribution! I'll take a look in the next week or so.

Sure, you could also find a WIP PR to integrate PMem optimized CacheLib with SecondaryCache in pmem/pmem-rocksdb-plugin#6