File-system and Block-layer Encryption:

Theory, Practice, and Improvement

Weigang Li
Wenqian Yu

Data Center Group

1
Intel
2017 Storage Developer Corporation
Conference. © Intel All Rights Reserved.
Agenda
 Data at-rest encryption
 File-system encryption
 Layered: eCryptfs, EFS
 Native: EXT4, ZFS
 Full-disk encryption
 dm-crypt
 Linux Kernel Crypto API
 Crypto Hardware Acceleration
 Benchmark and Optimization
 Key management – Linux keyring, LUKS
 Summary

2
2017 Storage Developer Conference. © Intel All Rights Reserved.
Data at-rest Encryption
Application-level • Flexible to define security strategy, algorithm, key
Database
Flexibility

management, etc.
Encryption
• Application dependent

eCryptfs EFS
• File based key management
File System • Transparent to application
Encryption • Support multiple users / keys
EXT4 ZFS

• One key for whole disk (volume)

Transparency

Full Disk Encryption dm-crypt

(FDE) software • Encrypt everything on disk
LUKS • OS-agnostic, Block-layer
Software

Hardware
Self Encrypting Drive • One Data Encryption Key (DEK) to encrypt the
(SED) whole disk, protected by Authentication Key (AK).
• Hardware based, secure key is kept in hard-drive

3
2017 Storage Developer Conference. © Intel All Rights Reserved.
Data at-rest Encryption
Application-level • Flexible to define security strategy, algorithm, key
Database
Flexibility

management, etc.
Encryption
• Application dependent

Our focus today

eCryptfs EFS
• File based key management
File System • Transparent to application
Encryption • Support multiple users / keys
EXT4 ZFS

• One key for whole disk (volume)

Transparency

Full Disk Encryption dm-crypt

(FDE) software • Encrypt everything on disk
LUKS • OS-agnostic, Block-layer
Software

Hardware
Self-encrypting Drive • One Data Encryption Key (DEK) to encrypt the
(SED) whole disk, protected by Authentication Key (AK).
• Hardware based, secure key is kept in hard-drive

4
2017 Storage Developer Conference. © Intel All Rights Reserved.
File System Encryption

5
2017 Storage Developer Conference. © Intel All Rights Reserved.
File system encryption
Application  File-system level encryption can be
implemented at different level:
 Fuse-based FS in User space
syscalls Fuse Enc/Dec
user  Layered FS on top of native FS
kernel
 In native FS, better performance
File System Layered
Enc/Dec
 Transparent to application.
 Per-file encryption and key management.
Enc/Dec Native

Block Layer FEK File File

hardware

Disk Enc EFEK

Master
key

6
2017 Storage Developer Conference. © Intel All Rights Reserved.
eCryptfs $ mount -t ecryptfs /secret /secret
Select key type to use for newly created files:
1) tspi
eCryptfs (upper)
2) openssl IV
IV
3) passphrase
 eCryptfs = Enterprise 4) pkcs11-helper
(page
(pagebased)
based)
Selection: 3
Cryptographic Filesystem. Passphrase: xxx
Select cipher: xxx R
 Layered file system. 1) aes: blocksize = 16; min keysize = 16; max keysize = 32
2) blowfish: blocksize = 8; min keysize = 16; max keysize = 56 N FEK Plaintext
3) des3_ede: blocksize = 8; min keysize = 24; max keysize = 24 G Page
 In Linux kernel since version 4) twofish: blocksize = 16; min keysize = 16; max keysize = 32
5) cast6: blocksize = 16; min keysize = 16; max keysize = 32 Extent
2.6.19. 6) cast5: blocksize = 8; min keysize = 5; max keysize = 16
Selection [aes]: 1
 Mount eCryptfs on top of a Select key bytes:
1) 16 Per page encryption
directory to protect it. 2) 32
3) 24
 Un-mount eCryptfs, the files in Selection [16]: 1
Enable plaintext passthrough (y/n) [n]: Enc Enc
lower FS are encrypted and Enable filename encryption (y/n) [n]:
Attempting to mount with the following options:
un-readable. ecryptfs_unlink_sigs
ecryptfs_key_bytes=16
ecryptfs_cipher=aes
 The Encrypted FEK (EFEK) is ecryptfs_sig=aa20c2d38cf280d5 FEKEK
Mounted eCryptfs
stored in the lower FS. auth-tok
EFEK Ciphertext
keyring Page
KDF

key source
Native FS (lower)
Ref: http://ecryptfs.sourceforge.net/ecryptfs.pdf
7
2017 Storage Developer Conference. © Intel All Rights Reserved.
EFS
 EFS = Encrypting File
System
 Filter driver layered on
top of NTFS

Source: https://technet.microsoft.com/library/bb457116.aspx#EJAA

8
2017 Storage Developer Conference. © Intel All Rights Reserved.
EFS (Cont.) Plaintext

Encryption
 Combination of public key RNG FEK
(RSA) and symmetric key
encryption (3DES, AES).
Ciphertext
 FEK is protected by user’s RSA
pub-key
public key. Encryption
ENC
 EFEK is decrypted by user’s EFEK

private key to unlock the DEC

encrypted file. pri-key RSA
Decryption On-disk
 FEK can be re-encrypted by
other’s pub-key to share the
encrypted file with other user FEK Decryption
w/o re-encrypting the file
content.
Plaintext

9
2017 Storage Developer Conference. © Intel All Rights Reserved.
EXT4 encryption
$ mkfs -t ext4 /dev/nvme1n1
RNG $ tune2fs -O encrypt /dev/nvme1n1
IV (page based)
$ mount /dev/nvme1n1 /mnt/ext4
Plaintext $ e4crypt add_key /mnt/ext4
Page Enter passphrase (echo disabled): xxx
FEK
inode Key with descriptor [775a2062517e439c] applied to /mnt/ext4.
nonce $ keyctl list @s
2 keys in keyring:
setkey
453787240: --alswrv 0 65534 keyring: _uid.0
Enc 1024073693: --alsw-v 0 0 logon: ext4:775a2062517e439c
Enc Ciphertext
Page
 In Linux kernel since version 4.1.
 Per-directory Master key.
Master
key inode  Per-file encryption key (FEK, derived
nonce
Key-ring from master key and nonce).
On-disk
 Support AES-256-XTS.

e4crypt / keyctl (Linux Kernel 4.12)

10
2017 Storage Developer Conference. © Intel All Rights Reserved.
ZOL encryption
$ zpool create -o ashift=13 testpool /dev/..
$ zfs create -o encryption=aes-128-ccm
IV testpool/tank
R Enter passphrase for 'tank': xxx
N Salt K Plaintext Enter again: xxx
G Block $ zfs get all testpool/tank
D FEK
Master F testpool/tank encryption aes-128-ccm local
key
IV

Enc  ZOL = ZFS On Linux

Keystore Ciphertext
Block
 http://zfsonlinux.org/
Wrapping
key Enc  ZOL encryption is not in mainline yet
IV  PR: https://github.com/zfsonlinux/zfs/pull/5769
KDF Salt  Support AES-CCM, AES-GCM
MAC
passphrase blkptr_t
file
uri Seal master key E(master key)
IV

key source MAC

On-disk
11
2017 Storage Developer Conference. © Intel All Rights Reserved.
Block Layer Encryption

12
2017 Storage Developer Conference. © Intel All Rights Reserved.
Block layer encryption
 Encrypt everything on the disk – one
Application key for whole disk (volume).
user  Hides file and directory information,
syscalls
kernel such as name and size.
File system
 OS-agnostic.

Block layer
Master
Encrypt Decrypt volume volume
key

hardware Wrapped
Enc
Disk key
Password

13
2017 Storage Developer Conference. © Intel All Rights Reserved.
dm-crypt $cryptsetup luksFormat -c aes-xts-plain64 -s 512 / dev/nvme3n1
$cryptsetup luksFormat -c aes-cbc-essiv:sha256 -s 256 /
dev/nvme3n1
Are you sure? (Type uppercase yes): YES
IV
(sectorbased)
(sector based) Enter passphrase: xxx
Plaintext Verify passphrase: xxx
Sector $cryptsetup open --type luks /dev/nvme3n1 dm-crypt-disk
R
Master (512B) Enter passphrase for /dev/nvme3n1: xxx
N
key $ lsblk
G
nvme3n1 259:1 0 745.2G 0 disk
└─dm-crypt-disk 253:3 0 745.2G 0 crypt
$ mkfs -t ext4 /dev/mapper/dm-crypt-disk
$ mount /dev/mapper/dm-crypt-disk /dm-crypt-disk/
setkey $ df -T
Enc Enc /dev/mapper/dm-crypt-disk ext4 1% /dm-crypt-disk
$ umount /dm-crypt-disk
$ cryptsetup close dm-crypt-disk

passphrase  In Linux kernel since version 2.6.

key-file
Key Slot
Ciphertext  Per-sector encryption.
Sector
(512B)  Support AES-XTS, AES-CBC.
LUKS HDR
 Single master key for whole disk.
cryptsetup
 Using LUKS for key management - 8 key
On-disk
slots.

14
2017 Storage Developer Conference. © Intel All Rights Reserved.
Linux Kernel Crypto API

15
2017 Storage Developer Conference. © Intel All Rights Reserved.
Linux Kernel Crypto API
Linux Kernel sub-system $ cat /proc/crypto:
 Data Transform
 Symmetric key ciphers: skcipher eCryptfs Ext4 …

 Asymmetric key ciphers: IPsec dm-crypt

akcipher
API call Callback
 AEAD ciphers
 Message digest Linux Kernel Crypto API
 Random number generation
Submit Callback
 Compression
Memory
 Crypto driver is registered and Engine Driver
selected based on its priority.
sg_list
Request Response
 Support asynchronous operation for Page

hardware acceleration.
DMA

Crypto Hardware

16
2017 Storage Developer Conference. © Intel All Rights Reserved.
Cryptographic Hardware
Acceleration

17
2017 Storage Developer Conference. © Intel All Rights Reserved.
 Security Benchmarks
Intel® QuickAssist Technology 110

100

90

80

70

60

50

40

30

20

10

0
RSA 2K IPSec SSL
Decrypt 1 Forwarding 3 WebProxy 1
(kOps/s) (Gbps) (Gbps)

Software-based OpenSSL with Intel® QAT

Big Data Benchmarks with Compression 2

SW Snappy Compression
Intel® QAT Compression

99
Hadoop run time
1. NGINX* and OpenSSL* connections/second. Conducted by Intel Applications Integration Team. Claim is actual performance measurement. reduced significantly
Intel® microprocessor. Processor: Intel® Xeon® processor Scalable family with C6xxB0 ES2
Performance tests use cores from a single CPU, Memory configuration:, DDR4–2400. Populated with 1 (16 GB) DIMM per channel, total of 6 DIMMs
Intel® QuickAssist Technology driver: QAT1.7.Upstream.L.0.8.0-37 Fedora* 22 (Kernel 4.2.7) BIOS:

87
PLYDCRB1.86B.0088.D09.1606011736
3. Cloudera* 5.4.2 with Snappy* Software vs. Intel® QuickAssist Technology hardware solution. Conducted by Intel Applications Integration Team.
Claim is actual performance measurement. Intel® Xeon® processor E5-2699 v4 (56 cores enabled) 256 GB DDR4 1.6 TB NVMe SSD 1 Intel® C6xxx-based card (24x)
10 Gbps CentOS* 6.7 w/ 2.6.32 kernel Cloudera* 5.4.2
QAT driver 0.9.1 Snappy* 1.1.2 (popular, fast compression codec)
One NameNode Eight DataNodes 10 Gbps network
2- 24 Core Intel(r) Xeon Scalable Platform -SP @1.8GHz, Single (UP) Processor configuration. Intel(r) C627 PCH with crypto acceleration capability (in x16 mode) Neon City platform. DDR4 2400MHz RDIMMs 6x16GB(total
96 GB), 6 Channels, 1 x Intel® Corporation Red Rock Canyon 100GbE Ethernet Switch in the x16 PCIe slot on Socket 0. 8 cache ways allocated for DDIO.
TERASORT TIME IN MINUTES LOWER IS BETTER

Intel® QuickAssist Technology integrates hardware acceleration for compute intensive workloads
Such as Bulk Cryptography, Public Key Exchange & Compression on Intel® Architecture Platforms
18
2017 Storage Developer Conference. © Intel All Rights Reserved.
Gen 2 Intel® QuickAssist Technology with Intel® Key Protection Technology (KPT)

Without Intel® KPT With Intel® KPT

DRAM DRAM

z
LBG KPT
Intel® C627 Chipset
with Intel® QAT
Intel® QuickAssist
Intel®
Unwrapping (Decrypt)
Technology PTT Secure
Channel

• Private key exposes in clear text • Private key is wrapped (encrypted)

• Key is not protected and unsafe • Key is protected and safe
• Subject to attacks • NOT subject to attacks

Private Key (PK) Symmetric Wrapping Key (SWK) Wrapped Private Key (WPK)

19
2017 Storage Developer Conference. © Intel All Rights Reserved.
1
Cryptographic hardware program model
Computing Wait Wait IO  Synchronous:
CPU  For high performance “single
Offload Complete Offload Complete
stream” accelerator
Hardware
 Simple program model
Engine

sync mode  Asynchronous:

 For multi-engines hardware
Computing Swap IO
 High throughput
CPU
Offload Complete  Complex control flow
Hardware
Engine

Engine

Engine

async mode

20
2017 Storage Developer Conference. © Intel All Rights Reserved.
Ext4 encryption with hardware accelerator
Extent Callback  Synchronous call causes
Page n performance drop!
Sequential
wakeup

Page 2 • add the encrypted page to

bio_vec Throughput(MB/s)
Page 1 • encrypt the next page
1000
800
600
all pages are encrypted
Linux Kernel Crypto API HW is not
400
fully utilized!
200
Submit BIO 0
Extent off s/w encryption qat-sync
Crypto Page vec encryption
Engine vec
dd if=/dev/zero of=/mnt/ext4/testfile bs=1M count=10000 oflag=sync
vec
Page
Intel® microprocessor. Processor: Intel® Xeon® E5-2699 v4 @ 2.20GHz @88 HT cores, Memory
bio_vec configuration:, DDR4–2400. Populated with 1 (16 GB) DIMM per channel, total of 4 DIMMs. 1X P3700
NVMe. Intel® QuickAssist Technology DH8950, in-tree driver in kernel 4.12

Page

21
2017 Storage Developer Conference. © Intel All Rights Reserved.
ZOL encryption with hardware accelerator
FIO Sequential Write to different number of NVMe drives

Throughput(MB/s) CPU utilization @88 HT cores (%)

5000 45
40
4000 35
30
3000 CPU utilization reduced
25
2000 20
15
1000 10
5
0
0
1 x nvme 2 x nvme 3 x nvme
1 x nvme 2 x nvme 3 x nvme
off s/w qat
off s/w qat
Intel® microprocessor. Processor: Intel® Xeon® E5-2699 v4 @ 2.20GHz @88 HT cores, Memory
 Block size (128KB) is “big enough” for one offloading. configuration:, DDR4–2400. Populated with 1 (16 GB) DIMM per channel, total of 4 DIMMs. 3X P3700
NVMe. Intel® QuickAssist Technology driver: QAT1.6 v2.6. Cent OS* 7.2 (Kernel 3.10)
ZOL 0.7.0 with PR (https://github.com/zfsonlinux/zfs/pull/5769). FIO-2.1.2, sequential write, 16 threads
 ZIO pipeline drives compression, checksums, data per NVMe disk.
ZOL encryption algorithm: AES-CCM-128

redundancy, and encryption, etc., with a pool of

threads (taskq) to improve performance.
 Hardware accelerator is fully utilized.

22
2017 Storage Developer Conference. © Intel All Rights Reserved.
Hardware offloading cost
Intel® QAT (DH8950) offloading cost
vs. Software cost (Intel® Xeon® E5-2699 v4 )
 Offloading cost:
18000  Create request descriptor
16000
 Interrupt handling, response
14000
polling
Cost (CPU Cycles)

12000
10000  Hardware* offloading cost is
8000
almost consistent: ~1600-1800
6000
4000
cycles.
2000  Choose right buffer size to
0
0 2000 4000 6000 8000 10000 12000 14000 16000 18000
get best performance with
Buffer Size(Bytes) reasonable offloading cost.
qat_aes_xts xts-aes-aesni  Software cost varies by buffer
Intel® microprocessor. Processor: Intel® Xeon® E5-2699 v4 @ 2.20GHz @88 HT cores, Memory
configuration:, DDR4–2400. Populated with 1 (16 GB) DIMM per channel, total of 4 DIMMs.
size.
* Intel® QuickAssist Technology DH8950, in-tree kernel driver.
Benchmark code calls Kernel Crypto API (skcipher), compute average CPU time with 1000x API calls.
Cycle count as measured at Kernel Crypto (skcipher) API for AES-XTS algorithm.

23
2017 Storage Developer Conference. © Intel All Rights Reserved.
dm-crypt with hardware acceleration
dm-crypt Page
Page(4KB)
(4KB)  dm-crypt calls Kernel Crypto API for data
(for each page) Sector
512B
Sector
512B
encryption, it can integrate with a
iv iv hardware accelerator seamlessly.
When 8 sectors
all done
Sector
 asynchronous call: sending 8x requests
512B
for one 4KB page in-parallel, no wait.
iv
 But: offloading cost is high -
x8 x8
 Encrypting a 4KB page  8x
API call Callback

Page
encryption requests.
Page(4KB)
(4KB)
 Encrypting 1MB data  2048x
Kernel Crypto API Sector Sector encryption requests.
512B 512B
Driver
Sector
Submit 512B
request x8 Callback x8

Accelerator Submit BIO

24
2017 Storage Developer Conference. © Intel All Rights Reserved.
dm-crypt optimization  Encrypt 8 sectors (sg_list) in a single call.
 Offloading cost is reduced greatly.
Page  Kernel Crypto API or accelerator hardware to
dm-crypt Page(4KB)
(4KB)
compute IV for each sector.
(for each page) Sector
512B
Sector
iv 512B Throughput (MB/s)
iv 900
Sector
512B
sg_list 800
iv 700
600
API call x1 Callback x1
500
400

Page 300
Page(4KB)
(4KB)
200
Kernel Crypto API
100
Sector Sector
Driver 512B 512B 0
w/o optimization with optimization
Submit Sector dd if=/dev/zero of=/dev/mapper/dm-c bs=1M count=10000 oflag=sync
512B
request x1 Callback x1 Note: experimental result, patch is not in upstream kernel yet.
Intel® microprocessor. Processor: Intel® Xeon® E5-2699 v4 @ 2.20GHz @88 HT cores, Memory configuration:, DDR4–2400. Populated
with 1 (16 GB) DIMM per channel, total of 4 DIMMs. 1X P3700 NVMe. Intel® QuickAssist Technology DH8950, in-tree driver in kernel
4.12
Accelerator Submit BIO

25
2017 Storage Developer Conference. © Intel All Rights Reserved.
Key Management

26
2017 Storage Developer Conference. © Intel All Rights Reserved.
Key management – keyring
keyctl
 In-kernel key management and
user
retention facility:
kernel
http://man7.org/linux/man-
pages/man7/keyrings.7.html
keyring

request key

setkey
Kernel
Linux Kernel Crypto API
sub-system

27
2017 Storage Developer Conference. © Intel All Rights Reserved.
Key management - LUKS
# cryptsetup luksDump /dev/nvme3n1
LUKS header information for /dev/nvme3n1
Version: 1
Cipher name: aes
Master Disk Format Cipher mode: xts-plain64
Hash spec: sha1
key Payload offset: 4096
slot#0 MK bits:
MK digest:
512
ed 36 e3 09 00 72 ea ba 24 04 6e 11 7e 69 1a 87 5a d1 32 57
MK salt: bc 41 51 8e e0 90 65 20 70 6d ef 2f 3f 80 06 a7…
slot#2 MK iterations: 48000
UUID: 0f6c5194-b299-4f3e-9f54-17dc67a62f04
Key Slot 0: ENABLED
Iterations: 192480
Salt: 62 4b 4d 54 b1 d5 0c 44 c2 a9 a0 d3 03 36 59 78…
Key material offset: 8
slot#7 AF stripes:
Key Slot 1: DISABLED
4000

Key Slot 2: DISABLED

Key Slot 3: DISABLED
Encryption Key Slot 4: DISABLED
Key Slot 5: DISABLED
Encrypted Key Slot 6: DISABLED
Key Slot 7: DISABLED
Data
 Linux Unified Key Setup, it is a specification (on-disk
format) for disk encryption.
Password
 Implemented in cryptsetup utility + dm-crypt (kernel
module) for disk encryption.
 Master key to encrypt / decrypt data.
Plaintext  Password to encrypt / unlock the master key.
Data  Support 8 key slots.

28
2017 Storage Developer Conference. © Intel All Rights Reserved.
Memory-based key management
 If data encryption / decryption is done in
Enc / Dec host CPU, keys (FEK, master key,
Library private key…) have to be exposed in
memory
 Cold boot (memory dump) attack
CPU
 Heartbleed-like attack
Memory  Possible solution
 SGX Enclave
 Total Memory Encryption (TME)
 Debug-register-based or cache-
based key storage
29
2017 Storage Developer Conference. © Intel All Rights Reserved.
HSM-based key management
Memory
 HSM = Hardware Security Module Crypto
Library
 KEK = Key Encryption Key Wrapped-key

 Session key (e.g., FEK) is generated by HSM

and wrapped by KEK, the wrapped-key can
CPU
send back to host memory. Buffer
 The wrapped-key is a “key handler” to request
HSM service.
 Key storage and data encryption / decryption Crypto Enc / Dec
Hardware Engine
are all kept inside device.
 Clear key material never leaves device.
Key hierarchy tree Key Gen
KEK

Key provision Root Key HSM

30
2017 Storage Developer Conference. © Intel All Rights Reserved.
Summary
 File-system and full-disk encryption are important software
technologies to implement data-at-rest encryption solution.
 Data encryption at different layer has its own pros and cons.
 Data encryption software shall consider how to best utilize the
hardware crypto accelerator underneath in order to get best
performance, e.g., buffer size, sync vs. async, offloading cost, etc.
 Besides data encryption, secure-key protection is even more
important for persistent data storage.
 Hardware based key management is a possible solution to protect
key leakage.

31
2017 Storage Developer Conference. © Intel All Rights Reserved.
References
 http://ecryptfs.sourceforge.net/ecryptfs.pdf
 https://zfs.datto.com/slides/caputi.pdf
 https://technet.microsoft.com/library/bb457116.aspx#EJAA
 http://kernsec.org/files/lss2014/Halcrow_EXT4_Encryption.pdf
 www.intel.com/content/www/us/en/embedded/technology/quickassist/overview.html

32
2017 Storage Developer Conference. © Intel All Rights Reserved.