[ruby-core:105075] [Ruby master Bug#9659] crash in FIPS mode after unchecked algo->init_func failure
From:
"jeremyevans0 (Jeremy Evans)" <noreply@...>
Date:
2021-08-26 22:41:22 UTC
List:
ruby-core #105075
Issue #9659 has been updated by jeremyevans0 (Jeremy Evans).
Status changed from Feedback to Closed
The crash in FIPS mode was fixed 7 years ago. The reason this issue was reopened 7 years ago was because of digest/md5 using the low-level OpenSSL API instead of the EVP API. The digest extension removed the OpenSSL engine in commit:2e601c284c9b61c286aa031d91e5198c17b44f00, so the low-level OpenSSL API is no longer used (openssl/ossl_digest.c uses the EVP API). So this can be closed.
----------------------------------------
Bug #9659: crash in FIPS mode after unchecked algo->init_func failure
https://bugs.ruby-lang.org/issues/9659#change-93481
* Author: jared.jennings.ctr (Jared Jennings)
* Status: Closed
* Priority: Normal
* ruby -v: ruby 1.8.7 (2011-06-30 patchlevel 352) [x86_64-linux]
* Backport: 2.0.0: DONTNEED, 2.1: DONTNEED
----------------------------------------
This is just like #4944, but in the `digest` extension instead of the `openssl` extension.
On my host, which is configured for FIPS 140-2 compliance (this is a U.S. Government security standard), OpenSSL refuses to perform an MD5 checksum. It indicates this refusal when the digest algorithm initialization function is called: this function returns a 0 indicating failure instead of a 1 indicating success. But it's just a bunch of arithmetic; how can it fail? So the return code is ignored. But if the initialization fails, and we go on trying to use the algorithm, the Ruby interpreter crashes:
~~~
$ OPENSSL_FORCE_FIPS_MODE= ruby -rdigest -e "puts Digest::MD5.hexdigest('hi')"
md5_dgst.c(78): OpenSSL internal error, assertion failed: Digest MD5 forbidden in FIPS mode!
Aborted (core dumped)
~~~
The digest extension, in the `rb_digest_base_alloc`, `rb_digest_base_reset`, and `rb_digest_base_finish` functions, is ignoring the return code of `algo->init_func`. If OpenSSL is present at build time, `algo->init_func` works out to be the `MD5_Init` function from OpenSSL. This function, according to its man page, returns a 1 for success or 0 for failure.
I see the problem under Ruby 1.8.7 as patched by Red Hat; I can't easily build the trunk on my system, but it looks like in r43668 the return value still isn't being checked in these three places:
* source:ext/digest/digest.c@43668#L551
* source:ext/digest/digest.c@43668#L589
* source:ext/digest/digest.c@43668#L627
---Files--------------------------------
002-builtin-indicate-digest-failure.patch (10.4 KB)
001-detect-digest-failure.patch (2.12 KB)
003-digest-openssl-md5-use-evp-api.patch (1.8 KB)
--
https://bugs.ruby-lang.org/
Unsubscribe: <mailto:[email protected]?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-core>