irb(main):003:0> OpenSSL::PKey::DH.new(128)
OpenSSL::PKey::DHError: BN lib
from (irb):3:in initialize' from (irb):3:in new'
from (irb):3
from /bin/irb:12:in `'
irb(main):004:0>
Could you please provide better exception message? While it is fine that DH.new fails with short key, it is not obvious from the message what is the reason. Thank you.
I'm not sure whether this is possible at all - the message being generated is what OpenSSL itself generates at this point. I'd have to check if there is a reliable way to detect whether we are in FIPS mode or not. Still, I'd prefer if OpenSSL itself provided a better exception message.
Status changed from Assigned to Third Party's Issue
This is indeed a third party issue. The exception message (or better, the lack thereof) is generated by OpenSSL. They're fine for RSA, DSA and EC ("key too short"), but miserable for DH, agreed.
I really wouldn't want to start to improve OpenSSL error messages in the Ruby extension, that's not where this belongs IMHO. We could try to open an issue on the OpenSSL tracker, though.
Closing as TPI for now, please feel free to reopen if your opinions differ from mine.