Problem/Motivation
Right now 8.2.x and 8.3.x are using the paragonie/random_compat https://github.com/paragonie/random_compat 1.x branch in core.
But for modules like real_aes it has becomes an issue because it needs the 2.x branch and trying to install it using composer results in a conflict.
background
in 8.0.x we used Crypt::randomBytes, which was the same as drupal 7 drupal_random_bytes, and not the paragonie/random_compat library. it used openssl (because we thought it was equally good and faster, but turns out it was not equally good).
8.1.x we used version 1 of the library (v2 didn't exist then) (which had openssl as the final acceptable implementation)
8.2.0 was released http://cgit.drupalcode.org/drupal/commit/?id=fb83de52e58e8fb3303de7aa983...
in 8.2.x if we just update the library to v2, it will break some people's stuff (if the only generator the system had was openssl).
we can't do that because of our Backwards Compatibility promise.
8.3.x is in dev
Proposed resolution
Upgrade random_compat to latest version (v2.0.2),
for 8.2.x also add some fallback (for the case where openssl was used, and have the fallback be the drupal_random_bytes() logic because its worst case behavior is better than the worst case behavior of openssl)
8.3.x
a. have an update requirement, and drop the drupal_random_bytes (so we dont carry that code for the whole 8.x lifetime)
But that is a bad idea cause we said 8 was going to be backward compatible
b. use the same/similar solution in 8.3.x, or until php7 is required (like if php 5 is unsupported and has security problems)
So. b. (like 8.2.x)
Remaining tasks
User interface changes
Not really, an extra install requirement or a message on the site status page.
API changes
No
Data model changes
No
| Comment | File | Size | Author |
|---|---|---|---|
| #119 | random_compat-2763787-119.patch | 443 bytes | klausi |
| #104 | interdiff-2763787-104.txt | 829 bytes | tuutti |
| #104 | 2763787-104.patch | 10.37 KB | tuutti |
| #102 | increment-2763787-102.txt | 1.55 KB | pwolanin |
| #102 | 2763787-102.patch | 10.38 KB | pwolanin |
Comments
Comment #2
therealssj commentedComment #3
nerdsteinIts encouraging that the tests passed. However, we should quantify differences between the versions and try to isolate the potential impact to Drupal
Comment #4
talhaparacha commentedDrupal 8 core uses random_compat library when Crypt::randomBytes is called for generating a string of highly randomized bytes. The latest version of random_compat, i.e. 2.x, does not use OpenSSL as one of its sources. The removal is due to the associated unreliability of openssl_random_pseudo_bytes() as can be seen here:
openssl_random_pseudo_bytes() is not cryptographically secure
But the openssl_random_pseudo_bytes() removal actually got done in the random_compat release 1.3.0. So updating D8 core from 1.x to 2.x shouldn't cause any problems.
Instead of marking the issue fixed, let's have someone else review it too. For reference: random_compat CHANGELOG.
Comment #5
alexpottSee https://github.com/FriendsOfPHP/security-advisories/pull/146 for a discussion about the security implications.
So one issue that we might face is that Drupal 8 windows installs might break :(
Comment #6
alexpottHere's a patch that upgrades the library properly... the procedure is:
Comment #7
dawehnerFor maximum library compatibility we should IMHO use
"paragonie/random_compat": "~1.0|~2.0"Their internal API didn't changed, its simply that openssl got dropped. Let's assume you have a library which depends on 1.0, it will no longer be installable. I think in an ideal world both this library and core specifies
~1.0|~2.0.Comment #8
alexpott+1
Comment #9
dawehnerLet's go with
^1.0|^2.0instead ...Comment #10
nerdsteinAdded an updated patch for the testbot to munch on
Comment #11
nerdsteinMarking as needs review
Comment #13
nerdsteinLet's try this again...
Comment #14
talhaparacha commentedThe patch #13 looks good to me as per #7 & #9.
But the composer documentation specifies that "a double pipe ( || ) will be treated as a logical OR" and we're using a single pipe ( | ) in #13. I'm not sure whether it is done purposely or not, hence not marking this issue as RTBC.
Comment #15
dawehnerWhat I think is weird is that the hash in the composer.lock is not changed:
Comment #16
nerdsteinStill learning how this is set up in core... I ran `composer update` from the Drupal root. The hash changes were not applied. Further, the other changes represented in composer.lock from both alexpott and I's last patch now seems to be missing.
After updating core/composer.json, what steps do I need to do?
Comment #17
alexpott@dawehner the hash does not change when the version doesn't change - it's part of the fun of the merge plugin. That is to say, I think this only changes when /composer.json changes - not core/composer.json changes.
Comment #18
nerdsteinBased on #17, is the patch provided in #13 suitable? I'm happy to make whatever changes are needed.
Comment #19
dawehnerOh yeah you are right @alexpott
Comment #20
alexpottSo the one thing that gives me pause is #2768953: Prevent insecure Guzzle from being installed when using composer to manager your project dependencies - there we decided to prevent insecure Guzzle regardless of interoperability - is this not the same thing?
Comment #23
nerdsteinI don't have a ton of confidence regarding my knowledge of how core interacts with these external projects. But, if I read the related issue in #20, it would suggest we should just use 2.0 and move forward.
Is this correct? If so, I can work on another patch
Comment #24
nerdsteinOne other take on this, is that the upgrade of random_compat doesn't appear to be a security-related upgrade. This is just supporting compatibility with other projects that can use the new version. So, isn't it appropriate for us to support 1.x or 2.x? If so, the patch I uploaded should be fine if testbot stops barfing.
Comment #25
rlhawkRight, and see comment #5 regarding security implications.
Comment #28
traviscarden commentedLooks like we just need a reroll of the latest patch?
Comment #31
nerdsteinMarking as needs review now that #28 is passed
Comment #32
dawehnerI'm not 100% sure whether supporting both version is the right thing to do, but to be honest, both are kinda the same though anyway.
Comment #36
rlhawkHere's a re-rolled patch.
Comment #37
rlhawkComment #38
slasher13remove unrelated changes
Comment #39
rlhawkThanks. I'm not sure how those sneaked in there.
Comment #40
dawehnerBack to RTBC
Comment #42
rlhawkYet another patch re-roll.
Comment #43
pwolanin commentedI'm not sure I agree with leaving 1.0 in composer.json here.
If the API is unchanged, any dependency that required 1.0 could be upgraded to 2.0 trivially?
Maybe it doesn't matter since if using PHP 7 you will always use the built-in?
Comment #44
pwolanin commentedI also think this is a bug fix that should be applied to 8.2.x also
Comment #45
rlhawkNot allowing 1.x seems fine with me, but I'm not clear on all of the possible implications within other parts of core.
Comment #46
dawehnerWell, one reason to maybe leave 1.0 in there is that contrib modules / their dependencies might use 1.0 in their composer.json files. In case they do, we would have a conflict here,
much like we have it in core now.
Comment #47
klausiLooks good! We can discuss changing the version constraint in a follow-up, the current patch is most permissible and should not cause problems for contrib.
Comment #48
catchI think it's fair to leave 1.0 and 2.0 for now, since it's impossible until the patch lands to require 2.0 in a module, making it impossible not to is a hard break, even if composer.json is considered @internal.
We could have a follow-up (8.4.x?) to require the 2.0 branch maybe, by then there's plenty for modules to update their dependencies.
Comment #51
catchCommitted/pushed to 8.3.x and cherry-picked to 8.2.x. Thanks!
As a library update this is rc-eligible (especially since it's compat rather than providing a specific API).
Comment #52
dawehnerThanks a ton for cherry-picking to 8.2!!
Comment #53
catchComment #54
naveenvalechaProbably this issue would also be the part of the release note.
Comment #55
eric_a commentedGiven that drupal/core requires drupal/core-utility we should quickly follow-up to update its
paragonie/random_compat": "~1.0"requirement declaration. (core/lib/Drupal/Component/Utility/composer.json)(Or revert.)
We really need an issue to create a test to prevent this type of composer breakage.
Comment #56
maximpodorov commentedMy site no longer works after upgrading to drupal-8.2.0-rc2.
Exception: There is no suitable CSPRNG installed on your system in vendor/paragonie/random_compat/lib/random.php on line 185 #0 core/lib/Drupal/Component/Utility/Crypt.php(31): random_bytes(55)
Maybe the problem is random_bytes_openssl.php file was removed in paragonie/random_compat.
Comment #57
catchReverted from both branches for now.
Comment #60
dawehner@maximpodorov
Thanks a lot to catch this problem before the actual release!
Comment #61
alexpott@maximpodorov can you describe your system? Ie. operating system and php version.
Comment #62
maximpodorov commentedCentOS 7
PHP 5.6.26
All extensions required for D8 (https://www.drupal.org/docs/7/system-requirements/php) are installed.
Comment #63
maximpodorov commentedThe possible explanation is here:
https://github.com/paragonie/random_compat/issues/111
This new version of random_compat requires that PHP's open_basedir restriction must permit to read /dev/urandom.
Comment #64
alexpott@maximpodorov - any chance you could read https://github.com/paragonie/random_compat/issues/109 and paste the output of
here... thanks!
Comment #65
maximpodorov commentedarray(7) {
["version"]=> string(6) "5.6.26"
["open_basedir"]=> string(34) "my secret path to site :)"
["safe_mode"]=> bool(false)
["directory separator"]=> string(1) "/"
["mcrypt_create_iv exists"]=> bool(false)
["openssl_random_pseudo_bytes exists"]=> bool(true)
["readable /dev/urandom"]=> bool(false)
}
Comment #66
maximpodorov commentedLooks like it's time to switch from open_basedir to chroot. Or switch to PHP7 finally.
Comment #67
eric_a commentedDoesn't #65 in combination with #4 suggest that #56 would happen on 1.3.0 also?
Comment #68
pwolanin commentedLooks like 1.3.0 will use openssl and openssl_random_pseudo_bytes exists so it won't hard fail?
So maybe we should do the same as Drupal 7 patch and fall back to the hash generator?
Comment #69
yesct commentedThe direction here is still under discussion and the issue summary is not clearly identifying what novice task might be appropriate (yet). so removing the tag for now. https://drupal.org/core-mentoring/novice-tasks
Comment #70
MixologicFor reference, this is what the maintainer of this library has to say about versioning here:
https://twitter.com/ParagonIE/status/776096550103625728
Comment #71
pwolanin commentedOr for 8.2 we should fallback to openssl on exception, and re-commit to 8.3.x as is (after fixing core/lib/Drupal/Component/Utility/composer.json)?
Comment #72
yesct commentedtalking to @pwolanin at the drupalcon dublin sprint. these are notes from the conversation.
1.
there is an actual fix hinted at in #55
and that is that we should actually specify which version to match the change this issue is making to upgrade the version
---------------------
now something more complicated. what to do.
2.
for openssl there is evidence that people *are* getting repeated "random" strings. (this could make things less secure)
in the code in drupal 7 is/was (see the backport in #2550519: Crypt::randomBytes()/drupal_random_bytes() doesn't actually return cryptographically secure random bytes which is rtbc (right now)) not possible to get repeat random strings because it concats many "random" things *and* includes microtime, which unless the request is at exactly the same time will always give a different string.
in drupal_random_bytes()
So for 8.2.x we should do the upgrade to v2.0.2, and add a fallback on exception to something like drupal_random_bytes(). That would have a requirement error on the status page, so gives people on 8.2 a chance to notice they will need to improve their system, but keep it working
For 8.3.x we could just do the upgrade to v2.0.2 ... but that breaks the BC promise of 8.x
Comment #73
maximpodorov commented+1 for fallback + site status warning.
Comment #74
pwolanin commentedDiscussed in person with alexpott and think we concluded again that the path forward described by YesCT is the best option for BC
Comment #75
pwolanin commentedHere's a patch pulling in basically the same code from #2550519: Crypt::randomBytes()/drupal_random_bytes() doesn't actually return cryptographically secure random bytes
Comment #76
pwolanin commentedalexpott suggest putting a function inside the Crypt namespace for a test, since a function inside the namespace will be used in preference.
Comment #77
pwolanin commentedHere's with a unit test.
Comment #78
yesct commentedI will take a look at the patch now. (others can also. :) )
Comment #79
yesct commentedNotes from looking at the test interdiff.
typically, a blank line between the use (or namespace) and the comment block.
Forces throwing an exception in the test environment.
Test environments have the fallbacks, so force an exception so we can test the exception. Also, for php 7, this needs to be in the same namespace that the crypt class is in, so it gets preferentially called over the php 7 built in function.
Add type and description...
https://www.drupal.org/node/1354#param
@param int $count
Integer just to match the global function definition.
type the function (public, protected, or private)
https://www.drupal.org/node/608152#visibility
$count passed to exception, just so $count is "used" and IDEs dont complain... hm. ok. That would be a pretty silly comment to add to the code? Just noting it on the issue here.
Tests random byte generation fallback exception situations.
this is here to make sure that the overriding function is the one that was called, and not the global one.
we could add comments to explain that, and meet the doc block requirements.
Tests if the call to random_bytes() throws an exception, that the crypt::random_bytes() still a useful string of random bytes.
Just noting that this was copied from CryptTest::testRandomBytes()
I think the message can be a sentence that ends with a period.
Comment #80
yesct commentedProbably useful to say Drupal 7.x since in this same paragraph we are mentioning PHP 7.
------
read the rest of this, compared it to the drupal 7 code and looks good. the message text is still relevant too, so using the same is ok.
Comment #81
pwolanin commentedHere with code and comment cleanups
Comment #82
yesct commentedoh, I didn't realize I might not have had enough context lines in the review. sorry.
#79
2. was probably too wordy. ok.
4. ah, it can't have visibility cause it's not in a class. and function visibility is an object oriented thing.
---
tweaked some comments.
(todo to myself or someone else, might be nice to get a tests only patch to document proof of the behavior on the issue. I was gonna do that, but sprint room being closed. :) )
Comment #84
pwolanin commentedsome sporadic test fail
Comment #85
klausicatch should be on the next line.
Otherwise looks good. I'm hesitant to RTBC this because I really think we should not have this fallback at all. I'd rather see the site fail and make people fix their random number sources.
Comment #86
dawehnerSorry for this question.
Why is there once a third parameter and once not? Maybe we could document that
Just to be sure I would put
runTestsInSeparateProcesseson thereNote: you can use
$this->assertCount()Comment #87
pwolanin commentedAdded annotation
The long comment at the start of the catch mentions that the two hash() invocations are different. That's a suggestion from Solr Designer to improve the robustness.
I don't think
$this->assertCount()works to calculate a string length. Fixed the param ordering.Comment #88
mile23Comment #89
dawehner@pwolanin
Thank you for clarifying that
I am not 100% comfortable, but given we have done something similar in d7 ...
Comment #90
klausiAh, now that we reintroduce our own random number generator we forgot to bring back the preseeding of mt_rand(), thanks David Rothstein for discovering that in #2550519: Crypt::randomBytes()/drupal_random_bytes() doesn't actually return cryptographically secure random bytes.
We should not put it back in DrupalKernel, but rather do it once in the request cycle in Crypt::randomBytes(). See http://cgit.drupalcode.org/drupal/diff/core/lib/Drupal/Core/DrupalKernel...
And that is why you should not do your own random number generator, because you tend to forget stuff :(
Comment #91
xjmComment #92
pwolanin commentedYEs, we should reseed - but hopefully was can just call the function without giving it a seed value? I think YesCT was trying to find more info.
Comment #93
pwolanin commentedOk, adding the seed call into the fallback generator. And improving the requirements check based on Heine's change in #2550519: Crypt::randomBytes()/drupal_random_bytes() doesn't actually return cryptographically secure random bytes to look for open_basedir
Comment #94
yesct commentedclarifying in summary.
Comment #95
yesct commentedI re-read the entire patch. I think this has BC, tests, and appropriate messages (requirement on install, message on site status page (for upgraded sites)).
Note to self: the phase of the site on install is install, on upgrade is upgrade, and runtime for a site that is... running (so that is what it will be for an upgraded site that will let them see the message on the site status page).
Comment #96
alexpottDon't need two of the same namespaces... and general convention has been to put the function after the class so the namespace / use statement / class is as expected.
We also should annotate the test so that it only runs on php 5.5 and 5.6. I think we can just do https://phpunit.de/manual/current/en/incomplete-and-skipped-tests.html#i...
Comment #97
alexpottSo I think we want
@requires PHP 5Comment #98
alexpottAfaics @requires does not work - we need to compare the PHP version in the test and mark as skipped if PHP version > 5
Comment #99
pwolanin commentedThere are 2 different namespaces, but sounds like we need to re-order the code so the function is at the end of the file
As designed, I think the test will work on both PHP 5 and 7, but let's confirm
Comment #100
alexpott@pwolanin - yes the test will work but it is pointless on PHP 7. And sorry for the confusion about namespaces - that was because all the other examples of this pattern we put the class first.
Comment #101
pwolanin commented@alexpott - it's not pointless on 7 since the built-in function can also throw an exception.
http://php.net/random_bytes
"If an appropriate source of randomness cannot be found, an Exception will be thrown."
Comment #102
pwolanin commentedJust moving the function to the bottom
Comment #103
klausiSome minor stuff:
This has nothing to do with page load now. Could be "Ensure mt_rand() is reseeded before calling it the first time."
newline at the end of file missing.
I like the hook_requirements() implementation with the open basedir detection, that should give people a better hint.
Comment #104
tuutti commentedComment #105
pwolanin commentedlooks good, we should copy that wording change to the D7 issue also.
Comment #106
alexpottCommitted 2bccef8 and pushed to 8.3.x. Thanks!
Code style fixes on commit.
Comment #108
larowlanThis was reverted on 8.2.x by @catch, but there is no indication of why?
Needs re-commit to 8.2.x?
http://cgit.drupalcode.org/drupal/commit/composer.lock?id=3fda618323bef5...
Comment #109
maximpodorov commentedIt was reverted because it breaks sites with non-empty open_basedir setting which does not contain /dev directory.
Comment #110
klausiBut now we take care of that with the bad fallback generator, so we just need to backport this to 8.2.x.
Comment #111
larowlanComment #106 says
But it was only pushed to 8.3.x (see #107).
Retesting against 8.2.x now.
RTBC unless bot disagrees.
Comment #112
alexpottSorry my comment was wrong about the branch pushed to. I can't see any compelling reason to commit this to 8.2.x - @larowlan is there one beyond the general security improvements?
Comment #113
dawehnerWell, its not only that, but it also enables some contrib modules to deal with the dependency on random_compat
Comment #114
alexpott@dawehner hmmm but what is wrong with those contrib modules saying the same as what we do here...
"paragonie/random_compat": "^1.0|^2.0",?Comment #115
dawehnerExternal libraries don't do that: https://github.com/defuse/php-encryption/blob/master/composer.json#L23
Comment #116
xjmThis includes a major version change for a dependency. In semver and our allowed changes policy, only patch-level updates should be made in patch releases:
https://www.drupal.org/core/d8-allowed-changes#patch
Technically it's even not allowed in a minor, but I can see the case for specifically breaking the policy for security hardening. But I definitely don't think this should be backported to a patch release.
Comment #117
catchWe don't expose random_compat in our own API, so there's no reason we shouldn't change it in a minor release IMO. Not to change it would be to essentially commit us to forking random_compat 1.x and maintaining it until the end of Drupal 8's LTS release - i.e. it's not just a case of security hardening now, but being able to upgrade to patch releases later on if there are bugs. In terms of disruption it can be as disruptive not to update (as this issue shows) as it is to update.
Having said that, I also don't see a reason to do this in a patch release unless there's a pressing reason - and the contrib dependency isn't one for me given the first version of this patch broke someone's install; contrib can recommend this core patch and/or require 8.3.x. If there was a pressing reason later like a critical bug fix in 2.x that's not in 1.x we could re-open and do it then.
Comment #118
larowlanPersonally I don't see the harm in updating in a patch release (especially given we're going with the
^1.0|^2.0version constraint). As it stands we can't usedefuse/php-encryption(https://github.com/defuse/php-encryption - the best-of-breed php encryption library) with Drupal 8.2, as it requires the 2.x version.The random_compat docs explicitly mention this:
So I guess we have to wait until 8.3 to use php-encryption with Drupal 8, or apply this patch.
Comment #119
klausiOk, so let's do the minimal thing first for 8.2.x which we all can agree on: change the composer version constraint to at least allow higher versions of random_compat.
Comment #120
larowlanThanks @klausi
Comment #122
catchThat's a good idea updating composer.json and not composer.lock. Committed/pushed to 8.2.x, thanks!
Comment #124
xjm