update scorecard docs

internal/plugins/scorecard/init.go

@@ -165,7 +165,7 @@ func generate(operatorName, testImageTag, outputDir string) error {
 type jsonPatches []jsonPatchObject
 
 // jsonPatchObject is a JSON 6902 patch object specific to the scorecard's test configuration.
-// See https://tools.ietf.org/html/rfc6902 for details.
+// https://kubernetes-sigs.github.io/kustomize/api-reference/kustomization/patchesjson6902/ for details.
 type jsonPatchObject struct {
 	Op    string                     `json:"op"`
 	Path  string                     `json:"path"`

website/content/en/docs/advanced-topics/scorecard/custom-tests.md

@@ -3,7 +3,7 @@ title: Writing Custom Scorecard Tests
 weight: 50
 ---
 
-This guide outlines the steps which can be followed to extend the existing scorecard tests and implement operator specific custom tests. 
+This guide outlines the steps which can be followed to extend the existing scorecard tests and implement operator specific custom tests.
 
 ## Run scorecard with custom tests:
 
@@ -14,6 +14,18 @@ The following steps explain creating of a custom test image which can be used wi
 The sample test image repository present [here][custom_scorecard_repo] has the following project structure:
 
 ```
+$ tree .
+.
+...
+├── config
+|   ...
+│   └── scorecard
+│       ├── bases
+│       │   └── config.yaml
+│       ├── kustomization.yaml
+│       └── patches
+│           ├── basic.config.yaml
+│           └── olm.config.yaml
 ├── Makefile
 ├── bundle
 │   ├── manifests
@@ -41,10 +53,11 @@ The sample test image repository present [here][custom_scorecard_repo] has the f
         └── tests.go
 ```
 
+1. `config/scorecard` - Contains a kustomization for generating a config from a base and set of overlays.
 1. `bundle/` - Contains bundle manifests and metadata under test.
-2. `bundle/tests/scorecard/config.yaml` - Configuration yaml to define and run scorecard tests.
-3. `images/custom-scorecard-tests/cmd/test/main.go` - Scorecard test binary.
-4. `internal/tests/tests.go` -  Contains the implementation of custom tests specific to the operator.
+1. `bundle/tests/scorecard/config.yaml` - Configuration file generated by `make bundle` from the `config/scorecard` kustomization.
+1. `images/custom-scorecard-tests/cmd/test/main.go` - Scorecard test binary.
+1. `internal/tests/tests.go` -  Contains the implementation of custom tests specific to the operator.
 
 #### Writing custom test logic:
 
@@ -56,48 +69,73 @@ The `tests.go` file is where the custom tests are implemented in the sample test
 package tests
 
 import (
-	"github.com/operator-framework/operator-registry/pkg/registry"
-	scapiv1alpha3 "github.com/operator-framework/operator-sdk/pkg/apis/scorecard/v1alpha3"
+  "github.com/operator-framework/operator-registry/pkg/registry"
+  scapiv1alpha3 "github.com/operator-framework/operator-sdk/pkg/apis/scorecard/v1alpha3"
 )
 
 const (
-	CustomTest1Name = "customtest1"
+  CustomTest1Name = "customtest1"
 )
 
-// CustomTest1 
+// CustomTest1
 func CustomTest1(bundle registry.Bundle) scapiv1alpha3.TestStatus {
-	r := scapiv1alpha3.TestResult{}
-	r.Name = CustomTest1Name
-	r.Description = "Custom Test 1"
-	r.State = scapiv1alpha3.PassState
-	r.Errors = make([]string, 0)
-	r.Suggestions = make([]string, 0)
+  r := scapiv1alpha3.TestResult{}
+  r.Name = CustomTest1Name
+  r.Description = "Custom Test 1"
+  r.State = scapiv1alpha3.PassState
+  r.Errors = make([]string, 0)
+  r.Suggestions = make([]string, 0)
 
   // Implement relevant custom test logic here
 
-	return wrapResult(r)
+  return wrapResult(r)
 }
 ```
 
 ### Scorecard Configuration file:
 
-The [configuration file][config_yaml] includes test definitions and metadata to run the test. For the example `CustomTest1` function, the following fields should be specified in `config.yaml`.
+The [configuration file][config_yaml] includes test definitions and metadata to run the test.
+This file is constructed using a kustomization under `config/scorecard`, with overlays for test sets.
+
+For the example `CustomTest1` function, add the following to `config/scorecard/customtest1.config.yaml`.
 
 ```yaml
-stages:
-- tests:
-  - image: quay.io/username/custom-scorecard-tests:dev
+- op: add
+  path: /stages/0/tests/0
+  value:
+    image: quay.io/<username>/custom-scorecard-tests:latest
     entrypoint:
     - custom-scorecard-tests
     - customtest1
     labels:
       suite: custom
       test: customtest1
- ```
+```
 
 The important fields to note here are:
 1. `image` - name and tag of the test image which was specified in the Makefile.
-2. `labels` - the name of the `test` and `suite` the test function belongs to. This can be specified in the `operator-sdk scorecard` command to run the desired test.
+2. `labels` - the name of the `test` and `suite` the test function belongs to.
+This can be specified in the `operator-sdk scorecard` command to run the desired test.
+
+Next, add a [JSON 6902 patch][kustomize-patchJson6902] to your `config/scorecard/kustomization.yaml`:
+
+```yaml
+patchesJson6902:
+...
+- path: patches/customtest1.config.yaml
+  target:
+    group: scorecard.operatorframework.io
+    version: v1alpha3
+    kind: ScorecardConfiguration
+    name: config
+```
+
+Once you run `make bundle`, the `bundle/tests/scorecard/config.yaml` will be (re)generated with your custom test.
+
+If generating a config file outside of the on-disk bundle, you can run:
+```console
+$ kustomize build config/scorecard > path/to/config.yaml
+```
 
 **Note**: The default location of `config.yaml` inside the bundle is `<bundle directory>/tests/scorecard/config.yaml`. It can be overridden using the `--config` flag. For more details regarding the configuration file refer to [user docs][user_doc].
 
@@ -106,10 +144,10 @@ The important fields to note here are:
 The scorecard test image implementation requires the bundle under test to be present in the test image. The `apimanifests.GetBundleFromDir()` function reads the pod's bundle to fetch the manifests and scorecard configuration from desired path.
 
 ```Go
-	cfg, err := apimanifests.GetBundleFromDir(scorecard.PodBundleRoot)
-	if err != nil {
-		log.Fatal(err.Error())
-	}
+cfg, err := apimanifests.GetBundleFromDir(scorecard.PodBundleRoot)
+if err != nil {
+  log.Fatal(err.Error())
+}
 ```
 The scorecard binary uses `config.yaml` file to locate tests and execute the them as Pods which scorecard creates. Custom test images are included into Pods that scorecard creates, passing in the bundle contents on a shared mount point to the test image container. The specific custom test that is executed is driven by the config.yaml's entry-point command and arguments.
 
@@ -118,11 +156,11 @@ An example scorecard binary is present [here][scorecard_binary].
 The names with which the tests are identified in `config.yaml` and would be passed in the `scorecard` command, are to be specified here.
 
 ```Go
-...     
+...
 switch entrypoint[0] {
 case tests.CustomTest1Name:
-    result = tests.CustomTest1(cfg)
-    ...
+  result = tests.CustomTest1(cfg)
+  ...
 }
 ...
 ```
@@ -131,21 +169,21 @@ The result of the custom tests which is in `scapiv1alpha3.TestResult` format, is
 
 ```Go
 prettyJSON, err := json.MarshalIndent(result, "", "    ")
-	if err != nil {
-		log.Fatal("Failed to generate json", err)
-	}
+if err != nil {
+  log.Fatal("Failed to generate json", err)
+}
 fmt.Printf("%s\n", string(prettyJSON))
 ```
 
 The names of the custom tests are also included in `printValidTests()` function:
 
 ```Go
 func printValidTests() (result scapiv1alpha3.TestStatus) {
-...
-    str := fmt.Sprintf("Valid tests for this image include: %s,
-            tests.CustomTest1Name
-    result.Errors = append(result.Errors, str")
-...
+  ...
+  str := fmt.Sprintf("Valid tests for this image include: %s", tests.CustomTest1Name)
+  result.Errors = append(result.Errors, str)
+  ...
+}
 
 ```
 
@@ -166,7 +204,7 @@ docker push <repository_name>/<username>/<image_name>:tag
 The `operator-sdk scorecard` command is used to execute the scorecard tests by specifying the location of test bundle in the command. The name or suite of the tests which are to be executed can be specified with the `--selector` flag. The command will create scorecard pods with the image specified in `config.yaml` for the respective test. For example, the `CustomTest1Name` test provides the following json output.
 
 ```console
-operator-sdk scorecard <bundle_dir_or_image> --selector=suite=custom -o json --wait-time=32s --skip-cleanup=false
+$ operator-sdk scorecard <bundle_dir_or_image> --selector=suite=custom -o json --wait-time=32s --skip-cleanup=false
 {
   "metadata": {
     "creationTimestamp": null
@@ -187,7 +225,7 @@ operator-sdk scorecard <bundle_dir_or_image> --selector=suite=custom -o json --w
 
 ### Debugging scorecard custom tests
 
-The `--skip-cleanup` flag can be used when executing the `operator-sdk scorecard` command to cause the scorecard created test pods to be unremoved. 
+The `--skip-cleanup` flag can be used when executing the `operator-sdk scorecard` command to cause the scorecard created test pods to be unremoved.
 This is useful when debugging or writing new tests so that you can view
 the test logs or the pod manifests.
 
@@ -208,14 +246,14 @@ namespaces for your test but instead considers these resources
 to be outside its scope.  You can however implement whatever
 service accounts your tests require and then specify
 that service account from the command line using the service-account flag:
-```
-operator-sdk scorecard <bundle_dir_or_image> --service-account=mycustomsa
+```console
+$ operator-sdk scorecard <bundle_dir_or_image> --service-account=mycustomsa
 ```
 
 Also, you can set up a non-default namespace that your tests
 will be executed within using the following namespace flag:
-```
-operator-sdk scorecard <bundle_dir_or_image> --namespace=mycustomns
+```console
+$ operator-sdk scorecard <bundle_dir_or_image> --namespace=mycustomns
 ```
 
 If you do not specify either of these flags, the default namespace
@@ -231,9 +269,9 @@ https://github.com/operator-framework/operator-sdk/blob/master/pkg/apis/scorecar
 ### Accessing the Kube API
 
 Within your custom tests you might require connecting to the Kube API.  
-In golang, you could use the [client-go][client_go] API for example to 
-check Kube resources within your tests, or even create custom resources. Your 
-custom test image is being executed within a Pod, so you can use an in-cluster 
+In golang, you could use the [client-go][client_go] API for example to
+check Kube resources within your tests, or even create custom resources. Your
+custom test image is being executed within a Pod, so you can use an in-cluster
 connection to invoke the Kube API.
 
 
@@ -246,3 +284,4 @@ connection to invoke the Kube API.
 [user_doc]: /docs/advanced-topics/scorecard/scorecard/
 [scorecard_binary]: https://github.com/operator-framework/operator-sdk/blob/master/internal/scorecard/examples/custom-scorecard-tests/images/custom-scorecard-tests/cmd/test/main.go
 [sample_makefile]: https://github.com/operator-framework/operator-sdk/blob/master/internal/scorecard/examples/custom-scorecard-tests/Makefile
+[kustomize-patchJson6902]: https://kubernetes-sigs.github.io/kustomize/api-reference/kustomization/patchesjson6902/

website/content/en/docs/advanced-topics/scorecard/scorecard.md

@@ -30,20 +30,28 @@ require if the tests are designed for resource creation.
 
 ## Running the Scorecard
 
-1. Define a scorecard configuration file by running
+1. A default set of kustomize files should have been scaffolded by `operator-sdk init`.
+If that is not the case, run `operator-sdk init` as you would have to initialize your project
+and copy scaffolded files:
   ```sh
-  $ operator-sdk generate kustomize scorecard
+  $ TMP_PROJECT="$(mktemp -d)/<current-project-name>"
+  $ mkdir "$TMP_PROJECT"
+  $ pushd "$TMP_PROJECT"
+  $ operator-sdk init
+  $ popd
+  $ cp -r "$TMP_PROJECT"/config/scorecard ./config/
   ```
-  See the [config file section](#config-file) for an explanation of the configuration file format.
-1. Generate [bundle][quickstart-bundle] manifests and metadata for your Operator.
+The default config generated by this kustomization can be immediately run against your operator.
+See the [config file section](#config-file) for an explanation of the configuration file format.
+1. (Re)enerate [bundle][quickstart-bundle] manifests and metadata for your Operator.
 `make bundle` will automatically add scorecard annotations to your bundle's metadata,
 which is used by the `scorecard` command to run tests.
 1. Execute the [`scorecard` command][cli-scorecard]. See the [command args section](#command-args)
 for an overview of command invocation.
 
 ## Configuration
 
-The scorecard test execution is driven by a configuration file named `config.yaml`.
+The scorecard test execution is driven by a configuration file named `config.yaml`, generated by `make bundle`.
 The configuration file is located at the following location within your bundle directory (`bundle/` by default):
 ```sh
 $ tree ./bundle
@@ -56,23 +64,24 @@ $ tree ./bundle
 
 ### Config File
 
-A complete scorecard configuration file can be found [here][sample-config]
-and used for running the scorecard pre-defined tests that ship with the SDK.
-
-A sample of the scorecard configuration file may look as follows:
+The following YAML spec is an example of the scorecard configuration file:
 
 ```yaml
+kind: ScorecardConfiguration
+apiversion: scorecard.operatorframework.io/v1alpha3
+metadata:
+  name: config
 stages:
 - parallel: true
   tests:
-  - image: quay.io/operator-framework/scorecard-test:dev
+  - image: quay.io/operator-framework/scorecard-test:latest
     entrypoint:
     - scorecard-test
     - basic-check-spec
     labels:
       suite: basic
       test: basic-check-spec-test
-  - image: quay.io/operator-framework/scorecard-test:dev
+  - image: quay.io/operator-framework/scorecard-test:latest
     entrypoint:
     - scorecard-test
     - olm-bundle-validation
@@ -186,7 +195,7 @@ See an example of the JSON format produced by a scorecard test:
       "kind": "Test",
       "apiVersion": "scorecard.operatorframework.io/v1alpha3",
       "spec": {
-        "image": "quay.io/operator-framework/scorecard-test:dev",
+        "image": "quay.io/operator-framework/scorecard-test:latest",
         "entrypoint": [
           "scorecard-test",
           "olm-bundle-validation"
@@ -216,7 +225,7 @@ See an example of the text format produced by a scorecard test:
 
 ```
 --------------------------------------------------------------------------------
-Image:      quay.io/operator-framework/scorecard-test:dev
+Image:      quay.io/operator-framework/scorecard-test:latest
 Entrypoint: [scorecard-test olm-bundle-validation]
 Labels:
 	"suite":"olm"
@@ -250,7 +259,7 @@ Scorecard will execute custom tests if they follow these mandated conventions:
  * tests can obtain the bundle contents at a shared mount point of /bundle
  * tests can access the Kubernetes API using an in-cluster client connection
 
-See [here][custom-image] for an example of a custom test image written in golang.
+See [here][custom-image] for an example of a custom test image written in Go.
 
 Writing custom tests in other programming languages is possible
 if the test image follows the above guidelines.