Upgrade from version 3.0.44 or higher

Upgrade to the latest version of the StackRox Kubernetes Security Platform from version 3.0.44 or higher.

2 minute read

Use these instructions to upgrade the StackRox Kubernetes Security Platform to the latest version from version 3.0.44 or higher.

To upgrade from a different version, check the upgrade instructions for your current version.

To upgrade the StackRox Kubernetes Security Platform to the latest version, complete the following steps:

  1. Find out what’s new.

    When you scan an image based on CentOS 7 or RHEL 7 in the StackRox Kubernetes Security Platform version 3.0.46.0 or higher, StackRox Scanner returns more vulnerability results than before. Previously, StackRox Scanner only returned vulnerabilities with fixes available.

    To avoid disrupting build or deployment pipelines, make sure your enforced policies use the Fixed By policy attribute so they only match fixable vulnerabilities.

  2. Backup the Central database.
  3. Upgrade the Central cluster.
  4. Upgrade all secured clusters.

Back up the Central database

Before completing the upgrade, StackRox recommends that you back up the contents of the Central database.

To create a backup, you must first generate an API token.

To generate an API token:

  1. On the StackRox portal, navigate to Platform Configuration > Integrations.
  2. Scroll down to the Authentication Tokens category, and select API Token.
  3. Click Generate Token.
  4. Enter a name for the token and select the role as Admin.
  5. Select Generate.

Copy the generated token and securely store it. You won’t be able to view it again.

To backup the Central database, run the following commands:

  1. Navigate to the StackRox portal.
  2. Select CLI in the upper right hand corner, and then select your platform to download the roxctl binary.
  3. Set the ROX_API_TOKEN and CENTRAL_ADDRESS environment variables:
    Copy
    export ROX_API_TOKEN=<api-token>
    export CENTRAL_ADDRESS=<address>:<port-number>
  4. Run the backup command.
    • For the StackRox Kubernetes Security Platform version 3.0.55 or newer:
      Copy
      roxctl -e "$CENTRAL_ADDRESS" central backup
    • For the StackRox Kubernetes Security Platform version 3.0.54 or older:
      Copy
      roxctl -e "$CENTRAL_ADDRESS" central db backup

Upgrade the Central cluster

  • If you deploy images from a private image registry instead of stackrox.io, first push the new image into your private registry, and then replace your image registry in the following commands.

  • If you used Red Hat UBI-based images when you deployed the StackRox Kubernetes Security Platform, replace the image names for commands in this section with the following UBI-based image names:

    • For Central, Sensor, and Compliance use stackrox.io/main-rhel
    • For Scanner use stackrox.io/scanner-rhel and stackrox.io/scanner-db-rhel
    • For Collector use collector.stackrox.io/collector-rhel

Central services

To upgrade Central services:

  1. Run the following command to upgrade:

    Copy
    kubectl -n stackrox patch deploy/central -p '{"spec":{"template":{"spec":{"containers":[{"name":"central","env":[{"name":"ROX_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}}]}]}}}}'
    kubectl -n stackrox set image deploy/central central=stackrox.io/main:3.63.0
    Copy
    oc -n stackrox patch deploy/central -p '{"spec":{"template":{"spec":{"containers":[{"name":"central","env":[{"name":"ROX_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}}]}]}}}}'
    oc -n stackrox set image deploy/central central=stackrox.io/main:3.63.0

Download roxctl CLI

Download the roxctl command-line interface (CLI) for the StackRox Kubernetes Security Platform version 3.63.0.

  1. Download the roxctl CLI:

    1. By using your browser:

    2. Or by using curl and providing your stackrox.io credentials:

      • On Linux:
        Copy
        curl -O https://mirror.openshift.com/pub/rhacs/assets/3.63.0/bin/Linux/roxctl
      • On macOS:
        Copy
        curl -O https://mirror.openshift.com/pub/rhacs/assets/3.63.0/bin/Darwin/roxctl

After you download the roxctl CLI:

  1. Make the roxctl binary executable and add it to your PATH:

    Copy
    chmod +x roxctl
    sudo mv roxctl /usr/local/bin/roxctl

    If you prefer not to add the roxctl CLI to your PATH, use ./roxctl instead of roxctl for all commands.

  2. Verify that the roxctl version you installed is up-to-date. The most current version is 3.63.0:

    Copy
    roxctl version

StackRox Scanner

To upgrade StackRox Scanner:

  1. In the StackRox Kubernetes Security Platform version 3.0.58.0, we’ve made changes to the Scanner configuration. If you are upgrading to version 3.0.58.0, you must modify your scanner configuration files.

    If you’ve created custom scanner configurations, you must apply those changes before updating the scanner configuration file.

    Copy
    roxctl -e "$CENTRAL_ADDRESS" scanner generate
    kubectl apply -f scanner-bundle/scanner/02-scanner-03-tls-secret.yaml
    kubectl apply -f scanner-bundle/scanner/02-scanner-04-scanner-config.yaml
    Copy
    roxctl -e "$CENTRAL_ADDRESS" scanner generate
    oc apply -f scanner-bundle/scanner/02-scanner-03-tls-secret.yaml
    oc apply -f scanner-bundle/scanner/02-scanner-04-scanner-config.yaml
  2. Run the following commands to upgrade:

    Copy
    kubectl -n stackrox patch hpa/scanner -p '{"spec":{"minReplicas":2}}'
    kubectl -n stackrox set image deploy/scanner scanner=stackrox.io/scanner:2.17.4
    kubectl -n stackrox set image deploy/scanner-db db=stackrox.io/scanner-db:2.17.4
    kubectl -n stackrox set image deploy/scanner-db init-db=stackrox.io/scanner-db:2.17.4
    Copy
    oc -n stackrox patch hpa/scanner -p '{"spec":{"minReplicas":2}}'
    oc -n stackrox set image deploy/scanner scanner=stackrox.io/scanner:2.17.4
    oc -n stackrox set image deploy/scanner-db db=stackrox.io/scanner-db:2.17.4
    oc -n stackrox set image deploy/scanner-db init-db=stackrox.io/scanner-db:2.17.4

Verify deployment

  1. Check that the new pods have deployed:

    Copy
    kubectl get deploy -n stackrox -o wide
    kubectl get pod -n stackrox --watch
    Copy
    oc get deploy -n stackrox -o wide
    oc get pod -n stackrox --watch
  2. Check the Central logs:

    Copy
    kubectl logs -n stackrox deploy/central -c central
    Copy
    oc logs -n stackrox deploy/central -c central

    If the upgrade is successful, you will see output similar to the following:

    Copy
    $ kubectl logs -n stackrox deploy/central -c central
    No database restore directory found (this is not an error).
    Migrator: 2019/10/25 17:58:54: starting DB compaction
    Migrator: 2019/10/25 17:58:54: Free fraction of 0.0391 (40960/1048576) is < 0.7500. Will not compact
    badger 2019/10/25 17:58:54 INFO: All 1 tables opened in 2ms
    badger 2019/10/25 17:58:55 INFO: Replaying file id: 0 at offset: 846357
    badger 2019/10/25 17:58:55 INFO: Replay took: 50.324µs
    badger 2019/10/25 17:58:55 DEBUG: Value log discard stats empty
    Migrator: 2019/10/25 17:58:55: DB is up to date. Nothing to do here.
    badger 2019/10/25 17:58:55 INFO: Got compaction priority: {level:0 score:1.73 dropPrefix:[]}
    version: 2019/10/25 17:58:55.189866 ensure.go:49: Info: Version found in the DB was current. We’re good to go!
    Copy
    $ oc logs -n stackrox deploy/central -c central
    No database restore directory found (this is not an error).
    Migrator: 2019/10/25 17:58:54: starting DB compaction
    Migrator: 2019/10/25 17:58:54: Free fraction of 0.0391 (40960/1048576) is < 0.7500. Will not compact
    badger 2019/10/25 17:58:54 INFO: All 1 tables opened in 2ms
    badger 2019/10/25 17:58:55 INFO: Replaying file id: 0 at offset: 846357
    badger 2019/10/25 17:58:55 INFO: Replay took: 50.324µs
    badger 2019/10/25 17:58:55 DEBUG: Value log discard stats empty
    Migrator: 2019/10/25 17:58:55: DB is up to date. Nothing to do here.
    badger 2019/10/25 17:58:55 INFO: Got compaction priority: {level:0 score:1.73 dropPrefix:[]}
    version: 2019/10/25 17:58:55.189866 ensure.go:49: Info: Version found in the DB was current. We’re good to go!

If you are upgrading from the StackRox Kubernetes Security Platform version 3.0.45 or older, Central may take longer to start as it migrates to a new, more efficient database.

You have successfully upgraded StackRox Central.

Upgrade all secured clusters

After upgrading Central services, you must upgrade all secured clusters, including the Central cluster.

You can use automatic upgrades to finish the upgrade process. Once you have finished automatic upgrades of your secured clusters, skip to the clean up step.

To complete manual upgrades of each secured cluster running the StackRox Sensor and Collector, follow the instructions below.

Update admission controller

In the StackRox Kubernetes Security Platform version 3.0.55, we’ve made changes to our admission controller integration. If you are using an admission controller or want to take advantage of the admission controller integration’s new features, you must run some additional commands to upgrade the secured cluster.

If you are upgrading from the StackRox Kubernetes Security Platform version number between 3.0.44 and 3.0.54, and you aren’t using:

you must manually redeploy Sensor on all clusters.

To check if you are using an admission controller with the StackRox Kubernetes Security Platform, run the following command:

Copy
kubectl get validatingwebhookconfiguration stackrox
Copy
oc get validatingwebhookconfiguration stackrox

If you get an error, skip this section and continue with the instructions in the update images section.

Otherwise, if you don’t get any errors, follow these additional instructions:

  1. Delete the existing validating webhook configuration:

    Copy
    kubectl delete validatingwebhookconfiguration stackrox
    Copy
    oc delete validatingwebhookconfiguration stackrox
  2. If you want to allow any additional admission controller features moving forward, navigate to the Clusters configuration page, select the cluster you are updating, and enable the appropriate options.

  3. Obtain a new sensor bundle for the respective cluster. You can do so using the CLI:

    Copy
    roxctl sensor get-bundle <cluster-name>

    or via the Clusters configuration page. Extract the bundle (if necessary), and cd into the bundle directory.

  4. Create the admission-control deployment and related objects:

    Copy
    kubectl -n "stackrox" apply -f "admission-controller-secret.yaml" 
    kubectl -n "stackrox" apply -f "admission-controller-rbac.yaml" 
    kubectl -n "stackrox" apply -f "admission-controller-netpol.yaml" 
    kubectl -n "stackrox" apply -f "admission-controller-pod-security.yaml" 
    kubectl -n "stackrox" apply -f "admission-controller.yaml"
    Copy
    oc -n "stackrox" apply -f "admission-controller-scc.yaml"
    oc -n "stackrox" apply -f "admission-controller-secret.yaml" 
    oc -n "stackrox" apply -f "admission-controller-rbac.yaml" 
    oc -n "stackrox" apply -f "admission-controller-netpol.yaml" 
    oc -n "stackrox" apply -f "admission-controller-pod-security.yaml" 
    oc -n "stackrox" apply -f "admission-controller.yaml"

    By default, the --listenOnEvents option is set to false during the upgrade. It controls the deployment of the admission controller webhook, which listens for Kubernetes exec and portforward events. If you’re using OpenShift version 3.11, don’t set the --listenOnEvents option to true. Since these events aren’t available for OpenShift 3.11, enabling them causes errors.

Update OpenShift Security Context Constraints

Run the following command only if you’re using the StackRox Kubernetes Security Platform on OpenShift. Otherwise, skip this section and go to the Update other images section.

In the StackRox Kubernetes Security Platform version 3.0.57.2, we’ve made changes to our security context constraints. If you are upgrading to 3.0.57.2 or higher, run the following command:

Copy
oc patch --type merge scc scanner -p '{"priority":0}'

Update other images

  1. Run the following commands to update images and make other necessary configurations:

    Run the following commands only if you upgrade from the StackRox Kubernetes Security Platform version number between 3.0.44 and 3.0.54. Otherwise, skip this section and go to the Verify deployment section.

    Copy
    kubectl -n stackrox patch deploy/sensor -p '{"spec":{"template":{"spec":{"containers":[{"name":"sensor","env":[{"name":"POD_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}}],"volumeMounts":[{"name":"cache","mountPath":"/var/cache/stackrox"}]}],"volumes":[{"name":"cache","emptyDir":{}}]}}}}'
    kubectl -n stackrox set image deploy/sensor sensor=stackrox.io/main:3.63.0
    kubectl -n stackrox set image ds/collector compliance=stackrox.io/main:3.63.0
    kubectl -n stackrox set image ds/collector collector=collector.stackrox.io/collector:3.1.30-latest
    kubectl -n stackrox apply -f - <<EOF
    kind: ClusterRole
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: stackrox:review-tokens
      labels:
        app.kubernetes.io/name: stackrox
        auto-upgrade.stackrox.io/component: "sensor"
      annotations:
        owner: stackrox
        email: "support@stackrox.com"
    rules:
    - resources:
      - tokenreviews
      apiGroups: ["authentication.k8s.io"]
      verbs:
      - create
    ---
    kind: ClusterRoleBinding
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: stackrox:review-tokens-binding
      labels:
        app.kubernetes.io/name: stackrox
        auto-upgrade.stackrox.io/component: "sensor"
      annotations:
        owner: stackrox
        email: "support@stackrox.com"
    subjects:
    - kind: ServiceAccount
      name: sensor
      namespace: stackrox
    roleRef:
      kind: ClusterRole
      name: stackrox:review-tokens
      apiGroup: rbac.authorization.k8s.io
    EOF
    Copy
    oc -n stackrox patch deploy/sensor -p '{"spec":{"template":{"spec":{"containers":[{"name":"sensor","env":[{"name":"POD_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}}],"volumeMounts":[{"name":"cache","mountPath":"/var/cache/stackrox"}]}],"volumes":[{"name":"cache","emptyDir":{}}]}}}}'
    oc -n stackrox set image deploy/sensor sensor=stackrox.io/main:3.63.0
    oc -n stackrox set image ds/collector compliance=stackrox.io/main:3.63.0
    oc -n stackrox set image ds/collector collector=collector.stackrox.io/collector:3.1.30-latest
    oc -n stackrox apply -f - <<EOF
    kind: ClusterRole
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: stackrox:review-tokens
      labels:
        app.kubernetes.io/name: stackrox
        auto-upgrade.stackrox.io/component: "sensor"
      annotations:
        owner: stackrox
        email: "support@stackrox.com"
    rules:
    - resources:
      - tokenreviews
      apiGroups: ["authentication.k8s.io"]
      verbs:
      - create
    ---
    kind: ClusterRoleBinding
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: stackrox:review-tokens-binding
      labels:
        app.kubernetes.io/name: stackrox
        auto-upgrade.stackrox.io/component: "sensor"
      annotations:
        owner: stackrox
        email: "support@stackrox.com"
    subjects:
    - kind: ServiceAccount
      name: sensor
      namespace: stackrox
    roleRef:
      kind: ClusterRole
      name: stackrox:review-tokens
      apiGroup: rbac.authorization.k8s.io
    EOF

Verify deployment

Check that the updated pods have successfully deployed before continuing:

Copy
kubectl get deploy,ds -n stackrox -o wide
kubectl get pod -n stackrox --watch
Copy
oc get deploy,ds -n stackrox -o wide
oc get pod -n stackrox --watch

The upgrade process is now complete.

Rollback Central if upgrade fails

Beginning from the StackRox Kubernetes Security Platform version 3.0.58.0, you can rollback to a previous version of Central.

You can perform a normal rollback if, for some reason, the StackRox Kubernetes Security Platform upgrade fails or you can perform a forced rollback to a previous Central version anytime.

  • You can only perform a normal rollback if you upgrade from the StackRox Kubernetes Security Platform version 3.0.57 or newer.
  • To perform a forced rollback, you need the StackRox Kubernetes Security Platform version 3.0.58 or newer.
  • Before you can perform a rollback, you must have free disk space available on your persistent storage. The StackRox Kubernetes Security Platform uses disk space to keep a copy of databases during the upgrade. If the disk space isn’t enough to store a copy and the upgrade fails, you won’t be able to roll back to an earlier version.

Normal rollback

Run the following command to roll back to a previous version when an upgrade fails (before the Central service starts):

Copy
kubectl -n stackrox rollout undo deploy/central
Copy
oc -n stackrox rollout undo deploy/central

Forced rollback

You can use forced rollback to roll back to an earlier version (after the Central service starts).

Using forced rollback to switch back to a previous version might result in loss of data and functionality.

Run the following command to perform,

  • forced rollback to a specific version:

    1. Edit Central’s ConfigMap:

      Copy
      kubectl -n stackrox edit configmap/central-config
      Copy
      oc -n stackrox edit configmap/central-config
    2. Change the value of the maintenance.forceRollbackVersion key to the version to which you want to roll back:

      Copy
      data:
        central-config.yaml: |
          maintenance:
            safeMode: false
            compaction:
               enabled: true
               bucketFillFraction: .5
               freeFractionThreshold: 0.75
            forceRollbackVersion: <x.x.x.x>
        ...
  • forced rollback to previously installed version:

    Copy
    kubectl -n stackrox rollout undo deploy/central
    Copy
    oc -n stackrox rollout undo deploy/central

Clean up

For most browsers, you must reload the page and re-accept the certificate to continue using the StackRox portal.

Revoke the API token

For security reasons, StackRox recommends revoking the API token you used to complete the database backup.

To revoke an API token:

  1. On the StackRox portal, navigate to Platform Configuration > Integrations.
  2. Scroll down to the Authentication Tokens category, and select API Token.
  3. Select the checkbox in front of the token name you want to revoke.
  4. Select Revoke.
  5. On the confirmation dialog box, click Confirm.

The updated Sensors and Collectors will continue to report the latest data from each secured cluster.

The last time a Sensor contacted Central is visible in the Clusters view.

To view the Last check-in time:

  1. On the StackRox portal, navigate to Platform Configuration > Clusters.
  2. View the Last check-in time for each cluster.

If any sensor hasn’t checked in for more than five minutes, check the cluster logs for that Sensor to ensure that it’s operating as usual.

The displayed check-in time doesn’t update automatically. Reload the page to see the updates.

Contact StackRox support if you experience any issues.

Questions?

We're happy to help! Reach out to us to discuss questions, issues, or feature requests.

© 2021 StackRox Inc. All rights reserved.