Upgrade from version 2.5.31 through version 3.0.34

Upgrade to version 3.0.54 of the StackRox Kubernetes Security Platform from version 2.5.31 through version 3.0.34.

3 minute read

Use these instructions to upgrade the StackRox Kubernetes Security Platform to version 3.0.54.0 from version 2.5.31 through version 3.0.34.

To upgrade from a different version, check the upgrade instructions for your current version.

To upgrade the StackRox Kubernetes Security Platform to 3.0.54, complete the following steps:

  1. Find out what’s new.

    When you scan an image based on CentOS 7 or RHEL 7 in the StackRox Kubernetes Security Platform version 3.0.46.0 or higher, StackRox Scanner returns more vulnerability results than before. Previously, StackRox Scanner only returned vulnerabilities with fixes available.

    To avoid disrupting build or deployment pipelines, make sure your enforced policies use the Fixed By policy attribute so they only match fixable vulnerabilities.

  2. Prepare to upgrade.
  3. Backup the Central database.
  4. Upgrade the Central cluster.
  5. Upgrade all secured clusters.

Prepare to upgrade

Check policy enforcement

Starting from version 3.0.35, the StackRox Kubernetes Security Platform includes new language-specific vulnerability results when checking images and deployments against your security policies. If you have enabled enforcement on image vulnerability-based policies by integrating with a continuous integration (CI) system, enabling admission control, or using scale-to-zero enforcement, we recommend disabling enforcement before you upgrade so that you can view policy violations before re-enabling enforcement.

To disable language-specific vulnerability scanning you can set the value of the LANGUAGE_VULNS environment variable to false:

Copy
kubectl -n stackrox set env deploy/scanner LANGUAGE_VULNS=false
Copy
oc -n stackrox set env deploy/scanner LANGUAGE_VULNS=false

New image scan results and policy violations appear in the StackRox portal over a four-hour interval as images get rescanned with the updated version of StackRox Scanner.

To find out if your policies are enforced, navigate to Platform Configuration > System Policies and filter the view by Enforcement: Fail build and Enforcement: Scale to zero.

Back up the Central database

Before completing the upgrade, StackRox recommends that you back up the contents of the Central database.

To create a backup, you must first generate an API token.

To generate an API token:

  1. On the StackRox portal, navigate to Platform Configuration > Integrations.
  2. Scroll down to the Authentication Tokens category, and select API Token.
  3. Click Generate Token.
  4. Enter a name for the token and select the role as Admin.
  5. Select Generate.

Copy the generated token and securely store it. You won’t be able to view it again.

To backup the Central database, run the following commands:

  1. Navigate to the StackRox portal.
  2. Select CLI in the upper right hand corner, and then select your platform to download the roxctl binary.
  3. Set the ROX_API_TOKEN and CENTRAL_ADDRESS environment variables:
    Copy
    export ROX_API_TOKEN=<api-token>
    export CENTRAL_ADDRESS=<address>:<port-number>
  4. Run the backup command.
    • For the StackRox Kubernetes Security Platform version 3.0.55 or newer:
      Copy
      roxctl -e "$CENTRAL_ADDRESS" central backup
    • For the StackRox Kubernetes Security Platform version 3.0.54 or older:
      Copy
      roxctl -e "$CENTRAL_ADDRESS" central db backup

Upgrade the Central cluster

  • If you deploy images from a private image registry instead of stackrox.io, first push the new image into your private registry, and then replace your image registry in the following commands.

  • If you used Red Hat UBI-based images when you deployed the StackRox Kubernetes Security Platform, replace the image names for commands in this section with the following UBI-based image names:

    • For Central, Sensor, and Compliance use stackrox.io/main-rhel
    • For Scanner use stackrox.io/scanner-rhel and stackrox.io/scanner-db-rhel
    • For Collector use collector.stackrox.io/collector-rhel

Central services

The following commands are a cumulative set of commands that completes the upgrade of the StackRox Kubernetes Security Platform from 3.0.31 to 3.0.54.

You might see errors when you run these commands:

  • if you completed an upgrade that previously ran any of these commands, or
  • if you aren’t using some features of the StackRox Kubernetes Security Platform that these commands configure.

You can ignore these errors and proceed with the rest of the instructions.

To upgrade Central services:

  1. Create new default proxy secret:

    Copy
    kubectl create -f - <<EOF
    apiVersion: v1
    kind: Secret
    metadata:
      namespace: stackrox
      name: proxy-config
      labels:
        app.kubernetes.io/name: stackrox
      annotations:
        "helm.sh/hook": "pre-install"
    type: Opaque
    stringData:
      config.yaml: |-
        # # NOTE: Both central and scanner should be restarted if this secret is changed.
        # # While it is possible that some components will pick up the new proxy configuration
        # # without a restart, it cannot be guaranteed that this will apply to every possible
        # # integration etc.
        # url: http://proxy.name:port
        # username: username
        # password: password
        # # If the following value is set to true, the proxy wil NOT be excluded for the default hosts:
        # # - *.stackrox, *.stackrox.svc
        # # - localhost, localhost.localdomain, 127.0.0.1, ::1
        # # - *.local
        # omitDefaultExcludes: false
        # excludes:  # hostnames (may include * components) for which not to use a proxy, like in-cluster repositories.
        # - some.domain
        # # The following configuration sections allow specifying a different proxy to be used for HTTP(S) connections.
        # # If they are omitted, the above configuration is used for HTTP(S) connections as well as TCP connections.
        # # If only the `http` section is given, it will be used for HTTPS connections as well.
        # # Note: in most cases, a single, global proxy configuration is sufficient.
        # http:
        #   url: http://http-proxy.name:port
        #   username: username
        #   password: password
        # https:
        #   url: http://https-proxy.name:port
        #   username: username
        #   password: password
    EOF
    Copy
    oc create -f - <<EOF
    apiVersion: v1
    kind: Secret
    metadata:
      namespace: stackrox
      name: proxy-config
      labels:
        app.kubernetes.io/name: stackrox
      annotations:
        "helm.sh/hook": "pre-install"
    type: Opaque
    stringData:
      config.yaml: |-
        # # NOTE: Both central and scanner should be restarted if this secret is changed.
        # # While it is possible that some components will pick up the new proxy configuration
        # # without a restart, it cannot be guaranteed that this will apply to every possible
        # # integration etc.
        # url: http://proxy.name:port
        # username: username
        # password: password
        # # If the following value is set to true, the proxy wil NOT be excluded for the default hosts:
        # # - *.stackrox, *.stackrox.svc
        # # - localhost, localhost.localdomain, 127.0.0.1, ::1
        # # - *.local
        # omitDefaultExcludes: false
        # excludes:  # hostnames (may include * components) for which not to use a proxy, like in-cluster repositories.
        # - some.domain
        # # The following configuration sections allow specifying a different proxy to be used for HTTP(S) connections.
        # # If they are omitted, the above configuration is used for HTTP(S) connections as well as TCP connections.
        # # If only the `http` section is given, it will be used for HTTPS connections as well.
        # # Note: in most cases, a single, global proxy configuration is sufficient.
        # http:
        #   url: http://http-proxy.name:port
        #   username: username
        #   password: password
        # https:
        #   url: http://https-proxy.name:port
        #   username: username
        #   password: password
    EOF
  2. Create new permissions required for generating a diagnostic bundle:

    Copy
    kubectl create -f - <<EOF
    kind: Role
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: stackrox-central-diagnostics
      namespace: stackrox
      labels:
        app.kubernetes.io/name: stackrox
    rules:
      - apiGroups:
          - '*'
        resources:
          - '*'
        verbs:
          - get
          - list
    ---
    kind: ClusterRole
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: stackrox:central-diagnostics
      labels:
        app.kubernetes.io/name: stackrox
    rules:
      - apiGroups: ['']
        resources:
          - namespaces
        resourceNames:
          - stackrox
        verbs:
          - get
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata:
      name: stackrox-central-diagnostics
      namespace: stackrox
      labels:
        app.kubernetes.io/name: stackrox
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: Role
      name: stackrox-central-diagnostics
    subjects:
      - kind: ServiceAccount
        name: central
        namespace: stackrox
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
      name: stackrox:central-diagnostics
      labels:
        app.kubernetes.io/name: stackrox
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: stackrox:central-diagnostics
    subjects:
      - kind: ServiceAccount
        name: central
        namespace: stackrox
    EOF
    Copy
    oc create -f - <<EOF
    kind: Role
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: stackrox-central-diagnostics
      namespace: stackrox
      labels:
        app.kubernetes.io/name: stackrox
    rules:
      - apiGroups:
          - '*'
        resources:
          - '*'
        verbs:
          - get
          - list
    ---
    kind: ClusterRole
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: stackrox:central-diagnostics
      labels:
        app.kubernetes.io/name: stackrox
    rules:
      - apiGroups: ['']
        resources:
          - namespaces
        resourceNames:
          - stackrox
        verbs:
          - get
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata:
      name: stackrox-central-diagnostics
      namespace: stackrox
      labels:
        app.kubernetes.io/name: stackrox
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: Role
      name: stackrox-central-diagnostics
    subjects:
      - kind: ServiceAccount
        name: central
        namespace: stackrox
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
      name: stackrox:central-diagnostics
      labels:
        app.kubernetes.io/name: stackrox
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: stackrox:central-diagnostics
    subjects:
      - kind: ServiceAccount
        name: central
        namespace: stackrox
    EOF
  3. Add new configurations for configuring exposed endpoints:

    Copy
    kubectl create -f - <<EOF
    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: central-endpoints
      namespace: stackrox
      labels:
        app.kubernetes.io/name: stackrox
      annotations:
        "helm.sh/hook": "pre-install"
    data:
      endpoints.yaml: |
        # Sample endpoints.yaml configuration for StackRox Central.
        #
        # # CAREFUL: If the following line is uncommented, do not expose the default endpoint on port 8443 by default.
        # #          This will break normal operation.
        # disableDefault: true # if true, don't serve on :8443
        # endpoints:
        #   # Serve plaintext HTTP only on port 8080
        #   - listen: ":8080"
        #     # Backend protocols, possible values are 'http' and 'grpc'. If unset or empty, assume both.
        #     protocols:
        #       - http
        #     tls:
        #       # Disable TLS. If this is not specified, assume TLS is enabled.
        #       disable: true
        #   # Serve HTTP and  gRPC for sensors only on port 8444
        #   - listen: ":8444"
        #     tls:
        #       # Which TLS certificates to serve, possible values are 'service' (StackRox-generated service certificates)
        #       # and 'default' (user-configured default TLS certificate). If unset or empty, assume both.
        #       serverCerts:
        #.        - default
        #         - service
        #       # Client authentication settings.
        #       clientAuth:
        #         # Enforce TLS client authentication. If unset, do not enforce, only request certificates
        #         # opportunistically.
        #         required: true
        #         # Which TLS client CAs to serve, possible values are 'service' (CA for StackRox-generated service
        #         # certificates) and 'user' (CAs for PKI auth providers). If unset or empty, assume both.
        #         certAuthorities: # if not set, assume ["user", "service"]
        #           - service
    EOF
    
    kubectl -n stackrox patch deploy/central -p '
    {
      "spec": {
        "template": {
          "spec": {
            "containers": [{
              "name": "central",
              "volumeMounts": [{
                "name": "endpoints-config-volume",
                "mountPath": "/etc/stackrox.d/endpoints/",
                "readOnly": true
              }]
            }],
            "volumes": [{
              "name": "endpoints-config-volume",
              "configMap": {
                "name": "central-endpoints"
              }
            }]
          }
        }
      }
    }'
    Copy
    oc create -f - <<EOF
    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: central-endpoints
      namespace: stackrox
      labels:
        app.kubernetes.io/name: stackrox
      annotations:
        "helm.sh/hook": "pre-install"
    data:
      endpoints.yaml: |
        # Sample endpoints.yaml configuration for StackRox Central.
        #
        # # CAREFUL: If the following line is uncommented, do not expose the default endpoint on port 8443 by default.
        # #          This will break normal operation.
        # disableDefault: true # if true, don't serve on :8443
        # endpoints:
        #   # Serve plaintext HTTP only on port 8080
        #   - listen: ":8080"
        #     # Backend protocols, possible values are 'http' and 'grpc'. If unset or empty, assume both.
        #     protocols:
        #       - http
        #     tls:
        #       # Disable TLS. If this is not specified, assume TLS is enabled.
        #       disable: true
        #   # Serve HTTP and  gRPC for sensors only on port 8444
        #   - listen: ":8444"
        #     tls:
        #       # Which TLS certificates to serve, possible values are 'service' (StackRox-generated service certificates)
        #       # and 'default' (user-configured default TLS certificate). If unset or empty, assume both.
        #       serverCerts:
        #.        - default
        #         - service
        #       # Client authentication settings.
        #       clientAuth:
        #         # Enforce TLS client authentication. If unset, do not enforce, only request certificates
        #         # opportunistically.
        #         required: true
        #         # Which TLS client CAs to serve, possible values are 'service' (CA for StackRox-generated service
        #         # certificates) and 'user' (CAs for PKI auth providers). If unset or empty, assume both.
        #         certAuthorities: # if not set, assume ["user", "service"]
        #           - service
    EOF
    
    oc -n stackrox patch deploy/central -p '
    {
      "spec": {
        "template": {
          "spec": {
            "containers": [{
              "name": "central",
              "volumeMounts": [{
                "name": "endpoints-config-volume",
                "mountPath": "/etc/stackrox.d/endpoints/",
                "readOnly": true
              }]
            }],
            "volumes": [{
              "name": "endpoints-config-volume",
              "configMap": {
                "name": "central-endpoints"
              }
            }]
          }
        }
      }
    }'
  4. Run the following commands to upgrade:

    Copy
    kubectl label clusterrole/stackrox-central-psp app.kubernetes.io/name=stackrox
    kubectl -n stackrox label rolebinding/stackrox-central-psp app.kubernetes.io/name=stackrox
    kubectl label psp/stackrox-central app.kubernetes.io/name=stackrox
    kubectl -n stackrox patch deploy/central -p '{"spec":{"template":{"spec":{"containers":[{"name":"central","volumeMounts":[{"mountPath":"/run/secrets/stackrox.io/proxy-config/","name":"proxy-config-volume","readOnly":true}]}],"volumes":[{"name":"proxy-config-volume","secret":{"optional":true,"secretName":"proxy-config"}}]}}}}'
    kubectl -n stackrox patch deploy/central -p '{"spec":{"template":{"spec":{"securityContext":{"fsGroup":4000,"runAsUser":4000}}}}}'
    kubectl patch clusterrole/stackrox-central-psp --type='json' -p='[{"op": "replace", "path": "/rules/0/apiGroups", "value": ["policy", "extensions"]}]'
    kubectl -n stackrox patch deploy/central -p '{"spec":{"template":{"metadata":{"annotations":{"traffic.sidecar.istio.io/excludeInboundPorts":"8443"}}}}}'
    kubectl -n stackrox set image deploy/central central=stackrox.io/main:3.0.54.0
    Copy
    oc -n stackrox patch scc central -p '{"priority": 0}'
    oc label clusterrole/stackrox-central-psp app.kubernetes.io/name=stackrox
    oc -n stackrox label rolebinding/stackrox-central-psp app.kubernetes.io/name=stackrox
    oc label psp/stackrox-central app.kubernetes.io/name=stackrox
    oc -n stackrox patch deploy/central -p '{"spec":{"template":{"spec":{"containers":[{"name":"central","volumeMounts":[{"mountPath":"/run/secrets/stackrox.io/proxy-config/","name":"proxy-config-volume","readOnly":true}]}],"volumes":[{"name":"proxy-config-volume","secret":{"optional":true,"secretName":"proxy-config"}}]}}}}'
    oc -n stackrox patch deploy/central -p '{"spec":{"template":{"spec":{"securityContext":{"fsGroup":4000,"runAsUser":4000}}}}}'
    oc patch clusterrole/stackrox-central-psp --type='json' -p='[{"op": "replace", "path": "/rules/0/apiGroups", "value": ["policy", "extensions"]}]'
    oc -n stackrox patch deploy/central -p '{"spec":{"template":{"metadata":{"annotations":{"traffic.sidecar.istio.io/excludeInboundPorts":"8443"}}}}}'
    oc -n stackrox set image deploy/central central=stackrox.io/main:3.0.54.0
    oc -n stackrox patch scc/central -p '{"fsGroup": {"type": "MustRunAs", "ranges":[{"min":4000,"max":4000}]},"runAsUser":{"type":"MustRunAs", "uid":4000}}, "seLinuxContext":"MustRunAs"}'

If you have installed Istio in your cluster, run the following additional commands:

Copy
kubectl create -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: central-internal-no-istio-mtls
  namespace: stackrox
  labels:
    app.kubernetes.io/name: stackrox
  annotations:
    stackrox.io/description: "Disable Istio mTLS for port 443, since StackRox services use built-in mTLS."
spec:
  host: central.stackrox.svc.cluster.local
  trafficPolicy:
    portLevelSettings:
      - port:
          number: 443
        tls:
          mode: DISABLE
EOF
Copy
oc create -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: central-internal-no-istio-mtls
  namespace: stackrox
  labels:
    app.kubernetes.io/name: stackrox
  annotations:
    stackrox.io/description: "Disable Istio mTLS for port 443, since StackRox services use built-in mTLS."
spec:
  host: central.stackrox.svc.cluster.local
  trafficPolicy:
    portLevelSettings:
      - port:
          number: 443
        tls:
          mode: DISABLE
EOF

Download roxctl CLI

Download the roxctl command-line interface (CLI) for the StackRox Kubernetes Security Platform version 3.0.54.0.

  1. Download the roxctl CLI:

    1. By using your browser:

    2. Or by using curl and providing your stackrox.io credentials:

      • On Linux:
        Copy
        curl -O https://mirror.openshift.com/pub/rhacs/assets/3.63.0/bin/Linux/roxctl
      • On macOS:
        Copy
        curl -O https://mirror.openshift.com/pub/rhacs/assets/3.63.0/bin/Darwin/roxctl

After you download the roxctl CLI:

  1. Make the roxctl binary executable and add it to your PATH:

    Copy
    chmod +x roxctl
    sudo mv roxctl /usr/local/bin/roxctl

    If you prefer not to add the roxctl CLI to your PATH, use ./roxctl instead of roxctl for all commands.

  2. Verify that the roxctl version you installed is up-to-date. The most current version is 3.63.0:

    Copy
    roxctl version

StackRox Scanner

Beginning from the StackRox Kubernetes Security Platform version 3.0.39.0, StackRox Scanner uses the Horizontal Pod Autoscaler (HPA). See Scanner autoscaling for more details.

To upgrade StackRox Scanner:

  1. Remove the old version of Scanner:

    Copy
    kubectl -n stackrox delete cm/clair-config deploy/scanner
    Copy
    oc -n stackrox delete cm/clair-config deploy/scanner

    If you are running the StackRox Scanner v2 (preview), you must also run the following commands:

    Copy
    kubectl -n stackrox delete deploy scanner-v2 scanner-v2-db 
    kubectl -n stackrox delete svc scanner-v2 scanner-v2-db
    kubectl -n stackrox delete cm scanner-v2
    kubectl -n stackrox delete secret scanner-v2 scanner-v2-db
    kubectl -n stackrox delete sa scanner-v2 scanner-v2-db
    Copy
    oc -n stackrox delete deploy scanner-v2 scanner-v2-db 
    oc -n stackrox delete svc scanner-v2 scanner-v2-db
    oc -n stackrox delete cm scanner-v2
    oc -n stackrox delete secret scanner-v2 scanner-v2-db
    oc -n stackrox delete sa scanner-v2 scanner-v2-db
  2. Generate a new scanner bundle by using the roxctl CLI:

    Copy
    roxctl -e <central-endpoint>:<port> -p <central-password> scanner generate
  3. Deploy the new StackRox Scanner:

    Copy
    ./scanner-bundle/scanner/scripts/setup.sh
    kubectl apply -R -f ./scanner-bundle/scanner/
    Copy
    ./scanner-bundle/scanner/scripts/setup.sh
    oc apply -R -f ./scanner-bundle/scanner/

Scanner autoscaling is enabled by default when you upgrade the StackRox Kubernetes Security Platform to version 3.0.39 or higher. To opt-out of Scanner autoscaling, run the following commands:

Copy
kubectl -n stackrox delete hpa scanner
kubectl -n stackrox scale --replicas=1 deploy/scanner
Copy
oc -n stackrox delete hpa scanner
oc -n stackrox scale --replicas=1 deploy/scanner

Monitoring

Starting from version 3.0.40.0, we’ve removed StackRox Monitoring components.

If you have deployed Monitoring, run the following commands to remove all relevant objects:

Copy
kubectl -n stackrox delete deploy monitoring
kubectl -n stackrox delete svc/monitoring
kubectl -n stackrox delete svc/monitoring-loadbalancer
kubectl -n stackrox delete route/monitoring-mtls
kubectl -n stackrox delete cm influxdb telegraf telegraf-proxy
kubectl -n stackrox delete secret monitoring monitoring-client
kubectl -n stackrox delete netpol/allow-ext-to-monitoring
kubectl -n stackrox delete sa/monitoring
kubectl -n stackrox delete rolebinding/stackrox-monitoring-psp
kubectl delete clusterrole stackrox-monitoring-psp
kubectl delete psp stackrox-monitoring
kubectl -n stackrox patch deploy/central -p '
spec:
  template:
    spec:
      containers:
      - name: central
        volumeMounts:
        - mountPath: /run/secrets/stackrox.io/monitoring/certs
          "$patch": "delete"
      - name: telegraf
        "$patch": "delete"
      volumes:
      - name: monitoring-client-volume
        "$patch": "delete"
      - name: telegraf-config-volume
        "$patch": "delete"
'
Copy
oc -n stackrox delete deploy monitoring
oc -n stackrox delete svc/monitoring
oc -n stackrox delete svc/monitoring-loadbalancer
oc -n stackrox delete route/monitoring-mtls
oc -n stackrox delete cm influxdb telegraf telegraf-proxy
oc -n stackrox delete secret monitoring monitoring-client
oc -n stackrox delete netpol/allow-ext-to-monitoring
oc -n stackrox delete sa/monitoring
oc -n stackrox delete rolebinding/stackrox-monitoring-psp
oc delete clusterrole stackrox-monitoring-psp
oc delete psp stackrox-monitoring
oc delete scc monitoring
oc -n stackrox patch deploy/central -p '
spec:
  template:
    spec:
      containers:
      - name: central
        volumeMounts:
        - mountPath: /run/secrets/stackrox.io/monitoring/certs
          "$patch": "delete"
      - name: telegraf
        "$patch": "delete"
      volumes:
      - name: monitoring-client-volume
        "$patch": "delete"
      - name: telegraf-config-volume
        "$patch": "delete"
'

Verify deployment

  1. Check that the new pods have deployed:

    Copy
    kubectl get deploy -n stackrox -o wide
    kubectl get pod -n stackrox --watch
    Copy
    oc get deploy -n stackrox -o wide
    oc get pod -n stackrox --watch
  2. Check the Central logs:

    Copy
    kubectl logs -n stackrox deploy/central -c central
    Copy
    oc logs -n stackrox deploy/central -c central

    If the upgrade is successful, you will see output similar to the following:

    Copy
    $ kubectl logs -n stackrox deploy/central -c central
    No database restore directory found (this is not an error).
    Migrator: 2019/10/25 17:58:54: starting DB compaction
    Migrator: 2019/10/25 17:58:54: Free fraction of 0.0391 (40960/1048576) is < 0.7500. Will not compact
    badger 2019/10/25 17:58:54 INFO: All 1 tables opened in 2ms
    badger 2019/10/25 17:58:55 INFO: Replaying file id: 0 at offset: 846357
    badger 2019/10/25 17:58:55 INFO: Replay took: 50.324µs
    badger 2019/10/25 17:58:55 DEBUG: Value log discard stats empty
    Migrator: 2019/10/25 17:58:55: DB is up to date. Nothing to do here.
    badger 2019/10/25 17:58:55 INFO: Got compaction priority: {level:0 score:1.73 dropPrefix:[]}
    version: 2019/10/25 17:58:55.189866 ensure.go:49: Info: Version found in the DB was current. We’re good to go!
    Copy
    $ oc logs -n stackrox deploy/central -c central
    No database restore directory found (this is not an error).
    Migrator: 2019/10/25 17:58:54: starting DB compaction
    Migrator: 2019/10/25 17:58:54: Free fraction of 0.0391 (40960/1048576) is < 0.7500. Will not compact
    badger 2019/10/25 17:58:54 INFO: All 1 tables opened in 2ms
    badger 2019/10/25 17:58:55 INFO: Replaying file id: 0 at offset: 846357
    badger 2019/10/25 17:58:55 INFO: Replay took: 50.324µs
    badger 2019/10/25 17:58:55 DEBUG: Value log discard stats empty
    Migrator: 2019/10/25 17:58:55: DB is up to date. Nothing to do here.
    badger 2019/10/25 17:58:55 INFO: Got compaction priority: {level:0 score:1.73 dropPrefix:[]}
    version: 2019/10/25 17:58:55.189866 ensure.go:49: Info: Version found in the DB was current. We’re good to go!

If you are upgrading from the StackRox Kubernetes Security Platform version 3.0.45 or older, Central may take longer to start as it migrates to a new, more efficient database.

You have successfully upgraded StackRox Central.

Upgrade all secured clusters

The following commands are a cumulative set of commands that completes the upgrade of the StackRox Kubernetes Security Platform from 3.0.31 to 3.0.54.

You might see errors when you run these commands:

  • if you completed an upgrade that previously ran any of these commands, or
  • if you aren’t using some features of the StackRox Kubernetes Security Platform that these commands configure.

You can ignore these errors and proceed with the rest of the instructions.

After upgrading Central services, you must upgrade all secured clusters, including the Central cluster.

You can use automatic upgrades to finish the upgrade process. Once you have finished automatic upgrades of your secured clusters, skip to the clean up step.

To complete manual upgrades of each secured cluster running the StackRox Sensor and Collector, follow the instructions below.

In the StackRox Kubernetes Security Platform version 3.0.40, we’ve reduced the computational load on Central by moving a few capabilities from Central to Sensor. Therefore, you must upgrade all your Sensor deployments.

Update admission controller

In the StackRox Kubernetes Security Platform version 3.0.41, we’ve made changes to our admission controller integration. If you are using admission controller you must run some additional commands to upgrade the secured cluster.

To check if you are using an admission controller with the StackRox Kubernetes Security Platform, run the following command:

Copy
kubectl get validatingwebhookconfiguration stackrox
Copy
oc get validatingwebhookconfiguration stackrox

If you get an error, skip this section and continue with the instructions in the update images section.

Otherwise, if you don’t get any errors, follow these additional instructions:

  1. Delete the existing validating webhook configuration:

    Copy
    kubectl delete validatingwebhookconfiguration stackrox
    Copy
    oc delete validatingwebhookconfiguration stackrox
  2. If you want to allow the admission controller to enforce on updates moving forward, navigate to the Clusters configuration page, select the cluster you are updating, and enable the Configure Admission Controller Webhook to listen on updates option.

  3. Obtain a new sensor bundle for the respective cluster. You can do so using the CLI:

    Copy
    roxctl sensor get-bundle <cluster-name>

    or via the Clusters configuration page. Extract the bundle (if necessary), and cd into the bundle directory.

  4. Create the TLS secrets for the admission-control service:

    Copy
    kubectl create secret -n "stackrox" generic admission-control-tls --from-file="admission-control-cert.pem" --from-file="admission-control-key.pem" --from-file="ca.pem"
    kubectl -n "stackrox" label secret/admission-control-tls 'auto-upgrade.stackrox.io/component=sensor'
    Copy
    oc create secret -n "stackrox" generic admission-control-tls --from-file="admission-control-cert.pem" --from-file="admission-control-key.pem" --from-file="ca.pem"
    oc -n "stackrox" label secret/admission-control-tls 'auto-upgrade.stackrox.io/component=sensor'
  5. Create the admission-control deployment and related objects:

    Copy
    kubectl apply -f admission-controller.yaml
    Copy
    oc apply -f admission-controller.yaml

Update images

  1. Run the following commands to update images and make other necessary configurations:

    Copy
    kubectl label clusterrole/stackrox-sensor-psp app.kubernetes.io/name=stackrox
    kubectl label clusterrole/stackrox-collector-psp app.kubernetes.io/name=stackrox
    kubectl -n stackrox label rolebinding/stackrox-sensor-psp app.kubernetes.io/name=stackrox
    kubectl -n stackrox label rolebinding/stackrox-collector-psp app.kubernetes.io/name=stackrox
    kubectl label psp/stackrox-sensor app.kubernetes.io/name=stackrox
    kubectl label psp/stackrox-collector app.kubernetes.io/name=stackrox
    kubectl patch clusterrole/stackrox-sensor-psp --type='json' -p='[{"op": "replace", "path": "/rules/0/apiGroups", "value": ["policy", "extensions"]}]'
    kubectl patch clusterrole/stackrox-collector-psp --type='json' -p='[{"op": "replace", "path": "/rules/0/apiGroups", "value": ["policy", "extensions"]}]'
    kubectl -n stackrox patch ds/collector -p '
    spec:
      template:
        spec:
          containers:
          - name: compliance
            securityContext:
              readOnlyRootFilesystem: true
              seLinuxOptions:
                type: "container_runtime_t"
            volumeMounts:
            - name: "etc-pki-volume"
              mountPath: "/etc/pki/ca-trust/"
            - mountPath: "/host/usr/sbin"
              name: "usr-sbin"
              readOnly : true
          volumes:
          - hostPath:
              path: "/usr/sbin"
            name: "usr-sbin"
          - emptyDir:
            name: "etc-pki-volume"
    '
    kubectl -n stackrox patch psp/stackrox-collector -p '{"spec":{"allowedHostPaths":[{"pathPrefix":"/","readOnly":true}]}}'
    kubectl -n stackrox patch ds/collector -p '{"spec":{"template":{"spec":{"containers":[{"name":"collector","volumeMounts":[{"mountPath":"/host/var/run/docker.sock","readOnly":true}]},{"name":"compliance","volumeMounts":[{"mountPath":"/host/var/lib","$patch":"delete"},{"mountPath":"/host/var/log/audit","$patch":"delete"},{"mountPath":"/host/usr/sbin","$patch":"delete"},{"mountPath":"/host/run","$patch":"delete"},{"mountPath":"/host/lib","$patch":"delete"},{"mountPath":"/host/usr/lib","$patch":"delete"},{"mountPath":"/host/etc","$patch":"delete"},{"mountPath":"/host/usr/bin","$patch":"delete"},{"mountPath":"/host/proc","$patch":"delete"},{"mountPath":"/host/var/run/docker.sock","$patch":"delete"},{"mountPath":"/host","name":"host-root-ro","readOnly":true}]}],"volumes":[{"name":"var-lib","$patch":"delete"},{"name":"var-log-audit","$patch":"delete"},{"name":"usr-sbin","$patch":"delete"},{"name":"run","$patch":"delete"},{"name":"lib","$patch":"delete"},{"name":"usr-lib","$patch":"delete"},{"hostPath":{"path":"/"},"name":"host-root-ro"}]}}}}'
    kubectl -n stackrox patch deploy/sensor -p '{"spec":{"template":{"metadata":{"annotations":{"traffic.sidecar.istio.io/excludeInboundPorts":"8443,9443"}}}}}'
    kubectl -n stackrox patch deploy/sensor -p '{"spec":{"template":{"spec":{"containers":[{"name":"sensor","resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"1Gi"}}}]}}}}'
    kubectl -n stackrox set image deploy/sensor sensor=stackrox.io/main:3.0.54.0
    kubectl -n stackrox set image ds/collector compliance=stackrox.io/main:3.0.54.0
    kubectl -n stackrox set image ds/collector collector=collector.stackrox.io/collector:3.1.10-latest
    Copy
    oc -n stackrox patch scc sensor -p '{"priority": 0}'
    oc -n stackrox patch scc collector -p '{"priority": 0}'
    oc label clusterrole/stackrox-sensor-psp app.kubernetes.io/name=stackrox
    oc label clusterrole/stackrox-collector-psp app.kubernetes.io/name=stackrox
    oc -n stackrox label rolebinding/stackrox-sensor-psp app.kubernetes.io/name=stackrox
    oc -n stackrox label rolebinding/stackrox-collector-psp app.kubernetes.io/name=stackrox
    oc label psp/stackrox-sensor app.kubernetes.io/name=stackrox
    oc label psp/stackrox-collector app.kubernetes.io/name=stackrox
    oc patch clusterrole/stackrox-sensor-psp --type='json' -p='[{"op": "replace", "path": "/rules/0/apiGroups", "value": ["policy", "extensions"]}]'
    oc patch clusterrole/stackrox-collector-psp --type='json' -p='[{"op": "replace", "path": "/rules/0/apiGroups", "value": ["policy", "extensions"]}]'
    oc -n stackrox patch ds/collector -p '
    spec:
      template:
        spec:
          containers:
          - name: compliance
            securityContext:
              readOnlyRootFilesystem: true
              seLinuxOptions:
                type: "container_runtime_t"
            volumeMounts:
            - name: "etc-pki-volume"
              mountPath: "/etc/pki/ca-trust/"
            - mountPath: "/host/usr/sbin"
              name: "usr-sbin"
              readOnly : true
          volumes:
          - hostPath:
              path: "/usr/sbin"
            name: "usr-sbin"
          - emptyDir:
            name: "etc-pki-volume"
    '
    oc -n stackrox patch psp/stackrox-collector -p '{"spec":{"allowedHostPaths":[{"pathPrefix":"/","readOnly":true}]}}'
    oc -n stackrox patch ds/collector -p '{"spec":{"template":{"spec":{"containers":[{"name":"collector","volumeMounts":[{"mountPath":"/host/var/run/docker.sock","readOnly":true}]},{"name":"compliance","volumeMounts":[{"mountPath":"/host/var/lib","$patch":"delete"},{"mountPath":"/host/var/log/audit","$patch":"delete"},{"mountPath":"/host/usr/sbin","$patch":"delete"},{"mountPath":"/host/run","$patch":"delete"},{"mountPath":"/host/lib","$patch":"delete"},{"mountPath":"/host/usr/lib","$patch":"delete"},{"mountPath":"/host/etc","$patch":"delete"},{"mountPath":"/host/usr/bin","$patch":"delete"},{"mountPath":"/host/proc","$patch":"delete"},{"mountPath":"/host/var/run/docker.sock","$patch":"delete"},{"mountPath":"/host","name":"host-root-ro","readOnly":true}]}],"volumes":[{"name":"var-lib","$patch":"delete"},{"name":"var-log-audit","$patch":"delete"},{"name":"usr-sbin","$patch":"delete"},{"name":"run","$patch":"delete"},{"name":"lib","$patch":"delete"},{"name":"usr-lib","$patch":"delete"},{"hostPath":{"path":"/"},"name":"host-root-ro"}]}}}}'
    oc -n stackrox patch deploy/sensor -p '{"spec":{"template":{"metadata":{"annotations":{"traffic.sidecar.istio.io/excludeInboundPorts":"8443,9443"}}}}}'
    oc -n stackrox patch deploy/sensor -p '{"spec":{"template":{"spec":{"containers":[{"name":"sensor","resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"1Gi"}}}]}}}}'
    oc -n stackrox set image deploy/sensor sensor=stackrox.io/main:3.0.54.0
    oc -n stackrox set image ds/collector compliance=stackrox.io/main:3.0.54.0
    oc -n stackrox set image ds/collector collector=collector.stackrox.io/collector:3.1.10-latest
  2. If you have deployed Monitoring, run the following commands to remove all relevant objects:

    Copy
    kubectl -n stackrox delete cm telegraf
    kubectl -n stackrox delete secret monitoring-client
    kubectl -n stackrox patch deploy/sensor -p '
    spec:
      template:
        spec:
          containers:
          - name: telegraf
            "$patch": "delete"
          volumes:
          - name: monitoring-client-volume
            "$patch": "delete"
          - name: telegraf-config-volume
            "$patch": "delete"
    '
    kubectl -n stackrox patch ds/collector -p '
    spec:
      template:
        spec:
          containers:
          - name: telegraf
            "$patch": "delete"
          volumes:
          - name: monitoring-client-volume
            "$patch": "delete"
          - name: telegraf-config-volume
            "$patch": "delete"
    '
    Copy
    oc -n stackrox delete cm telegraf
    oc -n stackrox delete secret monitoring-client
    oc -n stackrox patch deploy/sensor -p '
    spec:
      template:
        spec:
          containers:
          - name: telegraf
            "$patch": "delete"
          volumes:
          - name: monitoring-client-volume
            "$patch": "delete"
          - name: telegraf-config-volume
            "$patch": "delete"
    '
    oc -n stackrox patch ds/collector -p '
    spec:
      template:
        spec:
          containers:
          - name: telegraf
            "$patch": "delete"
          volumes:
          - name: monitoring-client-volume
            "$patch": "delete"
          - name: telegraf-config-volume
            "$patch": "delete"
    '

For any secured cluster in which you have installed Istio, run the following additional commands:

Copy
kubectl create -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: sensor-internal-no-istio-mtls
  namespace: stackrox
  labels:
    app.kubernetes.io/name: stackrox
  annotations:
    stackrox.io/description: "Disable Istio mTLS for port 443, since StackRox services use built-in mTLS."
spec:
  host: sensor.stackrox.svc.cluster.local
  trafficPolicy:
    portLevelSettings:
      - port:
          number: 443
        tls:
          mode: DISABLE
EOF
Copy
oc create -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: sensor-internal-no-istio-mtls
  namespace: stackrox
  labels:
    app.kubernetes.io/name: stackrox
  annotations:
    stackrox.io/description: "Disable Istio mTLS for port 443, since StackRox services use built-in mTLS."
spec:
  host: sensor.stackrox.svc.cluster.local
  trafficPolicy:
    portLevelSettings:
      - port:
          number: 443
        tls:
          mode: DISABLE
EOF

Verify deployment

Check that the updated pods have successfully deployed before continuing:

Copy
kubectl get deploy,ds -n stackrox -o wide
kubectl get pod -n stackrox --watch
Copy
oc get deploy,ds -n stackrox -o wide
oc get pod -n stackrox --watch

The upgrade process is now complete.

Clean up

For most browsers, you must reload the page and re-accept the certificate to continue using the StackRox portal.

Revoke the API token

For security reasons, StackRox recommends revoking the API token you used to complete the database backup.

To revoke an API token:

  1. On the StackRox portal, navigate to Platform Configuration > Integrations.
  2. Scroll down to the Authentication Tokens category, and select API Token.
  3. Select the checkbox in front of the token name you want to revoke.
  4. Select Revoke.
  5. On the confirmation dialog box, click Confirm.

The updated Sensors and Collectors will continue to report the latest data from each secured cluster.

The last time a Sensor contacted Central is visible in the Clusters view.

To view the Last check-in time:

  1. On the StackRox portal, navigate to Platform Configuration > Clusters.
  2. View the Last check-in time for each cluster.

If any sensor hasn’t checked in for more than five minutes, check the cluster logs for that Sensor to ensure that it’s operating as usual.

The displayed check-in time doesn’t update automatically. Reload the page to see the updates.

Contact StackRox support if you experience any issues.

Questions?

We're happy to help! Reach out to us to discuss questions, issues, or feature requests.

© 2021 StackRox Inc. All rights reserved.