Rolling out changes¶

A cluster runs what Git holds at the revision its fleet file pins. To ship a change, commit it and point fleet files at a revision that has it.

Onboarding a cluster¶

1. Import the cluster into ACM. The GitOpsCluster adds it to Argo CD (Install PolicyStack).
2. Add values/clusters/<cluster>.yaml with the elements this cluster runs beyond its other layers.

3. PR fleet/<cluster>.yaml to the default branch:

   # fleet/prod-east-1.yaml
   revision: main # (1)!
   valueFiles: # (2)!
     - environments/prod.yaml
     - datacenters/dc1.yaml
     - platforms/aws.yaml
   

   1. Branch, tag or SHA.
   2. Under values/, lowest precedence first. values/clusters/prod-east-1.yaml is added last (Order).

Within about 6 minutes there is one Application per element, named <element>-<cluster>. An element no layer enables still gets an Application, with no resources.

oc get applications.argoproj.io -n openshift-gitops | grep prod-east-1

For a cluster the hub builds, skip step 1 and add hub, version and install to the fleet file (Cluster lifecycle). The hub itself uses fleet/hubs/<hubName>.yaml and values/clusters/<hubName>.yaml (The hub).

Promoting and rolling back¶

A branch revision picks up every push. A tag or SHA changes only when revision does. Promote with a PR; its Render Diff comment shows the Policy changes.

 # fleet/prod-east-1.yaml
-revision: v1.4.0
+revision: v1.5.0

Roll back with git revert, or a PR that sets the old revision. Both need the usual fleet/ review (Protection).

• Application names don't include the revision, so Argo CD updates the same Policies in place.
• The Placement matches only the cluster's name label, so policies stay bound during the sync.
• An element added at the new revision gets an Application. One missing loses it and its Policies (Removing an element).

Dry run¶

Set global.remediationAction: inform in any layer (Global keys). Every Policy for clusters reading that layer, lifecycle included, reports instead of enforcing. To preview a revision on one cluster, add this in the same PR and remove it once the Policies look right:

# values/clusters/prod-east-1.yaml
global:
  remediationAction: inform # (1)!

1. enforce is rejected. Remove the key to go back to each template's own setting.

Disabling an element¶

Set enabled: false in a layer that outranks every layer enabling it (Order). environments/prod.yaml enables dns; to turn it off on one cluster:

# values/clusters/prod-east-1.yaml
stack:
  dns:
    enabled: false
  remediations:
    toggles:
      samples: false # (1)!

1. A toggle turns off one entry of an element.

The element renders nothing, sync prunes it from the hub (allowEmpty is set), and ACM removes its policies from the cluster (What happens on the managed cluster). A failed render prunes nothing: the Application shows ComparisonError / Unknown and the Policies stay.

Removing an element¶

Delete stack/<element>. When a cluster's revision no longer has it, the ApplicationSet deletes the Application and the resources finalizer deletes its Policies, PolicySets, Placement and PlacementBinding. Clusters pinned to older revisions keep it until their pin moves.

Applications are also deleted when:

A fleet file that doesn't parse, or a missing revision, deletes nothing (Revision).

Offboarding a cluster¶

Delete the fleet file in a PR. Its Applications and Policies go with it. If the file sets hub, choose what happens to the cluster:

What happens on the managed cluster¶

The policy goes away; what it created stays unless the template sets pruneObjectBehavior, is enforce, and that setting reached the cluster before the removal.

• POLICY033 rejects prune on a template with any dependency (waitForOperator, extraDependencies, the policy's dependencies, or a place after another template/policy under orderManifests/orderPolicies). ACM deletes a template while a dependency is unmet, such as during an operator upgrade, and prune would delete its objects (Validation).
• Keep the Placement's default tolerations. With placement.tolerations: [], a disconnected cluster drops out and loses its policies.

Pausing removals¶

Do this before an ACM upgrade or work on Argo CD cluster secrets. The ACM 2.17 upgrade can swap a cluster's secret for a renamed one and briefly drop its Applications and Policies.

# pause deletes; creates and updates continue
oc patch applicationset policystack -n openshift-gitops --type merge \
  -p '{"spec":{"syncPolicy":{"applicationsSync":"create-update"}}}'

# resume (helm upgrade does not remove it)
oc patch applicationset policystack -n openshift-gitops --type json \
  -p '[{"op":"remove","path":"/spec/syncPolicy/applicationsSync"}]'

Operator upgrades¶

Each operatorPolicies entry renders an OperatorPolicy named after its key (Operator Policy Options).

• ACM rejects Manual and subscription.installPlanApproval.
• Inform approves nothing; it flags an installed CSV missing from a non-empty versions.
• versions holds exact CSV names, no wildcards. Comma-separated names need ACM 2.14+, templates 2.13+.
• The hub's ACM and GitOps follow the same rules once their elements are on (Hand the operators to PolicyStack).

Elements ship unpinned. operatorPolicies is a map, so a layer sets only what it changes (Merge rules):

# values/environments/prod.yaml
stack:
  metallb:
    operatorPolicies:
      metallb: # (1)!
        versions:
          - <installed CSV> # (2)!
        subscription:
          startingCSV: <installed CSV> # (3)!

1. Must be a key the element defines. A new key fails POLICY032 (Validation) and the render.
2. oc get csv -n metallb-system on the managed cluster.
3. Without it, a fresh cluster resolves the channel head, which the pin blocks. Omit it when the pin is the head.

To bump, watch the OperatorPolicy status for an InstallPlan to update to [<csv>]. OLM picks that CSV from the upgrade graph; the policy only approves or holds it.

1. Add the new CSV next to the old one. Promote cluster by cluster (Promoting and rolling back).
2. Once all clusters run it, drop the old CSV, move startingCSV, and promote again.

Swapping old for new in one change makes every cluster NonCompliant until it upgrades.

Stop managing and Uninstall both fail the render unless entries naming the key in waitForOperator are turned off in the same layer.

# values/clusters/prod-east-1.yaml: uninstall metallb
stack:
  metallb:
    toggles:
      metallb-instance: false # (1)!
    operatorPolicies:
      metallb:
        complianceType: mustnothave # (2)!

1. The instance waits on metallb, so it goes off in the same layer.
2. removalBehavior changes what is deleted.