Kubernetes Cluster Management and Cloud Automation –

Verify the provider installed with kubectl get providers

kubectl get providers NAME                          INSTALLED   HEALTHY   PACKAGE                                                AGE provider-gcp-storage          True        True      xpkg.upbound.io/upbound/provider-gcp-storage:v0.35.0   8m56s upbound-provider-family-gcp   True        True      xpkg.upbound.io/upbound/provider-family-gcp:v0.36.0    8m50s

Create a Bucket for Each Managed Cluster

In this tutorial, we want Sveltos to coordinate with Crossplane to create a Google Storage Bucket for each managed cluster and then deploy an application in each managed cluster that uploads a file to the proper bucket.

This requires solving the following problems:

1. The operations must be run in a precise order. For each matching cluster, first, Projectsveltos must instruct Crossplane to create a Google Storage Bucket. Only after Crossplane is done creating the bucket Projectsveltos can proceed further deploying a Pod in the managed cluster that will upload a file to the bucket.
2. The Pod must be passed the URL of the Google Storage Bucket to use. Projectsveltos will fetch this information from the management cluster (the Crossplane Bucket instances) and will use it to instantiate a template representing a Pod that will upload a file to the storage bucket.

Let's build the Projectsveltos ClusterProfile YAML to achieve all that.

Crossplane Creating Google Storage Bucket

A Bucket instance is a Kubernetes custom resource (CR) that instructs Crossplane to create a Google Cloud Storage bucket. Crossplane will then update the status section of the CR with information about the bucket, such as its name, location, and URL.

apiVersion: storage.gcp.upbound.io/v1beta1 kind: Bucket metadata:   name: crossplane-bucket-default-clusterapi-workload   ... spec:   forProvider:     location: US     project: projectsveltos     publicAccessPrevention: inherited     storageClass: STANDARD   ... status:   atProvider:     ...     id: crossplane-bucket-default-clusterapi-workload     ...     url: gs://crossplane-bucket-default-clusterapi-workload

To instruct Projectsveltos to create a Bucket instance[1]:

apiVersion: config.projectsveltos.io/v1alpha1 kind: ClusterProfile spec:   ...   policyRefs:   - deploymentType: Local     kind: ConfigMap     name: bucket     namespace: default --- apiVersion: v1 kind: ConfigMap metadata:      name: bucket   namespace: default   annotations:     projectsveltos.io/template: "true" data:          bucket.yaml: |     apiVersion: storage.gcp.upbound.io/v1beta1     kind: Bucket     metadata:      name: crossplane-bucket-{{ .Cluster.metadata.namespace }}-{{ .Cluster.metadata.name }}      labels:        docs.crossplane.io/example: provider-gcp        clustername: {{ .Cluster.metadata.name }}        clusternamespace: {{ .Cluster.metadata.namespace }}     spec:       forProvider:         location: US       providerConfigRef:         name: default

Above is instructing Projectsveltos to:

1. Fetch the ConfigMap named bucket in the namespace default.
2. Take the content of the ConfigMap (the data section).
3. This content is a template for a Crossplane Bucket instance. Projectsveltos will instantiate the template before deploying it (the projectsveltos.io/template annotation is set).
4. The template refers to the namespace and name of the ClusterAPI-powered cluster. This ensures that Projectsveltos creates a single Crossplane Bucket instance for each matching cluster.
5. After the template is instantiated, Projectsveltos will deploy it to the management cluster (since the ClusterProfile.PolicyRefsdeploymentType is Local).

Once Projectsveltos creates a Bucket instance in the management cluster, Crossplane takes over and creates a Google Cloud Storage bucket. Projectsveltos then pauses and waits for Crossplane to finish creating the bucket.

Deploy Pod To Managed Cluster

When Crossplane finishes creating a Google Cloud Storage bucket, we need Projectsveltos to resume and deploy a Pod in the managed cluster. This Pod needs information about the bucket, such as its name and URL. Crossplane has added this information to the status section of the Bucket CR. 

apiVersion: config.projectsveltos.io/v1alpha1 kind: ClusterProfile metadata:   name: deploy-resources spec:   templateResourceRefs:   - resource:       apiVersion: storage.gcp.upbound.io/v1beta1       kind: Bucket       name: crossplane-bucket-{{ .ClusterNamespace }}-{{ .ClusterName }}     identifier: CrossplaneBucket

The YAML section that instructs Projectsveltos to fetch the Bucket CR is as follows:

In the templateResourceRefs section, add a resource reference to the Bucket CR. The resource reference should specify the following:

• The API version and kind of the Bucket CR.
• The name of the Bucket CR, which is a template that depends on the cluster namespace and name.
• An identifier for the resource reference. In this example, we are using the identifier CrossplaneBucket.

Projectsveltos will fetch all of the management cluster resources listed in the templateResourceRefs section before trying to deploy any resources to the managed cluster. In this specific example, we are telling Projectsveltos to fetch the Bucket CR instance, whose name is a template that depends on the cluster namespace and name.

Sveltos will react to changes to any resources listed in the templateResourceRefs section. Once the Bucket CR instance is updated by Crossplane, Sveltos will fetch it and then deploy the Pod in the managed cluster.

apiVersion: config.projectsveltos.io/v1alpha1 kind: ClusterProfile metadata:   name: deploy-resources spec:   ...   policyRefs:   ...   - deploymentType: Remote     kind: ConfigMap     name: uploader     namespace: default --- apiVersion: v1 kind: ConfigMap metadata:   name: uploader   namespace: default   annotations:     projectsveltos.io/template: "true" data:   ...   pod.yaml: |     apiVersion: v1     kind: Pod     metadata:       name: create-and-upload-to-gcs       namespace: default       annotations:         bucket: {{ (index .MgtmResources "CrossplaneBucket").status.atProvider.url }}     spec:       containers:       - name: uploader         image: google/cloud-sdk:slim         command: ["bash"]         args:           - "-c"           - |             echo "Hello world" > /tmp/hello.txt             gcloud auth activate-service-account --key-file=/var/run/secrets/cloud.google.com/service-account.json             gsutil cp /tmp/hello.txt gs://{{ (index .MgtmResources "CrossplaneBucket").metadata.name }}         volumeMounts:           - name: gcp-sa             mountPath: /var/run/secrets/cloud.google.com/             readOnly: true       volumes:         - name: gcp-sa           secret:             secretName: gcs-credentials

Above is the YAML section, which instructs Sveltos to deploy Pod in the managed cluster:

1. Add a reference to the uploader ConfigMap in the policyRefs section. The deploymentType is set to Remote because we want Sveltos to deploy the content of the ConfigMap in the managed cluster.
2. The ConfigMap data section contains a Pod expressed as a template (the projectsveltos.io/template annotation is set). This template references information from the Bucket CR that Sveltos fetched in the management cluster.
3. Sveltos will instantiate this template and deploy it to managed cluster.

Complete YAML

Following YAML is all that is required. A ClusterProfile and two ConfigMap instances.

Since the Pod also needs credentials to write to bucket, Sveltos is also deploying a Secret with GCP credentials that Pod will mount and use.

apiVersion: config.projectsveltos.io/v1alpha1 kind: ClusterProfile metadata:   name: deploy-resources spec:   clusterSelector: env=fv   templateResourceRefs:   - resource:       apiVersion: storage.gcp.upbound.io/v1beta1       kind: Bucket       name: crossplane-bucket-{{ .ClusterNamespace }}-{{ .ClusterName }}     identifier: CrossplaneBucket   - resource:       apiVersion: v1       kind: Secret       namespace: crossplane-system       name: gcp-secret     identifier: Credentials   policyRefs:   - deploymentType: Local     kind: ConfigMap     name: bucket     namespace: default   - deploymentType: Remote     kind: ConfigMap     name: uploader     namespace: default --- apiVersion: v1 kind: ConfigMap metadata:      name: bucket   namespace: default   annotations:     projectsveltos.io/template: "true" data:          bucket.yaml: |     apiVersion: storage.gcp.upbound.io/v1beta1     kind: Bucket     metadata:      name: crossplane-bucket-{{ .Cluster.metadata.namespace }}-{{ .Cluster.metadata.name }}      labels:        docs.crossplane.io/example: provider-gcp        clustername: {{ .Cluster.metadata.name }}        clusternamespace: {{ .Cluster.metadata.namespace }}     spec:       forProvider:         location: US       providerConfigRef:         name: default --- apiVersion: v1 kind: ConfigMap metadata:   name: uploader   namespace: default   annotations:     projectsveltos.io/template: "true" data:   secret.yaml: |     apiVersion: v1     kind: Secret     metadata:       name: gcs-credentials       namespace: default       annotations:         bucket: "{{ (index .MgtmResources "CrossplaneBucket").status.atProvider.url }}"     type: Opaque     data:       service-account.json: {{ $data:=(index .MgtmResources "Credentials").data }} {{ (index $data "creds") }}   pod.yaml: |     apiVersion: v1     kind: Pod     metadata:       name: create-and-upload-to-gcs       namespace: default       annotations:         bucket: {{ (index .MgtmResources "CrossplaneBucket").status.atProvider.url }}     spec:       containers:       - name: uploader         image: google/cloud-sdk:slim         command: ["bash"]         args:           - "-c"           - |             echo "Hello world" > /tmp/hello.txt             gcloud auth activate-service-account --key-file=/var/run/secrets/cloud.google.com/service-account.json             gsutil cp /tmp/hello.txt gs://{{ (index .MgtmResources "CrossplaneBucket").metadata.name }}         volumeMounts:           - name: gcp-sa             mountPath: /var/run/secrets/cloud.google.com/             readOnly: true       volumes:         - name: gcp-sa           secret:             secretName: gcs-credentials

• We created a ClusterProfile that instructs Sveltos to create a Bucket CR in the management cluster.
• We told Sveltos to wait for Crossplane to create a Google Cloud Storage Bucket, fetch the Bucket CR and then deploy a Pod in the managed cluster.
• The Pod will upload a file to the Bucket CR using the GCP credentials that Sveltos deployed in the managed cluster.

Summary

• ClusterAPI provides a consistent way to manage Kubernetes clusters across different infrastructure providers. This makes it easy to deploy and manage Kubernetes clusters in a variety of environments.
• Projectsveltos provides a way to manage Kubernetes add-ons and applications using GitOps principles. This makes it easy to automate the deployment and management of Kubernetes add-ons and to track changes to the cluster configuration.
• Crossplane provides a way to manage infrastructure resources using Kubernetes Custom Resource Definitions (CRDs). This makes it easy to manage infrastructure resources in a consistent way and to automate the provisioning and management of infrastructure resources.

By using ClusterAPI, Projectsveltos, and Crossplane together, you can get the following benefits:

• Consistency: You can manage Kubernetes clusters and infrastructure resources in a consistent way, regardless of the underlying infrastructure provider.
• Automation: You can automate the deployment and management of Kubernetes clusters and infrastructure resources, which can save time and effort.
• Track-ability: You can track changes to the cluster configuration and infrastructure resources, which can help you to troubleshoot problems and to audit changes.
• Reliability: You can improve the reliability of your Kubernetes clusters and infrastructure resources by automating the provisioning and management of these resources.

Support This Project

If you enjoyed this article, please check out the Sveltos GitHub repo and star the project if you found it helpful.

The GitHub repo is a great resource for getting started with Sveltos. It contains the code, documentation, and examples. You can also find the latest news and updates on the project on the GitHub repo.

If you have any feedback, bugs, or PRs, please feel free to contribute to the project. Your contributions will help make Sveltos even better.

And finally, please help spread the word about Sveltos by starring the repo or sharing it with your friends and colleagues. Thank you for reading!

[¹]: Sveltos needs to be granted permission to manage lifecycle of Crossplane Bucket instance. Add following RBACs.

kubectl edit clusterrole addon-controller-role-extra

- apiGroups:   - storage.gcp.upbound.io   resources:   - buckets   verbs:   - get   - list   - watch   - patch   - delete   - create