Here I automate to destroy GKE resource on weekend and re-provision it on Monday. I use terraform as IaC script, Terraform cloud as the backend, kubernetes job to run auto trigger apply/destroy, and terraformer to generate TF script from existing resource.

Create Terraform Script from Existing Resource

I use Terraformer as a tool to reverse terraform script (from existing resources → tf script)

Install terraformer :

1
2
3
4
export PROVIDER={all,google,aws,kubernetes} # in this case i choose google
curl -LO "https://github.com/GoogleCloudPlatform/terraformer/releases/download/$(curl -s https://api.github.com/repos/GoogleCloudPlatform/terraformer/releases/latest | grep tag_name | cut -d '"' -f 4)/terraformer-${PROVIDER}-linux-amd64"
chmod +x terraformer-${PROVIDER}-linux-amd64
sudo mv terraformer-${PROVIDER}-linux-amd64 /usr/local/bin/terraformer

Init Terraform and authentication service account :

export GOOGLE_APPLICATION_CREDENTIALS=credential.json
create main.tf
1. terraform { required_providers { google = { source = "hashicorp/google" version = "~> 4.0.0" } } } provider "google" { region = "asia-southeast1" zone = "asia-southeast1-a" project = "api-project-928654461237" }
Run terraform init

Run terraform import

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
takatux@takatux-legion:/mnt/d/LAB/Personal Projects/terraformer/existing-terraform$ terraformer import google --resources=gke --filter=name=[Your Cluster Name] --projects="[API Projects Here]" --regions="asia-southeast1"
2023/07/13 16:35:34 google importing project ....
2023/07/13 16:35:40 google importing... gke
2023/07/13 16:35:40 google done importing gke
2023/07/13 16:35:40 Number of resources for service gke: 21
2023/07/13 16:35:40 Refreshing state... google_container_cluster.tfer--
2023/07/13 16:35:41 Refreshing state... google_container_cluster.tfer--
...

16:35:47 Filtered number of resources for service gke: 21
2023/07/13 16:35:47 google Connecting....
2023/07/13 16:35:47 google save gke
2023/07/13 16:35:48 google save tfstate for gke

Generated Terraform script wiill be in generated directory. My directory looks like this

Generated TF Script

I had an issue when trying to use terraformer filter to get a specific cluster name. This such command doesn’t work : terraformer import google --resources=gke --filter=name=aruna-new-staging --projects="api-project-928654461237" --regions="asia-southeast1".

So I use terraformer import plan then custom the json files. It did the trick.

Create Cronjob to Automate Create and Destroy

Cronjob manifest (destroy.yaml) :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
apiVersion: batch/v1
kind: CronJob
metadata:
  name: destroy-new-staging
  namespace: cronjobs
spec:
  schedule: "0 21 * * 5"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: destroy-new-staging
            image: curlimages/curl
            imagePullPolicy: IfNotPresent
            command: ["/bin/sh", "-c"]
            args:
            - |
             curl --header "Authorization: Bearer $TOKEN" --header "Content-Type: application/vnd.api+json" --request POST --data @/etc/payload/destroy-workspace-new-staging-payload.json https://app.terraform.io/api/v2/runs
            env:
            - name: TOKEN
              valueFrom:
                secretKeyRef:
                  name: terraform-token
                  key: TERRAFORM_TOKEN
            volumeMounts:
            - name: payload-destroy
              mountPath: /etc/payload
          volumes:
            - name: payload-destroy
              configMap:
                name: payload-destroy
          restartPolicy: OnFailure
---
apiVersion: v1
data:
  destroy-workspace-new-staging-payload.json: "{\r\n    \"data\": {\r\n      \"attributes\": {\r\n        \"auto-apply\":
    true,\r\n        \"is-destroy\": true,\r\n        \"message\": \"test aruna\"\r\n
    \     },\r\n      \"type\":\"runs\",\r\n      \"relationships\": {\r\n        \"workspace\":
    {\r\n          \"data\": {\r\n            \"type\": \"workspaces\",\r\n            \"id\":
    \"ws-6cnxmEf3iysDxmRy\"\r\n          }\r\n        },\r\n        \"configuration-version\":
    {\r\n          \"data\": {\r\n            \"type\": \"configuration-versions\",\r\n
    \           \"id\": \"cv-zT4eQE4zcBEM9tAP\"\r\n          }\r\n        }\r\n      }\r\n
    \   }\r\n}"
kind: ConfigMap
metadata:
  name: payload-destroy
  namespace: cronjobs
---
apiVersion: v1
data:
  TERRAFORM_TOKEN: [YOUR TF TOKEN HERE]
kind: Secret
metadata:
  name: terraform-token
  namespace: cronjobs

#trigger manual : kubectl create job --from=cronjob/destroy-new-staging trigger-test-destroy -n cronjobs

Crontab

Job will run every 21:00 on Friday

Trigger cronjob in napoleon cluster

1
kubectl create job --from=cronjob/destroy-new-staging trigger-test-destroy-2 -n cronjobs

Terraform cloud result :

TF Cloud

Sometimes I got issue about missmatched configuration-version.

Try to run plan by updating the code on Git source, and then run this curl :

1
curl   --header "Authorization: Bearer $TOKEN"   https://app.terraform.io/api/v2/runs/run-LDt4gmpiLguXFJ89 | jq | grep "cv-"

Configuration Version

Then change the configuration-version ID in payload.json (configmap).

Apply the manifest changes then try to trigger the cronjob again :

1
kubectl create job --from=cronjob/destroy-new-staging trigger-test-destroy-2 -n cronjobs