Self hosted GHA on a GKE cluster

This issue has been tracked since 2021-12-06.

TL;DR

I was running a flow on Github that used Pulumi to deploy to a GKE cluster. We use this action to set the credentials so Pulumi can deploy on the GKE clusters.

As the GHA cost was increasing, I moved to a Digital Ocean cluster, and it worked the same, as in it could deploy to the GKE cluster perfectly.

As the DO cluster was unstable, I now moved to a GKE cluster for this, but here we do not appear to have access to the other GKE cluster.

It seems that the GKE credentials of the cluster override the ones set by this action.

Expected behavior

No response

Observed behavior

As the DO cluster was unstable, I now moved to a GKE cluster for this, but here we do not appear to have access to the other GKE cluster.

It seems that the GKE credentials of the cluster override the ones set by

Action YAML

deploy:
    runs-on: self-hosted
    needs:
      - build
    steps:
      - name: Checkout
        uses: actions/[email protected]

      - name: Install Pulumi CLI
        uses: pulumi/[email protected]

      - name: Setup Node 16
        uses: actions/[email protected]
        with:
          node-version: '16'

      - name: Get yarn cache directory path
        id: infovars
        run: |
          echo "::set-output name=dir::$(yarn config get cacheFolder)"
          echo "::set-output name=stackname::$(echo ${GITHUB_REF#refs/heads/} | awk -F'/' '{print $1}' )"

      - uses: actions/[email protected]
        id: yarn-cache
        with:
          path: ${{ steps.infovars.outputs.dir }}
          key: ${{ runner.os }}-yarn-${{ hashFiles('**/yarn.lock') }}
          restore-keys: |
            ${{ runner.os }}-yarn-

      - name: yarn install
        run: yarn install --immutable

      - name: Fetch secrets
        run: yarn secrets

      - name: Inject slug/short variables
        uses: rlespinasse/[email protected]

      - name: Configure AWS Credentials
        uses: aws-actions/[email protected]
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-region: ${{ env.AWS_REGION }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

      - name: Set up Cloud SDK
        uses: google-github-actions/[email protected]
        with:
          project_id: "settlemint-staging-projects"
          service_account_key: ${{ secrets.GOOGLE_CREDENTIALS }}
          export_default_credentials: true

      - name: Recover from cancellations
        working-directory: ./libs/infra/bpaas-console
        run: |
          pulumi stack select settlemint/${{steps.infovars.outputs.stackname}}
          pulumi stack export | jq "del(.deployment.pending_operations)" | pulumi stack import

      - name: Pulumi
        uses: pulumi/[email protected]
        with:
          command: up
          stack-name: settlemint/${{steps.infovars.outputs.stackname}}
          comment-on-pr: false
          refresh: true
          work-dir: ./libs/infra/bpaas-console
        env:
          PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}
          TAG: ${{ env.GITHUB_REF_SLUG }}-${{ github.sha }}

      - name: Recover from cancellations
        working-directory: ./libs/infra/bpaas-clustermanager
        run: |
          pulumi stack select settlemint/${{steps.infovars.outputs.stackname}}
          pulumi stack export | jq "del(.deployment.pending_operations)" | pulumi stack import

      - name: Pulumi Cluster Manager
        uses: pulumi/[email protected]
        with:
          command: up
          stack-name: settlemint/${{steps.infovars.outputs.stackname}}
          comment-on-pr: false
          refresh: true
          work-dir: ./libs/infra/bpaas-clustermanager
        env:
          PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}
          TAG: ${{ env.GITHUB_REF_SLUG }}-${{ github.sha }}

Additional information

We run pulumi on GKE somewhere else in a container where we do the following to make it work

export GOOGLE_APPLICATION_CREDENTIALS="$(mktemp).json"
# Check if GOOGLE_CREDENTIALS is base64 encoded
if [[ $GOOGLE_CREDENTIALS =~ ^([A-Za-z0-9+/]{4})*([A-Za-z0-9+/]{3}=|[A-Za-z0-9+/]{2}==)?$ ]]; then
  echo "$GOOGLE_CREDENTIALS"|base64 -d > $GOOGLE_APPLICATION_CREDENTIALS
  # unset for other gcloud commands using this variable.
  unset GOOGLE_CREDENTIALS
else
  echo "$GOOGLE_CREDENTIALS" > $GOOGLE_APPLICATION_CREDENTIALS
fi
cat $GOOGLE_APPLICATION_CREDENTIALS
gcloud auth activate-service-account --key-file=$GOOGLE_APPLICATION_CREDENTIALS

But I have no idea how to reproduce this using the action

sethvargo wrote this answer on 2021-12-06

Can you please share more about how your GKE cluster is configured? Is Workload Identity Federation enabled? What about metadata concealment?

I don't know exactly how Pulumi finds its authentication, and I couldn't find anything in their documentation about precedence or anything. It would be helpful to check the output of gcloud auth list and gcloud info after setup-gcloud. This will tell you what the currently-authenticated principal is.

- name: Set up Cloud SDK
  uses: google-github-actions/[email protected]
  with:
    project_id: "settlemint-staging-projects"
    service_account_key: ${{ secrets.GOOGLE_CREDENTIALS }}
    export_default_credentials: true

- name: Gcloud Info
  run: 'gcloud info'

- name: Gcloud Auth List
  run: 'gcloud auth list'
roderik wrote this answer on 2021-12-11

I have unfortunately destroyed the cluster and moved back to GHA where the above config works like a charm. Devs gotta keep releasing

sethvargo wrote this answer on 2021-12-23

Okay - I'm going to close this out since I can't reproduce this without getting the exact configuration of the GKE cluster.

More Details About Repo
Owner Name google-github-actions
Repo Name setup-gcloud
Full Name google-github-actions/setup-gcloud
Language TypeScript
Created Date 2019-11-05
Updated Date 2023-03-23
Star Count 1505
Watcher Count 65
Fork Count 548
Issue Count 11

YOU MAY BE INTERESTED

Issue Title Created Date Updated Date