Inconsistent but persistent auth errors starting today

This issue has been tracked since 2022-04-04.

TL;DR

We're experiencing ongoing random auth errors across Google services after authenticating with this action.

Expected behavior

We've had no issues with auth until today. We rolled out a new parallelized testing workflow earlier that's been working fine in testing up until it rolled out today and some of these auth errors are happening in non-parallelized jobs so I'm not sure it's related. Usually, this all works without a hitch, but Google Cloud's not showing any service outages on its status page.

Observed behavior

google-github-actions/setup-gcloud failed with: failed to execute command `gcloud --quiet auth login --cred-file /home/runner/work/unstoppable-domains-website/unstoppable-domains-website/gha-creds-a1b71b0c5598d105.json`: ERROR: gcloud crashed (OAuthError): ("Error code {'code': 503, 'message': 'The service is currently unavailable.', 'status': 'UNAVAILABLE'}", '{\n  "error": {\n    "code": 503,\n    "message": "The service is currently unavailable.",\n    "status": "UNAVAILABLE"\n  }\n}\n')

Error: 16 UNAUTHENTICATED: Failed to retrieve auth metadata with error: Error code [object Object]
    at Object.callErrorFromStatus (/home/runner/work/unstoppable-domains-website/unstoppable-domains-website/node_modules/@google-cloud/secret-manager/node_modules/@grpc/grpc-js/src/call.ts:81:24)
    at Object.onReceiveStatus (/home/runner/work/unstoppable-domains-website/unstoppable-domains-website/node_modules/@google-cloud/secret-manager/node_modules/@grpc/grpc-js/src/client.ts:334:36)
    at Object.onReceiveStatus (/home/runner/work/unstoppable-domains-website/unstoppable-domains-website/node_modules/@google-cloud/secret-manager/node_modules/@grpc/grpc-js/src/client-interceptors.ts:426:34)
    at Object.onReceiveStatus (/home/runner/work/unstoppable-domains-website/unstoppable-domains-website/node_modules/@google-cloud/secret-manager/node_modules/@grpc/grpc-js/src/client-interceptors.ts:389:[48](https://github.com/unstoppabledomains/unstoppable-domains-website/runs/5824013165?check_suite_focus=true#step:15:48))
    at /home/runner/work/unstoppable-domains-website/unstoppable-domains-website/node_modules/@google-cloud/secret-manager/node_modules/@grpc/grpc-js/src/call-stream.ts:276:24
    at processTicksAndRejections (node:internal/process/task_queues:78:11) {
  code: 16,
  details: 'Failed to retrieve auth metadata with error: Error code [object Object]',
  metadata: Metadata { internalRepr: Map(0) {}, options: {} },
  note: 'Exception occurred in retry method that was not classified as transient'
}

Error: google-github-actions/upload-cloud-storage failed with error code [object Object]

Action YAML

name: E2E tests
on:
  schedule:
    - cron: '0 0 * * *'
  push:
    branches:
      - master
      - staging
  pull_request:
    types:
      - opened
      - reopened
      - synchronize
  workflow_dispatch:
  # workflow_run:
  #   workflows: ['Build']
  #   types: ['completed']
concurrency:
  group: e2e_${{ github.ref }}_${{ github.event_name }}
  cancel-in-progress: true
env:
  GCP_PROJECT: unstoppable-domains-staging
  PACKAGE: e2e
  BAIL: false
jobs:
  setup:
    name: Setup workflow run
    uses: './.github/workflows/_setup.yml'
    with:
      trigger: ${{ github.event_name }}
      actor: ${{ github.actor }}
      title: ${{ github.event.pull_request.title }}

  pre-cache:
    name: Pre-cache frontend build
    needs: ['setup']
    if: ${{ needs.setup.outputs.skipped == 'true' || (needs.setup.outputs.e2e == 'true' && needs.setup.outputs.hotfix == 'false') }}
    uses: './.github/workflows/_precache.yml'
    with:
      environment: e2e

  test:
    name: End-to-End tests
    runs-on: ubuntu-latest
    needs: ['setup', 'pre-cache']
    if: ${{ needs.setup.outputs.skipped == 'true' || (needs.setup.outputs.e2e == 'true' && needs.setup.outputs.hotfix == 'false') }}
    outputs:
      failed: ${{ steps.tests.outcome != 'success' }}
      outcome: ${{ steps.tests.outcome }}
      started: ${{ steps.start_timer.outputs.started }}
      finished: ${{ steps.end_timer.outputs.finished }}
      duration: ${{ steps.end_timer.outputs.duration }}
      build: ${{ needs.pre-cache.outputs.cache-hit != 'true' }}
    strategy:
      fail-fast: false
      matrix:
        ci_node_total: [4]
        ci_node_index: [0, 1, 2, 3]
    permissions:
      id-token: write
      contents: read
      pull-requests: write
      issues: write
    services:
      postgres:
        image: 'postgres:11.6'
        ports: ['5432:5432']
        options: '--health-cmd pg_isready --health-interval 10s --health-timeout 5s
          --health-retries 5'
        env:
          POSTGRES_DB: unstoppable_website_test
      redis:
        image: 'redis:5.0'
        ports:
          - '6379:6379'
        options: '--entrypoint redis-server'
    steps:
      - name: Checkout repo
        uses: 'actions/[email protected]'

      - name: Configure Node.js ${{ secrets.NODE_VERSION }}
        uses: 'actions/[email protected]'
        with:
          node-version: ${{ secrets.NODE_VERSION }}
          cache: yarn

      - name: Retrieve project Yarn cache
        id: cache
        uses: 'actions/[email protected]'
        with:
          path: |
            ./node_modules
            **/node_modules
            !ci/node_modules
          key: ${{ runner.os }}-dev-${{ hashFiles('./yarn.lock') }}

      - name: Install dependencies
        if: ${{ steps.cache.outputs.cache-hit != 'true' }}
        run: yarn install --prefer-offline --frozen-lockfile 2> >(grep -v warning 1>&2)

      - name: Retrieve CI Yarn cache
        id: ci-cache
        uses: 'actions/[email protected]'
        with:
          path: ./ci/node_modules
          key: ${{ runner.os }}-ci-${{ hashFiles('./ci/yarn.lock') }}

      - name: Install CI dependencies
        if: ${{ steps.ci-cache.outputs.cache-hit != 'true' }}
        working-directory: ./ci
        run: yarn install --prefer-offline --frozen-lockfile 2> >(grep -v warning 1>&2)

      - name: Retrieve shared library build cache
        uses: 'actions/[email protected]'
        id: shared-cache
        with:
          path: shared/dist
          key: ${{ runner.os }}-shared-${{ hashFiles('./yarn.lock', './shared/src/**/*.ts', './shared/src/**/*.js', './shared/**/*.json') }}

      - name: Retrieve frontend build cache
        uses: 'actions/[email protected]'
        id: frontend-cache
        with:
          path: |
            client/build
            client/.next
            ~/.next
          key: ${{ runner.os }}-frontend-e2e-${{ hashFiles('./yarn.lock', './client/**/*.ts', './client/**/*.tsx', './client/**/*.js', './client/**/*.jsx', './client/**/*.json') }}

      - name: Retrieve Cypress cache
        uses: 'actions/[email protected]'
        id: cypress_cache
        with:
          path: ~/.cache/Cypress
          key: ${{ runner.os }}-cache-cypress-9.5.3

      - name: Install Cypress
        if: ${{ steps.cypress_cache.outputs.cache-hit != 'true' }}
        working-directory: ./e2e
        run: yarn cypress install

      - name: Login to Google Cloud
        id: auth
        uses: 'google-github-actions/[email protected]'
        with:
          workload_identity_provider: ${{ secrets.WORKLOAD_ID_PROVIDER_STAGING }}
          service_account: ${{ secrets.SERVICE_ACCOUNT_STAGING }}

      - name: Configure Google Cloud
        uses: 'google-github-actions/[email protected]'

      - name: Run TypeORM migrations
        env:
          APP_ENV: e2e
          GOOGLE_APPLICATION_CREDENTIALS: ${{ steps.auth.outputs.credentials_file_path }}
        run: 'yarn workspace backend db:migration:run'

      - name: Generate UUID
        id: uuid
        run: echo "::set-output name=uuid::$(uuidgen)"

      - name: Run E2E tests
        id: tests
        uses: 'cypress-io/[email protected]'
        continue-on-error: true
        env:
          NODE_ENV: development
          APP_ENV: e2e
          API_BASE_URL: 'http://localhost:8080'
          CLIENT_URL: 'http://localhost:3000'
          CYPRESS_PROJECT_ID: 9g89i4
          CYPRESS_RECORD_KEY: ${{ secrets.CYPRESS_RECORD_KEY }}
          RECORD: ${{ github.event_name == 'schedule' }}
          KNAPSACK_PRO_TEST_SUITE_TOKEN_CYPRESS: ${{ secrets.KNAPSACK_E2E }}
          KNAPSACK_PRO_CI_NODE_TOTAL: ${{ matrix.ci_node_total }}
          KNAPSACK_PRO_CI_NODE_INDEX: ${{ matrix.ci_node_index }}
          KNAPSACK_PRO_TEST_FILE_PATTERN: '${{ github.workspace }}/e2e/cypress/integration/*.ts'
          KNAPSACK_PRO_FIXED_QUEUE_SPLIT: true
        with:
          command: '${{ github.workspace }}/ci/node_modules/.bin/knapsack-pro-cypress --configFile ${{ github.workspace }}/e2e/cypress.ci.json --headless --reporter ${{ github.workspace }}/ci/node_modules/mochawesome --reporter-options html=false,json=true,reportDir=${{ github.workspace }}/e2e,reportFilename=test-report-${{ steps.uuid.outputs.uuid }},overwrite=false --browser chrome --spec ${{ github.workspace }}/e2e/cypress/integration/**/*.ts --record=${{ env.RECORD }}'
          install: false
          start: 'yarn workspace backend start:e2e, yarn workspace client start:e2e'
          wait-on: 'http://localhost:8080/api/alive, http://localhost:3000'
          working-directory: ${{ github.workspace }}/e2e

      - name: Upload test results to GitHub
        uses: actions/[email protected]
        continue-on-error: true
        with:
          name: ${{ env.PACKAGE }}_${{ github.sha }}
          path: |
            ${{ env.PACKAGE }}/test-report*
          retention-days: 1

  results:
    name: Validate and publish test results
    uses: './.github/workflows/_results.yml'
    needs: ['test', 'setup']
    if: ${{ needs.setup.outputs.skipped == 'true' || (needs.setup.outputs.e2e == 'true' && needs.setup.outputs.hotfix == 'false') }}
    with:
      package: e2e
      failed: ${{ needs.test.outputs.failed }}
      bailed: false
      install: ${{ needs.setup.outputs.cache-hit != 'true' }}
      build: ${{ needs.test.outputs.build }}
      sha: ${{ github.sha }}
      started: ${{ needs.setup.outputs.started }}
      outcome: ${{ needs.test.outputs.outcome }}
      report: 'https://storage.googleapis.com/unstoppable-devops/ci-artifacts/${{ github.run_id }}/e2e/test-report.html'
    secrets:
      WORKLOAD_ID_PROVIDER_STAGING: ${{ secrets.WORKLOAD_ID_PROVIDER_STAGING }}
      SERVICE_ACCOUNT_STAGING: ${{ secrets.SERVICE_ACCOUNT_STAGING }}
      NODE_VERSION: ${{ secrets.NODE_VERSION }}
      SONAR_TOKEN: ${{ secrets.SONARCLOUD_TOKEN }}
      CC_TEST_REPORTER_ID: ${{ secrets.CC_TEST_REPORTER_ID }}

Log output

##[debug]Evaluating condition for step: 'Login to Google Cloud'
##[debug]Evaluating: success()
##[debug]Evaluating success:
##[debug]=> true
##[debug]Result: true
##[debug]Starting: Login to Google Cloud
##[debug]Register post job cleanup for action: google-github-actions/[email protected]
##[debug]Loading inputs
##[debug]Evaluating: secrets.WORKLOAD_ID_PROVIDER_STAGING
##[debug]Evaluating Index:
##[debug]..Evaluating secrets:
##[debug]..=> Object
##[debug]..Evaluating String:
##[debug]..=> 'WORKLOAD_ID_PROVIDER_STAGING'
##[debug]=> '***'
##[debug]Result: '***'
##[debug]Evaluating: secrets.SERVICE_ACCOUNT_STAGING
##[debug]Evaluating Index:
##[debug]..Evaluating secrets:
##[debug]..=> Object
##[debug]..Evaluating String:
##[debug]..=> 'SERVICE_ACCOUNT_STAGING'
##[debug]=> '***'
##[debug]Result: '***'
##[debug]Loading env
Run google-github-actions/[email protected]
##[debug]Using workload identity provider "***"
##[debug]ID token url is https://pipelines.actions.githubusercontent.com/JIdNBcQ6kMWGQ1Emvk4eWkVClMzQPdc1GeXypChXNSBsqbvPVO/00000000-0000-0000-0000-000000000000/_apis/distributedtask/hubs/Actions/plans/99435d5c-744f-40ef-8ace-52f2f419aeed/jobs/f72e1371-574f-5793-8b43-f4df44fd5814/idtoken?api-version=2.0&audience=https%3A%2F%2Fiam.googleapis.com%2F***
::add-mask::***
##[debug]Creating credentials file
Created credentials file at "/home/runner/work/unstoppable-domains-website/unstoppable-domains-website/gha-creds-a1b71b0c5598d105.json"

::set-output name=credentials_file_path::/home/runner/work/unstoppable-domains-website/unstoppable-domains-website/gha-creds-a1b71b0c5598d105.json
##[debug]steps.auth.outputs.credentials_file_path='/home/runner/work/unstoppable-domains-website/unstoppable-domains-website/gha-creds-a1b71b0c5598d105.json'

::set-output name=project_id::unstoppable-domains-staging
##[debug]steps.auth.outputs.project_id='unstoppable-domains-staging'
##[debug]Node Action run completed with exit code 0
##[debug]CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE='/home/runner/work/unstoppable-domains-website/unstoppable-domains-website/gha-creds-a1b71b0c5598d105.json'
##[debug]GOOGLE_APPLICATION_CREDENTIALS='/home/runner/work/unstoppable-domains-website/unstoppable-domains-website/gha-creds-a1b71b0c5598d105.json'
##[debug]GOOGLE_GHA_CREDS_PATH='/home/runner/work/unstoppable-domains-website/unstoppable-domains-website/gha-creds-a1b71b0c5598d105.json'
##[debug]CLOUDSDK_PROJECT='unstoppable-domains-staging'
##[debug]CLOUDSDK_CORE_PROJECT='unstoppable-domains-staging'
##[debug]GCP_PROJECT='unstoppable-domains-staging'
##[debug]GCLOUD_PROJECT='unstoppable-domains-staging'
##[debug]GOOGLE_CLOUD_PROJECT='unstoppable-domains-staging'
##[debug]Finishing: Login to Google Cloud

Additional information

No response

selfagency wrote this answer on 2022-04-05
0

@sethvargo We're totally blocked here so it would be greatly appreciated if you could find time this morning to help me out. Thanks.

sethvargo wrote this answer on 2022-04-05
0

Hi @selfagency

Can you try adding "token_format: access_token" to the auth step temporarily? If the Action proceeds beyond the auth step, it means there's an issue with the upstream service. However, if the Action fails at the auth step (after adding that token), it means the issue is in the authentication handshake, and the specific error message will give us more information.

selfagency wrote this answer on 2022-04-05
0

@sethvargo

##[debug]Evaluating condition for step: 'Login to Google Cloud'
##[debug]Evaluating: success()
##[debug]Evaluating success:
##[debug]=> true
##[debug]Result: true
##[debug]Starting: Login to Google Cloud
##[debug]Register post job cleanup for action: google-github-actions/[email protected]
##[debug]Loading inputs
##[debug]Evaluating: secrets.WORKLOAD_ID_PROVIDER_STAGING
##[debug]Evaluating Index:
##[debug]..Evaluating secrets:
##[debug]..=> Object
##[debug]..Evaluating String:
##[debug]..=> 'WORKLOAD_ID_PROVIDER_STAGING'
##[debug]=> '***'
##[debug]Result: '***'
##[debug]Evaluating: secrets.SERVICE_ACCOUNT_STAGING
##[debug]Evaluating Index:
##[debug]..Evaluating secrets:
##[debug]..=> Object
##[debug]..Evaluating String:
##[debug]..=> 'SERVICE_ACCOUNT_STAGING'
##[debug]=> '***'
##[debug]Result: '***'
##[debug]Loading env
Run google-github-actions/[email protected]
##[debug]Using workload identity provider "***"
##[debug]ID token url is https://pipelines.actions.githubusercontent.com/JIdNBcQ6kMWGQ1Emvk4eWkVClMzQPdc1GeXypChXNSBsqbvPVO/00000000-0000-0000-0000-000000000000/_apis/distributedtask/hubs/Actions/plans/bedb98a3-f043-4f9e-88[2](https://github.com/unstoppabledomains/unstoppable-domains-website/runs/5838469587?check_suite_focus=true#step:13:2)7-c75a2c08e90b/jobs/a9[3](https://github.com/unstoppabledomains/unstoppable-domains-website/runs/5838469587?check_suite_focus=true#step:13:3)157a9-a0f6-5af8-a70b-8506ae[4](https://github.com/unstoppabledomains/unstoppable-domains-website/runs/5838469587?check_suite_focus=true#step:13:4)7[5](https://github.com/unstoppabledomains/unstoppable-domains-website/runs/5838469587?check_suite_focus=true#step:13:5)1b2/idtoken?api-version=2.0&audience=https%3A%2F%2Fiam.googleapis.com%2F***
::add-mask::***
##[debug]Creating credentials file
Created credentials file at "/home/runner/work/unstoppable-domains-website/unstoppable-domains-website/gha-creds-9007291eaec30215.json"

::set-output name=credentials_file_path::/home/runner/work/unstoppable-domains-website/unstoppable-domains-website/gha-creds-9007291eaec30215.json
##[debug]steps.auth.outputs.credentials_file_path='/home/runner/work/unstoppable-domains-website/unstoppable-domains-website/gha-creds-9007291eaec30215.json'

::set-output name=project_id::unstoppable-domains-staging
##[debug]steps.auth.outputs.project_id='unstoppable-domains-staging'
Warning: Overwriting existing environment variable GCP_PROJECT (was: "unstoppable-domains-staging")
##[debug]Creating access token
::add-mask::***

::set-output name=access_token::***
##[debug]steps.auth.outputs.access_token='***'

::set-output name=access_token_expiration::2022-04-05T18:4[6](https://github.com/unstoppabledomains/unstoppable-domains-website/runs/5838469587?check_suite_focus=true#step:13:6):56Z
##[debug]steps.auth.outputs.access_token_expiration='2022-04-05T18:46:56Z'
##[debug]Node Action run completed with exit code 0
##[debug]CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE='/home/runner/work/unstoppable-domains-website/unstoppable-domains-website/gha-creds-900[7](https://github.com/unstoppabledomains/unstoppable-domains-website/runs/5838469587?check_suite_focus=true#step:13:7)2[9](https://github.com/unstoppabledomains/unstoppable-domains-website/runs/5838469587?check_suite_focus=true#step:13:9)1eaec302[15](https://github.com/unstoppabledomains/unstoppable-domains-website/runs/5838469587?check_suite_focus=true#step:13:15).json'
##[debug]GOOGLE_APPLICATION_CREDENTIALS='/home/runner/work/unstoppable-domains-website/unstoppable-domains-website/gha-creds-9007291eaec30[21](https://github.com/unstoppabledomains/unstoppable-domains-website/runs/5838469587?check_suite_focus=true#step:13:21)5.json'
##[debug]GOOGLE_GHA_CREDS_PATH='/home/runner/work/unstoppable-domains-website/unstoppable-domains-website/gha-creds-9007291eaec30215.json'
##[debug]CLOUDSDK_CORE_PROJECT='unstoppable-domains-staging'
##[debug]CLOUDSDK_PROJECT='unstoppable-domains-staging'
##[debug]GCLOUD_PROJECT='unstoppable-domains-staging'
##[debug]GCP_PROJECT='unstoppable-domains-staging'
##[debug]GOOGLE_CLOUD_PROJECT='unstoppable-domains-staging'
##[debug]Finishing: Login to Google Cloud

Then, after running google-github-actions/[email protected], in two out of four identical runners, I get:

##[debug]Evaluating: steps.auth.outputs.credentials_file_path
##[debug]Evaluating Index:
##[debug]..Evaluating Index:
##[debug]....Evaluating Index:
##[debug]......Evaluating steps:
##[debug]......=> Object
##[debug]......Evaluating String:
##[debug]......=> 'auth'
##[debug]....=> Object
##[debug]....Evaluating String:
##[debug]....=> 'outputs'
##[debug]..=> Object
##[debug]..Evaluating String:
##[debug]..=> 'credentials_file_path'
##[debug]=> '/home/runner/work/unstoppable-domains-website/unstoppable-domains-website/gha-creds-f88746354[2](https://github.com/unstoppabledomains/unstoppable-domains-website/runs/5839334393?check_suite_focus=true#step:12:2)fb167a.json'
##[debug]Result: '/home/runner/work/unstoppable-domains-website/unstoppable-domains-website/gha-creds-f88746[3](https://github.com/unstoppabledomains/unstoppable-domains-website/runs/5839334393?check_suite_focus=true#step:12:3)5[4](https://github.com/unstoppabledomains/unstoppable-domains-website/runs/5839334393?check_suite_focus=true#step:12:4)2fb167a.json'
##[debug]Evaluating condition for step: 'Run TypeORM migrations'
##[debug]Evaluating: success()
##[debug]Evaluating success:
##[debug]=> true
##[debug]Result: true
##[debug]Starting: Run TypeORM migrations
##[debug]Loading inputs
##[debug]Loading env
Run yarn workspace backend db:migration:run
  yarn workspace backend db:migration:run
  shell: /usr/bin/bash -e {0}
  env:
    CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE: /home/runner/work/unstoppable-domains-website/unstoppable-domains-website/gha-creds-f887463[5](https://github.com/unstoppabledomains/unstoppable-domains-website/runs/5839334393?check_suite_focus=true#step:12:5)42fb1[6](https://github.com/unstoppabledomains/unstoppable-domains-website/runs/5839334393?check_suite_focus=true#step:12:6)[7](https://github.com/unstoppabledomains/unstoppable-domains-website/runs/5839334393?check_suite_focus=true#step:12:7)a.json
    GOOGLE_APPLICATION_CREDENTIALS: /home/runner/work/unstoppable-domains-website/unstoppable-domains-website/gha-creds-f[8](https://github.com/unstoppabledomains/unstoppable-domains-website/runs/5839334393?check_suite_focus=true#step:12:8)87463542fb167a.json
    GOOGLE_GHA_CREDS_PATH: /home/runner/work/unstoppable-domains-website/unstoppable-domains-website/gha-creds-f887463542fb167a.json
    CLOUDSDK_CORE_PROJECT: unstoppable-domains-staging
    CLOUDSDK_PROJECT: unstoppable-domains-staging
    GCLOUD_PROJECT: unstoppable-domains-staging
    GCP_PROJECT: unstoppable-domains-staging
    GOOGLE_CLOUD_PROJECT: unstoppable-domains-staging
    APP_ENV: e2e
##[debug]/usr/bin/bash -e /home/runner/work/_temp/c[9](https://github.com/unstoppabledomains/unstoppable-domains-website/runs/5839334393?check_suite_focus=true#step:12:9)6e0527-df69-4e39-aaaa-fe29c93fc39f.sh
yarn workspace v1.22.18
yarn run v1.22.18
$ yarn typeorm migration:run
$ ts-node -r tsconfig-paths/register -T ./lib/cli.ts migration:run
Error during migration run:
Error: 16 UNAUTHENTICATED: Failed to retrieve auth metadata with error: Error code [object Object]
    at Object.callErrorFromStatus (/home/runner/work/unstoppable-domains-website/unstoppable-domains-website/node_modules/@google-cloud/secret-manager/node_modules/@grpc/grpc-js/src/call.ts:81:24)
    at Object.onReceiveStatus (/home/runner/work/unstoppable-domains-website/unstoppable-domains-website/node_modules/@google-cloud/secret-manager/node_modules/@grpc/grpc-js/src/client.ts:334:36)
    at Object.onReceiveStatus (/home/runner/work/unstoppable-domains-website/unstoppable-domains-website/node_modules/@google-cloud/secret-manager/node_modules/@grpc/grpc-js/src/client-interceptors.ts:426:34)
    at Object.onReceiveStatus (/home/runner/work/unstoppable-domains-website/unstoppable-domains-website/node_modules/@google-cloud/secret-manager/node_modules/@grpc/grpc-js/src/client-interceptors.ts:389:48)
    at /home/runner/work/unstoppable-domains-website/unstoppable-domains-website/node_modules/@google-cloud/secret-manager/node_modules/@grpc/grpc-js/src/call-stream.ts:276:24
    at processTicksAndRejections (node:internal/process/task_queues:78:[11](https://github.com/unstoppabledomains/unstoppable-domains-website/runs/5839334393?check_suite_focus=true#step:12:11)) {
  code: [16](https://github.com/unstoppabledomains/unstoppable-domains-website/runs/5839334393?check_suite_focus=true#step:12:16),
  details: 'Failed to retrieve auth metadata with error: Error code [object Object]',
  metadata: Metadata { internalRepr: Map(0) {}, options: {} },
  note: 'Exception occurred in retry method that was not classified as transient'
}
error Command failed with exit code 1.
}
sethvargo wrote this answer on 2022-04-05
0

Hi @selfagency - that tells me that the authentication step is succeeding (you're getting an auth token), but then Cloud Storage is rejecting the token. If this is a recurring issue, I would recommend opening an issue with Google Cloud support. 503 is generally a server-side issue.

selfagency wrote this answer on 2022-04-06
0

@sethvargo I opened an issue with Google Cloud Support and they told me that debugging Workflow Identity Federation issues is out of scope.

sethvargo wrote this answer on 2022-04-06
0

Hi @selfagency the issue is not Workload Identity Federation (we proved that when adding token_format: access_token did not create an error. From the logs output, Cloud Storage is returning HTTP 503 responses, and unfortunately that's not something we can actually see or debug on this team.

sethvargo wrote this answer on 2022-04-20
0

Hi @selfagency - were you able to resolve this?

selfagency wrote this answer on 2022-04-20
0

Unfortunately we're still getting random errors. Google Cloud support thinks it's GitHub's fault. GitHub's OIDC team is now investigating after I asked an executive for help.

sethvargo wrote this answer on 2022-04-20
0

Okay I'll leave this open for now. I do think the issue is outside of this library, since it's just calling the upstream APIs.

selfagency wrote this answer on 2022-04-22
0

Mischa (GitHub Support)
Apr 22, 2022, 4:06 PM UTC

Hello daniel,

Thank you for your patience while I check in with Engineering.

Our Engineering team relayed that since the google-github-actions/auth Action is able to successfully get a token, they do not suspect that the issue stems from the OIDC flow from the GitHub side. A 503 error from the upstream API indicates a problem with the service in question being temporarily unavailable, which we would not be able to dig into further on our end.

Regards,

Michael Lee
Enterprise Support Engineer

sethvargo wrote this answer on 2022-04-22
0

That response makes sense. The logs clearly shows auth is getting GitHub's OIDC token and it's exchanged for a GCP auth credential.

gustavovalverde wrote this answer on 2022-04-29
0

I'm not sure if this is related, but we're recently having authentication issues when pushing images, and we haven't touch a single thing on the GCP side https://github.com/ZcashFoundation/zebra/runs/6222308696?check_suite_focus=true#step:9:2120

sethvargo wrote this answer on 2022-05-16
0

Hi @selfagency

Is this still happening? Do you have any additional information?

More Details About Repo
Owner Name google-github-actions
Repo Name auth
Full Name google-github-actions/auth
Language TypeScript
Created Date 2021-09-16
Updated Date 2023-03-24
Star Count 573
Watcher Count 16
Fork Count 116
Issue Count 3

YOU MAY BE INTERESTED

Issue Title Created Date Updated Date

Popular Frameworks
Programming Languages

Copyright © Codesti | Contact