Karma logs "History query failed x509: certificate signed by unknown authority" when using selfsigned cert

This issue has been tracked since 2021-11-12.

Running karma in a standalone docker container v0.93 (same problem in v0.92)
Polling prometheus and alertmanagers that are installed in a kubernetes 1.22.2 kluster with prometheus-operator.

We are using our own certifcate and have configured karma to use our own ca-cert bundle, and that works since the cert error for reading the alertmanager disaper when we use our own cert bundle.

The prometheus uses the same cert issued from the same CA (they use the same ingress)

Starting the container with (using our own registry)
docker run -d --name karma -p 8080:8080 -e CONFIG_FILE="/srv/karma/karma-config.yaml" -v /srv/karma/karma-config.yaml:/srv/karma/karma-config.yaml:ro -v /srv/karma/cert/ca-bundle.crt:/srv/karma/cert/ca-bundle.crt:ro library/prymitive/karma:v0.93

But we get periodically in the log
level=error msg="History query failed" error="failed to query Prometheus for label names: Get \"https://prometheus.mycluster.mysubdomain.mydomain.com/api/v1/labels?end=1636729693.3684304&match%5B%5D=%7B__name__%3D%22ALERTS_FOR_STATE%22%7D&start=1636729393.3684301\": x509: certificate signed by unknown authority" labels={"alertname":"TargetDown","job":"node-exporter","namespace":"monitoring","prometheus":"monitoring/kube-prometheus-stack-prometheus","service":"kube-prometheus-stack-prometheus-node-exporter","severity":"warning"} uri=http://prometheus..mysubdomain.mycluster.mydomain.com worker=29

We have not been able to use docker exec on the krama container too figure out where to place our ca-bundle on golang PATH too test if that helps.https://golang.org/src/crypto/x509/root_linux.go

We have the following config on the karma server

alertmanager:
  interval: 60s
  servers:
    - name: mycluster-mysubdomain-mydomain-com
      uri: https://prometheus.mycluster.mysubdomain.mydomain.com/alertmanager
      healthcheck:
        visible: true      
        filters:
          prom1:
            - alertname=Watchdog
      timeout: 10s
      proxy: true
      readonly: false
      tls:
        ca: /srv/karma/cert/ca-bundle.crt
        #insecureSkipVerify: true
        
annotations:
  default:
    hidden: false
  hidden:
    - help
  visible: []
custom:
  css: /custom.css
  js: /custom.js
debug: false
filters:
  default:
    - "@receiver=by-cluster-service"
karma:
  name: karma
labels:
  color:
    static:
      - job
    unique:
      - cluster
      - instance
      - "@receiver"
  keep: []
  strip: []
listen:
  address: "0.0.0.0"
  port: 8080
  prefix: /
log:
  config: false
  level: info
silences:
  comments:
    linkDetect:
      rules:
        - regex: "(DEVOPS-[0-9]+)"
          uriTemplate: https://jira.example.com/browse/$1
receivers:
  keep: 
    - myreceiver-to-keep
  strip:
    - myreceiver-to-strip
`silenceForm:`
  strip:
    labels:
      - job
ui:
  refresh: 30s
  hideFiltersWhenIdle: true
  colorTitlebar: false
  minimalGroupWidth: 420
  alertsPerGroup: 5
  collapseGroups: collapsedOnMobile

What have we done wrong?

dabarabash wrote this answer on 2021-11-20

We have got the same error

prymitive wrote this answer on 2021-11-27

tls config for each alertmanager applies only to communication with that alertmanager, it doesn't affect Prometheus queries in any way.
This will require adding config options for setting per Prometheus server TLS options.

prymitive wrote this answer on 2021-12-16

Adding new options via #3806

More Details About Repo
Owner Name prymitive
Repo Name karma
Full Name prymitive/karma
Language TypeScript
Created Date 2018-09-09
Updated Date 2023-03-17
Star Count 1921
Watcher Count 33
Fork Count 166
Issue Count 2

YOU MAY BE INTERESTED

Issue Title Created Date Updated Date