-
Notifications
You must be signed in to change notification settings - Fork 124
Description
The problem
I installed the plugin (v3.4.9) for the first time in grafana (v12.2.0) and configured the datasource following the instructions at the Grafana Labs page. After doing so, I have verified that the data from clickhouse can be accessed via the 'explore' page in Grafana, as well as in dashboards with panels that use the datasource plugin (all working as expected).
Alerting, however, is not working properly. I'm able to create alerts using the datasource, and the alert condition itself is verified correctly (showing 'Firing' when satisfying the triggering condition and 'Normal' when not) but the alert state does not change from 'Normal' to 'Firing'. Moreover, a certain alert triggers with alertname=datasourceError when the rule tries to be evaluated. Simultaneously, log lines indicating an error related to the datasource appear (viewable in the 'findings' section below). This only happens when Alert Evaluation uses the plugin. All my dashboards using the plugin works as intended.
My setup
Grafana v12.2.0
vertamedia-clickhouse-datasource v3.4.9
ClickHouse server version 25.3.8
What has been checked
The ClickHouse database is running as expected.
Grafana is running as expected.
The database user has the correct privileges to execute the defined queries.
Database is actually receiving query, executes and returns data to grafana.
No changes have been made in config. Neither grafana or ClickHouse.
What has been tried
Recreating the datasource
Creating the datasource through grafana operator
Recreating the alert rule
Downgrading the version of the plugin
Making a recording rule with the same query (doest not record and returns the same error)
Findings
Whenever the alerting rule is evaluated I see these lines in the log:
logger=ngalert.sender.router rule_uid=afgb1te4ysveoe org_id=1 t=2026-03-17T16:54:14.510422414Z level=info msg="Sending alerts to local notifier" count=1 logger=ngalert.scheduler rule_uid=afgb1te4ysveoe org_id=1 version=2
fingerprint=21351f07c0921115 now=2026-03-17T16:54:10Z rule_uid=afgb1te4ysveoe org_id=1 t=2026-03-17T16:54:13.14184637Z level=error msg="Failed to evaluate rule" attempt=2 max_attempts=3 next_attempt_in=1.364583571s error="the result-set has errors that can be retried: data source not found"
Query being used in the alert rule
SELECT sum(bytes_on_disk) AS value FROM clusterAllReplicas('{cluster}', system.detached_parts)