Add troubleshooting of vm metrics missing #958
+440
−0
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Jian Wang <jian.wang@suse.com>
github-actions
bot
requested review from
akashraj4261,
dariavladykina and
jillian-maroket
|
martindekov
reviewed
Jan 22, 2026
Contributor
martindekov
left a comment
martindekov
left a comment
There was a problem hiding this comment.
Looks good Jian, I like the exact sequence, left minor comment regarding a place where it's ambiguous what we wait for, but apart from that no other comments, thanks!
| rancher-monitoring-operator 0 35s | ||
| ``` | ||
|
|
||
| Wait a while |
Contributor
There was a problem hiding this comment.
Can we be specific for what should the user wait? Here I assume we wait a while for the service account, but as I am reading it as if I am fixing it and not familiar with the issue I am confused what should I wait before proceeding with the deletion of all below
ihcsim
reviewed
Jan 22, 2026
| https://github.com/harvester/harvester/issues/8565 | ||
| ## Cluster Metrics are available but Vitural Machine Metrics are missing |
Contributor
There was a problem hiding this comment.
Suggested change
| ## Cluster Metrics are available but Vitural Machine Metrics are missing | |
| ## Harvester Dashboard stops reporting virtual machine metrics after upgrade |
Comment on lines
+686
to
+688
| $ kubectl get servicemonitor -A | ||
| NAMESPACE NAME AGE | ||
| cattle-monitoring-system prometheus-kubevirt-rules 24s // is missing |
Contributor
There was a problem hiding this comment.
Out-of-box in 1.6.1 has other non-kubevirt service monitors:
$ kubectl get servicemonitor -A
NAMESPACE NAME AGE
cattle-monitoring-system prometheus-kubevirt-rules 24m
harvester-system service-monitor-cdi 25m
longhorn-system longhorn-prometheus-servicemonitor 25m
We can make it more precise with something like:
$ kubectl get servicemonitor -A -lprometheus.kubevirt.io=true
No resources found
| When the `rancher-monitoring` add-on is enabled, the cluster metrics are available but the virtual machine metrics are missing | ||
| The ServiceMonitor object `prometheus-kubevirt-rules/cattle-monitoring-system` is missing, when you create it manually, it is deleted quickly. |
Contributor
There was a problem hiding this comment.
Suggested change
| The ServiceMonitor object `prometheus-kubevirt-rules/cattle-monitoring-system` is missing, when you create it manually, it is deleted quickly. | |
| The `prometheus-kubevirt-rules` ServiceMonitor is missing in the `cattle-monitoring-system` namespace. This object cannot be manually re-added as the KubeVirt operator will automatically delete it. |
|
|
||
| ### Root Cause | ||
|
|
||
| When `kubevirt` is newly installed or upgraded, `kubevirt` generates a new configmap object to store the configurations, the `ServiceMonitor` is enabled or disabled according to the existing of the configured `ServiceAccount` object. |
Contributor
There was a problem hiding this comment.
Suggested change
| When `kubevirt` is newly installed or upgraded, `kubevirt` generates a new configmap object to store the configurations, the `ServiceMonitor` is enabled or disabled according to the existing of the configured `ServiceAccount` object. | |
| When `kubevirt` is newly installed or upgraded, `kubevirt` generates a new configmap object to store the configurations. A race condition exists within the KubeVirt operator that may cause the ServiceMonitor configuration to not be included in the configmap if the `rancher-monitoring-operator` service account is missing in the `cattle-monitoring-system` namespace at the time of install or upgrade. |
|
|
||
| ### Workaround | ||
|
|
||
| 1. Confirm the issue |
Contributor
There was a problem hiding this comment.
Suggested change
| 1. Confirm the issue | |
| The workaround involves ensuring the `rancher-monitoring-operator` service account exists, removing orphaned configmaps and restarting the KubeVirt operator. | |
| 1. Confirm the issue |
|
|
||
| 3. Ensure the serviceaccount is existing | ||
|
|
||
| This object is created when Harvester cluster is installed and should not be removed. If it does not exist, create it manually. |
Contributor
There was a problem hiding this comment.
Suggested change
| This object is created when Harvester cluster is installed and should not be removed. If it does not exist, create it manually. | |
| This object is created when Harvester cluster is installed and should not be removed. If it does not exist, create it manually. Otherwise, skip ahead to step 4 below. |
ihcsim
reviewed
Jan 22, 2026
| ### Issue Description | ||
| When the `rancher-monitoring` add-on is enabled, the cluster metrics are available but the virtual machine metrics are missing |
Contributor
There was a problem hiding this comment.
Suggested change
| When the `rancher-monitoring` add-on is enabled, the cluster metrics are available but the virtual machine metrics are missing | |
| After an upgrade, the Harvester dashboard stops reporting virtual machine metrics. The cluster metrics remain available. Disabling and re-enabling the add-on does not resolve the problem. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem:
Solution:
Add troubleshooting of vm metrics missing
Related Issue(s):
harvester/harvester#9674
Test plan:
Additional documentation or context