Not showing recommandation with AWS managed prometheus

[cx-asafd]

Describe the bug
Can see recommendation from eks manages prom

To Reproduce
Steps to reproduce the behavior:
krr simple -p "https://aps-workspaces.eu-west-1.amazonaws.com/workspaces/..../api/v1/query" --eks-managed-prom -n eu

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots

[cx-asafd]

[ERROR] No Prometheus or metrics service found

[jakesky-centrica]

Facing a similar issue

│    24. │ prometh… │ prometh… │ 0    │ 0        │ DaemonS… │ node-e… │          │ unset   │ unset -> │         │ unset -> │ unset   │
│        │          │          │      │          │          │         │          │ -> ?    │ ?        │         │ ?        │ -> ?    │
│        │          │          │      │          │          │         │          │ (Prome… │ (Promet… │         │ (Promet… │ (Prome… │
│        │          │          │      │          │          │         │          │ not     │ not      │         │ not      │ not     │
│        │          │          │      │          │          │         │          │ found)  │ found)   │         │ found)   │ found) 

                    Caused by HTTPError: 403 Client Error: Forbidden for url:                                                         
                    https://aps-workspaces.eu-west-2.amazonaws.com/workspaces/ws-xxxxx/api   

python3.10 krr.py simple --eks-managed-prom -p "https://aps-workspaces.eu-west-2.amazonaws.com/workspaces/ws-xxxxx" -v --eks-managed-prom-region eu-west-2


 _____       _               _          _  _______  _____
|  __ \     | |             | |        | |/ /  __ \|  __ \
| |__) |___ | |__  _   _ ___| |_ __ _  | ' /| |__) | |__) |
|  _  // _ \| '_ \| | | / __| __/ _` | |  < |  _  /|  _  /
| | \ \ (_) | |_) | |_| \__ \ || (_| | | . \| | \ \| | \ \
|_|  \_\___/|_.__/ \__,_|___/\__\__,_| |_|\_\_|  \_\_|  \_\



Running Robusta's KRR (Kubernetes Resource Recommender) 1.6.0-dev
Using strategy: Simple
Using formatter: table

[11:08:53] WARNING  The timeframe duration provided is insufficient and will be overridden with 2.016. Kindly adjust     runner.py:236
                    --timeframe_duration to a value equal to or greater than 2.016.                                                   
           DEBUG    Found 10 clusters: arn:aws:eks:eu-west-1:927431155644:cluster/ex-complete,                         __init__.py:353                               
                    arn:aws:eks:eu-west-2:xxxxx:cluster/xxxxx                                                                  
           DEBUG    Current cluster: arn:aws:eks:eu-west-2:xxxxx:cluster/xxxxx                                  __init__.py:354
           DEBUG    Configured clusters: []                                                                            __init__.py:356
           INFO     Using clusters: ['arn:aws:eks:eu-west-2:xxxxx:cluster/xxxxx']                                 runner.py:190
           INFO     Listing scannable objects in arn:aws:eks:eu-west-2:xxxxx:cluster/xxxxx                       __init__.py:62
           DEBUG    Namespaces: *                                                                                       __init__.py:63
           DEBUG    Resources: *                                                                                        __init__.py:64
[11:08:54] DEBUG    Listing Deployments in arn:aws:eks:eu-west-2:xxxxx:cluster/xxxxx                            __init__.py:154
           DEBUG    Listing Rollouts in arn:aws:eks:eu-west-2:xxxxx:cluster/xxxxx                               __init__.py:154
           DEBUG    Listing StatefulSets in arn:aws:eks:eu-west-2:xxxxx:cluster/xxxxx                           __init__.py:154
           DEBUG    Listing DaemonSets in arn:aws:eks:eu-west-2:xxxxx:cluster/xxxxx                             __init__.py:154
           DEBUG    Listing Jobs in arn:aws:eks:eu-west-2:xxxxx:cluster/xxxxx                                   __init__.py:154
           DEBUG    Rollout API not available in arn:aws:eks:eu-west-2:xxxxx:cluster/xxxxx                      __init__.py:195
           DEBUG    Found 0 Job in arn:aws:eks:eu-west-2:xxxxx:cluster/xxxxx                                    __init__.py:166
           DEBUG    Found 3 StatefulSet in arn:aws:eks:eu-west-2:xxxxx:cluster/xxxxx                            __init__.py:166
           DEBUG    Found 22 Deployment in arn:aws:eks:eu-west-2:xxxxx:cluster/xxxxx                            __init__.py:166
           DEBUG    Found 6 DaemonSet in arn:aws:eks:eu-west-2:xxxxx:cluster/xxxxx                              __init__.py:166
           INFO     Connecting to Prometheus for arn:aws:eks:eu-west-2:xxxxx:cluster/xxxxx     prometheus_metrics_service.py:63
                    cluster                                                                                                           
           INFO     Using Prometheus at                                                               prometheus_metrics_service.py:79
                    https://aps-workspaces.eu-west-2.amazonaws.com/workspaces/ws-xxxxx for cluster arn:aws:eks:eu-west-2:124978694494:cluster/xxxxx                                     
           DEBUG    Prometheus not found: Couldn't connect to Prometheus found under                                      loader.py:68
                    https://aps-workspaces.eu-west-2.amazonaws.com/workspaces/ws-xxxxx                
                    Caused by HTTPError: 403 Client Error: Forbidden for url:                                                         
                    https://aps-workspaces.eu-west-2.amazonaws.com/workspaces/ws-xxxxx/api             
                    /v1/query)                                                                                                        
           INFO     Connecting to Victoria for arn:aws:eks:eu-west-2:xxxxx:cluster/xxxxx       prometheus_metrics_service.py:63
                    cluster                                                                                                           
           INFO     Using Victoria at                                                                 prometheus_metrics_service.py:79
                    https://aps-workspaces.eu-west-2.amazonaws.com/workspaces/ws-009aa349-df06-485c-8                                 
                    ed2-670fe67af9e6 for cluster arn:aws:eks:eu-west-2:xxxxx:cluster/xxxxx                                     
[11:08:55] DEBUG    Victoria Metrics not found: Couldn't connect to Victoria Metrics found under                          loader.py:68
                    https://aps-workspaces.eu-west-2.amazonaws.com/workspaces/ws-xxxxx                 
                    Caused by PrometheusNotFound: Couldn't connect to Prometheus found under                                          
                    https://aps-workspaces.eu-west-2.amazonaws.com/workspaces/ws-xxxxx                 
                    Caused by HTTPError: 403 Client Error: Forbidden for url:                                                         
                    https://aps-workspaces.eu-west-2.amazonaws.com/workspaces/ws-xxxxx/api             
                    /v1/query))                                                                                                       
           INFO     Connecting to Thanos for arn:aws:eks:eu-west-2:xxxxx:cluster/xxxxx cluster prometheus_metrics_service.py:63
           INFO     Using Thanos at                                                                   prometheus_metrics_service.py:79
                    https://aps-workspaces.eu-west-2.amazonaws.com/workspaces/ws-xxxxx for cluster arn:aws:eks:eu-west-2:xxxxx:cluster/xxxxx                                     
           DEBUG    Thanos not found: Couldn't connect to Thanos found under                                              loader.py:68
                    https://aps-workspaces.eu-west-2.amazonaws.com/workspaces/xxxxx                
                    Caused by PrometheusNotFound: Couldn't connect to Prometheus found under                                          
                    https://aps-workspaces.eu-west-2.amazonaws.com/workspaces/ws-xxxxx                 
                    Caused by HTTPError: 403 Client Error: Forbidden for url:                                                         
                    https://aps-workspaces.eu-west-2.amazonaws.com/workspaces/ws-xxxxx/api             
                    /v1/query))                                                                                                       
           ERROR    No Prometheus or metrics service found    

kubectl version
Client Version: v1.28.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.3-eks-4f4795d

Profile
Using the default profile

Expected behaviour - Concise error handling to highlight Prometheus is not connected due to permission issue.

Terminology - EKS managed Prometheus references should be termed Amazon Managed Prometheus.

[aantn]

@cx-asafd and @jakesky-centrica, thanks for reporting.

Is the issue definitely a permission one then, and not a discovery issue? If so, does adding the params here solve it?

Regarding the terminology, you're correct of course. @pavangudiwada can you please update the terminology from EKS managed Prometheus to Amazon Managed Prometheus?

[johankvint]

I'm facing the same issue.

krr simple --prometheus-url "https://aps-workspaces.eu-west-1.amazonaws.com/workspaces/ws-xxxxxxxx" --eks-managed-prom                                                                                                                                                                                           

 _____       _               _          _  _______  _____
|  __ \     | |             | |        | |/ /  __ \|  __ \
| |__) |___ | |__  _   _ ___| |_ __ _  | ' /| |__) | |__) |
|  _  // _ \| '_ \| | | / __| __/ _` | |  < |  _  /|  _  /
| | \ \ (_) | |_) | |_| \__ \ || (_| | | . \| | \ \| | \ \
|_|  \_\___/|_.__/ \__,_|___/\__\__,_| |_|\_\_|  \_\_|  \_\


Thanks for using Robusta KRR. If you have any questions or feedback, please feel free to reach out to us at
https://github.com/robusta-dev/krr/issues

Watch our latest video to optimize your workloads and save costs: https://www.youtube.com/watch?v=TYRA2QcDIuI



Running Robusta's KRR (Kubernetes Resource Recommender) v1.22.0
Using strategy: Simple
Using formatter: table

[14:13:37] WARNING  The timeframe duration provided is insufficient and will be overridden with 2.016. Kindly adjust --timeframe_duration to a value equal to or greater than 2.016.          runner.py:340
           INFO     Using clusters: ['yyyyyy']                                                                                                                                                runner.py:284
[14:13:38] INFO     Prometheus URL is specified, will not auto-detect a metrics service                                                                                                       loader.py:55
           INFO     Trying to connect to Prometheus for yyyyyy cluster                                                                                                                        prometheus_metrics_service.py:69
           INFO     Using Prometheus at https://aps-workspaces.eu-west-1.amazonaws.com/workspaces/ws-xxxxxx for cluster yyyyy                                                       prometheus_metrics_service.py:98
           INFO     Prometheus not found: Couldn't connect to Prometheus found under https://aps-workspaces.eu-west-1.amazonaws.com/workspaces/ws-xxxxxx                                      loader.py:67
                    Caused by HTTPError: 403 Client Error: Forbidden for url: https://aps-workspaces.eu-west-1.amazonaws.com/workspaces/ws-xxxxx/api/v1/query) 
           ERROR    Wasn't able to connect to any Prometheus service in yyyyy cluster                                                                                                         runner.py:65
                    Try using port-forwarding and/or setting the url manually (using the -p flag.).                                                                                           
                    For more information, see 'Giving the Explicit Prometheus URL' at https://github.com/robusta-dev/krr?tab=readme-ov-file#usage                                             
           INFO     Listing scannable objects in yyyyyy                                                                                                                              __init__.py:96
[14:13:39] ERROR    An unexpected error occurred                                                                                                                                              runner.py:353
                    Traceback (most recent call last):                                                                                                                                                                                                                 
                      File "robusta_krr/core/runner.py", line 346, in run                                                                                                                                                                                              
                      File "robusta_krr/core/runner.py", line 296, in _collect_result                                                                                                                                                                                  
                    AttributeError: 'NoneType' object has no attribute 'get_cluster_summary'      

However if I use awscurl

awscurl -X POST --region eu-west-1 --service aps "https://aps-workspaces.eu-west-1.amazonaws.com/workspaces/ws-xxxxx/api/v1/query" -d 'query=QUERY' --header 'Content-Type: application/x-www-form-urlencoded'

{"status":"success","data":{"resultType":"vector","result":[]}}

So it is not a question about permissions.

EDIT: My issue is related to this #406
I'm able to connect successfully when using "static" access key, but not when assuming a role

[aantn]

Updating for anyone else looking at this - @johankvint found a problem with the 'static' access key flags, and we're fixing them in https://github.com/robusta-dev/krr/pull/407/files

Regarding session tokens, that will still need #406 which isn't ready yet

[elia-bracci-hs]

Have you already released a new version with the fix?

[aantn]

Hi @elia-bracci-hs, not yet. But you can run this from the source code with python/poetry to test.