THREESCALE-12133 optimize configuration reload #1564

tkan145 · 2026-01-19T04:01:25Z

What

NOTE:

Try to download the flamegraph file and and open it in your browser (interactive graph).
Latency was measured by reloading the static file, adding 2-3 seconds to the result if loading from a portal.
JSON encoding/decoding still consumes a significant portion of the CPU, but latency has decreased to an acceptable level, and it requires more extensive changes, so I decided to defer the refactor to the next PR.

Before:

File read

bash-5.2# accessed-files -p `pgrep -f 'nginx: worker'` -t 60        
Neither -r nor -w options are specified.                            
bash-5.2# accessed-files -p `pgrep -f 'nginx: worker'` -r -t 60     
Tracing 3599829 (/usr/local/openresty/nginx/sbin/nginx)...          
Please wait for 60 seconds.                                         
                                                                    
=== Top 10 file reads ===                                           
#1: 117280 times, 233684323 bytes reads in file apicast-policy.json.
#2: 44 times, 31357920 bytes reads in file config.json.

Latency

HTTP                                                                                
http_req_duration..............: med=42.46ms p(95)=48.14ms p(99)=2.37s p(99.9)=2.76s
  { expected_response:true }...: med=42.46ms p(95)=48.14ms p(99)=2.37s p(99.9)=2.76s
http_req_failed................: 0.00%  0 out of 9000                               
http_reqs......................: 9000   47.464397/s

Flamegraph

After:

File read

bash-5.2# accessed-files -p `pgrep -f 'nginx: worker'` -r -t 60
Tracing 3605605 (/usr/local/openresty/nginx/sbin/nginx)...     
Please wait for 60 seconds.                                    
                                                               
=== Top 10 file reads ===                                      
#1: 44 times, 31357920 bytes reads in file config.json.

Latency

HTTP                                                                                
http_req_duration..............: med=42.66ms p(95)=44.6ms p(99)=68.14ms p(99.9)=205.02ms
  { expected_response:true }...: med=42.66ms p(95)=44.6ms p(99)=68.14ms p(99.9)=205.02ms
http_req_failed................: 0.00%  0 out of 9147                                   
http_reqs......................: 9147   49.677724/s

Flamegraph

Verification steps

Install 3scale
Setup products, applications, backend, etc...

Details

#!/bin/bash

# Get the ADMIN_URL and ADMIN_ACCESS_TOKEN from apimanager and system-seed secret
DOMAIN=$(oc get routes console -n openshift-console -o json | jq -r '.status.ingress[0].routerCanonicalHostname' | sed 's/router-default.//')
ADMIN_ACCESS_TOKEN=$(oc get secret system-seed -n 3scale-test -o jsonpath="{.data.ADMIN_ACCESS_TOKEN}"| base64 --decode)
oc project 3scale-test
# Create the required secrets for Accounts, products and backends. 
oc apply -f - <<EOF
---
apiVersion: v1
kind: Secret
metadata:
  name: mytenant
type: Opaque
stringData:
  adminURL: https://3scale-admin.$DOMAIN
  token: $ADMIN_ACCESS_TOKEN
EOF
# user secret
oc apply -f - <<EOF
---
apiVersion: v1
kind: Secret
metadata:
  name: myusername01
stringData:
  password: "123456"
EOF
# Developer User
oc apply -f - <<EOF
---
apiVersion: capabilities.3scale.net/v1beta1
kind: DeveloperUser
metadata:
  name: developeruser01
  namespace: 3scale-test
  annotations:
    "insecure_skip_verify": "true"
spec:
  developerAccountRef:
    name: developeraccount01
  email: myusername01@example.com
  passwordCredentialsRef:
    name: myusername01
  providerAccountRef:
    name: mytenant
  role: admin
  username: myusername01
EOF
sleep 30
# TODO check for developer user completed
oc apply -f - <<EOF
---
apiVersion: capabilities.3scale.net/v1beta1
kind: DeveloperAccount
metadata:
  name: developeraccount01
  namespace: 3scale-test
  annotations:
    "insecure_skip_verify": "true"
spec:
  orgName: 3scale-test
  providerAccountRef:
    name: mytenant
EOF
deploy httpbin and use it as the backend
oc new-project httpbin
oc new-app quay.io/trepel/httpbin
oc get svc
oc scale deployment/httpbin --namespace httpbin --replicas=1 
oc project 3scale-test

# create backend
oc apply -f - <<EOF
---
apiVersion: capabilities.3scale.net/v1beta1
kind: Backend
metadata:
  name: backend1-cr
  namespace: 3scale-test
  annotations:
    "insecure_skip_verify": "true"
spec:
  mappingRules:
    - httpMethod: GET
      increment: 1
      last: true
      metricMethodRef: hits
      pattern: /
    - httpMethod: POST
      pattern : "/"
      metricMethodRef: hits
      increment: 1    
  name: backend1
  privateBaseURL: 'http://httpbin.httpbin.svc:8080'
  systemName: backend1
EOF

TOTAL_PRODUCTS=600

for ((n=1; n<=$TOTAL_PRODUCTS; n++))
do
# Product
oc apply -f - <<EOF
---
apiVersion: capabilities.3scale.net/v1beta1
kind: Product
metadata:
  name: product$n-cr
  namespace: 3scale-test
  annotations:
    "insecure_skip_verify": "true"
spec:
  applicationPlans:
    plan01:
      name: "My Plan 01"
  deployment:
    apicastHosted:
      authentication:
        userkey:
          authUserKey: token
  name: product$n
  backendUsages:
    backend1:
      path: /
  mappingRules:
    - httpMethod: GET
      pattern : "/"
      metricMethodRef: hits
      increment: 1
    - httpMethod: POST
      pattern : "/"
      metricMethodRef: hits
      increment: 1
  policies:
    - name: headers
      version: builtin
      enabled: true
      configuration:
        set_headers:
          - name: "echo1"
            value: "test"
            value_type: "plain"
    - name: headers
      version: builtin
      enabled: true
      configuration:
        set_headers:
          - name: "echo2"
            value: "test"
            value_type: "plain"
    - name: headers
      version: builtin
      enabled: true
      configuration:
        set_headers:
          - name: "echo3"
            value: "test"
            value_type: "plain"
    - name: logging
      version: builtin
      enabled: true
      configuration:
        custom_logging: "[{{time_local}}] {{host}}:{{server_port}} {{remote_addr}}:{{remote_port}} \"{{request}}\" {{status}} {{body_bytes_sent}} ({{request_time}}) {{post_action_impact}} AND {{upstream_response_time}}"
    - name: headers
      version: builtin
      enabled: true
      configuration:
        set_headers:
          - name: "echo4"
            value: "test"
            value_type: "plain"
    - name: headers
      version: builtin
      enabled: true
      configuration:
        set_headers:
          - name: "echo5"
            value: "test"
            value_type: "plain"
    - name: apicast
      version: builtin
      enabled: true
      configuration: {}
EOF

# application
oc apply -f - <<EOF
---
apiVersion: capabilities.3scale.net/v1beta1
kind: Application
metadata:
  name: application$n-cr
  namespace: 3scale-test
  annotations:
    "insecure_skip_verify": "true"
spec:
  accountCR: 
    name: developeraccount01
  applicationPlanName: plan01
  productCR: 
    name: product$n-cr
  name: testApp
  description: further testing
EOF
# TODO proxy promote
oc apply -f - <<EOF
---
apiVersion: capabilities.3scale.net/v1beta1
kind: ProxyConfigPromote
metadata:
  name: product$n-v1-production
  namespace: 3scale-test
  annotations:
    "insecure_skip_verify": "true"
spec:
  productCRName: product$n-cr
  production: true
  deleteCR: true
EOF
done


sleep 30
echo Product Route: 
echo "https://$(oc get routes | grep product1 |grep production| awk '{print $2}')" 
echo
echo User_key: 
echo $(curl -s -X 'GET' "https://3scale-admin.$DOMAIN/admin/api/applications.xml?access_token=$ADMIN_ACCESS_TOKEN&page=1&per_page=500&service_id=3" -H 'accept: */*' | grep -oP '<user_key>\K[^<]+' | sed 's/\s//g')

Once the script finish, visit the admin portal and wait for all products to be populated (going to take a while)
Create an token
Inside APIcast repo, move into dev-environments folder

▲ lib/3scale/APIcast cd dev-environments/http-proxy-plain-http-upstream/

Update the docker-compose.yml file as follow. Update <token> and <3scale_admin_portal> to match values from previous step.

diff --git a/dev-environments/http-proxy-plain-http-upstream/docker-compose.yml b/dev-environments/http-proxy-plain-http-upstream/docker-compose.yml
index f7b311a5..5032bc13 100644                                                                                                                     
--- a/dev-environments/http-proxy-plain-http-upstream/docker-compose.yml                                                                            
+++ b/dev-environments/http-proxy-plain-http-upstream/docker-compose.yml                                                                            
@@ -12,11 +12,12 @@ services:                                                                                                                       
     - two.upstream                                                                                                                                 
     environment:                                                                                                                                   
       THREESCALE_CONFIG_FILE: /tmp/config.json                                                                                                     
-      THREESCALE_DEPLOYMENT_ENV: staging                                                                                                           
-      APICAST_CONFIGURATION_LOADER: lazy                                                                                                           
+      THREESCALE_PORTAL_ENDPOINT: "https://<token>:<3scale_admin_portal>"                                                                          
+      THREESCALE_DEPLOYMENT_ENV: production                                                                                                        
+      APICAST_CONFIGURATION_LOADER: boot                                                                                                           
       APICAST_WORKERS: 1                                                                                                                           
-      APICAST_LOG_LEVEL: debug                                                                                                                     
-      APICAST_CONFIGURATION_CACHE: "0"                                                                                                             
+      APICAST_LOG_LEVEL: notice                                                                                                                    
+      APICAST_CONFIGURATION_CACHE: "15"                                                                                                            
     expose:                                                                                                                                        
       - "8080"                                                                                                                                     
       - "8090"

Start the gateway

make gateway IMAGE_NAME=quay.io/3scale/apicast:latest

Get the gateway's IP

 ▲ lib/3scale/3scale-operator docker inspect \                                                         
  -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' http-proxy-plain-http-upstream-gateway-1

In another terminal, create k6 script with the following content

import http from 'k6/http';
import { check, sleep } from 'k6';

export const options = {
  stages: [
    { duration: '3m', target: 100 },
    { duration: '6m', target: 200 },
    { duration: '2m', target: 0 },
  ],
  thresholds: {
    http_req_duration: ['p(95)<500'], 
    http_req_failed: ['rate<0.01'], 
  },
};

export function setup() {
  // This runs once for the whole test run, after the init context and before default
  const msg = `[${new Date().toISOString()}] Starting load test`;
  console.log(msg);
}

export default function () {
  const url = 'http://<APICast_gateway_IP>:8080/echo?user_key=<user_key>'
  const params = {
    headers: {
      'Host': '<hostname>'
      'Content-Type': 'application/json',
    },
  };

  const res = http.get(url, params);
  if(res.timings.duration > 1000) {
    const msg = `[${new Date().toISOString()}] Response time above 1s: ${String(res.timings.duration)} ms`;
    console.log(msg);
  }

  check(res, {
    'status 200': (r) => r.status === 200
  });

  sleep(1);
}

Replace:

APICast_gateway_IP, user_key with value from previous steps
Visit the admin portal and replace the hostname with the hostname of one product of your choice
Start k6 test

k6 run --summary-trend-stats="med,p(95),p(99),p(99.9)" script.js

Once the test is finish, stop the gateway

CTRL-C

Checkout this branch and build a new runtime-image

make runtime-image IMAGE_NAME=quay.io/3scale/apicast:reload-patch

Start the gateway again with new image

make gateway IMAGE_NAME=quay.io/3scale/apicast:reload-patch

Start the test again, you should see the delay drop to sub 1s

k6 run --summary-trend-stats="med,p(95),p(99),p(99.9)" script.js

Previously, every time the policy chain was rebuilt, the gateway would look for the manifests file on the disk. This caused a delay for incoming requests. Since the manifests are static and do not change at runtime, it is more efficient to cache them at the module level. This caching will speed up the lookup process.

Creating a json schema validator is somewhat expensive, so we cache this step with a local LRU cache

…m portal

akostadinov · 2026-01-20T14:15:24Z

gateway/src/apicast/policy_loader.lua

    if manifest then
      return load_path, manifest.configuration
    else
      insert(failures, load_path)


Reading lua for the first time so excuse me if I miss something.

Whenever the manifest is not found, here we insert a failure somewhere. But if it is missing, should we continue with the rest of the method? Shouldn't we just return to avoid further processing?

Perhaps this would not be performance critical because we don't expect errors to happen often. But it occurred to me to ask you if that makes sense.

akostadinov · 2026-01-20T15:17:22Z

gateway/src/apicast/policy_chain.lua

-    ):check(self)
+  if manifests then
+    PolicyOrderChecker.new(manifests):check(self)
+  end


A nit, it looks like the method is only called in gateway/src/apicast/configuration.lua without parameters. So perhaps makes sense to simplify by removing the argument from it as well the if clause.

Probably the idea was to use the manifest cache somewhere?

In any case - why is this change implemented? 🤔

It seems to me that:

It always loads all policies (on line :196) regardless of whether manifests is passed or not.

If manifests is passed, and is not null (as @akostadinov pointed out - this does not happen at this point), then the policy order check will be performed 2 times. Is that intended?

akostadinov

These are some excellent improvements! Just a couple of minor suggestions that you can ignore if they don't make sense to you.

Now learning a little bit more about apicast architecture, I think that your suggestion of looking at increment updates makes a lot of sense so we can discuss the options to implement.

mayorova · 2026-01-21T14:26:04Z

gateway/src/apicast/policy_loader.lua

+local tab_new = require('table.new')
+local isempty = require('table.isempty')
+
+local manifests_cache = tab_new(32, 0)


Why 32? A question without understanding (yet, at least) how the whole thing works.
We have about 50 policies, which also might be growing over time. I understand that a single customer will probably not use all of them, but is there a big difference in reserving a bit more (just in case) ?

Actually, reading further - I can see that when all policies are loaded, this cache is being replaced by the dictionary (or hash, or whatever it's called in Lua).
But then - shouldn't we allocate hash slots instead? tab_new(0, 32) or also potentially bump 32 to a bigger number?
Maybe it's nitpicking, not sure.

mayorova · 2026-01-21T14:58:58Z

gateway/src/apicast/policy_loader.lua

  return format('%s/?.lua', load_path)
 end

+local function get_manifest(name, version)


A few things:

Maybe name this method as get_cached_manifest or something like that, so the intention is clearer, as there are quite a few with similar names: get_manifest, load_manifest, read_manifest, it's not always clear what they do exactly.

I was thinking if it could make sense to have name:version as the key of the cache so we can avoid the loop. But well, I guess the idea is for cache to match exactly the structure of the manifests object that is loaded from filesystem, so it's easier to operate. There probably won't be many versions of the same policy anyway, so probably premature optimization.

mayorova · 2026-01-21T16:14:36Z

gateway/src/apicast/configuration_loader/remote_v2.lua

-    local service = configuration.parse_service(proxy_config.content)
-
-    -- We always assign a oidc to the service, even an empty one with the
-    -- service_id, if not on APICAST_SERVICES_LIST will fail on filtering


Maybe we need to leave this comment to explain the reason?

mayorova · 2026-01-21T16:16:09Z

gateway/src/apicast/configuration_loader/remote_v2.lua

-
-    -- We always assign a oidc to the service, even an empty one with the
-    -- service_id, if not on APICAST_SERVICES_LIST will fail on filtering
-    local oidc = self:oidc_issuer_configuration(service)


Looks like the oidc_issuer_configuration function is not called from anywhere anymore, so probably can be deleted?

mayorova · 2026-01-21T16:29:47Z

I haven't actually built/run the image, but I left some comments 🙃

tkan145 added 4 commits January 19, 2026 13:58

perf: reuse PolicyOrderChecker

ad0342d

perf: cache the JSON schema validator

771ea26

Creating a json schema validator is somewhat expensive, so we cache this step with a local LRU cache

perf: avoid construct the full policy chain when parsing response fro…

2a13c8e

…m portal

tkan145 requested a review from a team as a code owner January 19, 2026 04:01

akostadinov reviewed Jan 20, 2026

View reviewed changes

akostadinov approved these changes Jan 20, 2026

View reviewed changes

mayorova reviewed Jan 21, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

THREESCALE-12133 optimize configuration reload #1564

THREESCALE-12133 optimize configuration reload #1564

tkan145 commented Jan 19, 2026 •

edited

Loading

Uh oh!

akostadinov Jan 20, 2026

Uh oh!

akostadinov Jan 20, 2026

Uh oh!

mayorova Jan 21, 2026

Uh oh!

akostadinov left a comment

Uh oh!

mayorova Jan 21, 2026

Uh oh!

mayorova Jan 21, 2026

Uh oh!

mayorova Jan 21, 2026

Uh oh!

mayorova Jan 21, 2026

Uh oh!

mayorova Jan 21, 2026

Uh oh!

mayorova commented Jan 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

THREESCALE-12133 optimize configuration reload #1564

Are you sure you want to change the base?

THREESCALE-12133 optimize configuration reload #1564

Conversation

tkan145 commented Jan 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Before:

File read

Latency

Flamegraph

After:

File read

Latency

Flamegraph

Verification steps

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

akostadinov left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mayorova commented Jan 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

tkan145 commented Jan 19, 2026 •

edited

Loading