ci(ZENKO-5142): ignore flaky test failures on re-run via JUnit merging and check_retries

[Copilot]

• Remove the raw-reports.xml python call — only junit-merged.xml is generated per attempt
• Update download loop to fetch junit-merged.xml from previous attempts (consistent with what's now uploaded)
• Merge step is now a single python3 line as suggested

---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

[francoisferrand]

@copilot Is it really how ´check_retries' behaves? My understanding was that it looked in the report only, not the previous runs. Can you check how it really behaves?

Also we really want to keep artifacts of each run, so we need to keep the "run" suffix somehow

[Copilot]

@copilot Is it really how ´check_retries' behaves? My understanding was that it looked in the report only, not the previous runs. Can you check how it really behaves?

Also we really want to keep a...

You're right on both counts — check_retries only deduplicates <testcase> entries within the same XML file (same suite, same name/classname/file key). It doesn't look at previous GitHub Check runs at all. The previous implementation was incorrect.

Fixed in 6ae738a:

1. New .github/scripts/merge-junit-reports.js (Node.js, no external deps): merges multiple JUnit XML files by combining <testsuite> elements that share the same name across files — so a test that failed in attempt 1 and passed in attempt 2 ends up as two <testcase> entries in the same suite. check_retries: true in mikepenz/action-junit-report then detects the duplicate and keeps the passing result.

2. Per-attempt artifact names (junit-merged-{job}-attempt-N): each run attempt uploads its merged report under a unique name, so all attempts' artifacts are preserved. The next attempt downloads attempt-N-1 via actions/download-artifact@v4 run-id: ${{ github.run_id }}.

3. check_name: ${{ inputs.stage }} (which includes run_attempt) is restored, so each attempt creates a separate, independently visible GitHub Check run.

[francoisferrand]

@copilot do we actually need a bash step : can't we simply use a javascript expression directly in the next step argument: ${{ github.run_attempt -1 }} ?

[Copilot]

Removed the bash step in 06abf96 — ${{ github.run_attempt - 1 }} is used directly in the artifact name.

[francoisferrand]

• is this needed: iill continue-on-error really "ignore" the failure completely, and let the whole job/workflow complete successfully?
• if it is, should we not simply move the Publish test report step down here (and remove its continue-on-error ?

[Copilot]

Done in 06abf96. continue-on-error: true does completely suppress the failure for the job conclusion, which is why the extra "Fail if tests have failures" step was needed as a workaround. Moved "Publish test report" to the very end (after "Upload artifacts"), removed continue-on-error and the workaround step. Added if: always() so it still runs when earlier archiving steps fail.

[francoisferrand]

@copilot Publish summary only if the tests in this job actually failed (or if this is not the first run, as it means we retried: so there may be flakies)

[Copilot]

Done in 5a16b0a — job_summary: ${{ job.status != 'success' || github.run_attempt > 1 }}.

[francoisferrand]

@copilot keep this at the beginning, so the report is displayed earlier.
it can fail, followup steps are marked as "always()" anyway

[Copilot]

Done in ab9b374 — "Publish test report" is now immediately after "Upload merged JUnit report", with continue-on-error: true so the archiving steps still run. Added if: always() to "Upload results" and "Upload artifacts" for the same reason.

[francoisferrand]

@copilot the JS script is way to complex.

download every report with a single bash loop : for report in {1..${{ github.run_attempt - 1 }}} ; do curl ...

[Copilot]

Done in b64308c. The download is now a plain bash step:

for attempt in $(seq 1 ${{ github.run_attempt - 1 }}); do
  curl -f -s -u "user:password" \
    "{link}/data/{job}.${attempt}/raw-reports.xml" \
    -o "/tmp/junit-previous/attempt-${attempt}.xml" \
    || echo "Attempt ${attempt} not available (skipped)"
done

merge-junit-reports.js now only handles XML merging — no HTTP, no auth.

[francoisferrand]

@copilot

this script is still way too long and complex. Python is available "natively" in GHA, may use a script like this:

#!/usr/bin/env python3
"""
merge-reports.py - Safe JUnit XML merger
Usage: python3 merge-reports.py output.xml input1.xml input2.xml ...
"""

import xml.etree.ElementTree as ET
import sys

def merge_reports(output_file, input_files):
    """Safely merge JUnit XML reports"""
    
    # Create root element
    root = ET.Element('testsuites')
    
    total_tests = 0
    total_failures = 0
    total_errors = 0
    total_skipped = 0
    
    for file in input_files:
        try:
            tree = ET.parse(file)
            source_root = tree.getroot()
            
            # Handle both testsuites and testsuite root elements
            if source_root.tag == 'testsuites':
                testsuites = source_root.findall('testsuite')
            else:
                testsuites = [source_root]
            
            for suite in testsuites:
                root.append(suite)
                
                # Aggregate stats
                total_tests += int(suite.get('tests', 0))
                total_failures += int(suite.get('failures', 0))
                total_errors += int(suite.get('errors', 0))
                total_skipped += int(suite.get('skipped', 0))
        
        except ET.ParseError as e:
            print(f"Error parsing {file}: {e}", file=sys.stderr)
            sys.exit(1)
    
    # Set aggregated stats on root
    root.set('tests', str(total_tests))
    root.set('failures', str(total_failures))
    root.set('errors', str(total_errors))
    root.set('skipped', str(total_skipped))
    
    # Write output
    tree = ET.ElementTree(root)
    ET.indent(tree, space="  ")  # Pretty print
    tree.write(output_file, encoding='utf-8', xml_declaration=True)
    print(f"✓ Merged {len(input_files)} files -> {output_file}")

if __name__ == '__main__':
    if len(sys.argv) < 4:
        print(f"Usage: {sys.argv[0]} <output.xml> <input1.xml> <input2.xml> ...", file=sys.stderr)
        sys.exit(1)
    
    output = sys.argv[1]
    inputs = sys.argv[2:]
    merge_reports(output, inputs)

[Copilot]

Done in 76e87fd. Replaced the JS script with a Python script based on your suggestion (stdlib xml.etree.ElementTree, no external deps). The actions/github-script and npm install steps are gone — the "Merge JUnit reports" step is now a plain bash step calling python3 merge-junit-reports.py twice: once for raw-reports.xml (current attempt only) and once for junit-merged.xml (all attempts combined).

[francoisferrand]

@copilot too complex and hard to read, makes this simpler

[Copilot]

Simplified in 4b9d805. The Python script now expands glob patterns internally, so the bash step is just two calls:

python3 merge-junit-reports.py raw-reports.xml '${{ inputs.junit-paths }}'
python3 merge-junit-reports.py junit-merged.xml '${{ inputs.junit-paths }}' '/tmp/junit-previous/*.xml'

No shopt, no arrays, no conditionals.

[francoisferrand]

@copilot why 2 calls? Can't we make a single one?

[Copilot]

Done in f88291e. The script now takes output_dir current_glob [previous_glob] and writes both files internally, so the bash step is a single call:

python3 merge-junit-reports.py \
  /tmp/artifacts/data/${{ inputs.stage }} \
  '${{ inputs.junit-paths }}' '/tmp/junit-previous/*.xml'

[francoisferrand]

@copilot why handle this here?
can't we just process every file that is passed in argument (no globbing, just let bash handle wildcard expansion in the gha step)

[Copilot]

Done in ee270a5. Removed glob/os from the Python script — it now just takes plain file paths as sys.argv[2:]. Bash handles glob expansion in the action step:

shopt -s nullglob
python3 "$SCRIPT" "$STAGE/raw-reports.xml" ${{ inputs.junit-paths }}
python3 "$SCRIPT" "$STAGE/junit-merged.xml" ${{ inputs.junit-paths }} /tmp/junit-previous/*.xml

shopt -s nullglob ensures the second call works safely on attempt 1 (empty /tmp/junit-previous/).

[francoisferrand]

@copilot merge both line, let bash expand both wildcards

Suggested change

	python3 "$SCRIPT" "$STAGE/raw-reports.xml" ${{ inputs.junit-paths }}
	python3 "$SCRIPT" "$STAGE/junit-merged.xml" ${{ inputs.junit-paths }} /tmp/junit-previous/*.xml
	python3 "$SCRIPT" "$STAGE/junit-merged.xml" ${{ inputs.junit-paths }} /tmp/junit-previous/*.xml

[Copilot]

Done in 3b3e9df — removed the raw-reports.xml line. The download loop now fetches junit-merged.xml from each previous attempt (consistent with what's uploaded), so the merge step is a single call.