Skip to content

Add 64 integration and coverage tests for ValidationPipeline#940

Draft
Copilot wants to merge 2 commits intomasterfrom
copilot/improve-testing-strategies
Draft

Add 64 integration and coverage tests for ValidationPipeline#940
Copilot wants to merge 2 commits intomasterfrom
copilot/improve-testing-strategies

Conversation

Copy link

Copilot AI commented Mar 14, 2026

The PeasyAI ValidationPipeline lacked tests for its public API, cross-stage integration, error recovery, and several validator-specific behaviors, leaving core functionality implicitly trusted rather than verified.

New test file: tests/test_pipeline_and_integration.py

End-to-end pipeline flow (TestPipelineIntegration)

  • Stage 1 (post-processor) fixes propagate correctly into Stage 2 (validators) — e.g., var x: int = 0; fixed before InlineInitValidator runs
  • original_code is never mutated; fixes_applied and validators_run are populated correctly

Pipeline customisation (TestPipelineCustomisation)

  • add_validator / remove_validator mechanics
  • apply_fixes=False — issues still reported, code unchanged, fixes_applied empty
  • include_test_validators=True adds TestFileValidator; custom constructor validator list works

PipelineResult public API (TestPipelineResultAPI)

  • errors / warnings property filtering by severity
  • to_review_dict() backward-compat shape (INFO issues surface in "warnings" key)
  • summary() field presence

Convenience helpers (TestValidatePCodeConvenience)

  • validate_p_code() and create_default_pipeline() round-trip
  • is_test_file=True full flow: post-processor generates test declarations; separate apply_fixes=False path exercises TestFileValidator directly

Validator-specific gaps

  • SyntaxValidator: }; detection, auto_fixable flag, fix correctness, occurrence count in message
  • MachineStructureValidator: goto to undefined state (ERROR), self-loop goto (valid), empty state body (INFO), multi-machine independence

Input validation (TestInputValidationResult, TestDesignDocValidatorEdgeCases)

  • InputValidationResult.success() / .failure() factories
  • DesignDocValidator: exact MIN_LENGTH, exceeding MAX_LENGTH, all required sections present suppresses section warnings, extract_metadata edge cases

Error recovery (TestPipelineErrorHandling)

  • Crashing validator does not propagate; recorded as INFO issue; subsequent validators still execute

File/project helpers (TestValidateFileAndProject)

  • validate_file() reads from disk; PTst path triggers test-file mode in post-processor
  • validate_project() validates all .p files; cross-file context enables DuplicateDeclarationValidator

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

  • huggingface.co
    • Triggering command: /tmp/peasyai_venv/bin/pytest /tmp/peasyai_venv/bin/pytest tests/ -v --ignore=tests/pipeline --tb=short -q (dns block)
    • Triggering command: /tmp/peasyai_venv/bin/pytest /tmp/peasyai_venv/bin/pytest tests/ --ignore=tests/pipeline --ignore=tests/rag -v -q -j DROP (dns block)
    • Triggering command: /tmp/peasyai_venv/bin/pytest /tmp/peasyai_venv/bin/pytest tests/ --ignore=tests/pipeline --ignore=tests/rag -q (dns block)

If you need me to access, download, or install something from one of these locations, you can either:


🔒 GitHub Advanced Security automatically protects Copilot coding agent pull requests. You can protect all pull requests by enabling Advanced Security for your repositories. Learn more about Advanced Security.

Adds tests/test_pipeline_and_integration.py with 64 tests covering gaps
identified in the PeasyAI test suite:

- TestPipelineIntegration (7): Stage 1+2 end-to-end flow, inline-init
  fixed by post-processor, original_code preserved, validators_run list
- TestPipelineCustomisation (9): add/remove_validator, apply_fixes=False,
  include_test_validators, custom validator list
- TestPipelineResultAPI (9): errors/warnings properties, is_valid logic,
  summary format, to_review_dict shape and backward compat
- TestValidatePCodeConvenience (8): validate_p_code() and
  create_default_pipeline() helpers, full flow with is_test_file
- TestSyntaxValidatorBraceSemicolon (5): }; detection, auto-fix
- TestMachineStructureValidatorGoto (5): undefined/defined goto, self-
  loop, empty state INFO, multi-machine validation
- TestInputValidationResult (5): success()/failure() factory methods
- TestDesignDocValidatorEdgeCases (7): boundary lengths, all sections
  present, events mentioned, extract_metadata edge cases
- TestPipelineErrorHandling (3): crashing validator skipped gracefully,
  INFO issue recorded, other validators still run
- TestValidateFileAndProject (7): validate_file(), validate_project(),
  PTst path detection, cross-file DuplicateDeclarationValidator context

Co-authored-by: ankushdesai <12651260+ankushdesai@users.noreply.github.com>
Copilot AI changed the title [WIP] Improve testing methodologies and coverage Add 64 integration and coverage tests for ValidationPipeline Mar 14, 2026
Copilot AI requested a review from ankushdesai March 14, 2026 15:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants