[Relax][ONNX] Use ONNX Backend Tests to improve frontend coverage

[mshr-h]

This is my brain dump for a possible improvement to the Relax ONNX frontend tests.

Currently, many ONNX frontend tests in Relax manually construct small ONNX models and compare the imported Relax program against ONNX Runtime. This makes the test suite somewhat ad-hoc. As coverage grows, it becomes easy to miss ONNX spec corner cases, opset-specific behavior, and edge cases that are already covered by the official ONNX test suite.

ONNX provides an official Backend Test suite, including node-level tests for operator behavior:

• https://github.com/onnx/onnx/blob/main/docs/OnnxBackendTest.md

I think it would be worth exploring whether Relax's ONNX frontend tests can reuse those tests as a more systematic source of coverage.

The goal is not necessarily to expose TVM as a full ONNX backend. Rather, the idea is to use the ONNX Backend Tests as a testing mechanism for the importer:

• improve coverage of ONNX operator semantics
• reduce duplicated hand-written operator tests
• catch edge cases that are easy to miss with random-input tests
• align frontend correctness testing more closely with the ONNX specification
• keep TVM-specific frontend tests only for behavior that ONNX Backend Tests do not cover, such as importer options, parameter handling, name sanitization, and Relax-specific integration

Contributions, suggestions, or alternative ideas would be very welcome.

This issue is intended to collect thoughts and discuss whether this direction makes sense before working out a detailed design.

cc @KJlaccHoeUM9l @junrushao

[Aharrypotter]

Completely agree.

This will really clean up the relax/frontend/onnx workflow. By hooking into the official suite, we can focus entirely on the actual lowering logic and avoid spending time hand-writing tests and making up fake data.

I’d love to take this on and submit a PoC PR for the basic runner.

cc @KJlaccHoeUM9l @junrushao

[Aharrypotter]

I've started working on this and have a preliminary PoC ready.

What it does:

• Introduces a test runner that feeds ONNX Backend Test node-level cases into relax.frontend.onnx.from_onnx
• Compiles the imported Relax module with the VM and compares against ONNX's expected protobuf outputs
• Restricts the runner to NodeModel tests only (filters out real/simple/PyTorch model tests via BackendTest._test_items with layout validation)
• Adds onnx_backend pytest marker and SKIP_SLOW_TESTS support

Current status:

• 2406 node tests collected
• Representative subset (add, relu, gemm, conv, top_k, optional_get_element) passes; 1 xfail on test_top_k_cpu (dynamic k not yet supported)
• Full suite will be run in CI; known xfails are documented per importer gap category

Approach note: ONNX does not expose a public API to select only node tests, so the runner localizes _test_items access in one helper with assertions that fail collection if ONNX changes the internal layout.

I'll submit a Draft PR tonight for early feedback on the integration strategy. The PR will intentionally not fix importer gaps—those will be addressed in follow-ups so each change stays reviewable.

cc @KJlaccHoeUM9l @junrushao

[Aharrypotter]

Update: PR #19515 is up with a working implementation.

What it does

• Wraps the official ONNX Backend Test Suite (node-level tests) around relax.frontend.onnx.from_onnx
• Implements onnx.backend.base.Backend interface: import → DecomposeOpsForInference() → LegalizeOps() → detach_params() → tvm.compile() → VM execution
• Compares VM outputs against ONNX's protobuf reference outputs

Current numbers

• 72 operators with 388 test cases, all passing.
• No xfail markers — only operators where every ONNX node test passes are included
• Uncovered operators (cast, reduce, reshape, resize, quantization, etc.) are excluded via allowlist and can be added incrementally

Key design points

• NodeModel tests only; real/simple/PyTorch model tests filtered out
• SKIP_SLOW_TESTS=1 supported for fast local iteration
• onnx_backend pytest marker registered in conftest.py
• Existing test_frontend_onnx.py untouched — backend tests complement, not replace, TVM-specific tests

Feedback welcome.