OCPBUGS-97822: test/router: retry http2 test when response comes from wrong router by mkowalski · Pull Request #31367 · openshift/origin

mkowalski · 2026-07-06T13:17:26Z

Problem

The http2 test deploys a dedicated router shard with HTTP/2 enabled (ingress.operator.openshift.io/default-enable-http2: true) and reaches it via the shard's wildcard DNS record (*.<namespace>.apps.<cluster-domain>). Until that record propagates, the route hostname still matches the cluster's *.apps wildcard (RFC 4592 wildcards match multiple labels), so requests are served by the default router, which also admits the test routes but has HTTP/2 disabled by default.

Such a response carries the expected status code and body, so the poll loop accepts it, and the post-loop protocol assertion fails with HTTP/1.1 != HTTP/2.0 — even though the shard would have served the request correctly moments later.

Evidence

Component Readiness regression #42853: The HAProxy router should pass the http2 tests failing at ~20% on 5.0 HyperShift AWS conformance (periodic-ci-openshift-hypershift-release-5.0-periodics-e2e-aws-ovn-conformance) since 2026-06-27.

9 of 10 sampled failures are identical and always on the first test case:

fail [github.com/openshift/origin/test/extended/router/http2.go:513]: {route:http2-custom-cert-edge frontendProto:HTTP/2.0 backendProto:HTTP/1.1 statusCode:200 useHTTP2Transport:true}
Expected
    <string>: HTTP/1.1
to equal
    <string>: HTTP/2.0

Status 200 + correct body + wrong protocol is exactly the signature of the default router (no h2) serving the request. The 10th failure is a plain DNS/LB propagation timeout at http2.go:509. The window likely widened on HyperShift with the AWSServiceLBNetworkSecurityGroup promotion (openshift/api#2838) changing shard LB provisioning latency, but the race itself has always been in the test.

Fix

Treat a response with an unexpected negotiated protocol or body as retryable inside the existing 5-minute poll loop, instead of asserting on the first 200.
Call client.CloseIdleConnections() before retrying so the next attempt re-resolves DNS and opens a fresh TLS connection instead of being pinned to the wrong router by the transport's keep-alive pool.
Move body reading into the loop; final assertions after the loop are kept.

For test cases that legitimately expect HTTP/1.1 the check is a no-op on the happy path.

Generated-by: AI (analysis + patch), reviewed by mko@redhat.com

/cc @frobware

Summary by CodeRabbit

Tests
- Improved HTTP/2 router conformance checks to verify both the response protocol and response body during each poll.
- Added safer response handling by reading and closing response bodies immediately, including on failed attempts.
- Retries now more reliably refresh the network path when the expected router or shard is not reached.

The http2 test deploys a dedicated router shard with HTTP/2 enabled and reaches it via the shard's wildcard DNS record (*.<namespace>.apps.<cluster-domain>). Until that record propagates, the route hostname still matches the cluster's *.apps wildcard (RFC 4592 wildcards match multiple labels), so requests are served by the default router, which also admits the test routes but has HTTP/2 disabled by default. Such a response carries the expected status code and body, so the poll loop accepts it, and the subsequent protocol assertion fails with HTTP/1.1 != HTTP/2.0 even though the shard would have served the request correctly moments later. This is visible as a flaky failure on 5.0 HyperShift AWS conformance (Component Readiness regression #42853, ~20% failure rate, consistently failing on the first test case with 'Expected HTTP/1.1 to equal HTTP/2.0'): fail [.../test/extended/router/http2.go:513]: {route:http2-custom-cert-edge frontendProto:HTTP/2.0 ...} Expected <string>: HTTP/1.1 to equal <string>: HTTP/2.0 Fix by treating a response with an unexpected negotiated protocol or body as retryable inside the poll loop. Also drop pooled keep-alive connections before retrying so the next attempt re-resolves DNS and opens a fresh TLS connection instead of being pinned to the wrong router by the transport's connection pool. Generated-by: AI Signed-off-by: Mateusz Kowalski <mko@redhat.com>

openshift-merge-bot · 2026-07-06T13:17:29Z

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: automatic mode

openshift-ci-robot · 2026-07-06T13:17:30Z

@mkowalski: This pull request explicitly references no jira issue.

Details

In response to this:

Problem

The http2 test deploys a dedicated router shard with HTTP/2 enabled (ingress.operator.openshift.io/default-enable-http2: true) and reaches it via the shard's wildcard DNS record (*.<namespace>.apps.<cluster-domain>). Until that record propagates, the route hostname still matches the cluster's *.apps wildcard (RFC 4592 wildcards match multiple labels), so requests are served by the default router, which also admits the test routes but has HTTP/2 disabled by default.

Such a response carries the expected status code and body, so the poll loop accepts it, and the post-loop protocol assertion fails with HTTP/1.1 != HTTP/2.0 — even though the shard would have served the request correctly moments later.

Evidence

Component Readiness regression #42853: The HAProxy router should pass the http2 tests failing at ~20% on 5.0 HyperShift AWS conformance (periodic-ci-openshift-hypershift-release-5.0-periodics-e2e-aws-ovn-conformance) since 2026-06-27.

9 of 10 sampled failures are identical and always on the first test case:
fail [github.com/openshift/origin/test/extended/router/http2.go:513]: {route:http2-custom-cert-edge frontendProto:HTTP/2.0 backendProto:HTTP/1.1 statusCode:200 useHTTP2Transport:true}
Expected
   <string>: HTTP/1.1
to equal
   <string>: HTTP/2.0
Status 200 + correct body + wrong protocol is exactly the signature of the default router (no h2) serving the request. The 10th failure is a plain DNS/LB propagation timeout at http2.go:509. The window likely widened on HyperShift with the AWSServiceLBNetworkSecurityGroup promotion (openshift/api#2838) changing shard LB provisioning latency, but the race itself has always been in the test.

Fix

Treat a response with an unexpected negotiated protocol or body as retryable inside the existing 5-minute poll loop, instead of asserting on the first 200.

Call client.CloseIdleConnections() before retrying so the next attempt re-resolves DNS and opens a fresh TLS connection instead of being pinned to the wrong router by the transport's keep-alive pool.

Move body reading into the loop; final assertions after the loop are kept.

For test cases that legitimately expect HTTP/1.1 the check is a no-op on the happy path.

Generated-by: AI (analysis + patch), reviewed by mko@redhat.com

/cc @frobware

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

coderabbitai · 2026-07-06T13:18:24Z

Walkthrough

This PR modifies the HTTP/2 router conformance test to read and store the response body during the polling loop rather than after it. Polling now retries on protocol or body content mismatches, and closes idle connections before retrying. Post-poll assertions use the stored body.

Changes

HTTP/2 router test polling update

Layer / File(s)	Summary
Body capture and validation during polling `test/extended/router/http2.go`	Adds a shared `body` variable, moves response body reading into the poll attempt with close-and-retry on status/protocol/content mismatches, and updates post-poll assertions to use the stored body instead of a separate `ReadAll`/`Close`.

Estimated code review effort: 2 (Simple) | ~10 minutes

🚥 Pre-merge checks | ✅ 15

✅ Passed checks (15 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names	✅ Passed	The only Ginkgo titles in http2.go are static literals; no dynamic or run-specific values appear in It/Describe names.
Test Structure And Quality	✅ Passed	The test stays focused on router HTTP/2 conformance, uses deferred cleanup for the shard plus namespace teardown, and all cluster waits have explicit timeouts.
Microshift Test Compatibility	✅ Passed	No new MicroShift-unsafe test was added; the existing router test is already gated by [apigroup:config.openshift.io], [apigroup:image.openshift.io], and [apigroup:operator.openshift.io].
Single Node Openshift (Sno) Test Compatibility	✅ Passed	http2.go has no SNO skip because it makes no multi-node assumptions; it only checks router/DNS/protocol behavior on a single route shard.
Topology-Aware Scheduling Compatibility	✅ Passed	Only test/extended/router/http2.go changed; no manifests, controllers, or scheduling constraints were added.
Ote Binary Stdout Contract	✅ Passed	PASS: The PR only changes an It-block polling loop in test/extended/router/http2.go; no main/init/TestMain/BeforeSuite stdout writes or fmt.Print/klog issues were added.
Ipv6 And Disconnected Network Test Compatibility	✅ Passed	PASS: The changed test only uses cluster-internal route/DNS names and contains no IPv4 literals, IPv4-only parsing, or public internet dependencies.
No-Weak-Crypto	✅ Passed	The PR only changes HTTP/2 router test polling; no MD5/SHA1/DES/RC4/3DES/Blowfish/ECB, custom crypto, or secret comparisons were introduced.
Container-Privileges	✅ Passed	Only test/extended/router/http2.go changed; no container/K8s manifests or privilege-related settings (privileged, hostNetwork, hostPID, hostIPC, SYS_ADMIN, allowPrivilegeEscalation) were introduced.
No-Sensitive-Data-In-Logs	✅ Passed	New logs only emit route config, HTTP protocol values, and a test-server body string; no passwords, tokens, PII, or customer data are introduced.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the main change: retrying the HTTP/2 router test when the response comes from the wrong router.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

openshift-ci · 2026-07-06T13:19:41Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mkowalski

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~test/extended/router/OWNERS~~ [mkowalski]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

mkowalski · 2026-07-06T13:23:04Z

Filed https://issues.redhat.com/browse/OCPBUGS-97822 for this (Component Readiness regression 42853, release blocker).

openshift-ci-robot · 2026-07-06T13:29:40Z

@mkowalski: This pull request references Jira Issue OCPBUGS-97822, which is invalid:

expected the bug to target the "5.0.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Problem

The http2 test deploys a dedicated router shard with HTTP/2 enabled (ingress.operator.openshift.io/default-enable-http2: true) and reaches it via the shard's wildcard DNS record (*.<namespace>.apps.<cluster-domain>). Until that record propagates, the route hostname still matches the cluster's *.apps wildcard (RFC 4592 wildcards match multiple labels), so requests are served by the default router, which also admits the test routes but has HTTP/2 disabled by default.

Such a response carries the expected status code and body, so the poll loop accepts it, and the post-loop protocol assertion fails with HTTP/1.1 != HTTP/2.0 — even though the shard would have served the request correctly moments later.

Evidence

Component Readiness regression #42853: The HAProxy router should pass the http2 tests failing at ~20% on 5.0 HyperShift AWS conformance (periodic-ci-openshift-hypershift-release-5.0-periodics-e2e-aws-ovn-conformance) since 2026-06-27.

9 of 10 sampled failures are identical and always on the first test case:
fail [github.com/openshift/origin/test/extended/router/http2.go:513]: {route:http2-custom-cert-edge frontendProto:HTTP/2.0 backendProto:HTTP/1.1 statusCode:200 useHTTP2Transport:true}
Expected
   <string>: HTTP/1.1
to equal
   <string>: HTTP/2.0
Status 200 + correct body + wrong protocol is exactly the signature of the default router (no h2) serving the request. The 10th failure is a plain DNS/LB propagation timeout at http2.go:509. The window likely widened on HyperShift with the AWSServiceLBNetworkSecurityGroup promotion (openshift/api#2838) changing shard LB provisioning latency, but the race itself has always been in the test.

Fix

Treat a response with an unexpected negotiated protocol or body as retryable inside the existing 5-minute poll loop, instead of asserting on the first 200.

Call client.CloseIdleConnections() before retrying so the next attempt re-resolves DNS and opens a fresh TLS connection instead of being pinned to the wrong router by the transport's keep-alive pool.

Move body reading into the loop; final assertions after the loop are kept.

For test cases that legitimately expect HTTP/1.1 the check is a no-op on the happy path.

Generated-by: AI (analysis + patch), reviewed by mko@redhat.com

/cc @frobware

Summary by CodeRabbit

Tests

Improved HTTP/2 router conformance checks to verify both the response protocol and response body during each poll.

Added safer response handling by reading and closing response bodies immediately, including on failed attempts.

Retries now more reliably refresh the network path when the expected router or shard is not reached.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

mkowalski · 2026-07-06T13:32:39Z

/jira refresh

openshift-ci-robot · 2026-07-06T13:32:46Z

@mkowalski: This pull request references Jira Issue OCPBUGS-97822, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (5.0.0) matches configured target version for branch (5.0.0)
bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @melvinjoseph86

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-merge-bot · 2026-07-06T13:40:08Z

Scheduling required tests:
/test e2e-aws-csi
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi

openshift-ci · 2026-07-06T20:05:07Z

@mkowalski: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-vsphere-ovn-upi	`7b6974b`	link	true	`/test e2e-vsphere-ovn-upi`
ci/prow/e2e-aws-ovn-serial-1of2	`7b6974b`	link	true	`/test e2e-aws-ovn-serial-1of2`
ci/prow/e2e-aws-csi	`7b6974b`	link	true	`/test e2e-aws-csi`
ci/prow/e2e-aws-ovn-fips	`7b6974b`	link	true	`/test e2e-aws-ovn-fips`
ci/prow/e2e-aws-ovn-microshift-serial	`7b6974b`	link	true	`/test e2e-aws-ovn-microshift-serial`
ci/prow/e2e-aws-ovn-serial-2of2	`7b6974b`	link	true	`/test e2e-aws-ovn-serial-2of2`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jul 6, 2026

openshift-ci Bot requested a review from frobware July 6, 2026 13:17

openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 6, 2026

mkowalski changed the title ~~NO-ISSUE: test/router: retry http2 test when response comes from wrong router~~ OCPBUGS-97822: test/router: retry http2 test when response comes from wrong router Jul 6, 2026

coderabbitai Bot approved these changes Jul 6, 2026

View reviewed changes

openshift-ci Bot added the ready-for-human-review Indicates a PR has been reviewed by automated tools and is ready for human review label Jul 6, 2026

openshift-ci-robot added the jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. label Jul 6, 2026

openshift-ci-robot added jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. and removed jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Jul 6, 2026

openshift-ci Bot requested a review from melvinjoseph86 July 6, 2026 13:38

Uh oh!

Conversation

mkowalski commented Jul 6, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Evidence

Fix

Summary by CodeRabbit

Uh oh!

openshift-merge-bot Bot commented Jul 6, 2026

Uh oh!

openshift-ci-robot commented Jul 6, 2026

Problem

Evidence

Fix

Uh oh!

coderabbitai Bot commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Uh oh!

openshift-ci Bot commented Jul 6, 2026

Uh oh!

mkowalski commented Jul 6, 2026

Uh oh!

openshift-ci-robot commented Jul 6, 2026

Problem

Evidence

Fix

Summary by CodeRabbit

Uh oh!

mkowalski commented Jul 6, 2026

Uh oh!

openshift-ci-robot commented Jul 6, 2026

Uh oh!

openshift-merge-bot Bot commented Jul 6, 2026

Uh oh!

openshift-ci Bot commented Jul 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mkowalski commented Jul 6, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Jul 6, 2026 •

edited

Loading