#784 - Support Retry-After in FetcherBolt

[rzo1]

Honour the Retry-After HTTP response header by delaying the next fetch from the affected internal queue. The header value is parsed as either a number of seconds or an HTTP date. A new fetcher.max.retry.after config caps the honoured delay (-1, the default, honours it as-is).

For all changes

• Is there a issue associated with this PR? Is it referenced in the commit message?

• Does your PR title start with #XXXX where XXXX is the issue number you are trying to resolve?

• Has your PR been rebased against the latest commit within the target branch (typically main)?

• Is your initial contribution a single, squashed commit?

• Is the code properly formatted with mvn git-code-format:format-code -Dgcf.globPattern="**/*" -Dskip.format.code=false?

For code changes

• Have you ensured that the full suite of tests is executed via mvn clean verify?
• Have you written or updated unit tests to verify your changes?
• If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
• If applicable, have you updated the LICENSE file, including the main LICENSE file?
• If applicable, have you updated the NOTICE file, including the main NOTICE file?

[jnioche]

thanks @rzo1
my concern is that this would cause unnecessary tuple timeouts. If we come across a retry after, we should probably purge the queue instead.
A more robust approach would be to implement it via the host stream (see #867) and have a host aware spout implementation. I had made a start in the branch 851

[rzo1]

I had made a start in the branch 851

Contains SOLR changes only, so most likely gone.

--

I agree, that purging the queue for the given host might be beneficial. Let me later look into that in more depth again.

[jnioche]

I had made a start in the branch 851

Contains SOLR changes only, so most likely gone.

was actually in branch 990, sorry

[rzo1]

Agreed, the in-queue hold is the fragile part. A long Retry-After (and the default is uncapped) would park the queue's siblings past topology.message.timeout.secs and trigger replays.

For this PR, I would take the purge route. On a Retry-After, re-emit the affected queue's pending items rather than holding them, mirroring the existing crawl-delay-too-long path at FetcherBolt.java:682 so the frontier reschedules them. One thing I'd like your take on: should the re-emitted URLs go out as Status.ERROR (reuses the existing path, but carries error/retry semantics), or should we set an explicit future nextFetchDate so the scheduler honors the exact back-off?

I'd treat the host stream / host-aware spout design (#867, your branch 990) as the proper long-term home for this, happy to follow up there once we settle the short-term behaviour. Does that split sound right to you?