fix(slack): Handle Slack API rate limits for bookmarks.add

[sentry]

This PR addresses the issue where Slack API bookmarks.add calls were failing due to too_many_requests errors, particularly during rapid API interactions (e.g., incident creation).

Changes:

• Configured the slack_sdk.WebClient to use RateLimitErrorRetryHandler with a max_retry_count of 1. This enables automatic retries with exponential backoff for all Slack API calls when a 429 (Too Many Requests) response is received.
• Added specific retry logic within the add_bookmark method to handle too_many_requests errors. This logic reads the Retry-After header from Slack's response, pauses for the specified duration, and retries the bookmarks_add call once. This mirrors the existing retry pattern for not_in_channel errors.

Reasoning:
The previous implementation lacked a mechanism to handle Slack API rate limits, causing bookmark creation to fail when multiple Slack operations occurred in quick succession. These changes ensure that bookmarks.add operations are resilient to temporary rate limiting, improving the reliability of incident setup.

Fixes FIRETOWER-BACKEND-7D

_{Comment @sentry <feedback> on this PR to have Autofix iterate on the changes.}

Co-authored-by: richard.gibert@sentry.io richard.gibert@sentry.io

[sentry]

Bug: The manual retry logic in add_bookmark combined with the SDK's RateLimitErrorRetryHandler creates duplicate retries, causing unintended, extended blocking on rate-limiting errors.
_{Severity: MEDIUM}

Suggested Fix

Remove the manual retry logic within the add_bookmark function. Rely solely on the SDK's RateLimitErrorRetryHandler for handling rate-limiting errors. Consider increasing the max_retry_count on the handler if more than one automatic retry is desired.

Prompt for AI Agent

Review the code at the location below. A potential bug has been identified by an AI
agent. Verify if this is a real issue. If it is, propose a fix; if not, explain why it's
not valid.

Location: src/firetower/integrations/services/slack.py#L393-L408

Potential issue: The `add_bookmark` function implements manual retry logic for
`"ratelimited"` errors, but the Slack client is also configured with a
`RateLimitErrorRetryHandler`. When a rate limit error (429) occurs, the SDK handler
retries once. If that also fails, the manual logic catches the error and retries by
calling `self.client.bookmarks_add()` again. This new call is treated as a fresh attempt
by the SDK, so its handler will retry again if it encounters another 429. This
interaction results in compounded retries and can block the synchronous function for up
to three times the `Retry-After` duration, impacting performance during incident
creation.

Also affects:

• src/firetower/integrations/services/slack.py:68~69