Explore more
...
Scenarios
Incomplete executions

Automatic retry of incomplete executions

4min

 checks the origin of every incomplete execution when it's created.  automatically retries incomplete executions that have been created because of:

  • RateLimitError
  • ConnectionError
  • ModuleTimeoutError

and incomplete executions that were created with the Break error handler with the automatic run completion enabled.

 takes the following steps when  retries incomplete executions:

  1.  schedules the incomplete execution retries.
  2.  retries the incomplete execution.
  3. Based on the incomplete execution retry result  schedules another attempt or marks the incomplete execution resolved.

Automatic retry scheduling

 schedules automatic retries with the incomplete executions backoff schedule. The backoff schedule prevents the situation when you would get the same error multiple times in a row.

For example, when you get the ConnectionError because the app is unavailable, it might take some time until it's back.  spaces out the retry attempts to get a successful retry even at a later time after the original error.

Error type

Retry schedule

  • RateLimitError
  • ConnectionError
  • ModuleTimeoutError
  1. 1 minute (1 minute after the original  run).
  2. 10 minutes (11 minutes after the original  run).
  3. 10 minutes (21 minutes after the original  run).
  4. 30 minutes (51 minutes after the original  run).
  5. 30 minutes (1 hour 21 minutes after the original  run).
  6. 30 minutes (1 hour 51 minutes after the original  run).
  7. 3 hours (4 hours 51 minutes after the original  run).
  8. 3 hours (7 hours 51 minutes after the original  run).

Error handled by the Break error handler

If you enable automatic  run completion in the error handler settings. The default is:

  • maximum number of retry attempts: 3
  • retry delay: 15 minutes

You can customize the defaults in the error handler settings.

Other error types usually require changes in the incomplete execution and manual resolving.  doesn't retry these error types automatically by default.

Automatic retry processing

After  schedules the retries,  runs the  again, starting with the module that caused the error.

For each , there is a limit of 3 incomplete execution retries running in parallel. If there are more incomplete executions scheduled from the same ,  retries them in batches of 3 after the previous batch finishes.

In addition, the retry doesn't start when the original  is running already.

The 3 parallel retries limit applies to retries from the same . When  retries incomplete executions from multiple , then each of them has their own limit.

This limitation is to prevent your  from getting follow-up rate limit errors if you are retrying a lot of incomplete executions at the same time.

For example:

You have a  that runs for 10 minutes every hour. There was a disruption of a third party service for 5 hours.

  • The  now has 5 incomplete executions scheduled for automatic retry.
  •  first waits until the original  finishes if it's running already. This takes 10 minutes if the  started just now.
  • After the  finishes,  retries the first 3 incomplete executions. This takes an additional 10 minutes (20 in total).
  •  retries the remaining 2 incomplete executions after the previous batch finishes. This takes another 10 minutes (30 in total).
  • After another 30 minutes (1 hour in total),  starts the  again according to the  schedule.

Automatic retry result

If a retry attempt succeeds,  marks the incomplete execution as Resolved and stops retrying.

If all of the retry attempts fail,  marks the incomplete execution as Unresolved. You can then retry the incomplete execution when the app that caused the error is available again or you can resolve the incomplete execution manually.