Configurations¶
This is the entire code example for the configurations tutorial.
We will see each config in detail with his own section in the next parts of this tutorial.
from fastapi_task_manager import Config
config = Config(
# --------- App config ---------
redis_key_prefix=__name__,
concurrent_tasks=2,
statistics_redis_expiration=432_000,
statistics_history_runs=30,
# --------- Redis config ---------
redis_host="localhost",
redis_port=6379,
redis_password=None,
redis_db=0,
# --------- Runner config ---------
poll_interval=0.1,
worker_service_name="fastapi-task-manager",
# --------- Streams config ---------
stream_block_ms=1000,
# --------- Leader election config ---------
leader_heartbeat_interval=3.0,
leader_retry_interval=5.0,
# --------- Reconciliation config ---------
reconciliation_interval=30,
# --------- Retry / backoff config ---------
retry_backoff=1.0,
retry_backoff_max=60.0,
retry_backoff_multiplier=2.0,
# --------- Running heartbeat config ---------
running_heartbeat_interval=3.0,
)
App Configuration¶
Redis Key Prefix¶
# Code above omitted 👆
redis_key_prefix=__name__,
# Code below omitted 👇
👀 Full file preview
from fastapi_task_manager import Config
config = Config(
# --------- App config ---------
redis_key_prefix=__name__,
concurrent_tasks=2,
statistics_redis_expiration=432_000,
statistics_history_runs=30,
# --------- Redis config ---------
redis_host="localhost",
redis_port=6379,
redis_password=None,
redis_db=0,
# --------- Runner config ---------
poll_interval=0.1,
worker_service_name="fastapi-task-manager",
# --------- Streams config ---------
stream_block_ms=1000,
# --------- Leader election config ---------
leader_heartbeat_interval=3.0,
leader_retry_interval=5.0,
# --------- Reconciliation config ---------
reconciliation_interval=30,
# --------- Retry / backoff config ---------
retry_backoff=1.0,
retry_backoff_max=60.0,
retry_backoff_multiplier=2.0,
# --------- Running heartbeat config ---------
running_heartbeat_interval=3.0,
)
This configuration is used as custom prefix for all the keys used by the package in redis. Pay attention to make it unique if you are using the same redis instance for multiple applications on the same redis db.
Default value is __name__.
Concurrent Tasks¶
# Code above omitted 👆
concurrent_tasks=2,
# Code below omitted 👇
👀 Full file preview
from fastapi_task_manager import Config
config = Config(
# --------- App config ---------
redis_key_prefix=__name__,
concurrent_tasks=2,
statistics_redis_expiration=432_000,
statistics_history_runs=30,
# --------- Redis config ---------
redis_host="localhost",
redis_port=6379,
redis_password=None,
redis_db=0,
# --------- Runner config ---------
poll_interval=0.1,
worker_service_name="fastapi-task-manager",
# --------- Streams config ---------
stream_block_ms=1000,
# --------- Leader election config ---------
leader_heartbeat_interval=3.0,
leader_retry_interval=5.0,
# --------- Reconciliation config ---------
reconciliation_interval=30,
# --------- Retry / backoff config ---------
retry_backoff=1.0,
retry_backoff_max=60.0,
retry_backoff_multiplier=2.0,
# --------- Running heartbeat config ---------
running_heartbeat_interval=3.0,
)
This configuration is used to set the maximum number of concurrent tasks that can be executed by each worker.
Default value is 2.
Statistics Redis Expiration¶
# Code above omitted 👆
statistics_redis_expiration=432_000,
# Code below omitted 👇
👀 Full file preview
from fastapi_task_manager import Config
config = Config(
# --------- App config ---------
redis_key_prefix=__name__,
concurrent_tasks=2,
statistics_redis_expiration=432_000,
statistics_history_runs=30,
# --------- Redis config ---------
redis_host="localhost",
redis_port=6379,
redis_password=None,
redis_db=0,
# --------- Runner config ---------
poll_interval=0.1,
worker_service_name="fastapi-task-manager",
# --------- Streams config ---------
stream_block_ms=1000,
# --------- Leader election config ---------
leader_heartbeat_interval=3.0,
leader_retry_interval=5.0,
# --------- Reconciliation config ---------
reconciliation_interval=30,
# --------- Retry / backoff config ---------
retry_backoff=1.0,
retry_backoff_max=60.0,
retry_backoff_multiplier=2.0,
# --------- Running heartbeat config ---------
running_heartbeat_interval=3.0,
)
This configuration sets the TTL (in seconds) for statistics-related keys in Redis. After this time, the statistics data will expire and be automatically removed by Redis.
Default value is 432000 (5 days).
Statistics History Runs¶
# Code above omitted 👆
statistics_history_runs=30,
# Code below omitted 👇
👀 Full file preview
from fastapi_task_manager import Config
config = Config(
# --------- App config ---------
redis_key_prefix=__name__,
concurrent_tasks=2,
statistics_redis_expiration=432_000,
statistics_history_runs=30,
# --------- Redis config ---------
redis_host="localhost",
redis_port=6379,
redis_password=None,
redis_db=0,
# --------- Runner config ---------
poll_interval=0.1,
worker_service_name="fastapi-task-manager",
# --------- Streams config ---------
stream_block_ms=1000,
# --------- Leader election config ---------
leader_heartbeat_interval=3.0,
leader_retry_interval=5.0,
# --------- Reconciliation config ---------
reconciliation_interval=30,
# --------- Retry / backoff config ---------
retry_backoff=1.0,
retry_backoff_max=60.0,
retry_backoff_multiplier=2.0,
# --------- Running heartbeat config ---------
running_heartbeat_interval=3.0,
)
This configuration sets how many historical execution entries are kept per task in the statistics Redis Stream. Each entry contains both the run timestamp and duration. Older entries are automatically trimmed via XADD MAXLEN.
Default value is 500.
Redis Configuration¶
Redis Host¶
# Code above omitted 👆
redis_host="localhost",
# Code below omitted 👇
👀 Full file preview
from fastapi_task_manager import Config
config = Config(
# --------- App config ---------
redis_key_prefix=__name__,
concurrent_tasks=2,
statistics_redis_expiration=432_000,
statistics_history_runs=30,
# --------- Redis config ---------
redis_host="localhost",
redis_port=6379,
redis_password=None,
redis_db=0,
# --------- Runner config ---------
poll_interval=0.1,
worker_service_name="fastapi-task-manager",
# --------- Streams config ---------
stream_block_ms=1000,
# --------- Leader election config ---------
leader_heartbeat_interval=3.0,
leader_retry_interval=5.0,
# --------- Reconciliation config ---------
reconciliation_interval=30,
# --------- Retry / backoff config ---------
retry_backoff=1.0,
retry_backoff_max=60.0,
retry_backoff_multiplier=2.0,
# --------- Running heartbeat config ---------
running_heartbeat_interval=3.0,
)
The hostname or IP address of the Redis server. This is the only required configuration parameter with no default value.
Redis Port¶
# Code above omitted 👆
redis_port=6379,
# Code below omitted 👇
👀 Full file preview
from fastapi_task_manager import Config
config = Config(
# --------- App config ---------
redis_key_prefix=__name__,
concurrent_tasks=2,
statistics_redis_expiration=432_000,
statistics_history_runs=30,
# --------- Redis config ---------
redis_host="localhost",
redis_port=6379,
redis_password=None,
redis_db=0,
# --------- Runner config ---------
poll_interval=0.1,
worker_service_name="fastapi-task-manager",
# --------- Streams config ---------
stream_block_ms=1000,
# --------- Leader election config ---------
leader_heartbeat_interval=3.0,
leader_retry_interval=5.0,
# --------- Reconciliation config ---------
reconciliation_interval=30,
# --------- Retry / backoff config ---------
retry_backoff=1.0,
retry_backoff_max=60.0,
retry_backoff_multiplier=2.0,
# --------- Running heartbeat config ---------
running_heartbeat_interval=3.0,
)
The port number of the Redis server.
Default value is 6379.
Redis Password¶
# Code above omitted 👆
redis_password=None,
# Code below omitted 👇
👀 Full file preview
from fastapi_task_manager import Config
config = Config(
# --------- App config ---------
redis_key_prefix=__name__,
concurrent_tasks=2,
statistics_redis_expiration=432_000,
statistics_history_runs=30,
# --------- Redis config ---------
redis_host="localhost",
redis_port=6379,
redis_password=None,
redis_db=0,
# --------- Runner config ---------
poll_interval=0.1,
worker_service_name="fastapi-task-manager",
# --------- Streams config ---------
stream_block_ms=1000,
# --------- Leader election config ---------
leader_heartbeat_interval=3.0,
leader_retry_interval=5.0,
# --------- Reconciliation config ---------
reconciliation_interval=30,
# --------- Retry / backoff config ---------
retry_backoff=1.0,
retry_backoff_max=60.0,
retry_backoff_multiplier=2.0,
# --------- Running heartbeat config ---------
running_heartbeat_interval=3.0,
)
The password for authenticating with the Redis server. Set to None if your Redis instance does not require authentication.
Default value is None.
Redis DB¶
# Code above omitted 👆
redis_db=0,
# Code below omitted 👇
👀 Full file preview
from fastapi_task_manager import Config
config = Config(
# --------- App config ---------
redis_key_prefix=__name__,
concurrent_tasks=2,
statistics_redis_expiration=432_000,
statistics_history_runs=30,
# --------- Redis config ---------
redis_host="localhost",
redis_port=6379,
redis_password=None,
redis_db=0,
# --------- Runner config ---------
poll_interval=0.1,
worker_service_name="fastapi-task-manager",
# --------- Streams config ---------
stream_block_ms=1000,
# --------- Leader election config ---------
leader_heartbeat_interval=3.0,
leader_retry_interval=5.0,
# --------- Reconciliation config ---------
reconciliation_interval=30,
# --------- Retry / backoff config ---------
retry_backoff=1.0,
retry_backoff_max=60.0,
retry_backoff_multiplier=2.0,
# --------- Running heartbeat config ---------
running_heartbeat_interval=3.0,
)
The Redis database number to use. Redis supports multiple databases (0-15 by default).
Default value is 0.
Runner Configuration¶
These settings control the core task scheduling loop.
Poll Interval¶
# Code above omitted 👆
poll_interval=0.1,
# Code below omitted 👇
👀 Full file preview
from fastapi_task_manager import Config
config = Config(
# --------- App config ---------
redis_key_prefix=__name__,
concurrent_tasks=2,
statistics_redis_expiration=432_000,
statistics_history_runs=30,
# --------- Redis config ---------
redis_host="localhost",
redis_port=6379,
redis_password=None,
redis_db=0,
# --------- Runner config ---------
poll_interval=0.1,
worker_service_name="fastapi-task-manager",
# --------- Streams config ---------
stream_block_ms=1000,
# --------- Leader election config ---------
leader_heartbeat_interval=3.0,
leader_retry_interval=5.0,
# --------- Reconciliation config ---------
reconciliation_interval=30,
# --------- Retry / backoff config ---------
retry_backoff=1.0,
retry_backoff_max=60.0,
retry_backoff_multiplier=2.0,
# --------- Running heartbeat config ---------
running_heartbeat_interval=3.0,
)
The interval (in seconds) between coordinator scheduling cycles. Lower values mean tasks are picked up faster but increase Redis load.
Default value is 0.1.
Worker Service Name¶
# Code above omitted 👆
worker_service_name="fastapi-task-manager",
# Code below omitted 👇
👀 Full file preview
from fastapi_task_manager import Config
config = Config(
# --------- App config ---------
redis_key_prefix=__name__,
concurrent_tasks=2,
statistics_redis_expiration=432_000,
statistics_history_runs=30,
# --------- Redis config ---------
redis_host="localhost",
redis_port=6379,
redis_password=None,
redis_db=0,
# --------- Runner config ---------
poll_interval=0.1,
worker_service_name="fastapi-task-manager",
# --------- Streams config ---------
stream_block_ms=1000,
# --------- Leader election config ---------
leader_heartbeat_interval=3.0,
leader_retry_interval=5.0,
# --------- Reconciliation config ---------
reconciliation_interval=30,
# --------- Retry / backoff config ---------
retry_backoff=1.0,
retry_backoff_max=60.0,
retry_backoff_multiplier=2.0,
# --------- Running heartbeat config ---------
running_heartbeat_interval=3.0,
)
The service name used for worker identification. This appears in health check responses and logs to help identify which service a worker belongs to.
Default value is "fastapi-task-manager".
Streams Configuration¶
These settings control the Redis Streams-based task distribution system.
Stream Block Timeout¶
# Code above omitted 👆
stream_block_ms=1000,
# Code below omitted 👇
👀 Full file preview
from fastapi_task_manager import Config
config = Config(
# --------- App config ---------
redis_key_prefix=__name__,
concurrent_tasks=2,
statistics_redis_expiration=432_000,
statistics_history_runs=30,
# --------- Redis config ---------
redis_host="localhost",
redis_port=6379,
redis_password=None,
redis_db=0,
# --------- Runner config ---------
poll_interval=0.1,
worker_service_name="fastapi-task-manager",
# --------- Streams config ---------
stream_block_ms=1000,
# --------- Leader election config ---------
leader_heartbeat_interval=3.0,
leader_retry_interval=5.0,
# --------- Reconciliation config ---------
reconciliation_interval=30,
# --------- Retry / backoff config ---------
retry_backoff=1.0,
retry_backoff_max=60.0,
retry_backoff_multiplier=2.0,
# --------- Running heartbeat config ---------
running_heartbeat_interval=3.0,
)
The block timeout (in milliseconds) for XREADGROUP when consumers wait for new messages. Higher values reduce Redis round-trips but increase shutdown latency.
Default value is 1000.
Leader Election Configuration¶
FastAPI Task Manager uses distributed leader election via Redis to ensure that only one instance schedules tasks at a time, while all instances can execute them.
Leader Heartbeat Interval¶
# Code above omitted 👆
leader_heartbeat_interval=3.0,
# Code below omitted 👇
👀 Full file preview
from fastapi_task_manager import Config
config = Config(
# --------- App config ---------
redis_key_prefix=__name__,
concurrent_tasks=2,
statistics_redis_expiration=432_000,
statistics_history_runs=30,
# --------- Redis config ---------
redis_host="localhost",
redis_port=6379,
redis_password=None,
redis_db=0,
# --------- Runner config ---------
poll_interval=0.1,
worker_service_name="fastapi-task-manager",
# --------- Streams config ---------
stream_block_ms=1000,
# --------- Leader election config ---------
leader_heartbeat_interval=3.0,
leader_retry_interval=5.0,
# --------- Reconciliation config ---------
reconciliation_interval=30,
# --------- Retry / backoff config ---------
retry_backoff=1.0,
retry_backoff_max=60.0,
retry_backoff_multiplier=2.0,
# --------- Running heartbeat config ---------
running_heartbeat_interval=3.0,
)
The interval (in seconds) between leader lock renewals. The leader periodically renews its lock to signal it is still alive.
Default value is 3.0.
Leader Retry Interval¶
# Code above omitted 👆
leader_retry_interval=5.0,
# Code below omitted 👇
👀 Full file preview
from fastapi_task_manager import Config
config = Config(
# --------- App config ---------
redis_key_prefix=__name__,
concurrent_tasks=2,
statistics_redis_expiration=432_000,
statistics_history_runs=30,
# --------- Redis config ---------
redis_host="localhost",
redis_port=6379,
redis_password=None,
redis_db=0,
# --------- Runner config ---------
poll_interval=0.1,
worker_service_name="fastapi-task-manager",
# --------- Streams config ---------
stream_block_ms=1000,
# --------- Leader election config ---------
leader_heartbeat_interval=3.0,
leader_retry_interval=5.0,
# --------- Reconciliation config ---------
reconciliation_interval=30,
# --------- Retry / backoff config ---------
retry_backoff=1.0,
retry_backoff_max=60.0,
retry_backoff_multiplier=2.0,
# --------- Running heartbeat config ---------
running_heartbeat_interval=3.0,
)
The interval (in seconds) between leadership acquisition attempts for follower instances. When a worker is not the leader, it tries to acquire leadership at this interval.
Default value is 5.0.
Reconciliation Configuration¶
The reconciler detects and recovers stale or failed tasks. It runs only on the leader instance.
Reconciliation Interval¶
# Code above omitted 👆
reconciliation_interval=30,
# Code below omitted 👇
👀 Full file preview
from fastapi_task_manager import Config
config = Config(
# --------- App config ---------
redis_key_prefix=__name__,
concurrent_tasks=2,
statistics_redis_expiration=432_000,
statistics_history_runs=30,
# --------- Redis config ---------
redis_host="localhost",
redis_port=6379,
redis_password=None,
redis_db=0,
# --------- Runner config ---------
poll_interval=0.1,
worker_service_name="fastapi-task-manager",
# --------- Streams config ---------
stream_block_ms=1000,
# --------- Leader election config ---------
leader_heartbeat_interval=3.0,
leader_retry_interval=5.0,
# --------- Reconciliation config ---------
reconciliation_interval=30,
# --------- Retry / backoff config ---------
retry_backoff=1.0,
retry_backoff_max=60.0,
retry_backoff_multiplier=2.0,
# --------- Running heartbeat config ---------
running_heartbeat_interval=3.0,
)
The interval (in seconds) between reconciliation checks. The reconciler scans for overdue or stuck tasks at this interval.
Default value is 30.
Retry / Backoff Configuration¶
When a task fails, FastAPI Task Manager applies exponential backoff to delay re-execution. This prevents rapid failure loops and gives external dependencies time to recover.
Retry Backoff¶
# Code above omitted 👆
retry_backoff=1.0,
# Code below omitted 👇
👀 Full file preview
from fastapi_task_manager import Config
config = Config(
# --------- App config ---------
redis_key_prefix=__name__,
concurrent_tasks=2,
statistics_redis_expiration=432_000,
statistics_history_runs=30,
# --------- Redis config ---------
redis_host="localhost",
redis_port=6379,
redis_password=None,
redis_db=0,
# --------- Runner config ---------
poll_interval=0.1,
worker_service_name="fastapi-task-manager",
# --------- Streams config ---------
stream_block_ms=1000,
# --------- Leader election config ---------
leader_heartbeat_interval=3.0,
leader_retry_interval=5.0,
# --------- Reconciliation config ---------
reconciliation_interval=30,
# --------- Retry / backoff config ---------
retry_backoff=1.0,
retry_backoff_max=60.0,
retry_backoff_multiplier=2.0,
# --------- Running heartbeat config ---------
running_heartbeat_interval=3.0,
)
The initial backoff delay (in seconds) after a task failure. The first retry will be delayed by this amount.
Default value is 1.0.
Per-task override
This setting can be overridden on individual tasks via the retry_backoff parameter on @task_group.add_task().
Retry Backoff Max¶
# Code above omitted 👆
retry_backoff_max=60.0,
# Code below omitted 👇
👀 Full file preview
from fastapi_task_manager import Config
config = Config(
# --------- App config ---------
redis_key_prefix=__name__,
concurrent_tasks=2,
statistics_redis_expiration=432_000,
statistics_history_runs=30,
# --------- Redis config ---------
redis_host="localhost",
redis_port=6379,
redis_password=None,
redis_db=0,
# --------- Runner config ---------
poll_interval=0.1,
worker_service_name="fastapi-task-manager",
# --------- Streams config ---------
stream_block_ms=1000,
# --------- Leader election config ---------
leader_heartbeat_interval=3.0,
leader_retry_interval=5.0,
# --------- Reconciliation config ---------
reconciliation_interval=30,
# --------- Retry / backoff config ---------
retry_backoff=1.0,
retry_backoff_max=60.0,
retry_backoff_multiplier=2.0,
# --------- Running heartbeat config ---------
running_heartbeat_interval=3.0,
)
The maximum backoff delay (in seconds). The delay will never exceed this value, regardless of how many consecutive failures occur.
Default value is 60.0.
Per-task override
This setting can be overridden on individual tasks via the retry_backoff_max parameter on @task_group.add_task().
Retry Backoff Multiplier¶
# Code above omitted 👆
retry_backoff_multiplier=2.0,
# Code below omitted 👇
👀 Full file preview
from fastapi_task_manager import Config
config = Config(
# --------- App config ---------
redis_key_prefix=__name__,
concurrent_tasks=2,
statistics_redis_expiration=432_000,
statistics_history_runs=30,
# --------- Redis config ---------
redis_host="localhost",
redis_port=6379,
redis_password=None,
redis_db=0,
# --------- Runner config ---------
poll_interval=0.1,
worker_service_name="fastapi-task-manager",
# --------- Streams config ---------
stream_block_ms=1000,
# --------- Leader election config ---------
leader_heartbeat_interval=3.0,
leader_retry_interval=5.0,
# --------- Reconciliation config ---------
reconciliation_interval=30,
# --------- Retry / backoff config ---------
retry_backoff=1.0,
retry_backoff_max=60.0,
retry_backoff_multiplier=2.0,
# --------- Running heartbeat config ---------
running_heartbeat_interval=3.0,
)
The multiplier applied to the current delay after each consecutive failure. For example, with default settings: 1s, 2s, 4s, 8s, 16s, 32s, 60s (capped).
Default value is 2.0.
Running Heartbeat Configuration¶
While a task is executing, the worker periodically renews a heartbeat key in Redis. If a worker crashes, the key expires, signaling that the task is no longer being executed.
Running Heartbeat Interval¶
# Code above omitted 👆
running_heartbeat_interval=3.0,
# Code below omitted 👇
👀 Full file preview
from fastapi_task_manager import Config
config = Config(
# --------- App config ---------
redis_key_prefix=__name__,
concurrent_tasks=2,
statistics_redis_expiration=432_000,
statistics_history_runs=30,
# --------- Redis config ---------
redis_host="localhost",
redis_port=6379,
redis_password=None,
redis_db=0,
# --------- Runner config ---------
poll_interval=0.1,
worker_service_name="fastapi-task-manager",
# --------- Streams config ---------
stream_block_ms=1000,
# --------- Leader election config ---------
leader_heartbeat_interval=3.0,
leader_retry_interval=5.0,
# --------- Reconciliation config ---------
reconciliation_interval=30,
# --------- Retry / backoff config ---------
retry_backoff=1.0,
retry_backoff_max=60.0,
retry_backoff_multiplier=2.0,
# --------- Running heartbeat config ---------
running_heartbeat_interval=3.0,
)
The interval (in seconds) between heartbeat renewals while a task is executing. The worker renews the running key at this interval.
Default value is 3.0.