Skip to main content

Troubleshoot Workflow and Activity execution failures

View Markdown

This guide covers failures that happen while your Workflow and Activity code is running on a Worker: replay mismatches, oversized responses, unhandled exceptions, and Local Activities that outrun the Workflow Task heartbeat timeout. It applies to Workers connected to Temporal Cloud and to a self-hosted Temporal Service.

For alert thresholds and for durations, see Worker alerting. For metric definitions, see the Temporal SDK metrics reference.

temporal_workflow_task_execution_failed carries a failure_reason tag along with namespace, task_queue, and workflow_type. The reason decides what the Temporal Service does next, and the difference is worth knowing: two of the three retry forever, one terminates the Execution on the spot. Alert on each failure_reason separately.

Non-determinism error​

Metric: temporal_workflow_task_execution_failed with failure_reason=NonDeterminismError

Replay produced a different sequence of commands than the one recorded in Event History. The Worker noticed that the Workflow code it is running no longer matches what the Execution has already done.

Why it matters. Affected Executions stop making progress. The Temporal Service retries the Workflow Task over and over, loading up your Workflow Workers, and by default those Executions sit in Running status the whole time. This will not clear up on its own.

Triage.

  1. Identify the affected Workflow Executions. This metric does not carry a Workflow Id. Worker logs record the error with the Workflow Id and Run Id. In the Temporal UI you can also find affected Executions by querying the TemporalReportedProblems Search Attribute, which the Temporal Service sets on Executions experiencing repeated Workflow Task failures.
  2. Read the error. The WorkflowTaskFailed event in an affected Execution's Event History contains the message identifying exactly where replay diverged and which command was expected versus produced. This is the most direct signal for root cause.
  3. Determine whether this is a code change or a deploy artifact. Common causes:
    • A code change added, removed, or reordered commands such as Activity scheduling, Timers, Signals, or Child Workflows, without a versioning guard. Executions that built their History under the old code fail on the new code.
    • A rolling restart with old and new Worker versions briefly running together. Some Executions fail transiently and recover once the rollout completes. If your deploys routinely trigger this alert, lengthen its for duration past how long a rollout takes.
    • Changed Activity or Timer parameters in existing Workflow code without versioning.
  4. Roll back if it isn't clearing. If the errors started after a deploy and aren't going away, roll the Worker back. Affected Executions pick up again on their next Workflow Task retry once compatible code is running. Then add a proper versioning guard before you redeploy. See Versioning Workflows and Worker Versioning.
  5. Watch Worker pressure. All those retries add up. Cross-check Worker Task slots exhausted for worker_type=WorkflowWorker and Workflow Task execution latency high. Enough retry volume will saturate your capacity and start hurting healthy Executions on the same Task Queue.

gRPC message too large​

Metric: temporal_workflow_task_execution_failed with failure_reason=GrpcMessageTooLarge

The Workflow Task response was bigger than the gRPC message size limit. The Worker tried RespondWorkflowTaskCompleted and something rejected it: the gRPC library on the SDK side, a proxy or load balancer in the path, or the gRPC library on the Temporal Service side when it went to receive.

The Temporal Service never saw that original request, so the SDK follows up with RespondWorkflowTaskFailed and cause WORKFLOW_TASK_FAILED_CAUSE_GRPC_MESSAGE_TOO_LARGE. Replay would build the same oversized response every time, so the Service terminates the Workflow Execution instead of retrying it.

For the payload size limits themselves, and for the Activity Task version of this (which retries instead of terminating), see Troubleshoot the BlobSizeLimitError.

Why it matters. Affected Executions end immediately and permanently, with TERMINATED status and no retry. Whatever work was in progress is gone, and someone has to restart them by hand.

This is the one failure_reason on this page that kills Executions instead of retrying them, so give it a short for duration.

Triage.

  1. Identify the affected Executions. This metric does not carry a Workflow Id. Check Worker logs for Workflow Ids and Run Ids, then confirm the cause from the WorkflowTaskFailed and WorkflowExecutionTerminated events in Event History.
  2. Find what is oversized. The fix depends entirely on which part of the response is too large:
    • Oversized Activity inputs or outputs. Move the payload out of band: put it in blob storage and pass a reference through Event History instead. See External Storage for the pattern.
    • Accumulated Signals or Updates. A large number buffered into a single Workflow Task. Rate-limit senders or batch Signals.
    • Too many commands in one response. A Workflow scheduling a very large fan-out of Activities or Child Workflows in a single step. Break the fan-out into smaller batches across multiple Workflow Tasks.
  3. Fix and deploy before restarting anything. Terminated Executions do not retry. Restarting them before the cause is fixed means hitting the same limit and being terminated again. Once the corrected Worker is deployed and verified, restart the affected Executions from the Temporal UI or CLI.
Self-hosted Temporal Service

Check the Workflow terminate rate on your server dashboard. A spike alongside this metric confirms Executions are being terminated in bulk.

Workflow Task execution failures elevated​

Metric: temporal_workflow_task_execution_failed with failure_reason=WorkflowError

Workflow Tasks are failing steadily from unhandled exceptions and panics in Workflow code that the SDK catches and reports. WorkflowError is the catch-all reason. It covers thread pool exhaustion, unhandled exceptions thrown inside the Workflow function, and Data Converter errors.

Why it matters. The Temporal Service retries the Workflow Task. If the error is deterministic and shows up on every replay, the Execution is stuck retrying forever, burning Worker capacity and never getting healthy.

At high rates the retry pressure fills up your Workflow Worker slots and starts affecting healthy Executions on the same Task Queue. Unlike GrpcMessageTooLarge, the Temporal Service won't terminate the Execution for you, so this gets worse the longer you leave it.

Triage.

  1. Identify the affected Executions. This metric does not carry a Workflow Id. Worker logs carry the Workflow Id, Run Id, and full stack trace. The WorkflowTaskFailed event in Event History carries the error message and type. The workflow_type tag on the metric narrows which Workflow is failing.
  2. Determine which failure mode this is. WorkflowError covers several:
    • Thread pool exhaustion (Java SDK). A RejectedExecutionException from a saturated Workflow thread pool. setMaxWorkflowThreadCount on WorkerFactoryOptions is too low for the number of concurrent Executions, so new Workflow Tasks get rejected before they run. Raise the thread count, and think about whether the Worker pool needs to scale out too.
    • Unhandled exception in Workflow code. A bug or an unexpected condition throws. If it reproduces on every replay, the Execution is stuck. The WorkflowTaskFailed event names the error.
    • Data Converter error. Something failed serializing or deserializing Workflow inputs, outputs, or Memo fields. Check your Data Converter and Payload Codec configuration.
  3. Check Worker thread and slot pressure. Cross-check Worker Task slots exhausted for worker_type=WorkflowWorker. Slot exhaustion and thread pool exhaustion tend to show up together under load, and a CPU-starved Worker is slower to finish Workflow Tasks, which makes both worse.
  4. Fix and redeploy. Affected Executions resume on their next Workflow Task retry once compatible code is running.

Workflow Task execution latency high​

Metric: temporal_workflow_task_execution_latency, tagged namespace, task_queue, and workflow_type

Workflow Tasks are taking too long to execute on the Worker. The default Workflow Task timeout is 10 seconds, so at or above that value the Temporal Service is actively timing out Workflow Tasks.

A batch workload where Workflow Tasks routinely run long can sit above this threshold all day without anything being wrong. Set the threshold from your own observed p99, and only treat the default as meaningful if your Workflows are latency-sensitive.

Why it matters. The Temporal Service writes WorkflowTaskTimedOut events to Event History and reschedules timed-out Tasks on the normal Task Queue. Each timeout forces a Sticky Execution cache eviction on the Worker holding the Execution, so the next Workflow Task for it requires a full cold replay.

If you run Local Activities, a Workflow Task timeout causes them to re-execute from scratch on the retried Task, because their results are not checkpointed between Workflow Task heartbeats. Non-idempotent Local Activities produce duplicate side effects with real business impact.

At scale this compounds: more timeouts cause more cold replays, cold replays drive latency higher, and higher latency causes more timeouts.

Triage.

  1. Check replay latency. Look at temporal_workflow_task_replay_latency. If it is high, the time is going into re-running Event History rather than into new commands. Large histories, a slow Data Converter during replay, or a high cache eviction rate forcing cold replays are the usual culprits.
  2. Check the Sticky Execution cache. A high forced-eviction rate causes a cold replay on every Workflow Task. See Sticky cache holding zero entries under load.
  3. Check Worker CPU. If replay latency is normal but execution latency is high, the time is going into new command execution. High CPU slows all code on the Worker.
  4. Check for blocking Workflow code. Workflow code must not perform blocking I/O, heavy computation, or synchronous non-Temporal calls. Any blocking call holds the Task slot and inflates this metric. In the Python SDK, verify that no async def Workflow code is blocking the event loop.
  5. Check for throttling on respond operations. See RESOURCE_EXHAUSTED on respond operations. The SDK holds the slot until the respond call succeeds, which inflates this metric even when your Workflow code finished quickly.

Activity execution failures elevated​

Metric: temporal_activity_execution_failed, tagged activity_type

Activities are failing outright at a sustained rate: returning failures, not timing out.

ApplicationFailure instances marked with category BENIGN don't increment this counter, so how well this metric tracks only the unexpected failures depends on how consistently your application marks the expected ones.

Why it matters. A high failure rate means a burst of retry Tasks. If your Workers can't keep up with the retry volume, the Activity Task backlog grows. See Activity schedule-to-start latency elevated. At scale, sustained retry bursts put real pressure on Task matching and the database underneath.

Triage.

  1. Identify which Activity is failing. The activity_type tag narrows it down. Worker logs for that type carry the error messages, stack traces, and associated Workflow Ids.
  2. Work out whether this is transient or a bug. A downstream outage, a network partition, or a database timeout will recover on its own, so watch whether the rate falls. A code bug won't.
  3. Check downstream service health. A struggling dependency is a common cause of sustained failure bursts. If it is throttling you, check that your Retry Policy has sensible backoff. Without it, your retries pile more pressure onto something that is already overloaded.
  4. Check schedule-to-start latency. A growing retry backlog shows up as elevated Activity schedule-to-start latency even after the failure rate drops.
  5. Mark expected failures as benign. If your design fails Activities on purpose, as polling patterns, Saga compensations, and flow control through exceptions all do, mark those ApplicationFailure instances with category BENIGN. That keeps them out of this metric and lets the alert track only the failures you didn't expect, without tuning a threshold per activity_type. Check that your SDK version supports the category before you rely on it.

One caveat: internal failures increment this counter no matter what category you set. Those are things like a context propagation error or a context timeout, rather than an Activity returning a failure.

Unregistered Activity invocation​

Metric: temporal_unregistered_activity_invocation, tagged activity_type, task_queue, and workflow_type

A Workflow scheduled an Activity that the Worker polling that Task Queue has no registered implementation for.

This metric is emitted by the Go SDK only.

Why it matters. The Activity can't run. It keeps getting retried against a Worker that has no implementation for it, until the Activity's scheduleToClose timeout expires, or forever if you haven't set one. Meanwhile the Workflow Execution waiting on it goes nowhere.

This is nearly always a deployment mistake rather than a runtime condition: Workflow code scheduling an Activity that the deployed Worker doesn't register. It won't fix itself.

Triage.

  1. Identify the Activity and the Task Queue. The activity_type and task_queue tags name both. The workflow_type tag identifies which Workflow is scheduling it.
  2. Check whether the Activity is registered on the right Worker. Confirm the Worker polling that Task Queue registers that Activity type. A common cause is registering the Activity on a Worker polling a different Task Queue.
  3. Check for a partial rollout. If Workflow code that schedules a new Activity deployed ahead of the Worker that implements it, some Workers will be running without the registration. Complete the rollout.
  4. Check for a renamed Activity. Changing an Activity's registered name while Executions are in flight leaves those Executions scheduling the old name. Register both names until the in-flight Executions drain, or use a versioning guard.

Local Activity latency exceeds the heartbeat timeout​

Metric: temporal_local_activity_execution_latency, tagged activity_type

A Local Activity is running past the Workflow Task heartbeat timeout, which defaults to 30 minutes.

How Workflow Task heartbeating works​

A Local Activity executes inside the Workflow Task rather than as a separately scheduled Activity Task. That means the Workflow Task stays open for as long as the Local Activity runs, which would normally exceed the Workflow Task timeout.

To keep the Task alive, the SDK sends Workflow Task heartbeats: repeated RespondWorkflowTaskCompleted calls that tell the Temporal Service work is still going and ask for more time. The Service goes along with this up to the Workflow Task heartbeat timeout. After that it times the Task out and reschedules it on the normal Task Queue.

Local Activities cannot heartbeat individually the way regular Activities can, and their results are not recorded in Event History between Workflow Task heartbeats. So when the Task is rescheduled, every Local Activity in it runs again from the beginning.

Why it matters. When the Temporal Service times out the heartbeating Workflow Task, the Local Activity re-executes from scratch. A non-idempotent Local Activity produces duplicate side effects with real business impact.

Any pending Signals, Updates, or other events are delayed until the retried Workflow Task completes, so end-to-end Execution latency rises significantly.

The Local Activity also occupies an executor slot for its entire duration. Several in this state at once can occupy every available slot, blocking new Local Activities from starting. See LocalActivityWorker slots.

Local Activities are designed for short, fast operations. A single attempt running for 30 minutes is a design problem, not a tuning problem.

Triage.

  1. Identify the affected Local Activity. The activity_type tag narrows it down. Worker logs for that type show what it is doing, how long individual attempts run, and the associated Workflow Ids.
  2. Find what it is blocked on. A Local Activity running this long is nearly always stuck on a downstream call: a slow service, a slow query, or a network call with a very generous timeout. Fix the dependency, or shorten that timeout so the Local Activity fails fast instead of hanging.
  3. Check whether a retry chain is accumulating. A high failure rate paired with an aggressive Retry Policy can push total elapsed time past the heartbeat timeout even when every individual attempt is short. Check temporal_local_activity_execution_failed for the same activity_type, and fix the underlying failure first.
  4. Check whether timeouts have already happened. By the time this fires, the Temporal Service may have timed out heartbeating Workflow Tasks already. Look for timeout errors in Worker logs and WorkflowTaskTimedOut events in Event History. If they are there, your Local Activities have already run twice, so check whether they are idempotent and clean up any duplicate side effects.
  5. Fix the design. If the work genuinely takes this long, convert it to a regular Activity with heartbeating, which is the correct primitive for long-running work. If it must stay a Local Activity, set a scheduleToCloseTimeout below the Workflow Task heartbeat timeout so it fails with a timeout error the Workflow can handle, rather than having the entire Workflow Task re-executed.