Mayfly roadmap

Copy Markdown View Source

What is done, what is being considered, and one proposal written out in full. Dates are when a decision was taken, not promises. Open an issue to argue for or against anything here.

Shipped (1.0.0-rc.1)

  • Runtime: Mayfly.Handler behaviour, mix release build with the Mayfly ERTS layers (OTP 27/28/29, 16 regions, arm64 + x86_64), bundled ERTS via Docker, container images (mix lambda.build --image).
  • Events and responses: Mayfly.Events decoders (API Gateway v1/v2, Function URL, ALB, SQS, SNS, S3, EventBridge, Kinesis, DynamoDB Streams), buffered and streamed responses, Function URL preludes.
  • Observability: JSON logs, EMF metrics, :telemetry, internal extension for the Telemetry API, X-Ray propagation.
  • Shutdown: Mayfly.Shutdown + the mayfly-shutdown external extension layer (SIGTERM hooks and log flush on environment shutdown).
  • Companions: mayfly_plug (Plug/Phoenix endpoints as handlers), mayfly_aws (Idempotency, DynamoDB, S3, SQS, SNS, SSM, Secrets Manager).
  • Scaffolding: mix lambda.new with SAM/Terraform/CDK templates, catalog- rendered layer ARNs; mix lambda.doctor, mix lambda.invoke.
  • Verified on real Lambda: every OTP major on both architectures, Managed Instances, response streaming, container images.

Considered and not done

  • Elixir precompiled layer (2026-10-09): measured 511 vs 543 ms median cold start, below the 50 ms bar for a second version axis. Numbers in guides/layers.md.
  • mix lambda.build --deploy: deployment belongs to IaC; mix lambda.new gives you SAM/Terraform/CDK instead.
  • SHUTDOWN for the internal extension: Lambda does not deliver it to internal extensions; solved with the external mayfly-shutdown layer.

Next

  1. Publish to Hex (mayfly, then mayfly_plug, mayfly_aws); cut 1.0.0 after the rc period.
  2. API Gateway HTTP API template for mix lambda.new (today: Function URL).
  3. mix lambda.doctor reads the IaC files and checks the layer ARN against the toolchain OTP.
  4. Durable Functions – see the PR-FAQ below. Decision pending feedback.

PR-FAQ: mayfly_durable – durable workflows for Elixir on Lambda

Working backwards document. Nothing here exists yet.

Press release

Elixir developers can now write multi-step, long-running workflows as plain functions on AWS Lambda. mayfly_durable, a companion package to Mayfly, turns a handler into a durable workflow: each step's result is checkpointed to DynamoDB, so a workflow that waits hours for a callback, retries a flaky API for days, or fans out to a thousand sub-tasks survives Lambda's 15-minute limit, cold starts and crashes. When the workflow resumes, completed steps are replayed from the checkpoint instead of re-executed. There is no new service to run and nothing to deploy besides a table: the orchestrator is your Lambda function.

defmodule Orders.Fulfil do
  use Mayfly.Durable.Workflow

  def run(%{"order_id" => id}, ctx) do
    payment = step(ctx, "charge", fn -> Payments.charge(id) end)
    :ok = step(ctx, "reserve", fn -> Stock.reserve(id) end)
    label = wait_for(ctx, "label-created", timeout: {:hours, 24})
    step(ctx, "notify", fn -> Mail.shipped(id, label) end)
    {:ok, %{payment: payment.id}}
  end
end

Each step/3 runs the function once and stores its return value; a second execution of the same workflow instance returns the stored value without calling the function. wait_for/3 checkpoints and ends the invocation; the workflow resumes when an external signal arrives (an HTTP call, an SQS message, a timer) and the orchestrator is re-invoked with the same instance id.

FAQ

Why not Step Functions? You should use Step Functions when the workflow is the product: visual, auditable, operated by people who do not read Elixir. mayfly_durable is for the other case, where the orchestration is a dozen lines of business logic that happen to span time, and expressing it in ASL JSON with a Lambda per state is the heavier tool. The two interoperate: a durable workflow step can start a Step Functions execution and wait_for its callback.

How is this different from AWS Lambda Durable Functions? AWS's durable functions (where available) are a managed feature with SDKs for a fixed set of runtimes; Elixir is not one of them. mayfly_durable gives Elixir the same programming model on the existing provided.al2023 runtime with user-owned state. If AWS ships Elixir support, this package should become an adapter.

What is stored? One DynamoDB item per workflow instance (pk = instance id) holding the step log: [{name, status, result_json, finished_at}], the pending wait (if any) with its deadline, and the original input. Results are JSON; large results should be put in S3 and referenced (same rule as Mayfly.Idempotency). The item has a TTL after completion.

What are the semantics? Steps are at-least-once: a crash between running a step and writing its checkpoint re-runs the step on resume. Step functions must therefore be idempotent or wrapped in Mayfly.Idempotency (same table works). Steps are deterministic by name: the orchestrator does not record control flow, only named results, so if branches that depend on step results are fine, but renaming a step mid-flight is a new step. Workflow code may not block on time (Process.sleep) across invocations; it uses wait_for/sleep_until, which end the invocation.

How does resumption work? The orchestrator function is invoked with %{"instance_id" => …} (plus an optional signal payload). Mayfly.Durable loads the item, runs run/2 from the top, short-circuits completed steps, and either finishes, or writes a new wait and returns {:ok, :waiting}. Signals come in through whatever already invokes the function: a Function URL (POST /signal/:instance_id/:name), an SQS message with the instance id, or an EventBridge Scheduler one-time schedule created by sleep_until/2 (the package creates it with the function's own ARN as target).

Fan-out? each(ctx, "items", items, fn item -> … end, concurrency: 50) invokes the same function asynchronously once per item with a child instance id and wait_fors the children; results are collected in the parent's step log. Children are ordinary workflow instances.

Where does the Lambda timeout go? Each invocation does as many steps as fit before Mayfly.Context.remaining_time_ms/1 drops below a margin, then checkpoints and re-invokes itself asynchronously. A workflow can therefore run for days; the longest single step must fit in one invocation.

What does it cost? DynamoDB reads/writes per step (on-demand: fractions of a cent per thousand steps), one EventBridge Scheduler schedule per timed wait, and Lambda invocations for each resume. No idle compute. A workflow waiting a week costs one schedule.

Failure handling? A step that raises fails the workflow after retries: attempts (exponential backoff via re-invocation, so retries do not consume Lambda time). The failure is recorded; on_failure/2 lets the workflow compensate (saga style). Dead workflows are queryable from the table.

Testing? Mayfly.Durable.Test drives a workflow in-process against a fake table: run_until_waiting/2, signal/3, advance_time/2, with the step log inspectable. No AWS needed.

Why a separate package? It depends on mayfly_aws (DynamoDB, Scheduler) and on :telemetry; the core must stay dependency-free. It also imposes a programming model that most functions do not need.

What would make us not build it? Little demand for multi-step workflows in Elixir-on-Lambda (most functions today are single-step event handlers), or AWS shipping Elixir support for its managed durable functions. We will build a prototype if three users describe a workflow they would run on it.

Open questions

  • Replay-by-name vs. replay-by-position: name is simpler to reason about; position (as in Temporal) allows loops without naming every iteration. Start with names plus an explicit each.
  • Scheduler vs. SQS delay queues for timers: Scheduler supports arbitrary delays and one-time schedules; SQS caps delays at 15 minutes. Scheduler.
  • Should wait_for support Mayfly.Events.HTTP natively so a Function URL can be the signal endpoint with zero code? Probably yes, behind an option.