Operations

Running Mayfly functions in production: what to alarm on, how to read the logs, what it costs, how to size Managed Instances, and what to do when something is wrong. Everything here uses only what the runtime already emits; nothing needs an agent or a vendor.

What the runtime emits

Alarms

Start with four alarms per function. Values are starting points; tune after a week of traffic.

FN=my-fn; TOPIC=arn:aws:sns:eu-central-1:123456789012:alerts

# 1. Error rate > 1 % over 5 minutes (math expression on Lambda's own metrics)
aws cloudwatch put-metric-alarm --alarm-name "$FN-error-rate" --alarm-actions $TOPIC \
  --evaluation-periods 2 --datapoints-to-alarm 2 --threshold 1 --comparison-operator GreaterThanThreshold \
  --treat-missing-data notBreaching \
  --metrics '[
    {"Id":"errors","MetricStat":{"Metric":{"Namespace":"AWS/Lambda","MetricName":"Errors","Dimensions":[{"Name":"FunctionName","Value":"'$FN'"}]},"Period":300,"Stat":"Sum"},"ReturnData":false},
    {"Id":"invocations","MetricStat":{"Metric":{"Namespace":"AWS/Lambda","MetricName":"Invocations","Dimensions":[{"Name":"FunctionName","Value":"'$FN'"}]},"Period":300,"Stat":"Sum"},"ReturnData":false},
    {"Id":"rate","Expression":"100 * errors / MAX([invocations, 1])","Label":"error %"}]'

# 2. p99 duration above 80 % of the timeout (timeout 30 s -> 24 s)
aws cloudwatch put-metric-alarm --alarm-name "$FN-p99-duration" --alarm-actions $TOPIC \
  --namespace AWS/Lambda --metric-name Duration --dimensions Name=FunctionName,Value=$FN \
  --extended-statistic p99 --period 300 --evaluation-periods 3 --threshold 24000 \
  --comparison-operator GreaterThanThreshold --treat-missing-data notBreaching

# 3. Throttles (any)
aws cloudwatch put-metric-alarm --alarm-name "$FN-throttles" --alarm-actions $TOPIC \
  --namespace AWS/Lambda --metric-name Throttles --dimensions Name=FunctionName,Value=$FN \
  --statistic Sum --period 60 --evaluation-periods 1 --threshold 0 --comparison-operator GreaterThanThreshold

# 4. Cold starts above 5 % of invocations (EMF from attach_invocation_metrics("MyApp"))
aws cloudwatch put-metric-alarm --alarm-name "$FN-cold-starts" --alarm-actions $TOPIC \
  --evaluation-periods 3 --threshold 5 --comparison-operator GreaterThanThreshold --treat-missing-data notBreaching \
  --metrics '[
    {"Id":"cold","MetricStat":{"Metric":{"Namespace":"MyApp","MetricName":"ColdStart","Dimensions":[{"Name":"FunctionName","Value":"'$FN'"}]},"Period":300,"Stat":"Sum"},"ReturnData":false},
    {"Id":"all","MetricStat":{"Metric":{"Namespace":"MyApp","MetricName":"Duration","Dimensions":[{"Name":"FunctionName","Value":"'$FN'"}]},"Period":300,"Stat":"SampleCount"},"ReturnData":false},
    {"Id":"pct","Expression":"100 * cold / MAX([all, 1])","Label":"cold start %"}]'

Worth adding when they apply:

• Memory headroom: MaxMemoryUsed (EMF via attach_platform_metrics/1) Maximum above 85 % of the configured size. Lambda does not expose this as a metric itself; Mayfly's extension does.
• Init errors: a Logs metric filter on { $.type = "platform.initReport" && $.record.status = "error" }; a function that fails in init/1 never produces an Errors datapoint for the handler, so the error-rate alarm stays quiet.
• Streaming: Duration is wall time until the last chunk; alarm on your own timing/4 metric for the part you control.
• Dead-letter / failure destinations: for async invokes, configure --dead-letter-config or an on-failure destination and alarm on the queue depth; Lambda retries twice and then drops the event.

Reading the logs

With the function's log format set to JSON every line is an object, so Logs Insights queries can filter on fields instead of regexes.

-- Errors with their stack traces, newest first
fields @timestamp, requestId, message
| filter level = "ERROR"
| sort @timestamp desc
| limit 50

-- Everything one invocation logged (yours + the platform records)
fields @timestamp, level, message, type, record.status
| filter requestId = "8f4c4018-c0a5-42dc-8c20-7ba4619311f0" or record.requestId = "8f4c4018-c0a5-42dc-8c20-7ba4619311f0"
| sort @timestamp asc

-- Cold starts: init duration distribution
filter type = "platform.initReport" and record.status = "success"
| stats count(), avg(record.metrics.durationMs), pct(record.metrics.durationMs, 50), pct(record.metrics.durationMs, 95) by bin(1h)

-- Billed duration and memory by hour (from platform.report)
filter type = "platform.report"
| stats sum(record.metrics.billedDurationMs) / 1000 as billed_seconds,
        max(record.metrics.maxMemoryUsedMB) as max_mb,
        count() as invocations by bin(1h)

-- Timeouts and runtime failures (the ones your handler never saw)
filter type = "platform.report" and record.status != "success"
| fields @timestamp, record.requestId, record.status, record.errorType

-- Per-tenant volume (tenantId is set from the Lambda-Runtime-Aws-Tenant-Id header)
stats count() by tenantId
| sort count() desc

-- Slow invocations with what they logged last
filter type = "platform.report" and record.metrics.durationMs > 5000
| fields record.requestId as rid, record.metrics.durationMs

Field names: Mayfly writes requestId, tenantId, level, message, timestamp plus every Logger.metadata key; Lambda's records have type and record.*. Your own metadata (Logger.info("x", order_id: 1)) becomes order_id.

Cost

Lambda bills GB-seconds (duration × memory) plus requests; the Erlang layer, EMF metrics and JSON logs change the picture only through log volume.

Measured on arm64 (see deployment for the raw numbers):

Rules of thumb:

• A function doing 1 M invocations/month at 10 ms × 512 MB costs about $0.20 + $0.08 = $0.28; 1 M cold starts at 550 ms would add ~$4.60. Cold starts, not warm duration, dominate the bill for low-traffic functions; for high-traffic ones memory size × duration does.
• arm64 is ~20 % cheaper per GB-second than x86_64 and Mayfly's cold starts are equal or better there. Default to arm64.
• CloudWatch Logs ingestion ($0.50/GB) is the hidden cost of chatty JSON logging. One 300-byte line per invocation at 1 M/month is 0.3 GB ($0.15); LOGLEVEL=warning in production plus an Errors alarm is cheaper than info-level everything. Set a retention (--log-retention in the IaC templates defaults to 14 days).
• EMF metrics cost per metric per month (first 10 000 free, then $0.30 each after tiers), not per datapoint; a Duration metric with the FunctionName dimension is one metric. High-cardinality dimensions (requestId, user ids) are a cost bug: use properties, not dimensions.
• The public Erlang layers cost you nothing; layer storage counts against the publisher's quota, not yours.
• Managed Instances bill the EC2 instances, not GB-seconds: a m7g.large (~$0.08/h on-demand) running 8 concurrent invocations replaces up to 8 × 2 GB of classic Lambda. It pays off above roughly 30–40 % sustained utilisation of that capacity; below, classic Lambda is cheaper.

Sizing Managed Instances

Mayfly starts one poller per AWS_LAMBDA_MAX_CONCURRENCY slot and the VM keeps all vCPUs as schedulers, so a Managed Instances environment is a normal multi-core BEAM node serving PerExecutionEnvironmentMaxConcurrency requests in parallel.

• Memory: ExecutionEnvironmentMemoryGiBPerVCpu × vCPUs is the environment's memory. The runtime itself needs ~90 MB; budget per-invocation working set × concurrency on top. 2 GiB/vCPU is a good start for API work.
• Concurrency per environment: start at vCPUs × 2 for I/O-bound handlers (the BEAM parks processes waiting on sockets cheaply) and vCPUs × 1 for CPU-bound ones. Verified: 8 concurrent invocations on one m7g.large (2 vCPU) finished in 0.13 s wall time.
• Instance type: m7g.large is the smallest sensible arm64 choice; MaxVCpuCount on the capacity provider caps spend. c7g for CPU-heavy, r7g for cache-heavy handlers.
• Deadlines: Managed Instances do not kill a handler at the function timeout. Long loops must check Mayfly.Context.remaining_time_ms/1 and stop themselves, or they keep a slot busy forever.
• State: init/1 state is shared read-only across the pollers. Anything mutable belongs in a named process (Agent, GenServer) started from init/1; it is one BEAM node, so OTP works as usual, including ETS caches shared by all concurrent invocations.
• Extensions: internal extensions cannot register for INVOKE on Managed Instances; Mayfly.Extension still receives platform.report through the Telemetry API, which is what attach_platform_metrics/1 uses.

Runbook

Error rate alarm fired. Logs Insights: filter level = "ERROR" and group by message prefix to see whether it is one exception type. Mayfly's error lines include errorType and the first stack frame; if the type is Runtime.InvalidResponse the handler returned something unencodable, if HandlerError it returned {:error, _} deliberately.

Latency alarm fired, errors flat. Check cold-start percentage (EMF ColdStart) first: a deploy or a traffic spike creates new environments. Then platform.report durations vs your own timing/4 metrics to see whether the time is yours or a downstream call's. If it is yours and CPU-bound, increase memory (more vCPU).

Function times out. platform.report with record.status = "timeout" and record.metrics.durationMs ≈ timeout. There is no handler log for the end because the environment was killed. If the handler blocks on a downstream call, set socket timeouts below the function timeout so the error is yours and carries context. On Managed Instances a timeout does not kill the handler (see above).

Init error / Runtime.InitError. The function never served a request; platform.initReport has status: error. The payload posted to /runtime/init/error is in the logs as an ERROR line with the exception from init/1. Common causes: a missing environment variable read with System.fetch_env!/1, a dependency's application failing to start (check included_applications / extra_applications), or the layer OTP mismatch message from bootstrap.

Mayfly: release was built for ERTS X but the layer provides Y. Build toolchain and layer disagree. mix lambda.doctor tells you which; fix the .tool-versions or the layer ARN in the IaC (doctor checks both).

Retries and duplicates. Async invokes retry twice; SQS redelivers after the visibility timeout; EventBridge retries for up to 24 h. If an operation must not run twice, wrap it in Mayfly.Idempotency.run/3 (companion package mayfly_aws) with a business key; see the idempotency guide.

Lost log lines at shutdown. Without the mayfly-shutdown layer Lambda freezes and discards the environment silently; buffered log handlers lose their tail. Attach the layer and register Mayfly.Shutdown hooks for anything that must flush (see observability).

Rolling back. Functions are versioned; alias live → version N. aws lambda update-alias --name live --function-version N-1 is the rollback; with SAM/CDK the stack rollback does the same. Layers are immutable, so a rollback of the function also rolls back the layer it pointed to.

Deploying a new OTP major. Switch the layer ARN and the toolchain together (mise use erlang@…, --otp in the IaC, or mix lambda.new --otp). Build, mix lambda.doctor, deploy to a new version, shift the alias. The runtime verifies the match at boot, so a half-switch fails loudly rather than running a mismatched ERTS.