CloudWatch Logs: read-only access from a local machine

This setup grants read-only access to farfalla's HTTP access log (and any other Lambda log group) from a local AWS CLI without storing long-lived production credentials. The path is cross-account sts:AssumeRole: an IAM user in the publica.la account assumes a scoped read-only role that lives in the production account.

It was originally set up during the 2026-05-04 SingleStore memory incident to trace traffic patterns through the access log. The same recipe applies to any engineer.

What it provides

• Read-only access to CloudWatch Logs in the production AWS account (375481448855).
• Scoped to /aws/lambda/vapor-farfalla* log groups via the role policy.
• All actions required by aws logs filter-log-events and CloudWatch Logs Insights.
• No new credentials on disk. The role is assumed via the engineer's existing personal IAM user.

The production account also hosts API Gateway access logs (/aws/apigateway/vapor-farfalla-production/access-log) and Lambda logs for sibling services (e.g. castoro). Widen the policy as needed.

Prerequisites

• An IAM user in account 314863550469 (publica.la), with access keys configured locally as the default profile (verify with aws sts get-caller-identity).
• Admin access to the production account 375481448855, either directly or via a teammate who can run Step 1.

Step 1: Create the role in the production account

This step runs once for the team, by anyone with prod admin credentials. One shared role is assumable by every authorized teammate. CloudTrail still records which IAM user called sts:AssumeRole, so the audit trail stays intact.

Sign in to the AWS IAM console in the production account (375481448855) and go to Roles → Create role.

Trusted entity

Pick Custom trust policy and paste the JSON below. The Principal.AWS array lists every authorized teammate's IAM user ARN in the publica.la account. Add or remove entries as needed before continuing.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::314863550469:user/fgilio@publica.la",
          "arn:aws:iam::314863550469:user/imilano@publica.la"
        ]
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Permissions

Skip the policy picker on the Add permissions screen and click Next. The inline policy is attached after the role exists.

Role details

• Role name: farfalla-logs-readonly
• Description (optional): Read-only CloudWatch Logs access for engineers, scoped to vapor-farfalla*.
• Maximum session duration: 2 hours.

Click Create role.

Attach the inline permissions policy

Open the new role and go to Permissions → Add permissions → Create inline policy → JSON. Paste the document below, then click Next, name it FarfallaLogsReadOnly, and save. It covers every action required by CloudWatch Logs Insights and the Logs CLI, scoped to farfalla log groups.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ListGroups",
      "Effect": "Allow",
      "Action": ["logs:DescribeLogGroups"],
      "Resource": "*"
    },
    {
      "Sid": "ReadFarfallaLogs",
      "Effect": "Allow",
      "Action": [
        "logs:DescribeLogStreams",
        "logs:FilterLogEvents",
        "logs:GetLogEvents",
        "logs:GetLogGroupFields",
        "logs:GetLogRecord",
        "logs:StartQuery",
        "logs:StopQuery",
        "logs:GetQueryResults"
      ],
      "Resource": [
        "arn:aws:logs:*:375481448855:log-group:/aws/lambda/vapor-farfalla*",
        "arn:aws:logs:*:375481448855:log-group:/aws/lambda/vapor-farfalla*:*"
      ]
    }
  ]
}

The role is ready. Each teammate now configures their local profile (Step 2).

Direct link to the role for any follow-up edits: farfalla-logs-readonly in the IAM console.

Adding or removing a teammate later

Open the role and go to Trust relationships → Edit trust policy. Update the Principal.AWS array and save. Changes take effect immediately on the next sts:AssumeRole call.

Step 2: Add the local profile

Each teammate adds this stanza to their ~/.aws/config:

[profile farfalla-prod]
role_arn       = arn:aws:iam::375481448855:role/farfalla-logs-readonly
source_profile = default
region         = us-east-1

Step 3: Smoke test

AWS_PROFILE=farfalla-prod aws sts get-caller-identity
AWS_PROFILE=farfalla-prod aws logs describe-log-groups \
  --query 'logGroups[?contains(logGroupName, `farfalla`)].[logGroupName,storedBytes]' \
  --output table

The first call should report an assumed-role/farfalla-logs-readonly ARN under account 375481448855. The second should list the /aws/lambda/vapor-farfalla* groups.

Common operations

Tail recent log events that match a pattern

AWS_PROFILE=farfalla-prod aws logs filter-log-events \
  --log-group-name "/aws/lambda/vapor-farfalla-production-d" \
  --filter-pattern '"http-access-log"' \
  --start-time $(($(date -u -j -f "%Y-%m-%dT%H:%M:%SZ" "2026-05-04T20:30:00Z" +%s) * 1000)) \
  --end-time   $(($(date -u -j -f "%Y-%m-%dT%H:%M:%SZ" "2026-05-04T20:31:00Z" +%s) * 1000)) \
  --max-items 5 \
  --query 'events[].message' --output text

Run a Logs Insights query

Save the query to a file (avoids shell-escaping pain):

fields @timestamp
| filter @message like /http-access-log/ and @message like /author=/
| parse @message '"x-forwarded-host":["*"]' as host
| stats count() as hits by host
| sort hits desc
| limit 30

Then start it, poll until done, fetch:

QID=$(AWS_PROFILE=farfalla-prod aws logs start-query \
  --log-group-name "/aws/lambda/vapor-farfalla-production-d" \
  --start-time $(date -u -j -f "%Y-%m-%dT%H:%M:%SZ" "2026-05-04T17:00:00Z" +%s) \
  --end-time   $(date -u -j -f "%Y-%m-%dT%H:%M:%SZ" "2026-05-04T21:30:00Z" +%s) \
  --query-string "$(cat query.txt)" \
  --query 'queryId' --output text)

until [ "$(AWS_PROFILE=farfalla-prod aws logs get-query-results --query-id $QID --query 'status' --output text)" = "Complete" ]; do
  sleep 5
done

AWS_PROFILE=farfalla-prod aws logs get-query-results --query-id $QID --output json

What's in the HTTP access log

App\Http\Middleware\HttpAccessLogger writes one JSON line per request to stderr, which Lambda forwards to CloudWatch. Each line wraps a payload with method, uri, aws_request_id, tenant_id, user_id, and a curated subset of headers (x-forwarded-host, x-forwarded-for, cf-ipcountry, user-agent, etc.).

Caveats from the request topology (User → Caddy → Cloudflare → API Gateway → Lambda):

• cf-ipcountry is the country of the Caddy → Cloudflare hop, not the real client. It identifies which Caddy region (US/BR/IE) handled the request.
• cf-connecting-ip is Caddy's IP for the same reason.
• The real client IP is the first entry in x-forwarded-for. Parse with parse xff "*," as client_ip in Logs Insights.
• x-forwarded-host is set by Caddy and reflects the original tenant domain. Reliable.

See caddy-https-servers.md for the full request lifecycle.

Widening the policy

To grant access to other log groups, open the role, edit the FarfallaLogsReadOnly inline policy (Permissions → click the policy name → Edit → JSON), and add the log group ARNs to the Resource array. Common candidates:

• /aws/apigateway/vapor-farfalla-production/access-log: API Gateway access logs (~50 GB at the time of writing).
• /aws/lambda/vapor-castoro-production*: castoro PDF-processing service.

Cleanup

To remove a single teammate, edit the trust policy on the role (see "Adding or removing a teammate later" under Step 1) and drop their ARN.

To retire the role entirely, open the role and click Delete. The inline policy is removed with it.

Each teammate removes the [profile farfalla-prod] stanza from ~/.aws/config.