We are running a Lambda function to pull events from AWS SQS. After running in production for a long time without any deployment, the Lambda function was suddenly unable to pull events from AWS SQS anymore.

Problem

Without any deployment or code change, the Lambda function lost all AWS service connections (e.g., SQS, SSM, etc.).

Debug flow

Network issue?

We tried to reach the sqs.us-east-2.amazonaws.com domain with urllib.request. The result showed that the SQS service was accessible, so we ruled out network issues.

SG outbound rule issue?

We tried adding “0.0.0.0/0” to the Lambda security group rule, but the Lambda function was still unable to pull events from SQS, so we ruled out security group outbound rules as well.

Dependency issue?

My senior checked the source code of the Lambda function and found that it did not lock the “boto3” & “botocore” versions. We suspected that the Lambda function was calling the runtime-level boto dependency instead, and AWS never guarantees that the dependencies from the Lambda runtime are matched and stable.

Resolution

We added version locks to the requirements.txt and bundled boto3 and botocore with specific versions, then deployed them with our Lambda function as follows:

# requirements.txt
boto3==1.20.0
botocore==1.20.0

After redeploying with locked dependencies, the Lambda function was able to connect to AWS services again.

Takeaway

Never trust the AWS Lambda function default or AWS-managed dependencies as they might be upgraded or patched silently. Always lock the dependency versions yourself to ensure consistent behavior.

Key lesson: Lambda runtime dependencies can change without notice, breaking existing functionality. Explicit dependency management prevents unexpected failures.

Bonus

We can see that even some AWS official example code does not lock boto3 and botocore versions (see references #6 and #7). Consider a novice engineer like myself copying and pasting, or following the quick start examples — their Lambda functions might still have a high chance of experiencing massive outages if those two packages become incompatible.

I think the warning emphasis in reference #2 is not prominent enough to catch developers’ attention.

References:

1. https://github.com/boto/boto3/issues/4462
2. https://docs.aws.amazon.com/lambda/latest/dg/python-package.html
3. https://docs.aws.amazon.com/lambda/latest/dg/python-layers.html#python-layer-paths
4. https://stackoverflow.com/questions/55695187/import-libraries-in-lambda-layers?noredirect=1&lq=1
5. https://docs.aws.amazon.com/lambda/latest/dg/packaging-layers.html
6. https://github.com/awsdocs/aws-doc-sdk-examples/blob/main/python/example_code/bedrock/requirements.txt#L2
7. https://github.com/awsdocs/aws-doc-sdk-examples/blob/main/python/example_code/auto-scaling/requirements.txt#L1

This post is amended and reviewed by claude.ai.

We are using ECS for an application that needs to pull an image from private ECR. After we imposed the SG outbound rule as follows:

resource "aws_security_group_rule" "old" {
  type              = "egress"
  from_port         = 443
  to_port           = 443
  protocol          = "tcp"
  cidr_blocks       = ["10.0.0.0/8", "128.0.0.0/8"]
  security_group_id = "sg-123456"
}

Problem

We found that all the ECS auto scale groups showed an error that the ECR images were inaccessible with a 4xx error.

Resolution

I asked my senior, and he mentioned that we might need to enable the aws s3 prefix list in the SG outbound rule in order to enable ECS to pull the image from private ECR. This is because ECR stores Docker image layers in S3, so blocking S3 access prevents image pulls.

data "aws_prefix_list" "s3" {
  name = "com.amazonaws.${var.region}.s3"
}

resource "aws_security_group_rule" "new" {
  type              = "egress"
  from_port         = 443
  to_port           = 443
  protocol          = "tcp"
  cidr_blocks       = ["10.0.0.0/8", "128.0.0.0/8"]
  prefix_list_ids   = [data.aws_prefix_list.s3.id]
  security_group_id = "sg-123456"
}

After we enabled the aws s3 prefix list, ECS was able to launch the auto scaling action again, and the error was resolved.

How we learned

For security reasons, we should deny all egress traffic by default, but when we move the SG rule from something like 0.0.0.0/0 to an allowlist model, we need to ensure all the required ports and rules are reviewed. In this case, when imposing the old rule, the auto scaling was not triggered so we never knew that without the aws s3 prefix list, ECS would not be able to pull the images from private ECR.

Key takeaway: ECR’s dependency on S3 for image layer storage means that restrictive security groups must include S3 access via managed prefix lists.

References:

1. https://docs.aws.amazon.com/AmazonECR/latest/userguide/vpc-endpoints.html
2. https://www.eddgrant.com/blog/2023/05/12/aws-connecting-to-a-private-ecr-repository-using-vpc-endpoints
3. https://docs.aws.amazon.com/vpc/latest/userguide/working-with-aws-managed-prefix-lists.html

The above post is amended and reviewed by claude.ai

My company needs to inject some shared libraries into the subdirectory. To avoid copy-and-paste issues, we are using simple symlinks to link the shared libraries to each subdirectory, ensuring they share the same version without needing to touch the git sub-module.

Expected

Each time we update the source code or shared libraries, Terraform should monitor the changes and apply them to remote.

Actual

symlink directory inside the subdirectory will always be the same.

Terraform state will only amended if the source code has been updated.

Resolution

We need to watch the original directory of the symlink and the source code.

Here is the sample code

resource "null_resource" "run"{
…
  triggers = {
    source_dir_sha = sha1(join("", [for f in fileset("dir/source_code", "**"): filesha1("source_code/${f}")]))
    symlink_dir_sha = sha1(join("", [for f in fileset("dir/shared_libs", "**"): filesha1("shared_libs/${f}")]))
  }
…
}

Why

We always thought Terraform would pick up the symlink like the real file.

However, it just considers the symlink as a link, any change from destination will not be watched by Terraform

Bonus

Always use `openssl sha1` to check the shasum in MacOS and Linux, which will return the same result. While `shasum` might not return the same result in difference OS.

Reference

1. https://stackoverflow.com/a/57931341/2876087
2. https://www.geeksforgeeks.org/tar-command-linux-examples/
3. How to installing sha1sum in MAC OS?
4. https://formulae.brew.sh/formula/md5sha1sum

I am working to build a Python container with a minimized system library.

I need to connect to AWS RDS as my project needs to connect to RDS as it’s persistent storage.

Problem

RDS recommends using an SSL connection to ensure the traffic is secured. The Python container from DockerHub does not embed the RDS CA cert, so I must inject it myself.

Resolution

Download all the required CA certs

# download aws ca cert
wget ttps://truststore.pki.rds.amazonaws.com/global/global-bundle.pem -O /tmp/global-bundle.pem

# thanks mozilla maitains this lovely ca cert lib
pip install certifi

cat $(python -m certifi) /tmp/my-cert.pem

build.Dockerfile

FROM python:3.13
LABEL authors="hey.alpha@pm.me"

ENV REQUEST_CA_BUNDLE="/etc/ssl/certs/ca-certificates.crt"

COPY "/tmp/my-cert.pem" "/usr/local/share/ca-certificates/extra/my-cert.crt"
COPY "/tmp/global-bundle.pem" "/usr/local/share/ca-certificates/extra/global-bundle.crt"

RUN update-ca-certificates

References

1. https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.SSL.html#UsingWithRDS.SSL.RegionCertificateAuthorities
2. https://github.com/certifi/python-certifi
3. https://askubuntu.com/questions/73287/how-do-i-install-a-root-certificate

I am currently looking to upload a lambda function to AWS.

It will trigger the build bash script, and generate a JSON file with the following.

{
  "path": "my_path",
  "zip_file": "/tmp/my-lambda-function.zip"
}

Problem

We are using the following terraform resources to invoke the customized build script (build.bash) to build the zip file.

data "external" "build_zip" {
    program = ["bash", "-c", "build.bash"]
}

resource "aws_lambda_function" "test_lambda" {
  ...
  source_code_hash = filesha256(data.external.build_zip.result.zip_file)
  ...
}

Expected

We expected a new lambda function should be deployed to remote for each trigger.

Actual

The terraform does not ensure the lambda function will be deployed every time and it becomes a random behavior, it might or might not be deployed to remote unless we check the code from the AWS console.

Possible reason

data "external" "build_zip" { <-- this might only trigger once
    program = ["bash", "-c", "build.bash"]
    # following statement does not support 
    triggers = {
        always_run = timestamp() <-- ensure it will trigger each time
    }
}

Resolution

resource "null_resource" "build_zip" {
    provisioner "local-exec" {
        command = "build.bash""
        interpreter = ["/bin/bash", "-c"]
    }

    triggers = {
        always_run = timestamp()
    }
}

data "local_file" "zip_file_location" {
  filename = "/tmp/my-lambda-function.zip"
  depends_on = [ null_resource.build_zip ]
}

resource "aws_lambda_function" "test_lambda" {
  ...
  source_code_hash = filesha256(data.local_file.zip_file_location)
  ...
  depends_on = [ data.local_file.zip_file_location ]
}

The above code can ensure build script will be triggered every time.

Better resolution ( only update when the code changes)

used sha1 to watch a directory

resource "null_resource" "build_zip" {
    provisioner "local-exec" {
        command = "build.bash""
        interpreter = ["/bin/bash", "-c"]
    }

    triggers = {
        dir_hash = sha1(join("", [for f in fileset("/tmp/my-dir/", "**"): filesha1("my-dir/${f}")]))
    }
}

data "local_file" "zip_file_location" {
  filename = "/tmp/my-dir/my-lambda-function.zip"
  depends_on = [ null_resource.build_zip ]
}

resource "aws_lambda_function" "test_lambda" {
  ...
  source_code_hash = filesha256(data.local_file.zip_file_location)
  ...
  depends_on = [ data.local_file.zip_file_location ]
}

Why

Some people might ask why don’t you use the preferred way from from AWS provider official documents like the following.

# https://registry.terraform.io/providers/hashicorp/aws/4.67.0/docs/resources/lambda_function
data "archive_file" "lambda" {
  type        = "zip"
  source_file = "lambda.js"
  output_path = "lambda_function_payload.zip"
}

As my company will inject some fields or shared private dependency libraries into the zip file, which does not commit to the GIT repo, we only fetch those resources from “build.bash” script.

reference

1. https://stackoverflow.com/questions/65836439/re-run-the-program-of-an-external-data-source-on-every-plan-refresh-state-te
2. https://stackoverflow.com/questions/65316834/call-to-function-file-failed-no-file-exists-terraform
3. https://stackoverflow.com/questions/51138667/can-terraform-watch-a-directory-for-changes

Dear [Data Protection Officer’s Name],

I hope this message finds you well.

My name is XXX, and I am writing to formally invoke my right under the UK General Data Protection Regulation (GDPR) to request a subject access request. I have been the main resident and head of household at the address below since 1997:

[Your Full Name]
[Your Full Address]

I would like to request information on all council-registered occupants for the address [Your Full Address] since 1997, including but not limited to my children, parents, partner, etc.

Additionally, I can provide further proof that my household members have appointed me to represent them in this matter, should that be necessary.

Please let me know if you require any further information or documentation to verify my identity and proceed with this request.

Thank you for your assistance.

Best regards,
XXX

Introduction

If you’re a British National (Overseas) (BNO) holder applying for settlement, you’ll need to gather a few important documents[1].

To make things easier, it’s a good idea to keep track of the list of documents recommended and accepted by the UK government for your settlement application.

Common Issue

Many people may not know that there are other documents besides bank statements, utility bills (such as electricity and gas), or NHS letters that can be used to show residency in the UK.

Solution

An effective alternative is to request a “Yearly Electoral Registration Confirmation Letter” from your local Electoral Registration Office. This applies even to those who do not have the right to vote in the UK.

Fees

While some offices may charge a fee for issuing confirmation letters for previous years, it is important to note that you are entitled to obtain the confirmation letter for the current year free of charge.

How to Proceed

To find your local council and Electoral Registration Office, you can visit the official government website or contact your local government authority for guidance.

Even if you are not a BNO holder, you are still able to get a letter with issue data from the local council as living proof.

Find your local council

I hope it helps people under the British National (Overseas) visa route.

Credit

1. Leader of Sutton Council and Sutton Councillor
2. All team members of Sutton Council

refer:

1. https://www.gov.uk/government/publications/eu-settlement-scheme-statement-of-intent/annex-a-documentary-evidence-of-continuous-residence-in-the-uk
2. https://www.gov.uk/government/publications/indefinite-leave-to-remain-calculating-continuous-period-in-uk/indefinite-leave-to-remain-calculating-continuous-period-in-uk-accessible

1. https://www.facebook.com/MinnieMaMaUK
2. https://t.me/+cyBXu0WDk-ljMWE9

Your rights

You have certain rights relating to the personal information we hold about you which are outlined below. None of these are absolute and are subject to various exceptions and limitations. You can exercise these rights at any time by contacting us using the contact details below. Your rights are:

• Right to be informed
• Right of access
• Right of rectification or erasure
• Right of erasure
• Right to portability
• Right to object
• Right to withdraw consent
• Right of complaint
• Right to opt-out of marketing communications

Sample Letter to invoke the right to request a data rectification and subject access request to Data protection officers

## Formal Request for Rectification and Subject Access Request under UK GDPR for Council Tax Account ABC12345678

[Officer's Name]
[Council Name]
[Address]
[City, Postal Code]

Dear Officer,

I am writing to formally request rectification and submit a subject access request under the UK GDPR for Council Tax Account ABC12345678. 
My purpose for contacting you is to ensure accurate and up-to-date information regarding the residence of my parents, Mr. HIJ and Ms. XYZ, at the address 'XXXX' since June 10, 2022. 
Currently, our address accommodates four individuals. I kindly request that you update the account to reflect these occupants accurately.

The revised list of occupants for our address is as follows:
1. Mr. ABC
2. Ms. EFG
3. Mr. HIJ
4. Ms. XYZ

We would greatly appreciate your prompt assistance in adding these named occupants to our council tax account.
This is essential as we require official documentation from the local council to validate our residence at this address for settlement purposes. 
It is important to note that, due to my parents' advanced age (both over 75), they are not eligible for National Insurance numbers. 
Consequently, the Council tax bill serves as a vital document to establish their residence status in the UK.

Your understanding of our situation and the significance of adding my parents as named occupants is deeply appreciated.

Additionally, I kindly request a cover letter, featuring the Council's logo, to be sent to our address. 
This letter should enumerate all the named occupants residing at the aforementioned address.

As a reminder, under the UK GDPR, individuals have the right to have inaccurate personal data rectified or completed if it is incomplete. 
An individual can make a request for rectification verbally or in writing, and the controller has one calendar month to respond to such a request.
I kindly request that you respond to my request within the stipulated time frame.

Thank you for your prompt attention to this matter. 
I look forward to receiving your response at your earliest convenience.

Best,

[Your Name]
[Your Address]
[Your Contact Information]
[Date]

Reference

A guide to individual rights

# Problem

Gitlab plans to remove inactive repo to save their operation cost. I currently heavily make use of the Gitlab CI/CD to automatic the website release.

If Gitlab deleted my repo and I will lose all my source code and Git commits.

# Resolution

In order to find a new place to host my repo, I go back to Github again as nowadays Monkey worker (CI/CD) is an exclusive feature for Gitlab any more.

# Bonus

Firebase static hosting does not come with an auto DDoS shield while I notice that Cloudflare offers a similar static hosting with a DDoS shield.

I really do not want to get a bill for USD 99,999 while I am under a DDoS attack.

# Downside of Cloudflare

Cloudflare supports many modern web frameworks like React.js, Vue.js and Gitbook, etc.

However, my website is a simple stupid pure HTML file. Under this condition, the auto integration feature from Cloudflare is not useful for me.

# How to deploy a static page at Cloudflare

You can run npm run build to unlock the power of the Cloudflare page

// As I am not able to do fancy web framework and deal with webpack

## importance

## package.json

// add this line
"scripts": {    "build": "sh ./build.sh"  }

## build.sh

#!/bin/bash

npm i -g html-minifier-terser
mv ./_public/index.html ./_public/raw.html
html-minifier-terser --collapse-whitespace --remove-comments --remove-optional-tags --remove-redundant-attributes --remove-script-type-attributes --remove-tag-whitespace --use-short-doctype --minify-css true --minify-js true ./_public/raw.html > ./_public/index.html
rm -rf ./_public/raw.html

## output

2022-08-05T09:21:02.862613Z Success: Finished cloning repository files
2022-08-05T09:21:03.413183Z Installing dependencies
2022-08-05T09:21:03.423879Z Python version set to 2.7
2022-08-05T09:21:06.555729Z Downloading and installing node v17.9.1...
2022-08-05T09:21:06.917886Z Downloading https://nodejs.org/dist/v17.9.1/node-v17.9.1-linux-x64.tar.xz...
2022-08-05T09:21:07.337016Z Computing checksum with sha256sum
2022-08-05T09:21:07.467035Z Checksums matched!
2022-08-05T09:21:12.412214Z Now using node v17.9.1 (npm v8.11.0)
2022-08-05T09:21:12.778737Z Started restoring cached build plugins
2022-08-05T09:21:12.794225Z Finished restoring cached build plugins
2022-08-05T09:21:13.258235Z Attempting ruby version 2.7.1, read from environment
2022-08-05T09:21:16.706224Z Using ruby version 2.7.1
2022-08-05T09:21:17.054047Z Using PHP version 5.6
2022-08-05T09:21:17.210059Z 5.2 is already installed.
2022-08-05T09:21:17.235592Z Using Swift version 5.2
2022-08-05T09:21:17.236164Z Started restoring cached node modules
2022-08-05T09:21:17.250575Z Finished restoring cached node modules
2022-08-05T09:21:17.748291Z Installing NPM modules using NPM version 8.11.0
2022-08-05T09:21:18.155965Z npm WARN config tmp This setting is no longer used.  npm stores temporary files in a special
2022-08-05T09:21:18.15634Z npm WARN config location in the cache, and they are managed by
2022-08-05T09:21:18.156519Z npm WARN config     [`cacache`](http://npm.im/cacache).
2022-08-05T09:21:18.556771Z npm WARN config tmp This setting is no longer used.  npm stores temporary files in a special
2022-08-05T09:21:18.557123Z npm WARN config location in the cache, and they are managed by
2022-08-05T09:21:18.557367Z npm WARN config     [`cacache`](http://npm.im/cacache).
2022-08-05T09:21:18.689562Z 
2022-08-05T09:21:18.689844Z up to date, audited 1 package in 164ms
2022-08-05T09:21:18.690426Z 
2022-08-05T09:21:18.690651Z found 0 vulnerabilities
2022-08-05T09:21:18.697772Z NPM modules installed
2022-08-05T09:21:19.140946Z npm WARN config tmp This setting is no longer used.  npm stores temporary files in a special
2022-08-05T09:21:19.141316Z npm WARN config location in the cache, and they are managed by
2022-08-05T09:21:19.141763Z npm WARN config     [`cacache`](http://npm.im/cacache).
2022-08-05T09:21:19.159897Z Installing Hugo 0.54.0
2022-08-05T09:21:19.825418Z Hugo Static Site Generator v0.54.0-B1A82C61A/extended linux/amd64 BuildDate: 2019-02-01T10:04:38Z
2022-08-05T09:21:19.829217Z Started restoring cached go cache
2022-08-05T09:21:19.846978Z Finished restoring cached go cache
2022-08-05T09:21:19.990701Z go version go1.14.4 linux/amd64
2022-08-05T09:21:20.005434Z go version go1.14.4 linux/amd64
2022-08-05T09:21:20.008176Z Installing missing commands
2022-08-05T09:21:20.008448Z Verify run directory
2022-08-05T09:21:20.0086Z Executing user command: npm run build
2022-08-05T09:21:20.485151Z npm WARN config tmp This setting is no longer used.  npm stores temporary files in a special
2022-08-05T09:21:20.485492Z npm WARN config location in the cache, and they are managed by
2022-08-05T09:21:20.485657Z npm WARN config     [`cacache`](http://npm.im/cacache).
2022-08-05T09:21:20.5016Z 
2022-08-05T09:21:20.501987Z > sh ./build.sh
2022-08-05T09:21:20.502146Z 
2022-08-05T09:21:20.913386Z npm WARN config global `--global`, `--local` are deprecated. Use `--location=global` instead.
2022-08-05T09:21:20.91371Z npm WARN config tmp This setting is no longer used.  npm stores temporary files in a special
2022-08-05T09:21:20.913892Z npm WARN config location in the cache, and they are managed by
2022-08-05T09:21:20.914023Z npm WARN config     [`cacache`](http://npm.im/cacache).
2022-08-05T09:21:22.421422Z 
2022-08-05T09:21:22.421751Z added 24 packages, and audited 25 packages in 2s
2022-08-05T09:21:22.42191Z 
2022-08-05T09:21:22.422063Z 1 package is looking for funding
2022-08-05T09:21:22.422224Z   run `npm fund` for details
2022-08-05T09:21:22.422759Z 
2022-08-05T09:21:22.422967Z found 0 vulnerabilities
2022-08-05T09:21:22.875027Z Finished
2022-08-05T09:21:22.875725Z Note: No functions dir at /functions found. Skipping.
2022-08-05T09:21:22.876019Z Validating asset output directory
2022-08-05T09:21:23.367294Z Deploying your site to Cloudflare's global network...
2022-08-05T09:21:26.907478Z Success: Assets published!
2022-08-05T09:21:27.514049Z Success: Your site was deployed!

# To sum up

Cloudflare Page is a very great tool for a lazy engineer like myself to deploy a static s page with a DDoS shield and all the features provided by Cloudflare.

And npm run build able to unlock a very complex build script without working with Webpack which is likely to have breaking changes and have a lot of config files.

## Reference

1. https://news.itsfoss.com/gitlab-inactive-projects-policy/
2. https://www.reddit.com/r/Firebase/comments/mf3279/how_to_save_your_ass_from_a_ddos_99999_firestore/
3. https://developers.cloudflare.com/pages/framework-guides/
4. https://developers.cloudflare.com/pages/platform/build-configuration/#language-support-and-tools

Nowadays, using Serverless resources are the way to low down the infrastructure maintenance cost, engineer can focus on the business logic and bypass all the complex setting in the cloud.

However, there is no Blackmagic for any tech and we need to understand the life cycle of each service provider.

For instance, I set up a project to retrieve an API key from env var and expect the API key is stored at the secret manager.

In this way, It is expected that the API key will be automatically loaded to the latest version if I do not assign the version of the secret and the Lamda function at the application level does not need to care about how to reload or retrieve the newest one from the OS level.

However, It is not always true at the Azure function app at I do carry the expectation from another service provider like GCP cloud run which is Knative based and the secret is mount via injection each new version will auto trigger a deployment from the controller and I really do not need to care how can I handle it.

Lesson learn:

Never trust any Lamda function and dig into the detail or the tech involve under the hood

Reference:

1. https://github.com/MicrosoftDocs/azure-docs/issues/51708
2. https://docs.microsoft.com/en-us/azure/app-service/app-service-key-vault-references?tabs=azure-cli#rotation