🖼️ Serverless Image Processor with AWS Lambda

A fully automated serverless image processing system that automatically processes images uploaded to S3, creating multiple variants including compressed versions, format conversions, and thumbnails using AWS Lambda and Pillow.

🏗️ Architecture Overview

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   Upload S3     │───▶│  Lambda Function │───▶│  Processed S3   │
│    Bucket       │    │  + Pillow Layer  │    │     Bucket      │
└─────────────────┘    └─────────────────┘    └─────────────────┘
         │                       │                       │
         │              ┌─────────────────┐              │
         │              │   CloudWatch    │              │
         └──────────────│      Logs       │──────────────┘
                        └─────────────────┘

🔄 Processing Flow

1. Image Upload → User uploads image to S3 upload bucket
2. S3 Event Trigger → S3 automatically triggers Lambda function
3. Image Processing → Lambda processes image using Pillow library
4. Multiple Variants → Creates compressed, WebP, PNG, and thumbnail versions
5. Storage → Saves all variants to processed S3 bucket
6. Logging → Detailed processing logs in CloudWatch

✨ Features

• 🚀 Serverless Architecture - No servers to manage, pay only for usage
• 🖼️ Multiple Image Formats - JPEG, PNG, WebP support
• 📏 Smart Resizing - Automatic resizing for large images (max 4096px)
• 🗜️ Compression Options - Multiple quality levels (60%, 85%)
• 🖼️ Thumbnail Generation - 300x300 thumbnails automatically created
• 🔄 Format Conversion - Convert between JPEG, PNG, WebP
• 📊 Detailed Logging - Comprehensive CloudWatch logging
• 🛡️ Secure - Private S3 buckets with proper IAM policies
• 🏗️ Infrastructure as Code - Complete Terraform deployment
• 🐳 Docker-based Layer - Cross-platform Pillow layer building
• 💰 Cost Effective - Pay only for actual processing time

📁 Project Structure

Day-18/
├── 📂 lambda/                     # Lambda function code
│   ├── lambda_function.py         # Main processing logic
│   └── lambda_function_basic.py   # Fallback without PIL
├── 📂 scripts/                    # Deployment and build scripts
│   ├── build_layer_simple.sh     # Build Pillow layer (recommended)
│   ├── build_layer_docker.ps1    # PowerShell version
│   ├── deploy.sh                  # Full deployment script
│   └── destroy.sh                 # Cleanup script
├── 📂 terraform/                  # Infrastructure as Code
│   ├── main.tf                    # Core AWS resources
│   ├── variables.tf               # Input variables
│   ├── outputs.tf                 # Output values
│   ├── data-sources.tf            # Data sources
│   └── pillow_layer.zip          # Generated layer (after build)
├── 📄 README.md                   # This file
└── 📄 .gitignore                  # Git ignore rules

🚀 Quick Start

Prerequisites

• AWS CLI configured with appropriate permissions
• Terraform >= 1.0
• Docker (for building Pillow layer)
• Git Bash (Windows) or Bash (Linux/Mac)

1. Clone and Setup

git clone https://github.com/VanshShah174/serverless-image-processor-terraform-lambda.git
cd serverless-image-processor-terraform-lambda

2. Configure AWS Credentials

aws configure
# Enter your AWS Access Key ID, Secret Access Key, and region

3. Build Lambda Layer

Option A: Simple Build (Recommended)

cd scripts
./build_layer_simple.sh

Option B: PowerShell (Windows)

cd scripts
.\build_layer_docker.ps1

4. Deploy Infrastructure

# Full deployment
./deploy.sh

# Or manual deployment
cd ../terraform
terraform init
terraform plan
terraform apply

5. Test the System

# Upload a test image
aws s3 cp your-image.jpg s3://your-upload-bucket-name/

# Check processed results
aws s3 ls s3://your-processed-bucket-name/

# View logs
aws logs describe-log-groups --log-group-name-prefix "/aws/lambda"

📋 Configuration

Variables

Variable	Description	Type	Default
aws_region	AWS region for deployment	string	"us-east-1"
project_name	Project name prefix	string	"image-processor"
environment	Environment name	string	"dev"

Customization

Edit terraform/variables.tf to customize:

variable "project_name" {
  description = "Name prefix for all resources"
  type        = string
  default     = "my-image-processor"
}

variable "aws_region" {
  description = "AWS region"
  type        = string
  default     = "us-west-2"
}

🖼️ Image Processing Details

Input Formats Supported

• JPEG/JPG
• PNG
• BMP
• TIFF
• WebP

Output Variants Created

For each uploaded image photo.jpg, the system creates:

1. photo_compressed_abc123.jpg - High quality (85%) compressed JPEG
2. photo_low_abc123.jpg - Lower quality (60%) compressed JPEG
3. photo_webp_abc123.webp - WebP format (85% quality)
4. photo_png_abc123.png - PNG format (lossless)
5. photo_thumbnail_abc123.jpg - 300x300 thumbnail

Processing Features

• Smart Resizing - Images larger than 4096px are automatically resized
• Format Conversion - RGBA/LA images converted to RGB for JPEG compatibility
• Optimization - All outputs are optimized for size
• Metadata Preservation - Original filename and processing info stored in S3 metadata

💰 Cost Analysis

Estimated Monthly Costs (us-east-1)

Resource	Usage	Estimated Cost
Lambda Execution	1000 invocations/month (1GB, 30s avg)	~$0.20
Lambda Requests	1000 requests/month	~$0.20
S3 Storage	10GB stored images	~$0.25
S3 Requests	PUT/GET requests	~$0.10
CloudWatch Logs	1GB logs/month	~$0.50
Data Transfer	Minimal	~$0.05
Total Estimated		~$1.30/month

Cost Optimization Tips

• Lifecycle Policies - Archive old processed images to IA/Glacier
• Log Retention - Set CloudWatch log retention to 7-30 days
• Right-sizing - Adjust Lambda memory based on actual usage
• Batch Processing - Process multiple images in single invocation if possible

🔧 Advanced Configuration

Custom Image Processing

Modify lambda/lambda_function.py to add custom processing:

# Add custom variants
variants = [
    {'format': 'JPEG', 'quality': 95, 'suffix': 'high'},
    {'format': 'JPEG', 'quality': 50, 'suffix': 'mobile'},
    {'format': 'WEBP', 'quality': 90, 'suffix': 'webp-high'},
    # Add your custom variants
]

Environment-Specific Deployments

# Development
terraform workspace new dev
terraform apply -var="environment=dev"

# Production
terraform workspace new prod
terraform apply -var="environment=prod"

Custom Lambda Layer

Build layer with additional libraries:

# Modify build_layer_simple.sh
docker exec $CONTAINER_ID bash -c "
    pip install Pillow==10.4.0 opencv-python-headless boto3 -t /tmp/python/lib/python3.12/site-packages/
"

🔍 Monitoring & Troubleshooting

CloudWatch Metrics

Monitor these key metrics:

• Lambda Duration - Processing time per image
• Lambda Errors - Failed processing attempts
• Lambda Invocations - Total processing requests
• S3 Object Count - Images processed over time

Common Issues

1. PIL Import Error

[ERROR] Runtime.ImportModuleError: No module named 'PIL'

Solution: Rebuild the Lambda layer using ./build_layer_simple.sh

2. Lambda Timeout

[ERROR] Task timed out after 60.00 seconds

Solution: Increase timeout in terraform/main.tf:

resource "aws_lambda_function" "image_processor" {
  timeout = 300  # 5 minutes
}

3. Memory Issues

[ERROR] Runtime exited with error: signal: killed

Solution: Increase memory allocation:

resource "aws_lambda_function" "image_processor" {
  memory_size = 2048  # 2GB
}

Debugging

View Lambda Logs:

aws logs tail /aws/lambda/image-processor-dev-processor --follow

Test Lambda Function:

aws lambda invoke --function-name image-processor-dev-processor \
  --payload '{"test": true}' response.json

🛡️ Security Features

• Private S3 Buckets - No public access allowed
• IAM Least Privilege - Lambda has minimal required permissions
• Encryption - S3 server-side encryption enabled
• VPC Support - Can be deployed in VPC for additional security
• Resource Tagging - All resources properly tagged for governance

🚀 Performance Optimization

Lambda Optimization

• Memory Allocation - Optimized for image processing (1024MB default)
• Timeout - Set to 60 seconds for large images
• Runtime - Python 3.12 for best performance
• Layer Caching - Pillow layer cached across invocations

S3 Optimization

• Transfer Acceleration - Can be enabled for faster uploads
• Multipart Upload - Automatic for large files
• Versioning - Enabled for data protection
• Lifecycle Policies - Can be added for cost optimization

🧪 Testing

Unit Testing

# Test Lambda function locally
cd lambda
python -m pytest test_lambda_function.py

Integration Testing

# Upload test images
aws s3 cp test-images/ s3://your-upload-bucket/ --recursive

# Verify processing
aws s3 ls s3://your-processed-bucket/

Load Testing

# Upload multiple images simultaneously
for i in {1..10}; do
  aws s3 cp test-image.jpg s3://your-upload-bucket/test-$i.jpg &
done
wait

🔄 CI/CD Integration

GitHub Actions Example

name: Deploy Image Processor
on:
  push:
    branches: [main]
jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Setup Terraform
        uses: hashicorp/setup-terraform@v1
      - name: Build Layer
        run: ./scripts/build_layer_simple.sh
      - name: Deploy
        run: |
          cd terraform
          terraform init
          terraform apply -auto-approve

🧹 Cleanup

Destroy Infrastructure

# Using script
./scripts/destroy.sh

# Or manually
cd terraform
terraform destroy

Clean Docker Resources

# Remove unused Docker images
docker system prune -a

🤝 Contributing

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

📝 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

• AWS Lambda Documentation
• Pillow (PIL) Documentation
• Terraform AWS Provider
• Docker Documentation

📞 Support

If you encounter issues or have questions:

• 🐛 Issues: GitHub Issues
• 💬 Discussions: GitHub Discussions
• 📧 Email: vanshshah174@gmail.com

🎯 Roadmap

• Video Processing - Add support for video thumbnail generation
• AI Integration - Image recognition and tagging
• API Gateway - REST API for direct uploads
• Web Interface - Simple upload interface
• Batch Processing - Process existing S3 objects
• Custom Watermarks - Add watermark support
• EXIF Processing - Preserve/modify image metadata

---

⭐ Star this repository if it helped you build awesome serverless image processing! ⭐