AWS Lambda Logs HTTP Destination Extension

A Rust-based AWS Lambda extension that captures Lambda function logs and forwards them to a custom HTTP endpoint. This extension enables real-time log monitoring and analysis outside of the AWS CloudWatch ecosystem.

🚀 Features

• Automatic log capture: Intercepts all Lambda function logs
• HTTP forwarding: Sends logs to any custom HTTP endpoint
• Multiple authentication methods: Supports Basic Auth, Bearer Token, API Key, and no authentication
• Multi-architecture support: Compatible with x86_64 and ARM64
• Test mode: Allows testing without sending real data
• Enriched metadata: Includes Lambda function information (name, version, region, architecture, memory)
• Robust error handling: Detailed logging and failure management
• Flexible configuration: Complete configuration via environment variables

📋 Requirements

• Rust 1.70+
• AWS CLI (for deployment)
• cargo-lambda for Lambda builds
• jq for JSON response processing

🛠️ Installation

Install dependencies

# Install cargo-lambda
cargo install cargo-lambda

# Install cross-compilation targets
rustup target add x86_64-unknown-linux-gnu
rustup target add aarch64-unknown-linux-gnu

Local build

make build_local

Lambda build

# For x86_64
make build_lambda_x86

# For ARM64
make build_lambda_arm

⚙️ Configuration

Copy the example file and configure environment variables:

cp .env.example .env

Required environment variables

Variable	Description	Required
HTTP_ENDPOINT	Target HTTP endpoint URL	✅

Optional environment variables

Variable	Description	Default
HTTP_AUTH_TYPE	Authentication type (basic, bearer, apikey, none)	none
HTTP_AUTH_USERNAME	Username for Basic Auth	-
HTTP_AUTH_CREDENTIALS	Password/Token/API Key	-
HTTP_AUTH_HEADER_NAME	Header name for API Key	-
HTTP_TIMEOUT_SECONDS	HTTP timeout in seconds	30
HTTP_TEST_MODE	Test mode (true/false)	false

Configuration examples

Basic Authentication

export HTTP_ENDPOINT="https://api.example.com/logs"
export HTTP_AUTH_TYPE="basic"
export HTTP_AUTH_USERNAME="username"
export HTTP_AUTH_CREDENTIALS="password"

Bearer Token

export HTTP_ENDPOINT="https://api.example.com/logs"
export HTTP_AUTH_TYPE="bearer"
export HTTP_AUTH_CREDENTIALS="your-jwt-token"

API Key

export HTTP_ENDPOINT="https://api.example.com/logs"
export HTTP_AUTH_TYPE="apikey"
export HTTP_AUTH_HEADER_NAME="X-API-Key"
export HTTP_AUTH_CREDENTIALS="your-api-key"

🚀 Deployment

1. Build and deploy layer

# For x86_64
make build_lambda_x86
REGION=us-east-1 make deploy_cli_x86

# For ARM64
make build_lambda_arm
REGION=us-east-1 make deploy_cli_arm

2. Add permissions

# By organization
REGION=us-east-1 ORG_ID=your-org-id make add_permissions_x86

# By specific account
REGION=us-east-1 ACCOUNT_ID=123456789012 make add_permissions_by_account_x86

3. Configure your Lambda function

Add the layer to your Lambda function and configure environment variables:

aws lambda update-function-configuration \
  --function-name your-function \
  --layers arn:aws:lambda:us-east-1:your-account:layer:aws-lambda-logs-http-destination:1 \
  --environment Variables='{"HTTP_ENDPOINT":"https://api.example.com/logs","HTTP_AUTH_TYPE":"bearer","HTTP_AUTH_CREDENTIALS":"your-token"}'

📊 Data Format

Logs are sent in JSON format with the following structure:

{
  "logs": [
    {
      "timestamp": "2024-01-15T10:30:00Z",
      "level": "INFO",
      "message": "Log content",
      "source": "lambda_function",
      "function_name": "my-function",
      "function_version": "$LATEST",
      "function_memory_size": "128",
      "aws_region": "us-east-1",
      "architecture": "x86_64",
      "log_type": "function"
    }
  ],
  "source": "aws-lambda-extension",
  "timestamp": "2024-01-15T10:30:00Z"
}

🧪 Test Mode

To test the extension without sending real data:

export HTTP_TEST_MODE="true"

In test mode, the extension:

• Intercepts and processes logs normally
• Shows detailed information in logs
• Does not send data to the real HTTP endpoint
• Simulates successful sending

🔧 Makefile Commands

Command	Description
make build_local	Local build for development
make build_x86	Build for x86_64
make build_arm	Build for ARM64
make build_lambda_x86	Build and package layer for x86_64
make build_lambda_arm	Build and package layer for ARM64
make deploy_cli_x86	Deploy x86_64 layer
make deploy_cli_arm	Deploy ARM64 layer
make add_permissions_x86	Add permissions by organization (x86_64)
make add_permissions_arm	Add permissions by organization (ARM64)
make add_permissions_by_account_x86	Add permissions by account (x86_64)
make add_permissions_by_account_arm	Add permissions by account (ARM64)
make remove_x86_version	Remove specific version (x86_64)
make remove_arm_version	Remove specific version (ARM64)
make clean	Clean build artifacts

🏗️ Architecture

The extension consists of the following modules:

• main.rs: Entry point and extension configuration
• config.rs: Configuration and environment variable handling
• logs_extension.rs: Main log processor
• http_client.rs: HTTP client with authentication support
• log_transformer.rs: Lambda log to JSON format transformation

🔍 Troubleshooting

Extension logs

Extension logs appear in CloudWatch Logs with the [EXTENSION] prefix.

Common issues

1. Configuration error: Verify that HTTP_ENDPOINT is configured
2. HTTP timeout: Adjust HTTP_TIMEOUT_SECONDS if needed
3. Authentication issues: Verify credentials and auth type
4. Wrong architecture: Make sure to use the correct layer (x86_64 vs ARM64)

Debugging

Enable test mode to see the complete payload:

export HTTP_TEST_MODE="true"

📝 License

This project is licensed under the MIT License. See the LICENSE file for details.

🤝 Contributing

Contributions are welcome. Please:

1. Fork the project
2. Create a feature branch (git checkout -b feature/new-feature)
3. Commit your changes (git commit -am 'Add new feature')
4. Push to the branch (git push origin feature/new-feature)
5. Open a Pull Request

📞 Support

If you encounter any issues or have questions, please open an issue in the repository.