Integrate Application Logs to CloudWatch

This tutorial explains how to stream Kubernetes pod logs to AWS CloudWatch for full-text search capabilities. While Qovery provides log access through its Console and CLI using Loki, that solution has limitations for direct querying and full-text search functionality.

Loki is not a full-text search engine. It is a log aggregation system designed for Qovery’s interface rather than direct queries.

Prerequisites

Qovery cluster running on AWS EKS
Access to AWS IAM Console
Basic knowledge of Helm charts

Step 1: AWS Permissions Setup

Create IAM Policy

Create an IAM policy named fluent-bit-write-policy with these permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:DescribeLogStreams",
        "logs:PutRetentionPolicy",
        "logs:PutLogEvents"
      ],
      "Resource": "*"
    }
  ]
}

Create IAM User

Create an IAM user
Attach the fluent-bit-write-policy to the user
Generate access credentials for later use

Step 2: Deploy Fluent Bit with Helm

Add AWS EKS Helm Repository

In Qovery Console, navigate to your environment
Add a new Helm repository:
- Name: eks
- Kind: HTTPS
- URL: https://aws.github.io/eks-charts

Create Helm Service

Configure the Helm service with the following settings: Basic Configuration:

Chart: aws-for-fluent-bit
Version: 0.1.21
Enable cluster-wide resources: Yes

Values Override:

cloudWatch:
  enabled: true
  region: us-east-1  # Change to your region
  logGroupName: /aws/eks/fluent-bit

env:
  - name: AWS_ACCESS_KEY_ID
    valueFrom:
      secretKeyRef:
        name: fluent-bit-secrets
        key: AWS_ACCESS_KEY
  - name: AWS_SECRET_ACCESS_KEY
    valueFrom:
      secretKeyRef:
        name: fluent-bit-secrets
        key: AWS_SECRET_ACCESS_KEY

Step 3: Store AWS Secrets

Add two service-scoped secret variables:

AWS_ACCESS_KEY: Your IAM user access key ID
AWS_SECRET_ACCESS_KEY: Your IAM user secret access key

Always use secret variables (not regular environment variables) for AWS credentials to ensure they are encrypted.

Step 4: Deploy

Save your Helm configuration
Click the Play button to trigger deployment
Wait for the deployment to complete
Verify the pods are running successfully

Step 5: Access CloudWatch Logs

Using CloudWatch Logs Insight

Navigate to AWS CloudWatch Console
Go to Logs → Insights
Select the fluent-bit log group
Use CloudWatch Logs Insights query syntax to filter and search logs

Example Queries

Find all error logs:

fields @timestamp, @message
| filter @message like /ERROR/
| sort @timestamp desc
| limit 100

Search by application name:

fields @timestamp, kubernetes.pod_name, @message
| filter kubernetes.namespace_name = "your-namespace"
| sort @timestamp desc
| limit 100

Count logs by pod:

stats count() by kubernetes.pod_name
| sort count desc

Benefits of CloudWatch Integration

Full-Text Search

Perform complex queries across all your logs

Long-Term Retention

Configure custom retention policies up to 10 years

Advanced Analytics

Use CloudWatch Insights for log analysis

Alerting

Set up metric filters and alarms on log patterns

Troubleshooting

Logs not appearing in CloudWatch

Verify IAM permissions are correct
Check Fluent Bit pod logs for errors
Ensure AWS credentials are properly configured as secrets
Verify the log group name matches your configuration

Access denied errors

Confirm IAM policy includes all required permissions
Check that the IAM user credentials are valid
Verify the region in your configuration matches your cluster region

High costs

Configure log retention policies to reduce storage costs
Use log filtering to send only relevant logs
Consider using S3 archiving for older logs

View Logs

Access logs through Qovery Console

Helm Services

Deploy applications with Helm

Observability

Learn about observability options

Advanced Settings

Configure service advanced settings

Get Started

Platform Capabilities

How-to Guides

Tutorials

Troubleshooting

Useful Resources

Integrate Application Logs to CloudWatch

Prerequisites

Step 1: AWS Permissions Setup

Create IAM Policy

Create IAM User

Step 2: Deploy Fluent Bit with Helm

Add AWS EKS Helm Repository

Create Helm Service

Step 3: Store AWS Secrets

Step 4: Deploy

Step 5: Access CloudWatch Logs

Using CloudWatch Logs Insight

Example Queries

Benefits of CloudWatch Integration

Full-Text Search

Long-Term Retention

Advanced Analytics

Alerting

Troubleshooting

View Logs

Helm Services

Observability

Advanced Settings

​Prerequisites

​Step 1: AWS Permissions Setup

​Create IAM Policy

​Create IAM User

​Step 2: Deploy Fluent Bit with Helm

​Add AWS EKS Helm Repository

​Create Helm Service

​Step 3: Store AWS Secrets

​Step 4: Deploy

​Step 5: Access CloudWatch Logs

​Using CloudWatch Logs Insight

​Example Queries

​Benefits of CloudWatch Integration

Full-Text Search

Long-Term Retention

Advanced Analytics

Alerting

​Troubleshooting

​Related Documentation

View Logs

Helm Services

Observability

Advanced Settings

Prerequisites

Step 1: AWS Permissions Setup

Create IAM Policy

Create IAM User

Step 2: Deploy Fluent Bit with Helm

Add AWS EKS Helm Repository

Create Helm Service

Step 3: Store AWS Secrets

Step 4: Deploy

Step 5: Access CloudWatch Logs

Using CloudWatch Logs Insight

Example Queries

Benefits of CloudWatch Integration

Troubleshooting

Related Documentation