[Bug]: AWS (Amazon Web Services) connection is failing in OpenRAG version 0.4.0-dev5.

[SakshiDhoni-ibmer]

OpenRAG Version

0.4.0 dev5

Deployment Method

uvx (uvx openrag)

Operating System

26.3.1

Python Version

3.13

Affected Area

Ingestion (document processing, upload, Docling)

Bug Description

AWS (Amazon Web Services) connection test is failing in OpenRAG version 0.4.0-dev5, while the same configuration was working correctly in 0.4.0-dev2.
The system is unable to establish or verify a successful AWS connection despite valid credentials and proper configuration.

Steps to Reproduce

1. Navigate to the OpenRAG project directory
2. Add AWS credentials in the .env file:
   AWS_ACCESS_KEY_ID=YOUR_ACCESS_KEY
   AWS_SECRET_ACCESS_KEY=YOUR_SECRET_KEY
   AWS_DEFAULT_REGION=us-east-1
3. Restart all services.
4. Ensure the same credentials are available in the in AWS configuration Advance setup as well.
5. Attempt to test/verify AWS connection.

Expected Behavior

The AWS connection should be successfully established and verified, as it was in version 0.4.0-dev2.

Actual Behavior

Unable to verify successful connection, AWS option is not shown in add knowledge list to add document.

Relevant Logs

Screenshots

No response

Additional Context

No response

Checklist

• I have searched existing issues to ensure this bug hasn't been reported before.
• I have provided all the requested information.

[prasanthcaibmcom]

@ricofurtado Can you help take a look ?

[edwinjosechittilappilly]

AWS S3 and COS could only be activated in cloud SaaS environment wrt to product roadmap

[ricofurtado]

The AWS connector code has not changed between dev2 and dev5. I suspect the bug already existed in dev2.

[ricofurtado]

The bug was present from day one in PR #1168 and was never caught because testing was presumably done with admin/full-S3 credentials that do have s3:ListAllMyBuckets. It carried unchanged through dev2 → dev5.

@edwinjosechittilappilly do you have another another take in this issue?

[ricofurtado]

Pull request created in case we include it in the release 0.4.0
#1231

[ricofurtado]

There is a behaviour question(maybe product can anwer it?):

• If the credentials are valid but don’t have s3:ListAllMyBuckets, “Test connection” should still succeed?
  In this case, the UI should not block saving just because bucket listing failed.

• If bucket listing is unavailable, do we need to explain why in plain terms? E.g. “Credentials are valid, but this IAM user can’t list all buckets. Enter/select bucket names manually.”

-After saving, the connection should only be treated as usable if it has either:
ListBuckets permission, so auto-discovery can work, or
explicit bucket_names, so the connector can operate without bucket discovery.

So the practical fix is to split today’s combined step into two states:

• validate credentials
• try to discover buckets
• Bucket discovery should be best-effort, not a prerequisite for saving?
  If discovery fails with AccessDenied, we should still save the connection and guide the user into a manual bucket-name path instead of surfacing a generic failure?

[prasanthcaibmcom]

@SakshiDhoni-ibmer Please revalidate.

[edwinjosechittilappilly]

I agree @ricofurtado But @SakshiDhoni-ibmer can you send me the env that you had used ?
We had set it in such a way that s3 and AWS is enabled only if IBM_AUTH_ENABLED is True.
Can you check that once ?

[SakshiDhoni-ibmer]

AWS_ACCESS_KEY_ID=AKIA6ODU2BPZA6DVQBGU
AWS_SECRET_ACCESS_KEY=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
AWS_DEFAULT_REGION=us-east-1

@edwinjosechittilappilly this is my .env file data.
IBM_AUTH_ENABLED isn't in there. I'll add that now, revalidate the S3/AWS behavior, and get back to you with the results shortly. Thanks for pointing that out!

[SakshiDhoni-ibmer]

@edwinjosechittilappilly I’ve updated my .env file as suggested, but the AWS option is still not visible. Could you please take a look at my updated configuration to see if I missed anything?

openrag_env.txt

[SakshiDhoni-ibmer]

@ricofurtado I check that even aws s3 ls s3://bucket-name fails with AccessDenied. However, I can still cp files directly.