Cannot create PVC for ORDS access logs and missing retention policy support

[smithmiro7-ui]

I couldn’t find a dedicated place to submit feature requests, so I’m posting it here.

Based on our company policy, we are required to retain logs for 3 months. When running ORDS in a VM environment, our access logs grow to approximately 50 GB per month (we handle millions of requests per day).

Currently, there is no built-in way to control log retention, and there is also no clear support for using persistent storage for access logs in containerized environments.

Requested Features:

Log Retention Policy

Add a configuration option (e.g., retention flag/setting) for ORDS access logs

Allow automatic cleanup or rotation based on a defined retention period (e.g., 3 months)

PVC Support for Access Logs

Enable configuration to store ORDS access logs on a Persistent Volume Claim (PVC)

This is required for Kubernetes deployments to ensure logs are persisted and managed outside the container lifecycle

Use Case:

Retain logs for a fixed duration (e.g., 3 months)

Automatically delete or rotate older logs

Persist logs reliably in Kubernetes environments using PVCs

Prevent excessive disk usage in high-traffic systems

Concerns:
Ephemeral storage may fill up quickly due to large access logs (today, we use resource limits range to avoid this type of problem (CPU, RAM, and ephemeral storage).

Disk pressure on worker nodes could lead to pod eviction or instability (cluster wide effect)

Potential impact on other workloads running on the same nodes

[marcstef-oracle]

Thanks for the request — this is a valid use case.

At the moment, OrdsSrvs does not support PVC-based access logs or retention policies. A common approach in Kubernetes is to handle logs outside the container using external collection/retention mechanisms.

Have you already evaluated this approach in your environment?

Thanks,
Marco

[smithmiro7-ui]

@marcstef-oracle thank you for the explanation,
We are currently using Red Hat OpenShift Logging 6.4 and forwarding logs from stdout to Splunk.
ClusterLogForwarder is currently in use; however, it lacks the capability to forward logs from a specific file-based source.. While sidecar-based solutions exist for this purpose, they may introduce additional concerns related to ephemeral storage usage, which could potentially lead to disk pressure on the node.

We conducted a pressure test that generated approximately 400 requests per second, resulting in about 5 MB of logs per minute. This translates to roughly 7.2 GB per day, which is a significant amount of ephemeral storage.

If it is possible to redirect the logs to stdout, it would be a valid solution and would help avoid the risk of disk pressure.

From my understanding, kubelet’s default log retention applies only to stdout/stderr logs, which are written on the host under a specific path (/var/log/containers). This is where all container stdout logs are stored, and kubelet automatically rotates these log files by default.

OpenShift default configuration
https://access.redhat.com/solutions/4924281
containerLogMaxSize: the rotation is being done by default at ~ 50M
containerLogMaxFiles: maintaining 5 files per container
being 250M as maximum for container on the disk.

Kubernetes default configuration
https://kubernetes.io/docs/reference/config-api/kubelet-config.v1beta1/
containerLogMaxSize: the rotation is being done by default at ~ 10Mi
containerLogMaxFiles: maintaining 5 files per container
being 50Mi as maximum for container on the disk.

[marcstef-oracle]

Thanks for the update.

Just to clarify the current setup:

• general pod/application logs are already written to stdout
• ORDS access logs are a separate mechanism, to file, configured with standalone.access.log
• ORDS also provides standalone.access.log.retainDays

To better understand your use case:

• In your current setup, are ORDS access logs also sent to Splunk, or only the general runtime logs?
• How do you currently collect ORDS access logs (not kubernetes)?
• Have you already evaluated whether standalone.access.log.retainDays covers the retention requirement?

thank you

[vova123412]

Hi @marcstef-oracle, Thank you for following up..

A few clarifications from our side:

Regarding standalone.access.log.retainDays: ORDS itself does support this setting in its configuration files. However, in the current Oracle Database Operator CRD, I do not see standalone.access.log.retainDays exposed. In ordssrvs_types.go, I only see enable.standalone.access.log, I do not see standalone.access.log.retainDays represented there.

From a retention perspective, this is important for us because we have around 15 ORDS instances. In the worst case, keeping 1 day of access logs could consume roughly 105 GB of ephemeral storage, which is significant on nodes with 200 GB capacity and could become problematic during a full migration to k8s. If hourly retention were possible, or if standalone.access.log.retainDays could accept a decimal value, that would be much more practical for our use case.

On log shipping: because we are using ephemeral storage rather than a PVC, I have not found a workable way to forward these file-based access logs without some in-pod helper.(sidecar) At the moment, I see two possible approaches:

use the Splunk OpenTelemetry Collector, or
use a lightweight container(busybox) to read the log files and write them to stdout, then rely on OpenShift Logging to forward them to Splunk.

If the second option is considered valid, I think it would be preferable, because with OpenShift Logging via Vector or Fluentd we would keep flexibility to route the logs to multiple backends, such as Splunk or Datadog.

For context, in our current VM-based environment we use a Splunk agent that tails a specific filesystem path and forwards those ORDS access logs to Splunk.

[marcstef-oracle]

Thanks for the suggestions and the clarification. We will evaluate these enhancement requests for a future release after 2.2.

[marcstef-oracle]

Hi,

PVC persistence for ORDS access logs has been published in the ordssrvs_development branch.
Access logs can be stored on a PersistentVolume through the access log persistence configuration:
https://github.com/oracle/oracle-database-operator/blob/ordssrvs_development/docs/ordsservices/access_log.md

Log retention is supported through the ORDS setting standalone.access.log.retainDays.
Please note that this value is expressed in days, as defined by ORDS.

Please try it from the ordssrvs_development branch.

Thanks, Marco