v1.12.0

⚠️ Compatibility Notice

This version has upgraded ilogtail, after that container log collection is now supported under the containerd2 runtime.
However, this requires changes to existing deployment and configuration files, including environment variables, mount paths, and pipeline configurations. Without these changes, the system will not function properly.

For users on the default configuration, updates will be automatically applied with the new version.
For users with custom collection configurations, please carefully review and manually apply the required adjustments.

Changes include:

1. New environment variable:
   LOGTAIL_MODE=true
2. Mount path adjustments:
   • checkpoint: /usr/local/ilogtail/checkpoint → /usr/local/loongcollector/checkpoint
   • user_config: /usr/local/ilogtail/user_yaml_config.d → /usr/local/loongcollector/config/local
   • logger-config: /usr/local/ilogtail/plugin_logger.xml → /usr/local/loongcollector/conf/plugin_logger.xml
3. (Manual update required) Pipeline changes:
   • Add global.UsingOldContentTag = true to all pipelines
   • The legacy file collector file_log has been deprecated and replaced by input_file.
     Configuration reference: input-file documentation

Enhancements

• [BREAKING CHANGE] Log collection is now compatible with the containerd2 runtime (deployment and configuration changes required).

Bug Fixes

• Fixed an issue where service overview alert status was not filtered by namespace.
• Fixed incorrect display of anomaly data in fault reports.
• Added compatibility for apo-one-agent with Alpine Linux (musl-based) base images.
• Fixed Slice Out of Bound error in profile-agent on older Linux versions.

v1.10.1

⚠️ Compatibility Notice

This version introduces a cluster identifier in data collection to support cluster-level data isolation. If you're upgrading from a previous version:

• Existing data will be assigned to a "default (empty)" cluster.
• Newly collected data will be assigned to the configured cluster during installation.

While most features remain compatible and data display is unaffected, a few features that rely on historical data across clusters may show gaps or inconsistencies. Potentially affected features:

• Alert rules that depend on historical baselines
• AI-powered root cause analysis based on historical patterns

New Features

• Multi-Cluster Support: Enables data isolation and filtering by cluster. Users can specify the target cluster during installation to support precise queries in multi-cluster environments.
• AI-Powered Alert Diagnosis Direction: Automatically identifies potential root causes of alerts using large language models, guiding users quickly toward fault localization.
• Root Cause Analysis Report: After completing alert analysis, the system generates a structured root cause report to help users understand and trace the origin of issues.
• Enhanced Alert Analysis Workflow: Now supports detection of memory leak scenarios and offers deeper insights into issues like high CPU usage, memory pressure, and network latency on hosts.

Enhancements

• Faster Log Queries: Significantly improved the performance of full log retrieval, ensuring faster access to large datasets.
• Improved Alert Analysis Accuracy: Refined AI workflow strategies to enhance diagnosis precision and scenario coverage.
• Enhanced Data Group Management: Supports nested sub-groups to better reflect organizational structures; allows grouping by cluster, namespace, service name, and more for flexible data control.

Bug Fixes

• Fixed an issue where data could not be filtered properly on traditional servers without namespaces.
• Resolved an issue where the polaris metrics might be lost after extended operation.
• Fixed errors occurring when service endpoints contain special characters.
• Resolved time zone inconsistencies that caused baseline misalignment in incident reports.

v1.9.0

New Features

• Added alarm event filtering to help users quickly locate key alerts. Alarm detail descriptions have also been improved for better clarity.
  

• Alarms related to databases and middleware can now be correlated in alarm analysis, enhancing root cause accuracy in high-volume alert scenarios.

Enhancements

• Improved the styling of the left-side menu to ensure hover interactions align accurately with menu items, enhancing overall user experience.
  

• Improved display of the full logs page in small windows for better readability and usability.
  

• Refined log formatting in the error analysis workflow, making issue diagnosis more intuitive.

• Moved theme and language switch options to the “Preferences” menu in the top-right corner for a cleaner UI.

Bug Fixes

• Fixed an issue where alerts failed to trigger notifications due to incorrect validity checks.
• Fixed a problem where fault link data could not be retrieved when using the “Data Integration” installation method.
• Fixed an issue where process information was missing when collecting logs from traditional servers.
• Fixed an issue where applications without monitoring enabled would incorrectly show a "no data access" error.

Others

• Added support for Alibaba Cloud ARMS Java agent version 4.x.

Full Changelog: v1.8.0...v1.9.0

v1.8.0

New Features

• Light & Dark Mode Toggle: Introduced a theme toggle allowing users to switch between light and dark modes for a more personalized and comfortable user experience.

• Alert Details Page: Added a dedicated page for alert details, visualizing the full lifecycle of an alert—from triggering to resolution. Users can now navigate directly from alert notifications with a single click.

• Enhanced Root Cause Analysis for Alerts: Expanded root cause analysis capabilities for more alert types. Actionable insights and suggestions are provided to help users quickly diagnose and resolve issues.

Enhancements

• apo-otel-collector Stability Enhancements: Optimized queue configurations to reduce memory usage and prevent frequent out-of-memory (OOM) crashes.
• Support for cri-o Runtime Metadata: apo-otel-collector now supports enriching telemetry data with pod labels when using the cri-o container runtime, improving observability and context.

Bug Fixes

• Fixed an issue where adding new data sources in the Integration Center could occasionally result in an error.
• Resolved a problem where users might encounter frequent errors after session authentication expired.

Others

• SkyWalking Java Agent Compatibility: Improved compatibility with the SkyWalking Java agent. Versions 6.1 and above are now officially supported for trace-sidecar mode. Versions 8.4 and above are supported for trace-collector mode.

Full Changelog: v1.7.0...v1.8.0

v1.7.0

New Features

• Added role-based user permission control. Users can now be assigned different system permissions through role assignments.

Enhancements

• Workflow: Significantly improved the analysis speed of alert validity analysis and alert root cause analysis workflows, resulting in reduced token consumption. Enhanced the display of workflow execution pages.

• Alert Events: Added alert noise reduction rate metric and overview of alerts and resolved events. Optimized table display.

• Service Details: Improved page layout and display for better user experience.

Bug Fixes

• Fixed an issue where exception items did not automatically expand on the service details page.
• Fixed an issue that prevented calling large language models in offline deployment environments.
• Fixed multiple frontend components that were not functioning properly in offline environments.

Full Changelog: v1.6.0...v1.7.0

v1.6.0

⚠️ Breaking Changes

• Replaced HostPath with PV for persistence, improving maintainability. If you have created or modified workflows, it is recommended to back up PostgreSQL before upgrading; otherwise, workflow data will be reset.
• Data persistence changes: Grafana and the apo-backend database now use PostgreSQL by default. Please refer to the "Installation Guide" -> "Upgrade Guide" on the official website for data backup and upgrade instructions.
• Helm Charts configuration changes: The fixed port in the baseurl for workflows has been optimized to an editable port. During the upgrade, please modify the corresponding variables in the values file when upgrading.

New Features

• Further optimized the "Alert Validation Analysis" and "Alert Simple Root Cause Analysis" workflows, improving recognition accuracy and effectiveness.

• The service list in the Service Overview now supports sorting by different metrics. You can click the table headers to sort by the selected metric.
  

• Added support for jumping from TraceID in Fault Traces to Fault Logs.
  

• Added a TraceID filter condition in Fault Logs.
  

Enhancements

• Fault Traces now default to displaying fault data, with a new filter for "Slow & Error" state.
  

• OneAgent now supports automatic monitoring of services in newly created namespaces.

• Added configuration options for the frequency and sampling method of alert validation checks.

• The alert list now automatically updates with the latest data and statuses.

• Improved the layout of the workflow page.

Bug Fixes

• Fixed an issue in apo-backend where the polaris-analyzer memory was not being cleared in a timely manner.
• Fixed an issue where OneAgent's injection of Trace agents could overwrite JVM configurations.
• Fixed an issue where downstream services were missing in trace links during broken chain scenarios.
• Fixed an issue where dashboards in service details did not match the corresponding services.

Others

• Added the prometheus-remote-write receiver to apo-otel-collector.

Full Changelog: v1.5.2...v1.6.0

v1.5.1

Changelog

• Improve data loading performance for the Services Overview page
• Improved loading speed of the Workflows page
• Fixed empty charts when handling too many services in the Services Overview page
• Resolved authentication sync issue between Workflows and APO user sessions

Full Changelog: v1.5.0...v1.5.1

v1.5.0

New Features

• Workflows: A workflow orchestration platform designed for the observability domain, preset with various common workflows to help users solve everyday problems. Try it in Workflows tab or Alert Events tab.
• Alert Events: Add a tab for displaying alert events where workflows are integrated.
• Data Integration: APO now supports integrating multiple external data sources through a visual interface. The current version supports integrating trace data from existing OpenTelemetry and SkyWalking systems.
• Trace Sampling: The trace probe in APO OneAgent now has data sampling enabled by default, effectively reducing the data volume.

Enhancements

• Added support for obtaining request latency metrics for gateway-type applications when integrating with SkyWalking probes.

Bug Fixes

• Fixed an issue where odiglet on traditional servers was assigning the same service name 'java' to different Java applications.
• Fixed an issue in alarm analysis where sorting by fault contribution would sometimes fail due to missing service information labels in the metrics.

New Contributors

• @CMT0123 made their first contribution in #202

Full Changelog: v1.4.0...v1.5.0

v1.4.0

New Features

• Introduced the "Team" feature, now users can be grouped into teams.
• Introduced the "Data Group" feature, now different teams or users can be granted different data group viewing permissions.

Enhancements

• Optimized the display style of nodes in the upstream and downstream dependency topology.

Bug Fixes

• Fixed the issue of incorrect instance name format in traditional server environments.

Full Changelog: v1.3.0...v1.4.0

v1.3.0

⚠️ Breaking Change

• The Helm charts for integrating with external single-node VictoriaMetrics have changed.

If you previously integrated with external VictoriaMetrics during the installation of APO, please refer to the "Production Environment Deployment Recommendations" documentation to update your existing helm values file before using helm upgrade, otherwise it may lead to unusable metric data.

New Features

• Support for integrating external alerts into the APO platform, automatically associating relevant applications, and performing alert diagnostics through alert analysis functionality.

Enhancements

• Support for associating database/middleware alerts with relevant services.
• Experimental Feature: Support for using external VictoriaMetrics clusters during installation.

Full Changelog: v1.2.0...v1.3.0