cc-lib

Common ClusterCockpit golang packages providing reusable components for building HPC monitoring and metric collection applications.

Overview

cc-lib is a collection of Go packages developed for the ClusterCockpit project. These packages provide essential functionality for:

• Metric Collection: Receivers for various protocols (IPMI, Redfish, Prometheus, etc.)
• Data Processing: Message processing pipelines, resampling, and transformations
• Data Storage: Sinks for InfluxDB, NATS, Prometheus, and more
• Configuration: Flexible configuration management with validation
• Utilities: Caching, logging, topology detection, and helper functions

The library is designed to be modular, allowing you to use individual packages as needed in your own projects.

Packages

Core Messaging & Processing

Package	Description
ccMessage	Message types and protocols for metrics, logs, events, and control messages
messageProcessor	Expression-based message processing and transformation pipeline
schema	JSON schema definitions and validation for ClusterCockpit data structures

Metric Collection

Package	Description
receivers	Metric receivers for IPMI, Redfish, Prometheus, and other protocols
ccTopology	System topology detection and hardware information gathering

Data Storage & Output

Package	Description
sinks	Metric sinks for InfluxDB, NATS, Prometheus, HTTP, and file output
resampler	Data resampling and aggregation utilities

Configuration & Logging

Package	Description
ccConfig	Configuration file management with hot-reloading support
ccLogger	Structured logging with multiple output levels

Utilities

Package	Description
lrucache	Thread-safe LRU cache with TTL support and HTTP middleware
hostlist	Hostlist expansion for compact host specifications (e.g., node[1-10])
ccUnits	Unit conversion and handling for metrics
util	Common utility functions and helpers
runtime	Runtime environment setup, privilege dropping, and systemd integration

Installation

go get github.com/ClusterCockpit/cc-lib/v2

Requirements:

• Go 1.24.0 or higher

Quick Start

Using the LRU Cache

import "github.com/ClusterCockpit/cc-lib/v2/lrucache"

cache := lrucache.New(1000) // maxmemory in arbitrary units

value := cache.Get("key", func() (interface{}, time.Duration, int) {
    // Compute expensive value
    result := fetchFromDatabase()
    return result, 10 * time.Minute, len(result)
})

Expanding Hostlists

import "github.com/ClusterCockpit/cc-lib/v2/hostlist"

hosts, err := hostlist.Expand("node[1-10],gpu[1-4]")
// Returns: [gpu1, gpu2, gpu3, gpu4, node1, node2, ..., node10]

Creating Messages

import "github.com/ClusterCockpit/cc-lib/v2/ccMessage"

msg, err := ccMessage.NewMessage(
    "temperature",
    map[string]string{"hostname": "node01", "type": "node"},
    map[string]string{"unit": "degC"},
    map[string]interface{}{"value": 45.2},
    time.Now(),
)

Using Configuration Management

import "github.com/ClusterCockpit/cc-lib/v2/ccConfig"

config := ccConfig.New()
config.AddFile("config.json")

// Access configuration
value := config.Get("key")

// Watch for changes
config.Watch(func() {
    log.Println("Configuration changed")
})

Documentation

• API Documentation: pkg.go.dev/github.com/ClusterCockpit/cc-lib/v2
• Package READMEs: Each package has its own README with detailed documentation and examples

Package Documentation

• ccConfig - Configuration management
• ccLogger - Logging utilities
• ccMessage - Message types and protocols
• ccTopology - System topology detection
• ccUnits - Unit conversion
• hostlist - Hostlist expansion
• lrucache - LRU cache with TTL
• messageProcessor - Message processing
• receivers - Metric receivers
• runtime - Runtime environment setup
• schema - JSON schema validation
• sinks - Metric sinks
• util - Utility functions

Testing

Run all tests:

go test ./...

Run tests with coverage:

go test -cover ./...

Run tests for a specific package:

go test -v ./lrucache

Contributing

Contributions are welcome! Please feel free to submit issues and pull requests.

Development

1. Clone the repository
2. Make your changes
3. Run tests: go test ./...
4. Submit a pull request

Projects Using cc-lib

• cc-metric-collector - Metric collection daemon
• cc-metric-store - Metric storage backend
• ClusterCockpit - Web interface and monitoring system

License

This project is licensed under the MIT License - see the LICENSE file for details.

Copyright (C) NHR@FAU, University Erlangen-Nuremberg.

Acknowledgments

Developed by the National High Performance Computing (NHR) center at FAU.

Additional contributors:

• Holger Obermaier (NHR@KIT)