Skip to content

Commit a176bd7

Browse files
committed
fix: correct data format parsing and make start time parameter required (#2)
* fix: correct data format parsing and make start time parameter required * fix: treat error in the main file by properly checking the return value of file.Close()
1 parent 95f46f9 commit a176bd7

7 files changed

Lines changed: 357 additions & 169 deletions

File tree

README.md

Lines changed: 26 additions & 101 deletions
Original file line numberDiff line numberDiff line change
@@ -42,42 +42,31 @@ cd enemeter-data-processing
4242
make build
4343
```
4444

45-
## Data Format
46-
47-
ENEMETER data is provided in CSV format with four columns:
48-
49-
1. **Time Delta (milliseconds)** - Time elapsed since the last measurement
50-
2. **Temperature (millicelsius)** - System temperature
51-
3. **Voltage (microvolts)** - Battery voltage
52-
4. **Current (nanoamperes)** - System current (positive when charging, negative when discharging)
53-
54-
Example:
55-
```
56-
0000052051,0004815430,-1275169536,0000023192
57-
0000000047,0004815430,-1275169536,0000023192
58-
0000000047,0004815790,-1276443264,0000023192
59-
```
60-
6145
## Basic Usage
6246

63-
Process a CSV file with default settings:
47+
Process a CSV file with required parameters:
6448

6549
```bash
66-
./enemeter-data-processing --input=data/data.csv
50+
./enemeter-data-processing process --input=data/esp32.csv --start="2023-04-01 08:00:00"
6751
```
6852

53+
The `--start` parameter specifies the starting time of the measurements and is required.
54+
**Important:** You must include both the date AND time of day in the format "YYYY-MM-DD HH:MM:SS".
55+
6956
Save results to a text file:
7057

7158
```bash
72-
./enemeter-data-processing --input=data/data.csv --output=report.txt
59+
./enemeter-data-processing process --input=data/esp32.csv --start="2023-04-01 08:00:00" --output=esp32_report.txt
7360
```
7461

7562
## Command-line Options
7663

77-
### Input/Output Options
64+
### Required Parameters
65+
- `--input=<path>`: Path to the input CSV file
66+
- `--start=<time>`: Start time for measurements (format: YYYY-MM-DD HH:MM:SS) - must include time of day
7867

79-
- `--input=<path>`: Path to the input CSV file (required)
80-
- `--output=<path>`: Path to save the output report (optional)
68+
### Optional Parameters
69+
- `--output=<path>`: Path to save the output report
8170
- `--format=<text|json|csv>`: Output format (default: text)
8271

8372
### Processing Options
@@ -88,8 +77,8 @@ Save results to a text file:
8877

8978
### Time Filtering Options
9079

91-
- `--start=<time>`: Start time for filtering (format: YYYY-MM-DD[THH:MM:SS])
92-
- `--end=<time>`: End time for filtering (format: YYYY-MM-DD[THH:MM:SS])
80+
- `--start=<time>`: (Required) Start time for measurements (format: YYYY-MM-DD HH:MM:SS) - must include time of day
81+
- `--end=<time>`: End time for filtering (format: YYYY-MM-DD HH:MM:SS)
9382
- `--window=<duration>`: Time window to process (e.g., 1h, 30m, 24h)
9483

9584
### Data Filtering Options
@@ -123,123 +112,59 @@ Save results to a text file:
123112
Process a CSV file and display all metrics:
124113

125114
```bash
126-
./enemeter-data-processing --input=data/data.csv
115+
./enemeter-data-processing process --input=data/esp32.csv --start="2023-04-01 08:00:00"
127116
```
128117

129118
### Memory-Efficient Processing for Large Files
130119

131120
Use streaming mode with sampling for very large files:
132121

133122
```bash
134-
./enemeter-data-processing --input=big-data.csv --stream --sample=10
123+
./enemeter-data-processing process --input=big-data.csv --start="2023-04-01 10:15:30" --stream --sample=10
135124
```
136125

137126
This processes only every 10th record, reducing memory requirements.
138127

139-
### Filtering by Time
128+
### Time-Based Analysis
140129

141-
Process data from a specific time range:
130+
To analyze data from a specific time period:
142131

143132
```bash
144-
./enemeter-data-processing --input=data.csv --start="2025-04-01" --end="2025-04-02"
145-
```
146-
147-
Process the last 24 hours of data:
133+
# Analyze data between specific dates and times
134+
./enemeter-data-processing process --input=esp32.csv --start="2023-04-01 08:00:00" --end="2023-04-02 17:30:00"
148135

149-
```bash
150-
./enemeter-data-processing --input=data.csv --window=24h
136+
# Analyze data for a specific duration
137+
./enemeter-data-processing process --input=esp32.csv --start="2023-04-01 12:45:00" --window=24h
151138
```
152139

153140
### Extracting Specific Metrics
154141

155142
Get only temperature statistics in JSON format:
156143

157144
```bash
158-
./enemeter-data-processing --input=data.csv --metric=temperature --format=json
145+
./enemeter-data-processing process --input=esp32.csv --start="2023-04-01 08:00:00" --metric=temperature --format=json
159146
```
160147

161148
Extract hourly energy consumption in CSV format:
162149

163150
```bash
164-
./enemeter-data-processing --input=data.csv --metric=energy_by_hour --format=csv
151+
./enemeter-data-processing process --input=esp32.csv --start="2023-04-01 08:00:00" --metric=energy_by_hour --format=csv
165152
```
166153

167154
### Filtering by Data Values
168155

169156
Process only records with voltage between specified values:
170157

171158
```bash
172-
./enemeter-data-processing --input=data.csv --volt-min=3500000 --volt-max=4200000
159+
./enemeter-data-processing process --input=esp32.csv --start="2023-04-01 08:00:00" --volt-min=3500000 --volt-max=4200000
173160
```
174161

175162
Process only records where temperature is above a certain threshold:
176163

177164
```bash
178-
./enemeter-data-processing --input=data.csv --min-temp=25000
165+
./enemeter-data-processing process --input=esp32.csv --start="2023-04-01 08:00:00" --min-temp=25000
179166
```
180167

181168
## Output Examples
182169

183-
### Text Output (Default)
184-
185-
```
186-
========== ENEMETER DATA PROCESSING REPORT ==========
187-
Input File: data.csv
188-
Date: 2025-04-09 10:15:32
189-
Data Points: 1253
190-
Time Range: 2025-04-08 08:30:15 to 2025-04-09 08:30:10
191-
192-
ENERGY METRICS
193-
-------------
194-
Total Energy Consumed: 156.4578 joules
195-
Average Power: 0.0548 watts
196-
Peak Power: 0.1245 watts
197-
Estimated Energy per Day: 157.9825 joules
198-
Measurement Duration: 86395.00 seconds
199-
200-
TEMPERATURE STATISTICS
201-
---------------------
202-
Minimum Temperature: 22.45 °C
203-
Maximum Temperature: 28.95 °C
204-
Average Temperature: 24.75 °C
205-
206-
... additional sections ...
207-
```
208-
209-
### JSON Output
210-
211-
```json
212-
{
213-
"TotalJoules": 156.4578,
214-
"AveragePowerWatts": 0.0548,
215-
"PeakPowerWatts": 0.1245,
216-
"JoulesPerDay": 157.9825,
217-
"DurationSeconds": 86395.0,
218-
"TemperatureStats": {
219-
"MinTempCelsius": 22.45,
220-
"MaxTempCelsius": 28.95,
221-
"AvgTempCelsius": 24.75
222-
},
223-
...
224-
}
225-
```
226-
227-
### CSV Output
228-
229-
```
230-
Metric,Value
231-
TotalJoules,156.457800
232-
AveragePowerWatts,0.054800
233-
PeakPowerWatts,0.124500
234-
JoulesPerDay,157.982500
235-
DurationSeconds,86395.00
236-
...
237-
```
238-
239-
## Contributing
240-
241-
Contributions are welcome! Please feel free to submit a Pull Request.
242-
243-
## License
244-
245-
This project is licensed under the GPL License - see the LICENSE file for details.
170+
### Text Output (Default)

cmd/analyze-csv/main.go

Lines changed: 163 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,163 @@
1+
package main
2+
3+
import (
4+
"encoding/csv"
5+
"flag"
6+
"fmt"
7+
"math"
8+
"os"
9+
"strconv"
10+
)
11+
12+
func main() {
13+
inputFile := flag.String("input", "", "Path to the CSV file to analyze")
14+
sampleSize := flag.Int("samples", 10, "Number of sample rows to display")
15+
flag.Parse()
16+
17+
if *inputFile == "" {
18+
fmt.Println("Error: Please specify an input file with --input")
19+
os.Exit(1)
20+
}
21+
22+
file, err := os.Open(*inputFile)
23+
if err != nil {
24+
fmt.Printf("Error opening file: %v\n", err)
25+
os.Exit(1)
26+
}
27+
defer func() {
28+
if closeErr := file.Close(); closeErr != nil {
29+
fmt.Printf("Warning: Error closing file: %v\n", closeErr)
30+
}
31+
}()
32+
33+
reader := csv.NewReader(file)
34+
35+
fmt.Printf("Analyzing file: %s\n", *inputFile)
36+
fmt.Printf("Showing %d sample rows:\n\n", *sampleSize)
37+
38+
var timeDeltas []int64
39+
var voltages []int64
40+
var currents []int64
41+
var temps []int64
42+
43+
fmt.Println("RAW DATA SAMPLES:")
44+
fmt.Println("---------------------------------------------------------------")
45+
fmt.Printf("%-15s %-15s %-15s %-15s\n", "TIME_DELTA", "VOLTAGE", "CURRENT", "TEMP")
46+
fmt.Println("---------------------------------------------------------------")
47+
48+
for i := 0; i < *sampleSize; i++ {
49+
row, err := reader.Read()
50+
if err != nil {
51+
break
52+
}
53+
54+
if len(row) != 4 {
55+
fmt.Printf("Row %d has %d columns, expected 4\n", i+1, len(row))
56+
continue
57+
}
58+
59+
timeDelta, _ := strconv.ParseInt(row[0], 10, 64)
60+
voltage, _ := strconv.ParseInt(row[1], 10, 64)
61+
current, _ := strconv.ParseInt(row[2], 10, 64)
62+
temp, _ := strconv.ParseInt(row[3], 10, 64)
63+
64+
timeDeltas = append(timeDeltas, timeDelta)
65+
voltages = append(voltages, voltage)
66+
currents = append(currents, current)
67+
temps = append(temps, temp)
68+
69+
fmt.Printf("%-15d %-15d %-15d %-15d\n", timeDelta, voltage, current, temp)
70+
}
71+
72+
fmt.Println("\nVALUE RANGES:")
73+
fmt.Println("---------------------------------------------------------------")
74+
75+
minTimeDelta, maxTimeDelta := findMinMax(timeDeltas)
76+
minTemp, maxTemp := findMinMax(temps)
77+
minVoltage, maxVoltage := findMinMax(voltages)
78+
minCurrent, maxCurrent := findMinMax(currents)
79+
80+
fmt.Printf("Time Delta: Min=%d ms, Max=%d ms\n", minTimeDelta, maxTimeDelta)
81+
fmt.Printf("Temperature: Min=%d, Max=%d (Raw value)\n", minTemp, maxTemp)
82+
fmt.Printf(" Min=%.2f °C, Max=%.2f °C (Millicelsius)\n", float64(minTemp)/1000.0, float64(maxTemp)/1000.0)
83+
fmt.Printf(" Min=%.2f °C, Max=%.2f °C (Raw/100)\n", float64(minTemp)/100.0, float64(maxTemp)/100.0)
84+
85+
fmt.Printf("Voltage: Min=%d, Max=%d (Raw value)\n", minVoltage, maxVoltage)
86+
fmt.Printf(" Min=%.6f V, Max=%.6f V (Microvolts)\n", float64(minVoltage)/1000000.0, float64(maxVoltage)/1000000.0)
87+
fmt.Printf(" Min=%.6f V, Max=%.6f V (Millivolts)\n", float64(minVoltage)/1000.0, float64(maxVoltage)/1000.0)
88+
89+
fmt.Printf("Current: Min=%d, Max=%d (Raw value)\n", minCurrent, maxCurrent)
90+
fmt.Printf(" Min=%.6f A, Max=%.6f A (Nanoamperes)\n", float64(minCurrent)/1000000000.0, float64(maxCurrent)/1000000000.0)
91+
fmt.Printf(" Min=%.6f A, Max=%.6f A (Microamperes)\n", float64(minCurrent)/1000000.0, float64(maxCurrent)/1000000.0)
92+
93+
fmt.Printf("\nSUGGESTED UNITS FOR YOUR ESP32 DATA:\n")
94+
fmt.Printf("---------------------------------------------------------------\n")
95+
96+
suggestUnits(0, maxTemp, minVoltage, maxVoltage, minCurrent, maxCurrent)
97+
98+
fmt.Printf("ENEMETER DATA FORMAT INFORMATION:\n")
99+
fmt.Printf("---------------------------------------------------------------\n")
100+
fmt.Printf("Column order: TIME_DELTA, VOLTAGE, CURRENT, TEMP\n")
101+
fmt.Printf("Temperature: Values are in millicelsius (°C = value / 1000)\n")
102+
fmt.Printf("Voltage: Values are in microvolts (V = value / 1000000)\n")
103+
fmt.Printf("Current: Values are in nanoamperes (A = value / 1000000000)\n")
104+
}
105+
106+
func findMinMax(values []int64) (int64, int64) {
107+
if len(values) == 0 {
108+
return 0, 0
109+
}
110+
111+
min := values[0]
112+
max := values[0]
113+
114+
for _, val := range values {
115+
if val < min {
116+
min = val
117+
}
118+
if val > max {
119+
max = val
120+
}
121+
}
122+
123+
return min, max
124+
}
125+
126+
func suggestUnits(_, maxTemp, minVoltage, maxVoltage, minCurrent, maxCurrent int64) {
127+
// Suggest temperature units
128+
if maxTemp > 100000 {
129+
fmt.Println("Temperature: Values appear to be in raw ADC format")
130+
fmt.Println(" Consider using raw_value / 10 as °C")
131+
} else if maxTemp > 10000 {
132+
fmt.Println("Temperature: Values appear to be in millicelsius")
133+
fmt.Println(" Consider using raw_value / 1000 as °C")
134+
} else if maxTemp > 1000 {
135+
fmt.Println("Temperature: Values appear to be in centicelsius")
136+
fmt.Println(" Consider using raw_value / 100 as °C")
137+
} else {
138+
fmt.Println("Temperature: Values appear to be direct celsius readings")
139+
}
140+
141+
// Suggest voltage units
142+
if math.Abs(float64(minVoltage)) > 1000000 || math.Abs(float64(maxVoltage)) > 1000000 {
143+
fmt.Println("Voltage: Values appear to be in microvolts")
144+
fmt.Println(" Consider using raw_value / 1000000 as V")
145+
} else if math.Abs(float64(minVoltage)) > 1000 || math.Abs(float64(maxVoltage)) > 1000 {
146+
fmt.Println("Voltage: Values appear to be in millivolts")
147+
fmt.Println(" Consider using raw_value / 1000 as V")
148+
} else {
149+
fmt.Println("Voltage: Values appear to be direct voltage readings (V)")
150+
}
151+
152+
// Suggest current units
153+
if maxCurrent > 1000000 || minCurrent < -1000000 {
154+
fmt.Println("Current: Values appear to be in nanoamperes")
155+
fmt.Println(" Consider using raw_value / 1000000000 as A")
156+
} else if maxCurrent > 1000 || minCurrent < -1000 {
157+
fmt.Println("Current: Values appear to be in microamperes")
158+
fmt.Println(" Consider using raw_value / 1000000 as A")
159+
} else {
160+
fmt.Println("Current: Values appear to be in milliamperes")
161+
fmt.Println(" Consider using raw_value / 1000 as A")
162+
}
163+
}

0 commit comments

Comments
 (0)