Canadian Daily Weather Data

[jaderoca]

I made a small program for retrieving historical daily data for any one of the over 8000 weather stations in the Environment Canada database. Environment Canada supplies this data one year at a time, so I pull each year in tutn and append it all to a single CSV file for further analysis.

I'm sure that there are better ways to do what I've done, but I've tried my best to write the code the way I imagine idiomatic Rye would be written.

I'm proudest of my get-csv function, because I have one function that uses parameters to deal with source data, destination file, the differences between file structures supplied by Environment Canada, and whether or not data retrieved should create or replace the destination file or append to an existing file.

I think the biggest hack is getting data from a TableRow. I didn't see any way to get it directly and nothing I tried worked. I found that a TableRow can be converted to a string, so I did that and then used some simple string processing to get the data I needed.

Next up is sorting out git so that I get things properly named and organized!

https://github.com/jaderoca/Rye

[refaktor]

Great work!

Since we now have two fronts of communication open, to not duplicate everything. I will talk more about the code here. This is great example that I will use as a case on which I can improve things. Like the selection function I mentioned (instead of display), maybe add string interpolation " ... {someval} .... {otherval} ...", etc.

This is my, slightly tuned version of your code:

; Saves Environment Canada weather station list to a CSV file without extraneous comments

stations-source: https://collaboration.cmc.ec.gc.ca/cmc/climate/Get_More_Data_Plus_de_donnees/Station%20Inventory%20EN.csv
stations-filename: %StationList.csv
daily-source-str: "https://climate.weather.gc.ca/climate_data/bulk_data_e.html?format=csv&stationID=%v&Year=%d&Month=1&Day=14&timeframe=2&submit=Download+Data"

; downloads a CSV from and saves it to disc
; some CSVs have instructions or other stuff before the header row (column names), so we want to skip those
; sometimes we want to create or replace a file. Sometimes we want to just append to an existing file
; some source files are structured such that we need to add newlines when appending additional rows
get-csv: fn { source-uri destination-file skip-lines new-file needs-newlines } {
    ; download the list into a block with each line as a separate element
    source: Get source-uri |split newline
    
    ; When it's a new file, write the header row (writing like this will create or overwrite the file)
    if new-file {
        Write destination-file ( first rest\from source skip-lines )
    }

    ; open the file in append mode so that further writes will add to the file instead of overwriting it
    destination: Open\append destination-file

    for rest\from source 
	either new-file { inc skip-lines } { skip-lines } ; skip an extra line to avoid the header row when appending to a pre-existing file
    { ::row 
        either needs-newlines { newline ++ row } { row }
        |Write* destination
    }	
    Close destination
}

get-stations: fn { } {
	term/spin-it "Loading a list of stations" {
		get-csv stations-source 
        stations-filename 
        3 ; skip lines
        true ; start by creating a new file or overwriting the existing one
        true ; the source file is structured in a way that we need to add newlines to create rows
	}
}

; I thought about getting the list only if the user did not know the stationID for the desired weather station.
; However, I realized that having the list made it possible to check for a valid stationID to avoid errors or blank results.
; I'll still allow the user to enter a stationID directly, but I'll validate it against the list before hitting the internet.
; Having the list on hand also makes it easier for the user to locate a stationID by entering a station name
; instead of scrolling through over 8000 entries to find it.
    
; only get the list if we haven't got it
either stations-filename .Does-exist {
    print "The station list already exists. Do you want to refresh it?"
    display-selection [ false "No. Do not refresh the list of stations." true "Yes. Refresh the list of stations." ] 0
    |if { get-stations }
} {
    get-stations
}

; filter the station list to make it easier to pick the specific station of interest
station-name: input "Enter Station Name: "
print "" print ""

station-list: Load\csv %StationList.csv
|autotype 0.90
|columns? { "Name" "DLY First Year" :FirstYear "DLY Last Year" :LastYear "Station ID" :StationID }
|where-contains 'Name to-upper station-name |order-by 'LastYear 'desc

; display the filtered list for the user to choose from and retrieve the relevant data
station-row: display station-list

file-name: station-row -> "Name" |replace " " "_"
station-id: station-row -> "StationID"
first-year:: station-row -> "FirstYear" ; mod word because we'll be changing it to a user-selected value later
last-year:: station-row -> "LastYear" ; mod word because we'll be changing it to a user-selected value later

; build and display a list of years for the user to choose from
year-list: range first-year last-year

print "Select year to start:"
first-year:: display year-list

print "Select year to end:"
last-year:: display reverse year-list ; reverse the list to put most recent years at the top

; define the file name based on user selections, then go get the requested data
station-file: to-uri file-name ++ "_" ++ first-year ++
    ( either first-year = last-year { "" } { "-" ++ last-year } ) ++ "_Daily.csv"


for range first-year last-year { ::year
	term/spin-it join [ "Loading data for " file-name ", year " year ] {
		get-csv to-uri ( [ station-id year ] .format daily-source-str )
	    station-file
        either year = first-year { 0 } { 1 } ; column names start with the first line so we need to skip that when appending
        year = first-year ; true starts a new file, false appends to the existing file
        false ; the source file is structured in a way that doesn't require a newline to create rows
	}
}

; --- path to string already remove % prefix
print ( os/cwd? .to-string ++ "/" ++ to-string station-file ) ++ " has been created."
print "Most spreadsheet and database programs have the ability to import .CSV files as tables."

One thing that won't work with Rye currently on github yet is this line
get-csv to-uri ( [ station-id year ] .format daily-source-str )

format function was already there, but it could only accept single value, not it also accepts a block of values.

[jaderoca]

I've gone through this in detail and find lots of useful changes. Especially getting data out of a 'TableRow'!

And how did I miss join? Not that it's a major thing, but it's a great reminder for me to spend more time trying all the words you make available. That is such an obviously better way to do things.

The spinner is something I was interested in trying, so I'm very happy that you included an example in my code.

display-selection also looks quite useful.

Thanks for the encouragement and for taking the time to critique my code and offer suggestions.

[refaktor]

I mentioned you flutter console experiment ... one idea is to first make just regular text console in flutter, but then make functions like you used for selection and display use flutter UI elements to achieve that ... we will see how it comes together. It would not be a classic UI, but more conversational, like your terminal based app is. But that could work even better on mobile I think.

[jaderoca]

Flutter is still a pretty long way off for me. Even if the tooling has improved since it was introduced, going to mobile means offering much more than mere data downloads.

I've done a fair bit of spreadsheet work on mobile and it's not pleasant. That means a mobile app would have to include a range of useful analyses and/or some way to offer simple analytical tools.

I will get there, I'm sure, but a better understanding of rye and at least a foundational understanding of go and your code comes first. Given my other hobbies, I expect flutter to be a year or so in my future.

[refaktor]

The idea is that the same code you have now will also run with a flutter frontend and be GUIfied :) ... that's the idea. I'm not 100% certain how well it will come out, but I long think there is something there.

[jaderoca]

That would be amazing! Based on what you have already in the various forms of display, I think that is definitely achievable.

[refaktor]

I hope I will have some more complicated internal updates ready today to merge to github today. Then I can start working on these display / select etc functions.

[jaderoca]

👍👍

[jaderoca]

I have completed an almost complete rewrite. It turned out that Load\csv did not like the structure of the "daily data" file as downloaded from Environment Canada. That was going to make my get-csv function yet more complicated, so I just took what I've learned and did it all from nearly scratch.

There are some possibly questionable things I've done, like using files to help with my processing, but it works and there is less potential for errors.

Unless I find something major, I'll now move on to data analysis.

[jaderoca]

I've created a new repository for this project at https://github.com/jaderoca/HeatCost. I haven't made any changes to the code yet, but I realized that I shouldn't have repositories per language but per project. It was fine while I was just exploring Rye to see if I liked it, but it's not really sustainable given my long-term plans.