Build Open-Fleet Provisioning Page using the Golioth.io Fleet APIs

[keenanjohnson]

One of the promising device management solutions for a microcontroller based Frog Sensor is a platform called Golioth.

See the related issue on prototyping a Golioth ESP32 Device: Ribbit-Network/ribbit-network-frog-hardware#126

One of the key question marks at this point is how to implement the "open-fleet" concept with Balena.

Balena Open Fleet

Ribbit Network devices are currently managed as part of a Balena Open Fleet.

An OpenFleet is a project where members can join by simply adding their devices (no balenaCloud account required for users just wanting to add a device). The owner maintains the fleet as an open, non-commercial project. Open Fleets have no device limit, encouraging owners to try and get more users to support and grow their projects. Ribbit Network qualifies being an open-source, not-for-profit org.

A few key Ribbit Staff maintain access to the Balena backend and coordinate fleet-wide
updates, but anyone can view the source code and add their own device to the fleet if
they would like.

• Link to instructions to build your own Frog Sensor
• Balena Open-Fleet Join Instructions

Open Fleet in Golioth

Golioth currently doesn't have any capability for an open fleet. However, I have spoken to the team at Golitoh and they are excited about the idea and supporting Ribbit Network by offering a possibly reduced price.

However, in order to join the open-fleet in Golioth, we will need to build some software that calls into the Golioth APIs.

We should start by reviewing the Golioth APIs, and ensure that all the APIs are available to build a site that would allow people building their own Frog Sensors to join the Golioth Open-Fleet.

After that we should be empowered to build out that functionality!

[beriberikix]

👋 hello from Golioth! Let us know how we can help with the prototype/concept!

[keenanjohnson]

From a high level this is what we need someone to build a prototype to ensure that the Golioth API contains all the items needed to accomplish the following so that we can re-risk using an ESP32 device with Golioth as an open solution for our open-source network.

Brief Description
We need to create a prototype website / web service that allows anyone in the open-source community to provision and add an ESP32 device to a Golioth fleet.

High-Level Flow

• User goes to a ribbit network onboarding website
• User starts a provisioning process to add their new esp32 device to the Ribbit Fleet
• The Device is provisioned in a Golioth Fleet
• The user is provided with a prebuilt binary to flash the EPS32 with

[beriberikix]

Great! That high-level flow makes sense and should be do-able 🙌 Some data points:

• The REST API should enable the ribbit network onboarding website (RNOW) to do all the provisioning. Given a common Ribbit Golioth Project:
  • /v1/projects/{projectId}/devices can create a new "blank" device
  • /v1/projects/{projectId}/devices/{deviceId} can be used to upload device credentials
    • Note: As a convenience, the RNOW could generate credentials for users. We optionally do this in our console UI.

Before building the RNOW you can test this flow with the goliothctl commandline tool.

Now on the firmware side there will need to be more discussion. If the prebuilt binary will be a "golden" image, ie same for everyone, than one way to get credentials on the device is via the terminal (see the example.) The RNOW could also build a custom image that includes Golioth + Wi-Fi- credentials, but that will add non-trivial complexity.

Two other options that are also worth exploring:

1. Espressif has a Unified Provisioning library, which includes Wi-Fi or BLE provisioning and even mobile app SDKs (deps on hardware.) We haven't integrated it yet into our SDK but it should be possible in the future.
2. Certain Espressif devices, like the ESP32-S2 & ESP32-S3 have native usb and can be programmed via drag and drop (they show up like USB drives.) We could in theory use that as a way to get credentials on the device without opening a terminal or need a mobile app. Note: I'd personally recommend using the S3 over the classic ESP32. It is more capable for not much more money and Espressif is recommending it for new designs.

[todbott]

Hello @keenanjohnson , I just confirmed everything that @beriberikix wrote above! Using the API, I was able to add a new ESP32 to my Golioth dashboard, then begin receiving data from it.

I'll make a little mockup web portal in the next week or so, so we can see it in action. Very simple and cool!

[keenanjohnson]

Awesome! That is great @todbott ! Did you look into any of the challenges @beriberikix mentioned with flashing the prebuilt binary onto the device?

Perhaps it might be helpful to create a process flow diagram for the specific API calls needed + local command line commands and user actions to provision, flash, and join a device to a golioth fleet as a next step here @todbott ?

[beriberikix]

Charting out sounds like the right next step! Ya, it'll come down to which ESP32(s) you want to support. Also, do you expect anyone making their own frog sensor to be comfortable with the command line?

I can come up with even more ideas but I'll hold off until we answer some more questions 😊

[fosteman]

Also, do you expect anyone making their own frog sensor to be comfortable with the command line?

I'm thinking that it narrows down our potential user base substantially. Having a user-friendly interface and installation instructions and\or a ready plug-and-play product is crucial.

[todbott]

Thank you, here is a workflow I used late last week to add a new ESP32 to my Golioth.io dashboard and begin receiving data from it.

This workflow assumes that Ribbit Network already has a Golioth.io account and project. I used my own account when thinking of this onboarding flow--my project is named 'ribbit-practice', so that is the project name I'll use in the API calls below.

Getting the user a hardware ID

So, the user starts out with an ESP32 and a USB cable. They go to a special web portal, and click on "Join the fleet".

When they click on "Join the fleet", we can send a POST request to the Golioth API, this endpoint and this JSON:

/v1/projects/ribbit-practice/devices
{
  "name": "an arbitrary name for the new fleet member",
  "hardwareIds": [
    "a unique and random string"
  ]
}

As a result, we will receive a JSON response with a field called 'id'. A space for the new fleet member (and an associated ID) has now been created in our Golioth.io dashboard.

{
  "data": {
    "id": "6311a4752aa9e6655dabf779",
    ...
}

Getting the user a device identity and PSK

In order to send data to our Golioth.io dashboard (and also be available to receive any future firmware updates), users need to input an identity and PSK (pre-shared key) into their device, and the same identity and PSK have to be registered in our project. The identity and PSK can be anything, really. The necessary thing is to register them in our Golioth.io project, essentially letting the system know "You will be receiving data and doing firmware updates on a device with this identity and PSK in the future." So, another POST request to the API, with which we add an identity (string) and pre-shared key (string) for the new fleet member we created above:

/v1/projects/ribbit-practice/credentials
{
  "deviceId": "6311a4752aa9e6655dabf779",
  "type": "PRE_SHARED_KEY",
  "identity": "some kind of unique string",
  "preSharedKey": "some kind of unique string"
}

The identity (which we need to put in the code) now becomes the 'identity' in the JSON, with the suffix '@ribbit-practice' (the project name). The pre-shared key also needs to be put in the code.

From there? The hacky way forward:

Of course, a person could just copy-paste this ID and PSK into their code (along with their wi-fi credentials, like I did here:

    wifi_init("IODATA-cdbc80-2G", "8186895522346");
    wifi_wait_for_connected();
    golioth_client_t client =
            golioth_client_create("id@ribbit-practice", "psk");
    assert(client);

I was able to get my ESP32 (newly added to the fleet) to begin streaming data to the Golioth.io project dashboard, in this way.

Flashing is the real bottleneck

In this way, we could have a flow like this:

• User goes to web portal, receives an identity and PSK to put in their code
• User puts those in the code, and wifi credentials
• User downloads ESP-IDF, then runs idf.py build and idf.py flash to flash their code to the ESP32

The bottleneck here (the part that will make it very hard for users) is the need to download the ESP-IDF and install it. I thought perhaps we could just have a VM in the cloud somewhere that could take as input 1) the identity and PSK and 2) the user's wifi credentials, then build the necessary *.bin file (using idf.py build), and present them with that *.bin file... however, as far as I know flashing data to an ESP32 requires the ESP-IDF. So, it looks like the user will need that tool eventually, regardless of what they do.

[beriberikix]

Great writeup! My recommendation is to have a strongly opinionated happy path but still make alternatives possible. This is especially true when designing technical products for non-technical users. Trying to make something that is easy to use with a consumer-level setup experience AND supports multiple hardware variations is going to be tough.

Some more notes to ponder:

• Espressif makes a lot of different Wi-Fi devices: ESP8266, ESP32, -S2, -S3, -C3...and more on the way. S3 is the actual successor to the original ESP32 and is the most featureful.
• Initial flashing of the device will always be a challenge for non-technical users. Certain vendors pre-program their devboards with special bootloaders that make things like drag-and-drop programming possible
• The S2/S3 have native USB and drag-and-drop programming is possible
  • Internally we have a POC supporting a UF2 bootloader
• Some communities offer a desktop or web tool (via WebUSB) for non-technical users to initially program their devices
  • Internally we have a POC using Web technologies (BLE right now; USB next) for getting credentials (Wi-FI and Golioth,) similar to https://www.improv-wifi.com/.

So my recommendation would be:

• Recommend the ESP32-S3 as the happy path, with a specific devboard (like the Adafruit ESP32-S3 Feather.)
• Ensure that whichever devboard you pick is pre-flashed with a UF2-based bootloader
• Offer a prebuilt image and a link to download (Github Actions is your friend!)
• Implement a web-based setup flow similar to esp-web-tools to flash the pre-built image
• Offer documentation to setup devices "manually," ie building and flashing themselves, support for other boards. This is where the community could help.

[keenanjohnson]

This seems reasonable to me as a recommendation @beriberikix !

@todbott does that make sense to you? Would you have any time or interest in prototyping the web-based setup flow similiar to esp-web-tools to compliment the work you have already done? Completing this is the main blocker is deploying more Frogs right now as Raspberry Pis are nowhere to be found :(

[keenanjohnson]

Seems like @todbott is not able to continue on this, so this issue is open for anyone that has interest to tackle!