Simplify module type dependencies

[gordonbrander]

During the firmware flash process, we parse the module.json files from downloaded modules, but don't really use those definitions. This is because we must already have the module type described via fixture/db for the module type folder to have been downloaded in the first place. This is an unfortunate duplication.

Here's what I would like to do to simplify it:

• Define modules as we do today, but within module, have a "firmware_repository" field that contains a git or pio dependency descriptor. This replaces the "type" field.
• During install, fetch that dependency
• Read the module type from the dependency's module.json

This way we don't have to manually write module types in our fixture files. All we have to do is describe a type dependency for each module config. Specifying a type is unnecessary in this scheme. The type is described in the firmware's module.json. Only the repository needs to be described.

Module configs would look like this:

{
  "_id": "am2315_1",
  "environment": "environment_1",
  "firmware_repository": {
    "type": "git",
    "url": "https://github.com/OpenAgInitiative/openag_am2315.git",
    "branch": "0.0.1"
  }
}

(Branch is optional). Other module config details remain the same. cc @LeonChambers

[LeonChambers]

So this change would prevent us from maintaining a global database of approved firmware module types in the future. My idea was that at some point, we would have a cloud server whose firmware_module_type database was populated with metadata about a set of firmware module repositories that were approved through some review process. Then users could search in this database for things like "PID" and easily create an instance of that module in their own system simply by creating a firmware_module record in their own database that points to it. The metadata would be stored on that server mostly for search and display purposes, not actually for use at build time. Obviously this isn't possible right now because we don't have the infrastructure in place for it, and I agree that the way we are currently using the firmware_module_type database isn't actually useful. However, I'm also not sure that the proposed system makes it easier for the end user. Under the current system, the user just has to say that they want an "am2315" module and the information about where to download it from it stored in a pre-populated database. The user doesn't even have to know that the code actually comes from a particular branch of git repository at some specific url.

[gordonbrander]

The above is interesting, but I think it is orthogonal. Having a database of modules registered by name (PIP style) isn't blocked by also having a mechanism to fetch module from a fully qualified domain name. The first mechanism is just an alias for the second one. Or am I missing something?

[LeonChambers]

I agree that we could support both mechanisms. It seemed like what you were suggesting was to get rid of the current mechanism in favor of a new one

[gordonbrander]

@LeonChambers nah, what I'm after is the ability to load module_type defs from the module's own module.json file, rather than specifying it twice -- once in fixture, once in module.json (where the 2 versions might conflict).

Adding the repo-at-URL mechanism is a minimum viable way to get this working, and a name-based registry can be added on top of that. Also I think we want this type of functionality anyway, for module development (PIP has similar).

[LeonChambers]

Right. So having the info from the module.json file duplicated in a fixture for the firmware_module_type was my minimum viable solution before creating a name-based registry since I already had code for loading fixtures anyway. Your solution is probably better though and has potential uses cases for development. So as long as this is an addition to the current system and not a replacement, I'm all for it.

[LeonChambers]

Looking back on this, it is actually useful to have the information from the module.json file stored in the database. (I just wrote a software module that translates status codes coming from the arduino into status messages, and it needed to pull a lookup table for each firmware module from somewhere). However, I still agree that having the information both in a fixture and in the repo for the firmware module itself is messy. So, I'd like to implement something similar to this.

firmware_module_types already have a repository field so that the system knows where to download their code (and their module.json file) from. Thus, these records should be able to "update themselves" in some sense. We could write a script that pulls all of the firmware_module_types from the database, downloads their module.json files, and updates the records based on the downloaded data. This script could run every time openag_brain is started. With this in place, fixture files would just have to specify and ID and git URL for each firmware_module_type, and the system would figure out everything else itself. Fixtures would no longer have to be updated whenever a value is changed in the module.json file.

We could still add a firmware_repository field to the firmware_module record later if necessary. We would just have the same script go through the list of firmware_module records, fetch the module.json files from any with a value for firmware_repository and update those records with the fetched information. However, this would just be a convenience for developers so that they don't have to define firmware_module_types during testing.