Skip to content

Mesh-enabled Template Example Instantiation 1.2

This is available in 1.2.2 and later.

VeeaHub Distributed MQTT Application Summary

This distributed MQTT template application demonstrates the following:

  • Service Discovery using DNS Service (SRV) Record Set and Get. Applications running in the mesh that provide a new service can make their service's hostname and port number discoverable via DNS's SRV Record query. The application providing the service needs to call two Dbus methods (RegisterDnsHost & RegisterDnsSRV) to set up the SRV record. Then any client wanting to discover the service to retrieve the service's hostname and port can query DNS for the SRV record matching a specific Service FQDN to get that information.

  • MESH Secrets

    • The mesh's MQTT authorization username/password between the mesh broker and mesh client can be derived using the unique meshUUID key as a way to safely manage these secrets instead of being hardcoded in the docker image.

    • Configuration data such as port numbers, certificates and secrets can be provided in a JSON file (/usr/local/config/default/user-config.json) made accessible to the container that is volume mounted alongside the container and is not part of the image in any way for security concerns. Control Center Application's JSON schema provides this mechanism for Control Center uploaded apps. The same configuration is available for VHT users using WebDAV to be able to leverage common code for user-config.json processing. The mesh's broker config will rely on this configuration capability.

  • MQTT Bridge Mode- a MQTT bridge is a way to connect multiple MQTT Brokers. In this demonstration, there are two MQTT client/broker networks: one in the 'cloud' implemented with docker-compose upstream of the VeeaHub and one in a VeeaHub Mesh that implements a mesh broker and client. The MEN's MQTT broker is configured for 'bridge mode' using configuration options such as username/password, port number and certificate needed to connect to the cloud broker.

  • Cloud side MQTT client can discover all mqtt client nodes in the mesh and then can send targeted messages to a specific discovered client.

  • TLS is enabled between the cloud broker and mesh's bridge broker

Application overview

The application is comprised of two connected MQTT networks. The first MQTT network consisting of an MQTT broker and a client sits in the 'cloud'. The cloud here is a docker-compose file that contains a broker service and a client service. The cloud's broker service listens on two mqtt ports, one is encrypted using TLS and it connects with the bridge broker on the meshes MEN. The other port is not encrypted and listens for any cloud client. The cloud client's purpose in this demonstration is to first discover all the mesh mqtt clients using the topic 'req/all' where every mesh client is expected to subscribe and reply to this topic with its serial number. Then with the list of discovered mesh clients in hand, the cloud client can send a command message directly to one of the clients using the topic 'req/' where every client subscribes to this topic but using their serial number. The cloud client also collects example telemetry data sent out by the mesh clients.

The second MQTT network is in the VeeaHub mesh where the MQTT broker running on the MEN is configured for bridge mode and is connected to the cloud broker via TLS. Every VeeaHub in the mesh runs an MQTT client where each VeeaHub always responds to the 'node discovery' topic 'req/all' but will only respond to the second topic 'req/' if it matches their serial number. Additionally, all the mesh clients will randomly send out a simple example of telemetry data to be collected by the cloud client.

The picture below shows the two MQTT networks divided into two groups:

  • MQTT broker and client network representing the 'cloud-side'

  • MQTT broker and client network representing the 'mesh side'.

Application Components

The following is a breakdown of the main components used in this application.

Service Record Discovery

For a service to be discoverable a DNS Service Record (SRV Record) needs to be created and we can do this using two D-Bus API commands:

  • io.veea.VeeaHub.Networking.RegisterDnsHost()

  • io.veea.VeeaHub.Networking.RegisterDnsSRV()

see D-Bus API Documentation - io.veea.VeeaHub.Networking

In our example the mesh broker we configure a DNS hostname, DNS Service Record name and DNS Service Record port:

  • hostname - 'dmqtt-broker-application.service.veeamesh.local'

  • service name -'dmqtt'

  • service port - 1883

  • service domain - "service.veeamesh.local"

  • service protocol- "tcp"

Mesh Mqtt clients who want to connect to the Mesh's MQTT Broker but need an IP address and port can query DNS providing a specific Service Fully Qualified Domain Name (FQDN). This FQDN is specific and based on how the DNS Service Record was created in the mesh broker. The prepended underscores of the service name and service protocol required on the Services FQDN below are defined by the Service record.

  • "_dmqtt._tcp.service.veeamesh.local"

The client simply requests from DNS the SRV record providing a specific FQDN and can pull out the target(hostname) and port number.

BASH

  # get SRV/Service record
  self.log.info("Requesting DNS SRV Record")
  SERVICE_FQDN = "_dmqtt._tcp.service.veeamesh.local"
  rdata_list = dns.resolver.resolve(SERVICE_FQDN, "SRV")
  mesh_broker_hostname = str(rdata.target).rstrip(".")
  mesh_broker_port = rdata.port

Deriving Mesh Secrets using unique MeshUUID

For an MQTT client to connect with an MQTT broker there is username and password authentication that needs to happen first. The username/password needs to be added to the broker password file on the broker side and all the mqtt clients need to provide the same username/password to authenticate. The problem is how to configure this common username/password for the broker and clients across the mesh. One thing we don't want to do is configure these secrets into the image files which is not secure and can be determined. The solution we demonstrate here is that any node in the mesh can generate the same username/password secret using the meshUUUID as a seed and encrypting it with openssl to generate our shared secret this is common only across the mesh.

  • io.veea.VeeaHub.Info.MeshUUID()

see D-Bus API Documentation - io.veea.VeeaHub.Info

Mesh Broker Configuration using user-config.json

There are several configuration settings that the Mesh's Broker configuration needs before the Mesh broker can come online. There is a JSON file that the Mesh broker's container has access to. The file is /usr/local/config/defaults/user-config.json. For security reasons, this JSON file is NOT part of the image but it is installed into the VeeaHub alongside the container as a docker volume mount. The detail with this is that the installing mechanism of this volume that contains the user-config.json is different depending if the application is being deployed to the Control Center as a package subscription or if you're deploying your application using VHT. Details about this in the 'mesh side' section of this document.

MQTT Bridge - Mesh Broker Bridge Mode

A MQTT bridge lets you connect two MQTT brokers. They are generally used for sharing messages between the MQTT networks. In this demonstration, we will configure the Mesh's MQTT Broker to be the MQTT bridge and it will connect to our demonstration Cloud MQTT Broker. The Mesh's broker configuration needs several parameters to make it all work. Below is a snippet of the template file which when merges in user-config.json config will generate Mesh's /etc/mosquitto/mosquitto.conf.

BASH

```bash listener {{ MQTT_MESH_BROKER_PORT }} allow_anonymous false password_file /etc/mosquitto/passwd tls_version tlsv1.2