Protocol Gateway Template and Example #1964

b-abel · 2020-05-15T18:57:31Z

Implementing a production-ready protocol gateway currently involves a considerable effort and requires advanced knowledge of Hono and AMQP 1.0 (in addition to knowledge of the device-side protocol).
To simplify this, a useful example would be helpful.

What I have implemented here is two things:

a template in the form of an abstract base class that provides a protocol gateway that offers MQTT on the device side and connects northbound to the AMQP adapter while being as generic as possible.
a ready-to-use example implementation that shows how this base class can be used in practice. The use case of this example is a protocol gateway that offers the MQTT API of the Azure IoT Hub and passes these messages to Hono.

Challenges for a generic usage of Hono with MQTT are:

Topics are freely selectable in MQTT and may or may not contain all kinds of information
MQTT doesn't have metadata
for a command response several values must be sent back from the command with the response (requestId and CorrelationId)

For the last point, there are 3 strategies on how to get the two IDs from the command into the response:

encode them into the topic and have the device send them back
encode them into the payload and have the device send them back
keep them in the protocol gateway and add it to the response

Correlation of command responses

There is no simple generic solution for this. When the values are sent to the device (1. and 2.), the device must support this and add the correct data to the response in the required manner. The 3rd strategy has the disadvantage that the protocol gateway cannot simply be scaled horizontally.

The base class is designed to leave this to the concrete implementation.

The Azure IoT Hub sample implementation uses the 1st strategy because the API implemented there contains a corresponding attribute that can be used for this purpose. Since Hono's Command & Control API requires two values for correlation, they are encoded into this one attribute and decoded accordingly on the way back.

The template supports multitenancy. For this to work, the subclass must determine the credentials of the corresponding gateway device for each connection establishment by a device.
The protocol gateway then uses these credentials to authenticate a new client to the AMQP adapter if the connection does not already exist for this client (because of already connected devices).

The template does not support unauthenticated device connections, but the authentication is performed by the derived class and could be skipped there.

Authentication is implemented here only exemplarily for a configurable demo device. This allows the protocol gateway to be tested against Hono's sandbox or any other Hono installation with minimal effort.

I will provide documentation and tests later.

ctron · 2020-05-18T08:27:21Z

I am wondering if this should go into the main repository, or maybe go to some "extras" repository. Maybe even a dedicated "demo-gateway" repository, which can be flagged in GitHub as "template repository".

b-abel · 2020-05-18T08:50:08Z

I am open to moving this to a separate repository. I don't know of any decisions yet regarding whether to create multiple repositories or continue to keep everything in the main repository. I also found this repository: https://github.com/eclipse/hono-extras but don't know what it is intended for.

sophokles73 · 2020-05-19T09:29:49Z

@b-abel can you fix the JavaDoc that prevents the Travis build to succeed?

b-abel · 2020-05-19T09:49:27Z

@sophokles73 I don't see a JavaDoc error in the latest Travis run, but an error with pulling docker images. (After the first commit there was a JavaDoc error, which I have already fixed). Or am I missing something?

sophokles73 · 2020-05-19T11:00:00Z

ah, you have renamed the PR ... I was under the impression that there are two separate PRs. My bad ...

b-abel · 2020-05-29T12:03:30Z

I added tests for the base class. I would add a little bit of documentation (Developer Guide for the template) and a small unit test for the Azure example.

b-abel · 2020-05-29T12:40:50Z

It looks like the new checkstyle configuration from master is used by Travis causing the build to fail. What are we going to do? Rebase?

…devices via MQTT and forwards them to the AMQP adapter. And vice versa. This is implemented in the new Maven module "hono-sdk". It also adds an example implementation in the "hono-example" module, the "Azure IoT Hub Protocol Gateway". Demo certificates are now created for the latter. Signed-off-by: Abel Buechner-Mihaljevic <[email protected]>

Signed-off-by: Abel Buechner-Mihaljevic <[email protected]>

…ADME. Signed-off-by: Abel Buechner-Mihaljevic <[email protected]>

Adds package-local methods to verify subscriptions and TLS options in AbstractMqttToAmqpProtocolGateway. Signed-off-by: Abel Buechner-Mihaljevic <[email protected]>

Change the visibility of MQTT endpoint handlers to private. In tests they can be invoked with (mocked) MQTT messages. The tests are improved because they are more decoupled from the implementation and the public interface is tested. Signed-off-by: Abel Buechner-Mihaljevic <[email protected]>

…mentation. Signed-off-by: Abel Buechner-Mihaljevic <[email protected]>

Signed-off-by: Abel Buechner-Mihaljevic <[email protected]>

b-abel · 2020-06-04T13:58:06Z

Force pushed to get rid of the checkstyle problems.

b-abel · 2020-06-05T10:18:45Z

I am wondering if this should go into the main repository, or maybe go to some "extras" repository. Maybe even a dedicated "demo-gateway" repository, which can be flagged in GitHub as "template repository".

@ctron I just realized that you probably meant the existing "hono-extras" repository. I agree that having this protocol gateway example in a separate repository is a good idea. How can I add it there?

custom_http_adapter.md and java_client_consumer.md had conflicting weights. Move custom_http_adapter.md to the (current) bottom of the list. Signed-off-by: Abel Buechner-Mihaljevic <[email protected]>

Signed-off-by: Abel Buechner-Mihaljevic <[email protected]>

sophokles73 · 2020-06-09T06:04:40Z

@b-abel @ctron I also believe that this should go into the hono-extras repository. One of the reasons is that I do not want people to think of it as an official component which gets the same level of support as the protocol adapters. I also do not want to foster expectations that we will add more protocol gateways over time. This should simply serve as an example project that people can take or leave. WDYT?

b-abel · 2020-06-09T09:02:32Z

@sophokles73 I agree that this should be in a separate repository, for the reasons you mentioned. And I don't see why we wouldn't want to use hono-extras for it. I'm happy to turn the PR against this repo. The question is whether there are already ideas about the directory structure there. And who does the review?

Signed-off-by: Abel Buechner-Mihaljevic <[email protected]>

…rConfig. Rename `AbstractMqttToAmqpProtocolGateway` to `AbstractMqttProtocolGateway` which expresses more clearly what s important and removes the secondary information that it connects to the AMQP protocol adapter. Rename `MqttGatewayServerConfig` to `MqttProtocolGatewayConfig` because it no longer contains only server configuration. Signed-off-by: Abel Buechner-Mihaljevic <[email protected]>

Signed-off-by: Abel Buechner-Mihaljevic <[email protected]>

Applied the changes in the CommandSubscriptionsManager of the MQTT adapter to the CommandSubscriptionsManager of the protocol gateway to keep them similar. Signed-off-by: Abel Buechner-Mihaljevic <[email protected]>

Signed-off-by: Abel Buechner-Mihaljevic <[email protected]>

…ACK). Signed-off-by: Abel Buechner-Mihaljevic <[email protected]>

example/mqtt-protocol-gateway-example/pom.xml

...-protocol-gateway-example/src/main/java/org/eclipse/hono/example/protocolgateway/Config.java

sophokles73 · 2020-06-16T05:50:29Z

@b-abel FMPOV the directory structure in the hono-extras is simple: one top level folder per extra. I do not think that we should have a single multi-project build but instead each top level folder should contain its own build. We also will need this because not every extra will be Java/Maven based. So I can imagine to use azure-mqtt-protocol-gateway as the top level folder name here. However, I think we should/could keep the additional demo cert in Hono's code base but maybe rename it to allow for it to be (re-)used more easily, i.e. without tying it to azure so tightly. WDYT?

b-abel · 2020-06-16T09:15:37Z

@sophokles73 Thank you for your feedback! Should I create the new PR in Hono-Extras now, or are we waiting for a statement from @ctron?

Do you only want to move the Azure IoT Hub Protocol Gateway, i.e. the sample implementation of the template, to hono-extras, or do you want to move the template itself?

As far as the directory structure is concerned: what you wrote makes total sense. We should probably keep the number of items (directories) at the top level of the repository as small as possible for ease of reference. How about a top-level protocol gateway directory to prepare for when another protocol gateway project is added in the future? That would be just to organize similar projects, each protocol gateway project should contain its own build. WDYT?

You're right, the certificate should generally be issued much better for a gateway.

…-gateway). Signed-off-by: Abel Buechner-Mihaljevic <[email protected]>

sophokles73 · 2020-06-16T14:52:31Z

Should I create the new PR in Hono-Extras now, or are we waiting for a statement from @ctron?

IMHO you should create the new PR. No need to wait, as @ctron already has proposed to move it to hono-extras.

Do you only want to move the Azure IoT Hub Protocol Gateway, i.e. the sample implementation of the template, to hono-extras, or do you want to move the template itself?

I think we should move everything.

How about a top-level protocol gateway directory to prepare for when another protocol gateway project is added in the future? That would be just to organize similar projects, each protocol gateway project should contain its own build. WDYT?

I do not expect nor want to maintain additional protocol gateways. This single one should be sufficient as an example for how to use the template, shouldn't it?

b-abel · 2020-06-16T15:11:58Z

I do not expect nor want to maintain additional protocol gateways. This single one should be sufficient as an example for how to use the template, shouldn't it?

Of course, is a single example for the usage of the template enough. I had other protocol gateways in mind (which do not use MQTT). So the top-level directory would be protocol-gateway and contain (1) the template, (2) the azure-mqtt-protocol-gateway. In the future, other protocol gateways (potentially written in another language and using other protocols) might or might not be added in this top-level directory as well.

I will create new PRs and then close this one.

b-abel · 2020-06-22T10:08:04Z

Replaced by a new PR on the hono-extras repository: eclipse-hono/hono-extras#3.

b-abel requested a review from sophokles73 as a code owner May 15, 2020 18:57

b-abel changed the title ~~Adds a base class for a Protocol Gateway that receives messages from …~~ Protocol Gateway Template and Example May 18, 2020

b-abel added 9 commits June 4, 2020 15:41

Fix HTML in JavaDoc.

5c9bebd

Signed-off-by: Abel Buechner-Mihaljevic <[email protected]>

Improve documentation of Azure IoT Hub protocol gateway example in RE…

8845bd2

…ADME. Signed-off-by: Abel Buechner-Mihaljevic <[email protected]>

Refactorings to improve testability.

caa558b

Adds package-local methods to verify subscriptions and TLS options in AbstractMqttToAmqpProtocolGateway. Signed-off-by: Abel Buechner-Mihaljevic <[email protected]>

Add startup and shutdown hooks for the derived protocol gateway imple…

6ffb76b

…mentation. Signed-off-by: Abel Buechner-Mihaljevic <[email protected]>

Add tests for AbstractMqttToAmqpProtocolGateway.

a58e1a7

Signed-off-by: Abel Buechner-Mihaljevic <[email protected]>

Remove trailing whitespaces from JavaDoc.

f78d0a1

Signed-off-by: Abel Buechner-Mihaljevic <[email protected]>

Replace Future#setHandler by onComplete.

6e74d0a

Signed-off-by: Abel Buechner-Mihaljevic <[email protected]>

b-abel force-pushed the mqtt_protocol_gateway branch from 06629cb to 6e74d0a Compare June 4, 2020 13:57

b-abel added 2 commits June 8, 2020 11:27

Change weights of developer guide documentation pages.

4833e80

custom_http_adapter.md and java_client_consumer.md had conflicting weights. Move custom_http_adapter.md to the (current) bottom of the list. Signed-off-by: Abel Buechner-Mihaljevic <[email protected]>

Fix Javadoc.

47d0f78

Signed-off-by: Abel Buechner-Mihaljevic <[email protected]>

b-abel added 5 commits June 9, 2020 11:32

Move the command acknowledgement timeout into MqttGatewayServerConfig.

332f00b

Signed-off-by: Abel Buechner-Mihaljevic <[email protected]>

Add documentation for the MQTT protocol gateway template.

11119d9

Signed-off-by: Abel Buechner-Mihaljevic <[email protected]>

Apply changes of commit 4a4b93f.

6d6d097

Applied the changes in the CommandSubscriptionsManager of the MQTT adapter to the CommandSubscriptionsManager of the protocol gateway to keep them similar. Signed-off-by: Abel Buechner-Mihaljevic <[email protected]>

Remove autowiring and replace it by Java-based configuration.

f6e4d7a

Signed-off-by: Abel Buechner-Mihaljevic <[email protected]>

b-abel added 4 commits June 15, 2020 09:30

Polish README.

9fe1ab8

Signed-off-by: Abel Buechner-Mihaljevic <[email protected]>

Add tests for mqtt protocol gateway example.

e2562c0

Signed-off-by: Abel Buechner-Mihaljevic <[email protected]>

Add link to the MQTT Protocol Gateway in the Concept page.

759299e

Signed-off-by: Abel Buechner-Mihaljevic <[email protected]>

Document that command is settled automatically (does not wait for PUB…

0de051d

…ACK). Signed-off-by: Abel Buechner-Mihaljevic <[email protected]>

sophokles73 reviewed Jun 16, 2020

View reviewed changes

example/mqtt-protocol-gateway-example/pom.xml Show resolved Hide resolved

...-protocol-gateway-example/src/main/java/org/eclipse/hono/example/protocolgateway/Config.java Show resolved Hide resolved

Issue demo cert for example-gateway (and no longer for azure-protocol…

a571ae6

…-gateway). Signed-off-by: Abel Buechner-Mihaljevic <[email protected]>

This was referenced Jun 18, 2020

Create demo certificate for example gateway. #2038

Merged

Protocol Gateway Template and Example eclipse-hono/hono-extras#3

Merged

b-abel closed this Jun 22, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Protocol Gateway Template and Example #1964

Protocol Gateway Template and Example #1964

b-abel commented May 15, 2020

ctron commented May 18, 2020

b-abel commented May 18, 2020

sophokles73 commented May 19, 2020

b-abel commented May 19, 2020

sophokles73 commented May 19, 2020

b-abel commented May 29, 2020

b-abel commented May 29, 2020

b-abel commented Jun 4, 2020

b-abel commented Jun 5, 2020

sophokles73 commented Jun 9, 2020

b-abel commented Jun 9, 2020

sophokles73 commented Jun 16, 2020

b-abel commented Jun 16, 2020

sophokles73 commented Jun 16, 2020

b-abel commented Jun 16, 2020

b-abel commented Jun 22, 2020

Protocol Gateway Template and Example #1964

Protocol Gateway Template and Example #1964

Conversation

b-abel commented May 15, 2020

ctron commented May 18, 2020

b-abel commented May 18, 2020

sophokles73 commented May 19, 2020

b-abel commented May 19, 2020

sophokles73 commented May 19, 2020

b-abel commented May 29, 2020

b-abel commented May 29, 2020

b-abel commented Jun 4, 2020

b-abel commented Jun 5, 2020

sophokles73 commented Jun 9, 2020

b-abel commented Jun 9, 2020

sophokles73 commented Jun 16, 2020

b-abel commented Jun 16, 2020

sophokles73 commented Jun 16, 2020

b-abel commented Jun 16, 2020

b-abel commented Jun 22, 2020