Refactor of deconz adapter to solve various problems

[manup]

This PR is a rewrite of the adapter and driver for ConBee I, II, III and RaspBee I/II and addresses following topics:

• Robustness in sending ZCL / ZDO commands:
  The old driver is rather optimistic and didn't handle errors in a defined manner. The rewrite is in par with the deCONZ reference implementation and handles error paths explicitly by tracking timeouts and transaction sequence numbers. Especially here the queue handling is streamlined now to never do "fire and forget" commands and having untracked Promises.
• Reconfiguration and channel change:
  The old driver is only re-configuring some of the network parameters and doesn't send ZDP channel change commands to move a network to a new channel if the configuration was changed. The PR handles this fully within the new state machine. Further any misconfiguration gets handled same as in deCONZ reference.
• Backup support, this also handles automatic migration between ConBee I, II, III and RaspBee I/II.

---

Not part of this PR is Zigbee Install Code support. Although at least the ConBee II firmware supports these with a hack, we would like to add proper firmware support first before adapting this in the driver in a future PR.

---

Some clarification is needed on APS level ACKs, sice these can block queues of any coordinator (not just ConBee) for offline and sleeping devices. If I understand the Adapter interface right, decision of using APS ACKs is left to the adapter? But imho that should only be decided by the z2m controller since the adapter has no knowledge of wherever end-devices are sleeping or certain devices are tricky to reach in the network, or simply offline. Currently I plan to maintain a simple in memory list for devices which don't yield responses or often timeout to turn APS ACKs on for these. But it would be better if not every adapter has to implement such things. @Koenkk I'd appreciate your thoughts on this.

As remark in deCONZ a device integration is made via a JSON file "Device Description File (DDF)" which also specifies when a sleeping device is awake, for example the Xiaomi devices which are "never" reachable without a button press, do in fact respond to ZCL commands once every hour then their periodic report is send to the coordinator and shortly after they do a MAC poll request to their parent to receive commands. In deCONZ we use this to schedule reconfiguration if needed, for example to configure a new sensitivity value for the vibration sensor.

DONE APS ACKs are now enabled for all unicast requests.

---

Personal wishlist for a future Adapter interface:
(to simplify adapter development a lot 😄 )
An adapter should only provide an interface for `APS Request, Confirmation and Indication. And handling of these moved into z2m controller. The adapters shouldn't know anything about ZCL, ZDO requests or tracking their responses. With that only z2m controller needs robust handling implemented in one place which works the same for every coordinator. Imho this would reduce the code size of adapters a lot and bugs could be ironed out more broadly in one place.

[Koenkk]

Thanks for the PR! The Deconz adapter really can use some love ❤️

It would be nice if the missing features can be implemented, then it can be moved back to Recommended.

If I understand the Adapter interface right, decision of using APS ACKs is left to the adapter?

Currently APS ack is always enabled, if not acked the message will just time-out. Note that on a higher level (above the adapter) a sendPolicy (docs) is supported (to e.g. send messages when the device checks-in).

Currently I plan to maintain a simple in memory list for devices which don't yield responses or often timeout to turn APS ACKs on for these.

Therefore I would not do this (to keep adapter implementations consistent).

An adapter should only provide an interface for `APS Request, Confirmation and Indication. And handling of these moved into z2m controller. The adapters shouldn't know anything about ZCL, ZDO requests or tracking their responses.

In the ideal world yes, but in reality vendors have implemented different levels of abstraction. E.g. the zStack adapters don't propagate the raw ZDO messages but only the parsed ones.

[manup]

Hi, thanks yes the refresh is much needed, I hope the most urgent issues can be resolved.

It would be nice if the missing features can be implemented, then it can be moved back to Recommended.

The PR made a bit progress on this:

• Backup is now implemented (with an empty device list untill Install Codes / Link Keys get refactored in firmware).
  The implementation and logic was aligned to the Ember adapter. From a users perspective I'm still not sure how to apply a backup other than to put the coordinator_backup.json in the data directory, the https://www.zigbee2mqtt.io/guide/adapters/#coordinator-backups is a bit vague here.
  In theory the backup should also allow to switch coordinator vendors (sans the Install Code / Link Keys) but I haven't tested this. All other parameters including mac address get synced, but there might be troubles when switching between Zigbee R21 - R23 coordinators.
• Channel Change works now too and I made a few tests to move the network around.
• Touchlink is currently not planned for near term.
• Install Codes will be added in a later PR after refactoring the firmware for ConBee/RaspBee 2 and 3.
• Changing the transmit power is something we do consider to add. Technically it's a small change. But the catch is that the firmware is already at the maximum allowed tx power and going above that can get vendors in legal trouble at least in the EU. So the update will eventually only provide a way to reduce the tx power, which still can be useful to limit sniffer visibility when joining devices and while the network key is transmitted.

Currently APS ack is always enabled, if not acked the message will just time-out. Note that on a higher level (above the adapter) a sendPolicy (docs) is supported (to e.g. send messages when the device checks-in).

Ah alright I'll enable the ACKs then, from observation in the sniffer there isn't much polling towards devices so it should be ok in usual operation. I might add a little quirk later on that tracks how many parallel APS unicast requests are running and ensure that at least 2 request slots are always usable without ACKs so that the queue never blocks for too long (iirc there are max. 12 parallel APS requests in current firmware).

[Koenkk]

From a users perspective I'm still not sure how to apply a backup other than to put the coordinator_backup.json in the data directory, the

Putting the coordinator_backup.json in the data directory is indeed the only way to restore a backup

[manup]

Update: The PR is now in shape and the topics above are implemented and ready for review. I've tested in my (small) setup with two routers and 3 sleeping end-devices also letting it run over a few days. Tests where done with ConBee I, II and III with latest stable firmware versions. Also made some tests to ensure that a setup handled by the former adapter doesn't get messed up by the PR.

[Koenkk]

Many thanks for this! Would you also mind updating the Attention parts of https://www.zigbee2mqtt.io/guide/adapters/deconz.html ?

[manup]

Many thanks for this! Would you also mind updating the Attention parts of https://www.zigbee2mqtt.io/guide/adapters/deconz.html ?

Yes I'll have a look and make a PR 👍

[Nerivec]

@manup I was looking at the deconz code earlier. Seems there are a few unused private in the driver class. It led me to the event emitting that's only used within the same class. Would be much better for perf to remove this and just trigger the functions directly. This is generating countless unnecessary emits.

zigbee-herdsman/src/adapter/deconz/driver/driver.ts

Lines 182 to 184 in a7fbacf

	this.tickTimer = setInterval(() => {
	this.tick();
	}, 100);

zigbee-herdsman/src/adapter/deconz/driver/driver.ts

Lines 200 to 202 in a7fbacf

	this.on(DRIVER_EVENT, (event, data) => {
	this.handleStateEvent(event, data);
	});

zigbee-herdsman/src/adapter/deconz/driver/driver.ts

Lines 257 to 259 in a7fbacf

	private emitStateEvent(event: DriverEvent, data?: DriverEventData) {
	this.emit(DRIVER_EVENT, event, data);
	}

[manup]

@Nerivec Thanks for looking into this. I only have a limited understanding of how nodejs works. My goal was to use emit because the event should be processed async. Meaning that they are queued but not pile up into larger call stacks.

It's a bit difficult to describe, the following toy example In C shows what I tried to do via emit:

static  int tasks = 0;

void task1() { tasks |= 2; }
void task2() { tasks |= 1; }

int main() {

   tasks = 1; // dummy queue event 1

  for (;;) // main loop
  {
      if (tasks & 1) {
        tasks &= ~1; // clear event 1
        task1();
      }

      if (tasks & 2) {
        tasks &= ~2; // clear event 2
        task2();
      }
  }
}

Note that neither task1 nor task2 call each other directly, they only "queue" up the events (an event here is simply a bit in tasks). The call stacks depths remain always low.

[Nerivec]

When you're doing this:

zigbee-herdsman/src/adapter/deconz/driver/driver.ts

Line 880 in a7fbacf

this.emitStateEvent(DriverEvent.Connected);

might as well just do this directly:

this.handleStateEvent(DriverEvent.Connected);

At the very least, that would remove the whole "emit to itself". Here's a start: master...Nerivec:zigbee-herdsman:sample-deconz

Also a lot of stuff appears to not be awaited. This is likely to cause improper states (like connecting vs connected, trying to do something after an error occurred, etc.). Things like this probably don't happen the way you intended (callback-driven):

zigbee-herdsman/src/adapter/deconz/driver/driver.ts

Lines 1189 to 1190 in a7fbacf

	// if written is false, we also need to wait for drain()
	this.serialPort.drain(); // flush

You should use the parser/writer instead of dealing with serial vs socket in-place (like in sendRequest).

Any reason you used an interval-based tick instead of just reacting on incoming (from serial/socket) & outgoing (from Controller>Adapter layer)?

You should be able to use the ASH layer of the ember adapter as model. It should be pretty close to the way you seem to have intended the driver to work (state-based approach).