Hue reset by serial number

[Mstrodl]

See: Koenkk/zigbee2mqtt#3492

I couldn't find any docs for these commands, but I worked off the excellent pcap files and analysis from @itavero in that issue. I'm also not really super well-versed in zigbee stuff, so maybe this is all totally the wrong to do this. Advice appreciated :)

I have tested this with a Philips Hue LED strip and it seems to work pretty well using an ember SiLabs MGM12P-based device running a fork of zigbee2mqtt with this feature added. The changes for other adapters are untested because I don't have any of them, but they seem like they should probably work.

Basically, we sweep every channel and broadcast this reset command, and if a device finds its serial number in that command, it will leave the network it was in and attempt to join the network matching the EPAN ID in the command. As a result, there's not a great way to know if a device actually received and acted on the message or not other than seeing it attempt to join the network on the normal channel some time later. If anyone has better ideas on how to do this, I'm all ears.

Let me know if the sendZclFrameInterPANBroadcastWithoutReply changes should be their own PR or somehow combined with sendZclFrameInterPANBroadcast (perhaps a timeout of null could make it not wait for a reply?)

[Nerivec]

The sendZcl function should use an extra arg instead of duplicating the whole function (like others do to flip "response" requirement). createHueResetRequestFrame should be removed, the create call can just be called on-site.

Custom clusters should not be added to ZH anymore (in ZHC instead), problem is, this is not triggered by converters... but the custom cluster overrides the Touchlink one, so, it's conflict-prone...

Needs testing on adapters that would/should support this, and also with other devices that would require the real touchlink cluster and could conflict with this one (I'm assuming some Philips -or Philips code using- devices should not use that custom cluster).
I won't be able to help on the testing, I don't have any device that fits the bill.

Overall, we want to avoid ZH being specific manufacturer-wise, but this feature is completely the opposite (and restricted to few devices too)... Not sure how to handle it for this case though 🤷

@chrivers any insights on the issue?

[Mstrodl]

Oops, only just saw your note after writing tests :)

The sendZcl function should use an extra arg instead of duplicating the whole function (like others do to flip "response" requirement). createHueResetRequestFrame should be removed, the create call can just be called on-site.

Ok. No big deal.

Custom clusters should not be added to ZH anymore (in ZHC instead), problem is, this is not triggered by converters... but the custom cluster overrides the Touchlink one, so, it's conflict-prone...
Needs testing on adapters that would/should support this, and also with other devices that would require the real touchlink cluster and could conflict with this one (I'm assuming some Philips -or Philips code using- devices should not use that custom cluster).

I think it will only resolve to the philips hue one when you have manufacturerCode set in the ZCL header... no? If not for that, they would conflict like you say.

For clarification: Hue bulbs support both the normal ZLL cluster and reset by serial number.

Overall, we want to avoid ZH being specific manufacturer-wise, but this feature is completely the opposite (and restricted to few devices too)... Not sure how to handle it for this case though 🤷

Fair. This is pretty useful functionality though. With that said, I have no idea what, if any, devices other than hue bulbs support this

[Nerivec]

The cluster conflict would be a problem if some Philips devices (or devices using the same manuf code) do not support the custom cluster. It would fail reading frames properly for cmd 0x00. I'm not sure if that manuf code is common (non-Philips devices would certainly fail, since this is custom Philips firmware only), other manuf aren't supposed to use assigned codes, but.... So, long shot, but possible...

The fact we'd need custom ZH-layer support, custom Z2M-layer support & custom frontend-layer support is... 😰
Let's wait for Koenkk's thoughts on this 😉

[Mstrodl]

Hmm. I would think this should be pretty much equivalent to the normal hue bridge sending this message, no? They just have a text field for serial number in the app and it'll seemingly do the same thing. Here's an article with some screenshots: https://www.trustedreviews.com/how-to/how-to-reset-philips-hue-bulbs-with-and-without-a-bridge-4145209

I don't know a ton about zigbee, so maybe I'm totally off-base here, but I'd think an unknown manufacturer-specific cluster broadcast shouldn't break other devices assuming everyone is conforming to the spec... right?

The fact we'd need custom ZH-layer support, custom Z2M-layer support & custom frontend-layer support is... 😰

Is there a way to do this without adding to zigbee-herdsman or zigbee2mqtt? In my experience, hue bulbs are pretty temperamental. They way I intend to use this is only via the mqtt interface. Is it a matter of policy to support every feature in the frontend?

[Koenkk]

First of all, thanks for the implementation!

@Nerivec is right on the fact that we want to prevent manufacturer specific stuff in zigbee-herdsman as much as possible. It would be the nicest if this can be completely implemented in zigbee-herdsman-converters (as this is were we put all the manu specific stuff).

My initial thoughts

1. We add a toZigbee converter which can send out this frame.
2. • Problem: toZigbee converters can only be called for paired devices/groups. A generic mechanism should be introduced to call a toZigbee converter without having an entity.
   • Proposal: add a new bridge call which can do this, e.g. topic: zigbee2mqtt/bridge/request/to_zigbee (I don't like this name, but I hope you get the point) payload {"converter_name: "hue_serial_reset", "payload": {"serial": "1234"}. This should be simple to implement and can be re-used later to do similar stuff like this
3. No changes needed on the frontend, we can document how to send the zigbee2mqtt/bridge/request/to_zigbee MQTT message via the frontend:

[Mstrodl]

I think I get your meaning.

Does this seem like a good starting point for a new philips_factory_reset converter? I am obviously not super familiar with ZHC and the architecture or whatever:
https://github.com/Koenkk/zigbee-herdsman-converters/blob/db1d6f2ee92b1c05ea24b01745819e785e8c3095/src/converters/toZigbee.ts#L248

There seems to be a need for an entity in all these though, how do you propose eliminating that requirement? One way I can think of is to add a new field to converters... Maybe convertDetachedSet: (adapter, key, value, meta) => {}. This adapter could either be ZH.Adapter, or some more limited wrapper class which only has a few methods: sendZclFrameInterPANBroadcast, setChannelInterPAN, restoreChannelInterPAN (maybe this should even be implicitly called after the converter for safety reasons? Or add a withInterPAN((setChannel) => {/* User code goes here */}) method?).

This would require some changes on the zigbee-herdsman side to expose this functionality publicly, though, since right now adapter is private in zigbee-herdsman's Controller. convertDetachedSet's first parameter could also be Controller and we add the inter-PAN methods I mention here to that instead.

I think this is going to be a bigger change, so I want to make sure I run the architecture decisions by you first lol.

[Nerivec]

Everything in ZHC is going to require a device/endpoint/group, this would require a major refactor I think.

What if we implement a bridge/request/raw that takes the arguments (with bridge/response/raw as needed):

interface RawPayload {
  ieeeAddress?: string;
  networkAddress?: number;
  groupId?: number;
  dstEndpoint?: number;
  srcEndpoint?: number; // defaults to `ZSpec.HA_ENDPOINT`
  interPan?: boolean; // defaults to false
  profileId?: number; // defaults to `ZSpec.HA_PROFILE_ID`
  clusterKey?: number | string;
  frame: {}; // omitted for brevity, contains the stuff needed for `Zcl.Frame.create`
  disableResponse?: boolean; // defaults to false
  timeout?: number; // defaults to 10000
}

and passes all these directly to ZH Controller sendRaw, which then calls the appropriate send function based on the args:

• sendZdo => if profileId is ZDO
• sendZclFrameToEndpoint => if ieeeAddress + dstEndpoint present
• sendZclFrameToGroup => if groupId present
• sendZclFrameToAll => if networkAddress is broadcast
• sendZclFrameInterPANToIeeeAddr => if interPan==true + ieeeAddress present
• sendZclFrameInterPANBroadcast => if interPan==true + ieeeAddress not present

Just a quick overview, need to refine the payload, call conditions & more, and a good payload validation on Z2M side.

The payload for a feature such as this PR could then be documented like others are, in the Z2M docs relevant pages.
We could then add a few "quick actions" in frontend to populate payload automatically (stored in ZHC and published in bridge/definitions).

Bonus: this could be mighty useful beyond this feature too, with the obvious being debugging purposes.

[Mstrodl]

This idea occurred to me too when we were talking earlier... I don't love the idea of having the logic and commands being totally at the whims of the consumer though. In this case, for example, the consumer would need to sweep the channels themselves and send a message to each. Additionally, kicking us back out of InterPAN mode between each message could cause some weird behavior if an interview for a new device happens before we send the next channel's message. We'd also need to add some extra handling for data types which can't be natively serialized as json like Buffer. I dunno. If it's what you guys want, we can do it this way, but it feels like more of an escape hatch than a good solution to support these devices.

I also don't love doing it with ZHC converters either though...

ZHC could have a separate "detached converters" export with some manufacturer specific commands which take in a ZH.Controller (which would gain functions for inter-PAN messages, which mostly are thin wrappers around Adapter). From there, Z2M can have a command to cue up one of the "detached converters". Would you prefer that to hacking it into the existing converter structure?

[Nerivec]

The raw topic allows far more advanced behaviors to derive from it though. We can of course have just payloads associated with a "name", but we can also have a "name" associated with a more advanced logic as needed.
E.g.
ZHC: src/converters/rawActions.ts

// publish whole in `bridge/definitions`
export const RAW_PAYLOADS: Record<string, RawPayload> = {
  xyz_trigger_custom_behavior: {}, // omitted for brevity
  // ...
}

// publish keys in `bridge/definitions`
export const RAW_ACTIONS: Record<string, () => Promise<void>> = {
  hue_serial_reset: (sendRaw: (args: RawPayload) => Promise<void>, args: Record<string, unknown>) => {
    // trigger more advanced logic
  }
  // ...
}

And we can trigger from Z2M in bridge extension using something like

import {RAW_ACTIONS} from "zigbee-herdsman-converters";

function raw(message: string | KeyValue): Promise<Zigbee2MQTTResponse<"bridge/response/raw">> {
  // message: {action?: string; args?: Record<string, unknown>, payload?: Record<string, RawPayload>};
  if (message.action) {
    // validation
    const func = RAW_ACTIONS[message.action];

    if (func) {
      return func(this.zigbee.sendRaw.bind(this.zigbee), message.args ?? {});
    }
  } else if (message.payload) {
    // validation
    return this.zigbee.sendRaw(message.payload);
  }
}

Again, just a quick write-up, needs more refining.
This allows maximum extensibility.
Integration in frontend would be easy, we can integrate buttons to hide actions/payloads, we can even extend the raw defs to include conditions (as in "can only trigger if") to show/hide buttons, prevent use, etc..

[Mstrodl]

For now, I can add just raw actions (and a hue reset action as a "patient zero" of sorts) (no custom/predefined payloads, etc. Those can be added later if you want them) to Z2M, ZH, and ZHC. Reasonable?

To expand a little more, I was thinking:

export const RAW_ACTIONS: Record<string, () => Promise<void>> = {
  hue_serial_reset: (context: RawContext, args: Record<string, unknown>) => {
    rawContext.withInterPAN((setChannel) => {
      for(const channel of scanChannels) {
        setChannel(channel);
      }
      rawContext.sendZCLBroadcastWhateverThisFunctionsForgettableNameIs(Zcl.Frame.create(/* ... */));
    });
  }
}

RawContext is just going to be a thin wrapper around Adapter, owned by ZH. Very similar to how ZH.Endpoint works today.

Only other thing I can think of is how to register the cluster. I could just create it in ZHC at the call-site (i.e. in the hue_serial_reset action) rather than actually registering it, but I assume that's not the "right" way to do it. Thoughts? Seems like existing ZHC support for custom clusters still have to be associated with a particular device.

Thanks!

[Koenkk]

@Nerivec I like it!

[Nerivec]

@Mstrodl I'll work out a base, we can refine what's needed as we go, I'll ping you to make sure implementing this particular custom will fit nicely, we'll use it as a template (I don't have any Hue device to test this with, so you'll have to 😉).