bluetooth: Add cmd validity checks at the start of hci_sock_ioctl()

bluetooth: Add cmd validity checks at the start of hci_sock_ioctl()

Previously, channel open messages were always sent to monitors on the first
ioctl() call for unbound HCI sockets, even if the command and arguments
were completely invalid. This can leave an exploitable hole with the abuse
of invalid ioctl calls.

This commit hardens the ioctl processing logic by first checking if the
command is valid, and immediately returning with an ENOIOCTLCMD error code
if it is not. This ensures that ioctl calls with invalid commands are free
of side effects, and increases the difficulty of further exploitation by
forcing exploitation to find a way to pass a valid command first.

Signed-off-by: Ruihan Li <[email protected]>
Co-developed-by: Marcel Holtmann <[email protected]>
Signed-off-by: Marcel Holtmann <[email protected]>
Signed-off-by: Luiz Augusto von Dentz <[email protected]>