Using adb directly runs it under your user, which will probably be unable to access the necessary USB device.
Starting the server manually under a privileged user is the easiest way to circumvent those restrictions if you don't want to fiddle with udev rules, which is the recommended solution, but is more work.
It only works for me with one of my two USB ports, and my Kobo ereader has the same issue. Not sure why, best guess is one might be USB 2.0 and the other 3.0
Not sure what the issue was, I did not debug it. I will try again and see if it works or not, and will debug it further if it does not work. Arch Linux or Void Linux definitely should offer the same or more (or better) support.