Are you already running Catalina? I observed a huge UI performance increase with the current Mac Mini on Catalina. The lagginess and stuttering you described bothered me to no end, but it feels (almost) completely fine on Catalina.

Edit: I have a 5k screen connected