This is done on the RD Broker server
This is who can connect in. After adding an RDWeb server, it added other general groups. In this example, I have locked it down to our Vendors and IT deparment
This is to illustrate where to actually "allow" users to get too
Setting the Servers for the Farm (place #1)
Put in the all servers for this farm. If you don't, you'll get a message below
Click on Server (Local) properties
Click on "View RD Gateway server farm members"
Add your servers to this farm.
Ensure that the Status is OK.
Make sure that you have RD Gateway Role installed on that particular server