Issue
I'm using rabbitmq as a message broker with a spring project that uses websockets with STOMP. I was looking on how many concurrent websocket connections (i.e. how many concurrent mobile and web users) I can have, I noticed there are 3 different places that affect that:
1) maxConnections in apache tomcat 9 config.
2) number of file descriptors which said to be checked by ulimit -n
3) number of ports allowed by the server machine (I saw that each new user that opens a websocket connection with my server, rabbitmq shows that the connection is using a new port).
So I'm a little lost:
(a) Does each websocket actually need a dedicated port? isn't there the concept of channels to utilize connections over available ports?
(b) is the number of channels related to number of file descriptors? and what value should I put (I mean can a single server node of 8 cores and 16 GB RAM handle a million file descriptors, meaning million concurrent websocket clients)?
(c) And most importantly, if the message broker uses a unique port for each client, then the first two configs (maxConnections and # of file descriptors) won't matter cause I'm limited to around 60K ports for TCP after modifying that on the OS?
(d) In summary, what is the major factor of max concurrent player I can have in my game (I have it on mobile and web app).
Can anybody please explain the full cycle to me including all pre-mentioned components (web server, OS of the web server, OS of the message broker server and the message broker itself)? and how come service like cloudamqp says they support up to 160K concurrent connections on a single node, while I see that each client (app user) is reserving a port?
Thanks a lot for explaining the process in advance.
Solution
(a) Does each websocket actually need a dedicated port? isn't there the concept of channels to utilize connections over available ports?
STOMP has no concept of channels (https://stomp.github.io/stomp-specification-1.2.html). Every STOMP client has at least one TCP connection associated with it.
(b) is the number of channels related to number of file descriptors? and what value should I put (I mean can a single server node of 8 cores and 16 GB RAM handle a million file descriptors, meaning million concurrent websocket clients)?
Channels are an AMQP-specific resource that aren't associated with file descriptors. AMQP / STOMP TCP connections are associated and each connection consumes one file descriptor. The number to use depends on the benchmarks you have run to see what your server(s) can handle.
(c) And most importantly, if the message broker uses a unique port for each client, then the first two configs (maxConnections and # of file descriptors) won't matter cause I'm limited to around 60K ports for TCP after modifying that on the OS?
Every TCP connection is identified by a four-tuple of source IP, source port, destination IP and destination port. Because of this, you are not limited to 60K ports. If your connections are all coming from the same IP address (like a load-balancer or proxy) then yes, you may run into this issue as only one source IP is used.
(d) In summary, what is the major factor of max concurrent player I can have in my game
Only you can answer this by running benchmarks that simulate real-world expected load. Maybe your server has a slow disk, that could be the "major factor". Or maybe it's hosted on an unreliable network. I suggest testing your code with a tool like toxiproxy to simulate what can go wrong.
NOTE: the RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.
Answered By - Luke Bakken