×

UniFi - How to Tune the Controller for High Number of UniFi Devices

Overview

This article will help users diagnose a UniFi Network Controller running under a high load, as well as describe modifications that can be done for optimal performance.

NOTES & REQUIREMENTS:
  • Applicable to the latest UniFi Controller release versions. 
  • This article describes advanced configuration options and should only be attempted by advanced users.
  • See the UniFi - Explaining the system.properties File article for more information on the system.properties file.
  • Create a backup using the UniFi - How to Create and Restore a Backup help center article and export/save your backup configuration to a separate device before following this article.

Table of Contents

  1. Introduction
  2. Symptom: High CPU Usage 
  3. Symptom: Heartbeat Missed or Slow to Provision
  4. Database Connection Tuning
  5. Conclusions
  6. Related Articles

Introduction

There are a few things to take into consideration when managing large UniFi installs with hundreds of devices and several sites connected to a single UniFi Network Controller. A Controller running under such a high load might encounter some issues if configured the same way as a Controller managing a much lighter load. There are some symptoms that can be recognized, diagnosed and worked on to improve the controller's performance. 

The sections below show how to modify the UniFi Controller system settings by altering or adding values to the system.properties file. See the UniFi - Explaining the system.properties File article for more information on this file.

WARNING: Before proceeding with the configurations outlined below, make sure to create a backup of your machine and Controller configurations. A typo error could break the system. See the UniFi - How to Create and Restore a Backup article for more information.

Symptom: High CPU Usage

One of the most important metrics to monitor, is CPU usage on your UniFi Network Controller. High CPU usage is the first indication that there is an issue. Unfortunately, there is no silver bullet for this and just increasing the CPU is not necessarily the answer.

Allocating Additional Memory
Before increasing the RAM allocation on your machine, first try increasing the `XMX` and `XMS` options. By default, the UniFi Network Controller has these set to 1GB.  The following lines set the xmx and xms to values to 2GB (2048MB):

unifi.xmx=2048
unifi.xms=2048

The changes above would increase the memory the UniFi Network Controller is allowed to consume from 1 to 2GB. The reason why not having enough memory could cause high CPU usage, is because the Java Virtual Machine would be spending too many CPU cycles collecting garbage to stay within the 1 GB of memory allocated to the Controller. So before moving to a machine with more CPU resources, it is recommended to max out the available memory on that machine with the above settings. Then observe if CPU usage decreases. If 2GB is not sufficient, administrators may want to raise the limit to 4-8GB depending on the scale. In this case it would simply be increasing the value above in increments of 1024, i.e. 4 GB= 4096. 

NOTE: The jstat -gcutil Java command can be used to check whether the memory allocation is sufficient on your machine. See the Oracle documentation for more information.

Enabling High Performance Java Garbage Collector
If increasing memory does not solve the problem, administrators may consider also adding this line to the system.properties file (note this only applies to Controllers other than Cloud Key):

unifi.G1GC.enabled=true

This enables a new Java Garbage Collector that can help optimize performance. However, if after these changes high CPU usage does continue after memory increase, a larger machine with more CPU cores and more memory to handle the workload may be necessary.

Changing Mongo Version/Engine
In persistent cases, administrators may consider updating MongoDB version to 3.2+ with WiredTiger as storage engine, as a means of better scaling the UniFi Network Controller. See the following links for instructions:

Symptom: Heartbeat Missed or Slow to Provision

No matter the number, all devices will try to inform back to the Controller. By default, the controller can handle 200 simultaneous connections from devices, so devices with heartbeats missed shouldn't be an issue unless a single Controller is managing thousands of devices. If it is only managing a few hundred devices the adjustment described in this section can be tried out, but it may not have the desired results. The number of simultaneous inform messages that can be processed can be set in the system.properties file by adding the following lines and adjusting the values to best suit your needs. It does not matter where these lines are added in the file.

inform.num_thread=200
inform.max_keep_alive_requests=100

The default value is 200 and the max_keep_alive_requests value should always be lower than than num_thread. Try adjusting up from there. An increase in device stability should be seen, and by pushing the configuration out to other devices, become even more stable. 

Database Connection Tuning

When running a large UniFi installation, it may be desired to run an external Mongo cluster to be able to scale the database independently from the UniFi Network Controller application. Discussion on that can be found here on our Community Beta forum (need access?). If high CPU usage is seen on the Mongo process, it can indicate the need of a bigger box or the need to separate the mongodb process as mentioned above. Once that is done, the following can be tuned to see if it results in better application performance: 

db.mongo.connections_per_host=100
db.mongo.threads_multiplier=5
This results in 500 threads that can be waiting for a Mongo connection. So keep in mind that more threads can mean higher CPU usage because the CPU has to context-switch between threads. It may allow for higher DB throughput, but only if the Mongo process is able to consume more CPU to serve requests faster. 

Conclusions

The best way to increase performance and provide stability for large installations is to ensure you have monitoring on your system resources. Beyond that, practices such as offloading the database workload, and increasing memory can allow the UniFi Network Controller to serve more clients and devices. This should be reflected in the resource usage and the performance of the UniFi Network Controller UI. 

Related Articles

UniFi - How to Create and Restore a Backup

UniFi - Explaining the system.properties File

UniFi - Configuring Multiple Sites

Was this article helpful?
29 out of 43 found this helpful
Can't find what you're looking for?
Visit our worldwide community of Ubiquiti experts for more answers
Visit the Ubiquiti Community