Tuning NGINX maps hashes

NGINX maps

NGINX maps allow you to create variables based on key-value pairs, and they can be used to improve the performance and scalability of your website by reducing the number of conditionals and lookups required to process requests. You can create maps using the map directive in the NGINX configuration file, and you can use them in various contexts, such as in the server, location, or http blocks.

To tune NGINX maps, you can consider the following factors:

• Size of the map: The size of the map can have an impact on performance, as larger maps may require more memory and processing time to evaluate. You should aim to keep the size of your maps as small as possible, and consider using multiple maps for different purposes if necessary.

• Structure of the map: The structure of the map can also impact performance, as certain structures may be more efficient than others. For example, using a hash table can be faster than using a linear search, especially for large maps. You can use the hash parameter of the map directive to specify the type of hash table to use.

• Use of variables: Using variables in your maps can improve the flexibility and maintainability of your configuration, but it can also increase the complexity and processing time of the map. You should use variables sparingly and consider the impact on performance when using them.

• Use of regular expressions: Using regular expressions in your maps can allow you to match patterns in your keys or values, but it can also increase the processing time of the map. You should use regular expressions sparingly and consider the impact on performance when using them.

By tuning your NGINX maps, you can improve the performance and scalability of your website and reduce the resources required to process requests.

Using maps

The ngx_http_map_module is a great feature of NGINX allowing you to complement the declarative nature of its configuration with dynamic variables creation.
You typically only use the map directive from this module. It allows the creation of new variables based on different values of other, built-in NGINX variables, that contain various request’s data, like HTTP header values or query parameters.

For one example, you may want to check out Performance-friendly way of blocking Referrer spam.

In there, we were able to extract the domain name found in the Referer HTTP header, all with the use of map, we created a new variable $http_referer_host.
And we further created another useful variable $bad_referer using another map that sets its value to 1 if the domain belongs to a domain from the list of known bad “spam” domains:

map $http_referer_host $bad_referer {
    hostnames;
    default       0;
    .bad-one.com   1;
    .another-spam.com 1;
    # etc. 4k+ more entries!
}

This map being substantially large and containing over 4K records, requires setting up the following NGINX configuration:

map_hash_bucket_size 128;

Let’s talk about why this is needed, and the algorithm of finding the right configuration.

Hash sizes

Aside from map, there are other little known directives of the ngx_http_map_module.
The in-memory hashes is what allows NGINX to quickly match with entries found in the map directive’s values.
The 2 little known directives are meant for setting the size of data tables which those hashes in memory. They are:

• map_hash_bucket_size, the default is 64 on most systems but varies based on CPU being used. Further down we will find the actual default value used on your system. This sets the bucket size for the map variables hash tables.
• map_hash_max_size, the default value being 2048. This sets the maximum size of the map variables hash tables.

In fact, these directives are little known for a reason. If NGINX starts up fine without them, that is with the default values of these directives, then you already run the optimal setup tailored to your maps data. This is because, NGINX does a some smart job at determining the best actual size for the data tables in memory:

During the start and each re-configuration nginx selects the minimum possible sizes of hash tables such that the bucket size that stores keys with identical hash values does not exceed the configured parameter (hash bucket size).

And also because NGINX, automatically sets the default for map_hash_bucket_size to match up with the CPU being used.

So you rarely want to touch those directives. But… what if your NGINX refuses to start or giving error about maps tuning?

Such a rare case is relatively isolated to having a large map with thousands of entries. And that’s exactly what we have in our referral spam approach “big map example”.

If we are not to add the previously mentioned map_hash_bucket_size 128;, NGINX will refuse to reload or start completely with this error:

nginx: [warn] could not build optimal map_hash, you should increase either map_hash_max_size: 4096 or map_hash_bucket_size: 64; ignoring map_hash_bucket_size

Next, here is how to determine the right configuration for any huge map and make NGINX start/reload fine with it.

Get the CPU cache line size

Before we proceed further, it helps to know that any of the 2 directives are measured in bytes. And their values should be aligned to the CPU cache line size.

You can easily find your processor’s cache line size with the following:

cat /sys/devices/system/cpu/cpu0/cache/index0/coherency_line_size
> 64

So 64 is automatic/default value of the map_hash_bucket_size on a given system.

Thus, the minimum value for map_hash_bucket_size on this specific machine, should be at least 64.
If we want to increase it further, aligning means that the next possible values are 128, 256, etc.

Values like 65, 100, etc. that don’t align to cache line size, can not be used.

The workflow of following NGINX advice correcly

So when we have the error about both directives having a problem like the above error, NGINX documentation gives an advice about setting up hash sizes:

Therefore, if nginx emits the message requesting to increase either hash max size or hash bucket size then the first parameter should first be increased.

Unfortunately, the NGINX advice uses some incorrect wording in its advice which creates a lot of confusion. The “either” should be “both … and …”.
Armed with that, here’s the correct advice:

Therefore, if nginx emits the message requesting to increase BOTH hash max size and hash bucket size then the first parameter should first be increased.

That is important because NGINX typically can give only two kind of errors related to maps asking you to adjust either one or both parameters.
The adjustments for when you have only one are different for when you have both.

Anyway, we follow that advice and add:

map_hash_max_size 4096;

Here we doubled, from the default, the maximum hash table size from 2048 to 4096.

But checking configuration with nginx -t leads us to a similar message with updated values:

nginx: [warn] could not build optimal map_hash, you should increase either map_hash_max_size: 2048 or map_hash_bucket_size: 64; ignoring map_hash_bucket_size

We must continue following NGINX advice and increase only the map_hash_max_size, when NGINX offers to increase both parameters.

For example, the next value we try is 4096, and we now get:

nginx: [warn] could not build optimal map_hash, you should increase either map_hash_max_size: 4096 or map_hash_bucket_size: 64; ignoring map_hash_bucket_size

So we double and double and keep trying to get a different, hopefully successful output.

Finally, after increasing further and further up to map_hash_max_size 16384;, we may reach nginx -t telling us a different kind of message:

nginx: [emerg] could not build map_hash, you should increase map_hash_bucket_size: 64

Note the difference to all the previous messages. It only mentions one parameter: map_hash_bucket_size. The above advice no longer applies.

And after you increase the mentioned parameter, you’ll notice now that having map_hash_bucket_size 128; likely solves the startup problem caused by your large map.

You may have a feeling that you’ve done everything NGINX told you and made an optimal setup. However, it may be not optimal as it is.

Going backwards

So our complete settings now, if we follow NGINX documentation’s advice and the output from NGINX are:

map_hash_max_size 16384;
map_hash_bucket_size 128;

But is this correct? Is it efficient? Both answers, are unfortunately “no”.

You may notice now, that gradually decreasing map_hash_max_size by half or removing it altogether will still make NGINX start and reload just fine and give no errors in nginx -t. But hey, why did it tell us to increase the value of it in the first place?
And why if we only use map_hash_bucket_size 128; it also no longer complains?

Another question coming in to consideration. Are these going to work any differently:

map_hash_max_size 16384;
map_hash_bucket_size 128;

… vs just map_hash_bucket_size 128;?

There is relatively simple answer to this: nothing is perfect in this world.

Let’s consider our referral spam map with a huge map consisting of 4k entries. It uses hostnames directive, and we specify domains with a leading dot. This surely has a lof unique values, but the default 2048 of map_hash_max_size fits with storing all the unique (non-colliding) values. It is important to understand that, for this point, values abc.com and xyz.com, while still being unique values, have a certain duplication of data, and thus colliding hash keys.

In fact, there’s a lot of duplication in that map. Many domains end with the same TLD, for example .com.
The bucket size needs to be bigger when we have a lot of duplication (read, some hashes) in maps data.

NGINX is trying to be smart for us, but not to the full extent. The advice from documentation is for when we have more unique data.
And then we increase “unique storage” for our map. But after all tries, NGINX sees that too many scans are still required to locate data by non-unique hashes, and offers us to increase the “non-unique” storage (the bucket size). Implementing “suggestion” logic of going backwards to find the right optimal values for these 2 parameters and a big map, would be tricky within NGINX itself. So it’s not there.

The map_hash_max_size, if set unnecessarily high, only allocates more memory for nothing.

If all the data fits in the map_hash_max_size, it is important to decrease map_hash_max_size or remove it altogether if possible, as a larger value results in higher memory usage.

Surely, there are cases when you do need map_hash_max_size and has to be increased, but such are going to be with a map data where there are many and many more unique data entries than our example.

Our spam domain map, despite being large, fits just fine within the default value, but due to duplication, requires a larger bucket. Thus we only need map_hash_bucket_size 128; and nothing else.

So generally speaking, all we need to do for a big map that causes NGINX startup issues, is to go through the tries and failures of increasing the max hash size, increasing the bucket size if needed, then attempting to decrease the max hash size if possible.

Rules of thumb

• You need a larger map_hash_max_size only when you have a lot of truly unique data within your map
• You need a larger bucket_size when you have a lot of duplication within the map values like is the case for big hostnames map which has a lot of repetitive TLDs like example.com, foo.com, etc.
• When nginx -t gives you information about adjusting both directives, increase only map_hash_max_size first, like in the advice, until you reach a different kind of message or success. When offered to increase bucket size, do so, until success. But don’t forget to attempt decreasing the max hash size after all, for a better memory footprint.

Literature

• ServerFault