Fork Bombs: Hardening Unix-Systems from Denial of Service (DoS) Attacks

Intro

A Denial-of-Service (DoS) attack is a malicious attempt to disrupt the availability of a computer, network, or website by overwhelming it with excessive traffic or resource utilization, rendering it unable to fulfill legitimate requests. Denial-of-Service (DoS) attacks come in various forms and exploit different weaknesses in a network or system to disrupt its availability. These attacks can range from protocol flooding (ICMP, SYN, UDP, HTTP, etc.), application-level intensity attacks, to even SlowLoris web attacks, which keep Web connections open for as long as possible.

With the growing media popularity of reporting on distributed-denial-of-service (DDoS) attacks, where large botnets compromised of millions of devices to overwhelm a single resource, it’s important to also be aware of other types of DoS attacks, including fork bombs, in order to better protect against them.

What is a fork bomb?

Fork bombs are a less well-known type of Denial-of-Service (DoS) attack which is particularly dangerous because they can cause a system to crash or become unresponsive. Fork bombs can be issued by any standard user and does not require any special privileges to bring down a computer. What makes a fork bomb so dangerous is the logic issued to the computer which causes an infinite loop of process generation.

A classic example of a fork bomb is one written in Unix shell :(){ :|:& };:, possibly dating back to 1999, which can be more easily understood as:

fork() {
    fork | fork &
}
fork

This is a shell function that recursively calls itself twice, once in the background (&) and once in the foreground (|). The function creates a new process each time it is called, until all available system resources are consumed, and the system becomes unresponsive.

The number of processes will rapidly increase within seconds. The exponential rate of process creation is what makes it so dangerous. A fork bomb is not “technically” an exploit, as the system processes the instructions exactly as intended, but it continuously generates new child processes, each of which in turn spawns more child processes, resulting in a dramatic increase in the number of processes consuming valuable processing power. This rapid depletion of system resources leads to a denial-of-service attack on the system as a whole.

If you’re interested in exploring a collection of different types of fork bombs, visit Aaronryank’s Github page for a comprehensive listing.

Protections and HArdening

Ulimit is useful on Unix-like systems as it allows caps to be placed on system resources.

As demonstrated in the image, the default maximum number of processes on this baseline Kali image is set to 7696, providing a level of protection against fork bombs.

If you have no limits set on your host (verified with ulimit -a or ulimit -u), you can set a limit with the following command where 2000 (arbitrary digit) is the maximum processes allowed for the user which is called a ‘soft limit’.

ulimit -u 2000

Keep in mind that with ulimit, a hard limit can be set as well with the following commands.

ulimit -Hu 2000

A soft limit is the limit that can temporarily be raised by the user until it reaches the value of the hard limit. A hard limit refers to the upper limit that can be set for a user, beyond which it cannot be raised by the user or any other process.

To validate and test the process caps on my account, I executed a fork bomb on a virtual machine and observed the rate-limiting mechanism taking effect within seconds, as indicated by the following system messages.

It is important to know that ulimit is considered temporary because its settings only apply to the current shell session and are not persisted after the session ends. This means that if a user logs out or the system is rebooted, the ulimit settings will not persist and will need to be set again.

This behavior is intentional and is designed to allow users to temporarily change the resource limits for their current session. For example, a user may need to increase the limit on the number of processes they can run for a specific task, but they do not want this increased limit to persist after they log out. The temporary nature of ulimit allows the user to make these changes without affecting the overall system settings.

However, if you want to make changes to the resource limits that persist even after a user logs out or the system is rebooted, you can modify the configuration files in /etc/security/limits.conf. These changes will take effect for all users on the system once the system is rebooted or the user logs out and back in.

Given the above explanation in /etc/security/limits.conf, the syntax for limiting the maximum processes to 2000 would be added to the end of the file in the format of:

kali hard nproc 2000

Let’s break down this command.

Kali is the username.
The type is hard indicating the maximum limit that can be set for a user, which cannot be increased later by that user or any other process.
nproc handles the maximum number of processes.
2000 is the value of the hard limit being set.

Baselining Process Caps

One common question regarding process caps is determining the appropriate limit for each user. This depends on the server’s resources and the functions performed by each account on the server. Imposing process caps without careful consideration could result in serious consequences on a production network, potentially causing disruptions in application performance. The solution to baseline the limits for each host and the accounts on them is through a little scripting.

It is recommended to run the following bash script using Cron on a regular basis, such as every minute or two. By letting the script run for a few weeks, it will capture and log only the highest process counts by each user, along with the corresponding timestamps, in the specified output location defined by the output_log variable. This will provide you valuable information for baselining process limits on a per-user and per-host basis.

Script

#!/bin/bash

# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
# SOFTWARE.

# Store the log file location in a variable
output_log="/root/user_process_count_monitor_$(hostname).log"

# Store the current date and time in a variable
current_time=$(date +"%Y-%m-%d %T")

# Get a list of all users
users=$(cut -d: -f1 /etc/passwd)

# Loop through each user and get their maximum number of processes
for user in $users; do
  # Use the ps command to get the number of processes for the current user
  processes=$(ps -u $user | wc -l)

  # Check if the user's log entry exists in the log file
  line_number=$(grep -n "^.*, $user," $output_log | cut -d: -f1)
  if [[ -n $line_number ]]; then
    # If the log entry exists, retrieve the highest process count for the user from the log file
    highest_processes=$(grep "^.*, $user," $output_log | awk -F',' '{print $3}')

    # Check if the current process count is higher than the highest process count in the log file
    if [[ $processes -gt $highest_processes ]]; then
      # If the current process count is higher, delete the user's log entry
      sed -i "${line_number}d" $output_log

      # Write the new highest process count to the log file
      echo "$current_time, $user, $processes" >> $output_log
    fi
  else
    # If the log entry does not exist, write the user's process count to the log file
    echo "$current_time, $user, $processes" >> $output_log
  fi
done

If you are curious of the output from this code, which was executing every 60 seconds for a week, please see the table below demonstrating the output file contents in output_log.

TimeStamp	User	Peak Process Count
2021-09-20 09:25:00	user2	26
2021-09-20 12:16:00	user1	13
2021-09-25 18:48:00	root	595

Peak User Process Count Monitor Script Contents from Output File

Conclusion

Fork bombs are a real threat to computers and can easily cause major disruption if proper precautions aren’t taken. This can be done by setting process limits, using rate-limiting tools, or monitoring system usage. By being proactive and taking these measures, you can help ensure your server stays secure and stable.