Echo 404 directly from nginx to improve performance

I am in charge of production servers serving static content for a website. Those servers are constantly being crawled by bots looking for potential exploits (which isn’t that much of a problem security-wise because no application can be reached behind the web server) but
generates thousands of 404 per day, sometimes per hour. I am looking into ways of blocking those requests but it’s tricky (you want to make sure you don’t block legitimate traffic and these bots are becoming more and more clever at looking like they’re legit) and is going to take me a while to find an acceptable solution.

In the meantime I would like to reduce the performance impact of serving those 404 pages. Indeed we’re using nginx which by default is configured to serve it’s 404 page from the disk (This can be changed using the error_page directive but in the end the 404 will either have to be served from disk or from another external source (e.g. upstream application which would be worst)) which isn’t ideal.

I ran a test with ab on my local machine with a basic configuration: in one case I echo a message directly from nginx so the disk isn’t touched at all, in the other case I hit a missing page and nginx serves its 404 from disk.

server {
  # [...] the default nginx stuff
  location / {  }
  location /this_page_exists {
     echo "this page was found";
  }
}

Here are the test results (my laptop has Intel(R) Core(TM) i7-2670QM + SSD in case you’re wondering why they are so high):

$ ab -n 500000 -c 1000 http://localhost/this_page_exists
Requests per second:    25609.16 [#/sec] (mean)$ ab -n 500000 -c 1000 http://localhost/this_page_doesnt_exists
Requests per second:    22905.72 [#/sec] (mean)

As you can see, returning a value with echo is 11% ((25609−22905)÷22905×100) faster than serving the 404 page from disk. Accordingly I would like to echo a simple 404 Page not Found string from nginx.

I tried many things so far but they all failed, essentially the idea was this:

location / {
  try_files $uri @not_found;
}
location @not_found {
  echo "404 - Page not found";
}

The problem is that as soon as the echo directive is used, the http response code is set to 200. I tried changing that by doing error_page 200 = 400 but that breaks the configuration.

How can I serve a 404 page directly from nginx? (without hacking the source which may be might next step)

Actually, by default, nginx generates a 404 response internally. It only serves a file from disk if you tell it to using an error_page directive. If you want to control the format of the 404 page, instead of echo "404 - page not found";, you can use return 404 "404 - page not found"; (assuming you’re using a somewhat recent version of nginx, I belive you need 0.9 or newer)

Check more discussion of this question.

Related posts:

1. ubuntu server 10.04 nginx 404 errors
2. nginx custom 404 error page for virtual host
3. Conditionally serve 404 file based on request url file extension in Nginx?
4. Nginx 404 Deploying Ruby on Rails Application with Phusion Passenger
5. Nginx redirect requests to sub-domains that do not exist to custom 404 page when wild card A record is set?