I just knew X-Forwarded-Proto in production

In the middle of a beautiful day, I deployed a change to production and expect everything to go well. Anxiety hit me well while waiting for the incoming request to come to my system. This is quite a big feature and I just design it end-to-end and expect internal users will have a smooth flow while they hit the API. But it turns out they didn’t. All incoming requests got an error 404 response code while it didn’t send any additional data as a response body. I was confused, I didn’t know why it returned a 404 response code. Logging didn’t help either as it didn’t give much helpful information. The code is well-tested in staging and that’s why I am quite confident that the feature will work well. Digging deeper into the issue and I found the root cause, it was caused by the X-Forwarded-Proto header. Let me tell you more below.

System Flow

Basically, I design a service that works similarly as an API gateway with the purpose of forwarding the request to another service. I forward not only the request body that coming in. I also forward all the request headers to the service to make sure that all additional data from the client will be forwarded to the service through the gateway.

Finding The Culprit

When the code was deployed to production, I didn’t expect the code won’t work since the code itself was already well-tested in staging environment. Then, when I saw the rollbar alert coming in, I bit surprised yet curious about why I encountered such issue. Even the log didn’t show any response body from the request made to the service. I try to dig deeper and try to reproduce the issue by sending all incoming request body from client and its header from Insomnia. As expected, the request failed. Then I try to disable the header request one by one to know whether any of incoming request headers affect the way system forward the request from client → gateway → service. Then VOILA! When I changed the X-Forwarded-Proto header from HTTP to HTTPS, the request is going through to service and returns 200 response code, which is the response code that I expected since I design the system. Then I try to understand what is the header definition, and why this header may impact the way my system behaves.

X-Forwarded-Proto definition from https://http.dev/x-forwarded-proto:

The HTTP X-Forwarded-Proto header is used to identify the original protocol used by a client to communicate with an intermediary. This is typically either HTTP or HTTPS. This is not needed in situations where the client connects directly to the server. However, when intermediaries exist, the server can only determine the protocol used between itself and the most recent proxy or load-balancer. The X-Forwarded-Proto preserves this information that will otherwise be lost.

This sentence hit me hard: However, when intermediaries exist, the server can only determine the protocol used between itself and the most recent proxy or load-balancer. The X-Forwarded-Proto preserves this information that will otherwise be lost.

I just realized that my system production setup is a bit different from the staging.

By taking the definition from http.dev and realizing the production setup, if I forward the X-Forwarded-Proto header with value of HTTP from the gateway, it means the proxy server will make a call to the service by using HTTP protocol instead HTTPS, which error 404 response code starting to make sense since all request to the service need to be under HTTPS protocol. It seems the proxy server can’t forward the request to the service and can’t find the particular route under HTTP protocol, hence 404 response code returned.

Conclusion

It’s pretty helpful as developers to be able to reproduce the issue by sending the EXACT same request body and headers to the API. We can know the root cause by trying to mimic the request by sending each of incoming request headers. I also learned the X-Forwarded-Proto for the very first time by facing this issue. Its behavior is quite unique for the request with intermediaries exist in between.