Stoutner

Programs that respect your privacy

The X-Requested-With Header

Information in this post regarding workarounds and long-term solutions has been superseded in Privacy Browser 3.12 by access to WebView’s DevTools. I leave the original post for historical reasons and because it might be of interest to some readers.

As mentioned is several places on this website and in the app, there are some negative implications of using Android’s WebView to render web pages in Privacy Browser. The purpose of this post is to describe one of these downsides, explain the mitigations that are currently being taken, and describe the permanent solution that will be implemented in the future.

The Background

When any app uses Android’s WebView to load a web page, WebView attaches an extra header, named X-Requested-With, with the value set to the application ID. X-Requested-With is not a standardized header, but it is commonly used as a flag to mark AJAX (Asynchronous JavaScript and XML) requests. In that sense, WebView’s use of the field for a different purpose can cause issues for some web pages.

The Problem

One of the web tracking technologies that Privacy Browser is designed to mitigate is browser fingerprinting. Any piece of information the browser sends a web server that makes it stick out from the crowd increases the ability of the web server to uniquely fingerprint the browser. Including the app ID in the header, especially as long as Privacy Browser has a small market share, increases the chance that the total information sent to the server is unique.

For the Standard version of Privacy Browser the app ID is “com.stoutner.privacybrowser.standard”. The following is a log from www.stoutner.com that shows what information Privacy Browser <= 1.8 transmits to a web server (note that the User Agent has been changed from the default to “PrivacyBrowser/1.0”).

GET / HTTP/1.1|Host:www.stoutner.com|Connection:keep-alive|Cache-Control:max-age=0|Upgrade-Insecure-Requests:1|User-Agent:PrivacyBrowser/1.0|Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8|Accept-Encoding:gzip, deflate|Accept-Language:en-US|X-Requested-With:com.stoutner.privacybrowser.standard
The Workaround

Google doesn’t want to make it easy to get rid of the X-Requested-With header. However, there is a mechanism for replacing header information. This doesn’t allow a program to stop sending the X-Requested-With header, but it does allow a program to replace the app ID with a null value. Beginning with Privacy Browser 1.9, the following information will be sent in the headers:

GET / HTTP/1.1|Host:www.stoutner.com|Connection:keep-alive|Upgrade-Insecure-Requests:1|User-Agent:PrivacyBrowser/1.0|x-requested-with:|Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8|Accept-Encoding:gzip, deflate|Accept-Language:en-US
The Problems with the Workaround

The first problem with this workaround is that sending X-Requested-With: is not the same as not sending it at all. There are only going to be a few browsers that send X-Requested-With: as a header, so for fingerprinting purposes this workaround is only slightly better than including the app ID.

The second problem is that the technique for overwriting WebView’s headers only works on the initial request to a website. Any dependent requests for resources (like images, CSS files, JavaScript files, etc.) use the default headers. So even in Privacy Browser >= 1.9, the log of a request for a resource will be as follows:

GET /wp-content/themes/twentysixteen/style.css?ver=4.5.3 HTTP/1.1|Host:www.stoutner.com|Connection:keep-alive|Cache-Control:max-age=0|User-Agent:PrivacyBrowser/1.0|Accept:text/css,*/*;q=0.1|Referer:https%3a//www.stoutner.com/|Accept-Encoding:gzip, deflate|Accept-Language:en-US|X-Requested-With:com.stoutner.privacybrowser.standard|If-None-Match:"10d4b-535f10e3fb580-gzip"|If-Modified-Since:Thu, 23 Jun 2016 12%3a18%3a46 GMT

This is a problem for every browser that uses Android’s WebView. For example, Lightning 4.3.3 behaves the same way as Privacy Browser >= 1.9: initial web requests include “X-Requested-With:” and resource requests include “X-Requested-With:acr.browser.lightning”.

Option to Use the Default Behavior

Even though the spec allows headers with null values, some websites don’t like it, usually causing them to fail to load. Privacy Browser 3.11 has the option to revert to the default behavior of sending the app ID, either app-wide or as a domain setting. Because this value is set when a URL is first loaded, and is not changed when a site is refreshed or loaded from the navigation history, changes to the X-Requested-With header may not take effect until after the app is restarted. Privacy Browser will restart automatically when changing the behavior app-wide in the settings. When changed in domain settings, the user will need to either restart the app or load the domain in question in a tab that doesn’t currently have it loaded.

The Long-Term Solution

There is a provision in WebView to bypass the default web loading functions by manually acquiring the information using HttpURLConnection and feeding it into WebView. This would allow the complete removal of the X-Requested-With header. However, making sure that all possible types of HTTP communication are handled correctly would be complex and any initial implementation would likely result in buggy behavior.

As a better solution, in the 4.x series Privacy Browser will switch from using Android’s default WebView to a custom rolling fork named Privacy WebView. This will allow for the complete removal of the X-Requested-With header.

Last updated

Soren Stoutner

14 responses to “The X-Requested-With Header”

  1. […] Privacy Browser now handles “mailto:” links on websites.  A couple of bugs were also fixed that caused the browser to crash when working with bookmarks with no favorite icon and the navigation drawer to sometimes stop working after viewing a full-screen video.  Progress was made regarding the “X-Requested-With” header, which was covered in more detail in a previous post. […]

  2. […] of implementation imposed by Android’s WebView that were described in the post about the “X-Requested-With” header. Specifically, the DNT header is only sent with the initial URL request, and not with resource […]

  3. Why not just comment out the relevant section of the source, build it under a different package name, and only use the system default as a last-ditch fallback? I’m also not entirely sure why the LD_PRELOAD trick with function overloading like tools like torsocks and proxychains use wouldn’t work here. Am I missing something?

    • The plan is to do exactly that with the creation of Privacy WebView in the 4.x series. But it is a non-trivial endeavor to create a rolling fork of WebView. For example, it is unlikely that F-Droid will be able to easily build a custom version of WebView because it does not use the build system that is commonly used by most Android apps, meaning that it will take effort and likely contributions to the F-Droid source code to be able to distribute a custom WebView through their platform. However, if you are aware of an easy way to do it you are welcome to submit a patch to the code.

    • A user could use a local proxy if they desired, but there are significant shortcomings to doing so.

      1. A local proxy cannot modify HTTPS traffic (the default) unless it unwinds the SSL connection, modifies the header, and reinitializes the HTTPS connection on the other side. That is, the proxy would have to perform a MITM (Man In The Middle) attack. Doing so would require installing a wildcard SSL certificate on the device for all domains on the internet, trusting that certificate, and having the proxy use that certificate for all internet traffic. If you look at examples on the internet of systems that have done things like that it doesn’t often end well from a security perspective.
      2. The two ways to run a local proxy are either to use a local port or to use Android’s VPN capabilities. Using a local port would likely not play well with Orbot, which would break Tor for those wanting to use it. Running as a VPN would eliminate the ability to run another VPN, which many users are already doing.
      3. Running a local proxy as a MITM would mean that Privacy Browser could not see the original certificate presented by the website, which would break SSL Certificate pinning.

      Instead, I plan to fix this the correct way as described at https://redmine.stoutner.com/issues/37 in the 4.x series with Privacy WebView.

  7. I just found that installing Android System WebView Beta gives you access to the WebView DevTools. There you cannot only disable the X-Requested-With header for good, but also disable TLSv 1.0/1.1 among other things.

  8. Exposing the DevUI on the stable Branch of WebView would be very much appreciated, because all Apps that are using it will benefit from any changes made and it would also be a unique characteristic of PrivacyBrowser.