The website fingerprinting (WF) attack on Tor shows that it is possible to detect which website a user is visiting with high accuracy just by looking at the traffic. One of members of the Tor project, Mike Perry, criticised this attack. Some of his main concerns were that the academic papers consider wrong adversary model and the base detection rate in the open-world WF attack would be insignificant because of high false positive rates.
We wanted to test how effective an adversary would be when deploying such attacks in the wild, rather than in highly advantageous lab conditions. We show that only if a user is browsing a specific version of a really popular webpage (as popular as Alexa top 10 sites) with only one tab on a specific Tor browser bundle (TBB) within a couple of days of the adversary then the adversary has a chance of knowing which webpage is being accessed. Collecting this level of information about a user is hard and on top of that this data has to be updated periodically, which makes a successful WF attack expensive. Given these evidences, how effective WF is on real world Tor users is still an open research question.
- Websites change over time. What if the adversary cannot access the same version of the website as the targeted users?
- What if the adversary’s assumption about a user’s browsing settings is different from reality?
- There are millions of websites in the world. What if the adversary doesn’t have a sample of the sites the users are browsing?
Websites change over time:
Webpages constantly change. This is reflected in the traffic traces and consequently in the accuracy of the WF attack. To test how quickly WF attack accuracy changes over time, we performed the attack with data collected at different points in time. As you can see in the following graph, the attack accuracy drops from 75% to just 50% if the data is only 10 days old, and the attack is completely useless for data that is 90 days old. Therefore, an adversary needs to collect data at least within 10 days of the targeted users visit to be at least 50% successful in the attack.
Data can be collected in many different ways:
There are many ways to browse a website through Tor. I will only talk about two settings: different number of tabs in the browser, and different versions of the Tor Browser Bundle (TBB).
We found that website fingerprinting cannot be successful if the user loads multiple pages at the same time using multiple tabs, which is beyond the control of the adversary. During our test, the accuracy of the attack dropped to below 10% if the adversary collected data through one tabbed browsing but the user used multi tabbed browsing. The picture below shows the accuracy of different WF classifiers proposed in academic research (gray bars) and the accuracy of the classifiers with two tabbed browsing (black bars).
There are different versions of TBB and browsing through different versions may change the traffic. What if the adversary uses the latest version but the user is browsing with an old version? We tested this with three Tor Browser Bundle versions, 2.4.7, 3.5 and 18.104.22.168, where we trained a model using data collected using one version and tested using data collected through another. We found that the accuracy of WF attack can as low as 6% if the adversary and the user do not have the same TBB version.
Too many websites to keep track:
There are many websites in the world, and thus it is reasonable to consider that the adversary can’t track all of them. What if the adversary does not have the website the user is browsing? It may happen either because the adversary hasn’t considered a page or the adversary has a different version of the page. To test this we performed WF attack on real users’ browsing data (80 users browsing data for 5 days, collected for Active Linguistic Authentication). We found that the academic machine learning model, trained on homepages of top Alexa sites, can identify only 3-56 webpages out of over 200 with very high false positives (over 300). The pages the classifier cannot detect were mostly specialized (e.g., dell.msn.com where training set contains msn.com) or personalized pages. So the adversary needs to train the system with every possible (or at least some) specialized page, otherwise the false positive rates would be too high for a reasonable detection rate.
Bayesian Detection Rate:
In Mike Perry’s post, he mentioned that the Bayesian detection rate (BDR) for WF attack is too low to be practical. One thing he overlooked is the prior of a page that is the probability of accessing a page. A popular page (for example, google.com) has higher probability of being accessed than an unpopular page (for example, a random page with Alexa rank over 500). We estimated the prior of 4 popular domains: google.com, facebook.com, twitter.com and wikipedia.org and of 4 unpopular domains (4 random pages with Alexa rank over 500) which is basically the following: out of all the urls 80 users browsed for 5 days, how many times page x was accessed? We found that for a popular page with high prior, the BDR is 53.1% but for an unpopular page the BDR is less than 15%. This means that an adversary would be wrong over 85% of the time. Note that even this is an upper bound because these pages are in Alexa. We suspect that the BDR for even less popular pages would be so low that it would render WF attacks ineffective.
Now, it can be argued that the adversary can collect more information about the targeted users to improve the attack accuracy, for example, by knowing which TBB version the users are browsing with, how many tabs they use and may be collect a couple of personalized pages. Collecting all these information is almost impossible and makes WF attack expensive. And if the adversary already knows so much about the targeted users, is website fingerprinting needed at all?
I want to emphasize that we are not saying that website fingerprinting attack is impossible but, rather, that the cost of actually performing a successful attack in the real world has been underestimated. There is a lot of value in understanding an attack from a theoretical perspective and developing defenses against a strong adversary, but it is important to estimate the cost of these attacks in the wild because that will help us understand how vulnerable a regular Tor user would be to this attack.