[Editor’s Note: This post is a part of a series on this blog, called Jixee Hotfix. It will feature real problems that our engineering team encounter on a weekly basis and the solutions they come up with to fix it. Posts are written by the engineers encountering the problems. This post was co-written by our CEO, Rishi Mathur and VP of Ops, Eric Norton.]
In this day and age software companies deal with hackers, DDOS and brute force server attacks on a daily basis. Here at Jixee we maintain strict security protocols and monitor network access 24/7. While hacking attempts are not new to us, we did pick up an interesting pattern last week and we want to share our findings with you. This will be the first of a two-part series describing the hackers’ approach and how we mitigate any future attempts.
How Did This Start?
Our infrastructure team deals with brute force attacks on a daily basis. All day long, our networks and clusters of servers absorb bogus requests and brute force attacks. At the end of the day our team looks at the logs and determines how many attacks took place and whether it is something to worry about.
Over the past few months, we noticed a steady increase in brute force attacks to our servers, particularly our SSH ports. A few weeks ago, we started to notice thousands of strange looking requests in our logs. These requests increased to the point where a few of our web nodes went offline, signaling the start of a DDOS attack. Now, DDOS attacks are quite common, but the nature and the country of origin of this attack really piqued our attention.
What We Discovered From the Attacks
Every request that comes to our web nodes has an identifier called a user agent. A user agent is a way of identifying a device that is trying to access our services. It tells us whether a request is coming from a mobile device, a web browser, etc. Here at Jixee, we are used to seeing the standard user agents like Safari, Firefox, Chrome, and others.
This is where things start to get interesting. At the start of the DDOS attack, we noticed that all the requests were originating from the same country. Additionally, the user agent was identifying itself as iOS and Android versions of the Facebook SDK (software development kit). After a little research, we determined that these requests are usually made to Facebook’s API.
So what does all of this mean? Essentially, Jixee experienced a coordinated attack by thousands of mobile devices using the Facebook SDK. These devices were leveraging the Facebook SDK to make thousands of requests per second to our services, in an attempt to bring Jixee down!
Possible Scenarios That Lead to Attacks
So what happened here? How does one (or many) coordinate such an attack? It’s difficult to come up with a definitive answer to the question “How did they do it?” without a healthy dose of speculation. So let’s take out our black hats, white hats, and tin foil caps, grab a tea and spitball some ideas.
First off, one of the biggest questions to ask, is why us? What would anyone gain by taking out Jixee? The truth is, probably nothing. It could be a test to see how secure our network is, and if one should bother trying to exploit it. It could be an angry teen who saw a post on a forum, didn’t like the responses, and decided that we were the lucky target of their angst. Whatever it was that motivated these attacks, it’s worth figuring out how they were accomplished, so we can better defend ourselves against future attacks. Let’s look at some potential scenarios and see if we can figure out how in the world they are pulling this off.
The most obvious reason this occurred is because thousands of mobile users unknowingly downloaded an infected app. We’ve heard the horror stories, read the news reports, but never in a million years would we EVER download a flashlight app, and then auto-update it. NEVER right? Well, you and I wouldn’t, but the millions of ‘new’ mobile device users that come online every day might. Once enough of these unassuming users’ devices are compromised, a botnet is born. A maniacal botnet master can now do whatever they please with these compromised devices. If a botnet master decides they want to throw a million requests at Jixee, just for the heck of it, they do it. The voiceless now have a voice, and they are called botnets.
Due to the nature of these requests, it could be true that a slew of mobile networks in this specific country were hijacked. And by a slew, we mean thousands. Could it be a coincidence that all of these attacks originated at mobile networks? We think not. The hijackers can compromise certain services of a mobile network to allow them to steer requests from their users in a certain direction.
Now, put on your tin foil cap, and consider this. In this age where data and intellectual property are king, would it not be advantageous to ‘get ahead’ in the world, if one could simply steal these assets? And, would it not benefit a nation to do just that? Edward Snowden tipped us off to the knowledge that all of our data is sucked into one huge NSA vacuum on a daily basis. Who is to say that the NSA is the only one doing this? It’s not outside the realm of possibility, since these attacks came from only one country, that this country sponsors the creation and use of botnets to pound on every web service’s door to see if they can break it down. We were lucky our door was bolted, but how many doors aren’t? Who else is vulnerable to seemingly random DDOS attacks from a particularly contentious country? Could your company be next?
Have you experienced this or something similar with your software? What do you think happened? Comment below with your opinions and conjecture and watch out for Part II next week.