The Recent DDOS Attacks Aimed at the Web server Hosting LiquidVPN has Pushed Back our Scheduled VPN Network Security Upgrade.
From June 1st 2014 to now (June 4th 2014) our web server has been the target of various distributed denial of service attacks effectively taking down our front end web server for a combined total of around 15 hours of downtime over the last 4 days. Currently the server is still being hit but some new equipment and services put in place over the last few days is mitigating most of the damage. Because the scheduled changes will force users to update their configuration files to connect we have decided to pus the upgrade back at least 3 extra days. We made the decision to push the upgrade back because the support issues that are sure to arise after the upgrade will require our technicians to be on hand to answer support requests and our web server to be online so users can download the latest configuration files.
The Attacks & Its Effects on our Network. The Lessons we Learned and the Silver Lining.
The first Distributed Denial of Service Attack Aimed at our Web Server Started June 1st 2014 at Around 1:30 PM EST. The attack only got up to 2000Mbps in size before we null routed the server IP stopping the attack from getting larger but also taking our website down for an hour or two. At this point we were not sure if the attack was aimed at us or if it were someone simply experimenting with a new found power. We decided to re-route the IP address and wait to see what happened.
The second DDOS attack started on Jun 1st 2014 at 8:44 PM EST. Again the attack got up to around 2000Mbps before we null routed the servers IP address to stop the attack from getting and larger. By null routing the IP we were able to keep the attack from bringing down any critical infrastructure located on the same switch. For instance a VPN authentication server or even some of our colo’s hardware. After we null routed the IP we realized the attack was meant for us. We began by reaching out to Incapsula a very well respected DDOS mitigation and CDN. We explained our infrastructure and got setup on their enterprise plan. We also reached out to several of our colo’s and upstream providers and have begun the process switching to a much more robust server located on a network segment with built in DDOS mitigation (more on this as the move comes closer). After setting up the DDOS mitigation we decided to re-route the affected IP address. The LiquidVPN website was down for about 6 hours while all of this was going on.
The third attack happened on June 3 2014 at 9:10 AM. One thing we (perhaps naively) were hoping was that the attacks were aimed at a domain name and not directly at the IP address because it takes time for IP address changes to propagate. This attack proved us wrong. It bypassed the DDOS mitigation in place because they still had the server IP in their DNS resolver. We did not let this attack reach 1000Mbps instead we null routed the IP address and then simply changed the servers IP addresses. The server was only down for a few minutes but DNS propagation took several hours. Meaning some people were not able to connect to the web server for up to 8 hours.
The fourth attack happened on June 3 2014 at 8:01 PM and has been happening off and on for the last 24 hours. After the server resisted the attack for several hours we began seeing a second type of DDOS that was exploiting open DNS resolver’s and networks that allow spoofed IP address DNS Amplification Attack the attacker was able to achieve up to 10,000Mbps. So far Incapsula has allowed the server to remained online. We dare not say its over yet. Until we are happy that the steps taken to protect the server against further attacks are working the way we want them too we do not want to begin going live with the VPN network enhancements. It is possible this is two separate attackers. The previous 3 attacks were CHARGEN attacks. Its hard to believe anyone would have this protocol internet accessible still.
The silver lining in all of this is that the overall health of the VPN network was virtually unaffected. We are finally getting a chance to focus on some aging infrastructure that was on the back burner. The new hardware and CDNs will give us the resources to implement some new features that were not possible on our current platform and we really got under some wankers skin.