Knowledgebase

Portal Home > Knowledgebase > Articles Database > clubuptime.com going down?

clubuptime.com going down?

Posted by nixell, 03-05-2011, 02:30 AM
Hello, Sorry if i make this thread in wrong subforum are clubuptime.com going down? i can't access their website. Thanks
Posted by Matt R, 03-05-2011, 02:30 AM
We're under a large scale DDOS attack of over 40gbps in size overall. Please refer to the below URL for more information. http://updates.clubuptime.com/index....e-ddos-attack/ Edit: This has taken out our phones, billing center, support desk, primary website, and a few VPS nodes on the same primary IP subnet at Softlayer. It only affects three client servers at this time. Second Edit: Just in case anyone is tempted to, DO NOT post your IP addresses of your VPS servers here on WHT. This is a very targeted attack and posting your IP addresses publicly may potentially make it worse. Last edited by Matt R; 03-05-2011 at 02:34 AM.
Posted by Yasirjazz, 03-05-2011, 02:40 AM
what attact is this? some one is doing it?
Posted by Matt R, 03-05-2011, 03:59 AM
Your guess is as good as ours, to be honest. We expect to have the VPS nodes back online shortly. Clearing your DNS cache will bring the main Club Uptime websites back online for you as well.
Posted by kjetterman, 03-05-2011, 11:57 AM
I'm sorry you guys are going through this Matt. What a pain in the butt. (To put it lightly) Hope it gets cleared up for you soon.
Posted by FiberFy, 03-05-2011, 02:08 PM
I hope you guys fix it Matthew, I've been your client for some days and I love my VPS. (But I'm still waiting for IPv6 on my vps, hehe )
Posted by FiberFy, 03-05-2011, 03:00 PM
Hello, Bradlay S: Unfortunately your being lied to. There was no major attack even close to 10gig today within Dal01-07 Hmm, quite strange, can you clarify this Matthew?
Posted by SSL-Nick, 03-05-2011, 04:05 PM
Hmm would like to know this to
Posted by Matt R, 03-05-2011, 04:09 PM
We're in Dal05, not Dal07. Refer to Softlayer Ticket ID numbers <> etc... Before accusing us of lying, get your own facts straight. Last edited by foobic; 03-06-2011 at 12:53 PM. Reason: BAQ-828136
Posted by A J C, 03-05-2011, 04:11 PM
My question is who is "Bradlay S"?
Posted by SSL-Nick, 03-05-2011, 04:14 PM
I think he is part of sales.
Posted by A J C, 03-05-2011, 04:17 PM
There's nobody called "Bradlay" working for us. Not only that, the chat shows the department name, not the name of the person you're talking to.
Posted by Aaronn, 03-05-2011, 04:17 PM
I think the only reason he stated that is due to keep the 'public annoucement' of another client not to provide details about them. Is this fixed yet?
Posted by Matt R, 03-05-2011, 04:18 PM
He's referring to Softlayer Sales, not Club Uptime sales.
Posted by Matt R, 03-05-2011, 04:18 PM
It's been fixed for about 15 hours now. Well, sort of. Softlayer Null-routed all of the affected IP addresses (an entire primary subnet). We swapped it out with another range which will remain hidden and undisclosed. Everything is back online and thus-far DOS free.
Posted by A J C, 03-05-2011, 04:20 PM
I guess that makes sense too.
Posted by Aaronn, 03-05-2011, 04:25 PM
Good Job!!!
Posted by Visbits, 03-05-2011, 05:13 PM
10G, that sucks... surely you've already started contacting ISP informing them of what their network has just done...?
Posted by Matt R, 03-05-2011, 05:15 PM
Not much we can do. Although a HUGE part of it has been coming from the Amazon EC2 cloud. It's too bad that they've not responded to a single abuse report in 15 hours....
Posted by SSL-Nick, 03-05-2011, 05:28 PM
I meant for SoftLayer
Posted by rondawes, 03-05-2011, 05:54 PM
My sites came back up early this morning but just went down again. Status Please? Softlayer Portal shows all green on all nodes and networks.
Posted by Matt R, 03-05-2011, 05:57 PM
The status page shouldn't be relied on right now -- The IP's will resolve internally on the Softlayer network but have been null-routed to the public. We've started seeing multi-gigabit attacks to all of Dallas and Seattle. Again. They're also hitting our external providers, DNS servers, updates blogs, etc... Whomever is doing this is certainly attempting to stop our business from operating at all.
Posted by SSL-Nick, 03-05-2011, 05:58 PM
Looks like they are under attack again. I suggest you watch http://updates.clubuptime.com/
Posted by SSL-Nick, 03-05-2011, 06:00 PM
If it's a competitor company it's extremely low.
Posted by Matt R, 03-05-2011, 06:00 PM
Also note that we've had to take our primary server offline intentionally. Updates.clubuptime.com will remain available (they've been hitting it as well) hopefully. clubuptime@gmail.com is the only current support outlet as our phone servers have been targeted as well.
Posted by A J C, 03-05-2011, 06:30 PM
Please do not contact the Gmail account mentioned there for ETAs on restoration of service. We understand this outage is frustrating, but answering e-mails asking for an ETA on the restoration of service distracts technical staff from working toward solving the problem. Thank you for your cooperation.
Posted by nixell, 03-05-2011, 08:09 PM
There any way to contact clubuptime?
Posted by ScottSwezey, 03-05-2011, 08:12 PM
Yes, there is. Last edited by ScottSwezey; 03-05-2011 at 08:14 PM. Reason: sig removed
Posted by quad3datwork, 03-05-2011, 08:29 PM
Got couple VPSs in your Seattle location and they are online. Hope everything works out ok.
Posted by Matt R, 03-05-2011, 08:33 PM
All of our Seattle VPS nodes are offline with the exception of one. Half of Dallas is offline, including our internal servers. WDC is 100% online at this time.
Posted by MyLabuan, 03-05-2011, 08:37 PM
Will you compensate for this downtime... Ticket ID : HEG-888-74425
Posted by Matt R, 03-05-2011, 08:43 PM
I'd recommend updating the ticket once our servers are back online. The server hosting the clubuptime.com website, helpdesk, and billing center is 100% offline right now for security as someone was trying to breach it's security. My best guess is that they hit all of our servers to distract us from the fact that they were trying to break into our main server that hosts our website. Until that's back online, we can't reference ticket ID's, view your billing information, etc...
Posted by Kevin K, 03-05-2011, 08:56 PM
I know this is a bad situation and certainly a targeted attack. Since there is nothing on your site we can refer to for updates, what is the status of this outage and what is being done to bring the servers back online network wise? Also what is gonna be done to prevent this in the future? This is 2 days in a row of this same prolonged outage and is causing some rather large concern. I understand this is an attack and out of your control at this point, but those of us that are paying for a service to be up need to know if we are left looking for a new provider to avoid being effected by these attacks.
Posted by Matt R, 03-05-2011, 09:09 PM
We definitely understand your concerns. Unfortunately, we simply can't provide any sort of ETA. We're working with Gigenet right now to work out a solution on getting the main clubuptime.com website back online so updates could be provided. Our normal updates server was hit so hard that Nginx has decided to choke and give out. We're going to be reloading the OS on that shortly to make it a static web page it serving php/mysql just seems to be too much as it's a targeted HTTP attack. As far as the nodes go, we're still investigating with Softlayer as to how the attackers managed to gain our new IP address information as it's not published in forward or reverse DNS information. We re-issued all IP Addresses last night between 12am and 6am on all host nodes and re-routed IP traffic. 12 hours later, the attack slammed all of those nodes again on their new primary IP addresses simultaneously alongside more servers, this time including Seattle and our updates server. Softlayer is refusing to remove the null-routes for 24 hours from the time that they were null-routed (again) as the attack is extremely large. All accounts will be receiving SLA credit, but I will not deny that if you need your servers back online and they are mission critical, it may be a good idea to migrate to another provider. Right now, we only have command line access to our affected systems over the Softlayer Private network. We can generate backup files for any clients needing one, although it may take quite some time to generate and deliver them due to the nature of the backend private network and other factors as we work on restoring service.
Posted by Rob_T, 03-05-2011, 09:09 PM
Just a suggestion, but rather than sticking the "updates" website on an easily-flooded linode, now looks like it might be the time to leverage a third party's huge infrastructure and host it on wordpress.com or be a bit more active with that twitter account. Good luck with getting through this.
Posted by Matt R, 03-05-2011, 09:16 PM
Fully agreed. It was designed to be there as a small off-site notification system. While we trust Softlayer fully, it would be our luck that the Softlayer facility containing clients of ours would also go down with the updates website on it. We never expected it to have as much traffic as it has today. In the end, we haven't provided many updates as there simply aren't many to provide. The attack stopped for about 10-12 hours and then came back twice as hard as it was yesterday. I wish there was more that we could say, but there simply isn't much that we (nor any provider) can do, really.
Posted by FiberFy, 03-05-2011, 09:58 PM
Matthew I asked you to clarify, I didn't acuse you of anything! It was another person who passed me the message. I am your client and I am satisfied with the services provided
Posted by 2kreative, 03-05-2011, 10:25 PM
Bummer I had 6hrs downtime earlier today and now about 4.5hrs and counting.. Did I miss an email notification about this?
Posted by zoogne, 03-05-2011, 10:40 PM
You've now definitely read the full thread.
Posted by FiberFy, 03-05-2011, 10:56 PM
And my server is offline too, in Seattle
Posted by chasebug, 03-05-2011, 11:02 PM
VPS still down. How much credits are we gonna get issued for these downtimes? The website don't work also: http://updates.clubuptime.com/ 502 Bad Gateway nginx/0.7.65
Posted by kjetterman, 03-05-2011, 11:22 PM
Yikes you guys! This is definitely a targeted attack. I wonder what they are after?
Posted by Matt R, 03-06-2011, 12:10 AM
The most that I know for sure is that they attempted to break into our SQL servers but failed. There are hundreds of thousands of TCP dump logs per hour that show failed attempts at MySQL connections. That's why our primary website is offline -- We took it offline to protect our client data just in case the Cisco Guard and CSF couldn't hold up. No data has been leaked as there is no ability for them to connect to our SQL servers. Doesn't stop them from trying, though...
Posted by efan0388, 03-06-2011, 12:24 AM
I've been a customer of your's for a couple months Matt and even though I didn't get the birthday special for some reason :p I'm still extremely happy with your services. That being said I'd rather you take your time and fix everything that needs to be fixed right. Even though this is a huge inconvenience it's not your fault. Just wanted to say thank you for working for us.
Posted by JDrago1926, 03-06-2011, 12:54 AM
Best of luck to Club Uptime with getting everything back in place. Attacks like this are horrible and it's a sad shame people do it. Hope this gets resolved quick.
Posted by SSL-Nick, 03-06-2011, 01:22 AM
I don't want to jump ship but getting lots of complaints. I guess node was down Friday (Client reports). Now today got a few complaints from clients about this. Thinking of purchasing a server for the week.
Posted by SSL-Nick, 03-06-2011, 01:43 AM
To all who wonder about a refund. I'ts a Network outage not a server as the servers are online but network is off.
Posted by nixell, 03-06-2011, 02:46 AM
hmm.. i hope it normal again, what type DDOS attack? are softlayer not provided DDOS protection?
Posted by Matt R, 03-06-2011, 02:47 AM
Softlayer provides protection for small to medium DDOS attacks at best. The DDOS attacks that hit us would have crippled most networks entirely.
Posted by no69_2007, 03-06-2011, 03:21 AM
I would suggest you not to post your every move here, because it will update the attackers as well.
Posted by FiberFy, 03-06-2011, 04:09 AM
It's a good suggestion, you guys should create a blog on blogspot.com for example ;=) Best luck dealing with these guys
Posted by dev_to, 03-06-2011, 05:27 AM
I really understand the situation but by now it's taking a little to long. I don't want to but i may need to switch to another provider and see how things work out with ClubUptime. I would suggest that you setup a update blog with blogspot or wordpress aswell. That's not much work and would keep us up to date.
Posted by A J C, 03-06-2011, 06:57 AM
To be honest, Softlayer are being quite tight-lipped about the whole affair. I went to bed around midnight GMT, and nearly 11 hours later the attack is still in progress. Someone clearly wants ClubUptime out of the picture for some reason, and by switching providers whoever is doing it is getting his or her way. I've decided to take everyone up on their suggestions and have set up an updates blog here. Please check it for further updates. Matt and I will be checking this thread periodically too.
Posted by BkWTom, 03-06-2011, 07:36 AM
It's such a shame that you experiencing a terrible DDoS attack like this. I wish you the best of luck with getting things back to normal!
Posted by StevenG, 03-06-2011, 08:19 AM
These things happen every now and then and it sucks. Stick with it, it'll pass. Good to know that these guys don't hide when things do go wrong anyway.
Posted by syaman, 03-06-2011, 09:20 AM
Is it just me, or is the new blog and the entire Wordpress.com now down too??
Posted by dev_to, 03-06-2011, 09:22 AM
No it's up and running for me =)
Posted by SSL-Nick, 03-06-2011, 09:29 AM
I spoke with a support tech from Softlayer and he was really nice. So yes this is a pretty big attack. I logged off because the support tech took 30 minutes to respond and was tired. (Forgot I had the chat open to) Last edited by foobic; 03-06-2011 at 12:56 PM. Reason: BAQ-828136
Posted by misspink, 03-06-2011, 09:30 AM
Good luck with this. I've been repeatedly hit with 3Gbps attacks and I know how annoying it is.
Posted by Matt R, 03-06-2011, 09:34 AM
I guess it's time to ring up Daniel Kracht in the morning. You would think that they would make an attempt to verify the account holder before giving away potentially confidential information... Edit: Sorry, but I'm having the mods remove the Ticket ID numbers. Softlayer should NOT be giving out any information without validating the account holder first. Last edited by Matt R; 03-06-2011 at 09:38 AM.
Posted by FiberFy, 03-06-2011, 09:41 AM
@Matt_R That's true, I'm with you. Let me know if you need any help Matt
Posted by SSL-Nick, 03-06-2011, 09:42 AM
Yes thats what I thought to. I told him the provider is under and an attack and would like to know if it is true. Provided ticket id's and they transfer me to support. Then he reads over (Even though I stated it pretty clear that I was getting hosting from another company that is on the network. Hope Softlayer can start using more protection. I know you guys ask for the safety question (Example: Whats last 4 digits of CC: ----). Before you start providing information.
Posted by LinuXice, 03-06-2011, 10:31 AM
My VPS in dallas hasn't been affected at all, no pingdom alerts so far. Anyway, CU site is offline this morning, just waiting for it come back to order a new vps,
Posted by rondawes, 03-06-2011, 10:38 AM
My VPS just came back online! Hope it stays that way.
Posted by bloger, 03-06-2011, 01:05 PM
yes the VPS come back online so this server stay online o or this server come to down
Posted by fshagan, 03-06-2011, 01:16 PM
Yeah, Wordpress.com was hit with a major DOS attack too. From Friday at TechCrunch: Wordpress.com hosts 18 milliion sites, and TechCrunch is one of the them. Mullenweg says most of the attacks originated in China. This can happen to anyone.
Posted by JDrago1926, 03-06-2011, 02:05 PM
Glad to hear a majority of the VPS servers are going back online. Seems as Club Uptime's site is still down, hopefully everything is resolved soon.
Posted by Matt R, 03-06-2011, 04:09 PM
All virtual machines are online at this time. We've left our server off intentionally for the time being. We're hopefully going to re-enable public access within the next few hours.
Posted by SSL-Nick, 03-06-2011, 04:26 PM
You should open a private site for clients only. Information would be included in the client email (when they purchase a product). That way we can access support while the main site is down or purchase servers ect.
Posted by misspink, 03-06-2011, 04:27 PM
What if the attacker is a client?
Posted by Matt R, 03-06-2011, 04:30 PM
We've considered it but it simply doesn't seem likely. Not impossible, but seeing as we've had nothing but happy clients (With the exception of a few resource abusers), we're fairly certain that it's not the case.
Posted by Matt R, 03-06-2011, 04:32 PM
We've put our updates page on the Wordpress network now (updates.wordpress.com). Updates.clubuptime.com simply tells people to go there now. We'd much rather do that than risk an attacker finding our "secret" client portal for information/stats and DDOS'ing that too. Wordpress is MUCH harder to take down. I hope.
Posted by SSL-Nick, 03-06-2011, 04:39 PM
I was thinking this to. As when the attacks started was when the E-mails were sent telling people to cancel minecraft.
Posted by ScottSwezey, 03-06-2011, 04:44 PM
Or if he was an ex-client was informed of the secret portal when he was a client. I suppose the secret things could be relocated every month or two, but that is a giant pain, costs lots of time for staff involved in moving it, and can be confusing for clients who were expecting to find it at location A but instead get nothing because it's now at location B.
Posted by Avesta-Aria, 03-06-2011, 04:57 PM
Whoever was investing in such attack had a goal to follow, its generally not going to be cheap to attack multiple hosts using different methods and in large scale! I wish you guys could find the person behind this and drag him to court to let him know that playing with legitimate businesses is not an easy game to attend to. Good luck guys. - Aria
Posted by Aaronn, 03-06-2011, 05:12 PM
Whats the status of everything? Has it stopped as peoples servers you said are coming back online?
Posted by SSL-Nick, 03-06-2011, 05:14 PM
Servers are back (At least mine are, so I can't speak for others). Main site is still down.
Posted by techjr, 03-06-2011, 05:18 PM
They have posted there update page with the latest information.... it has been linked to on page five and other pages...
Posted by A J C, 03-06-2011, 05:22 PM
I'm going to reiterate what I said earlier. The updates page has now moved (permanently) to the Wordpress network. You'll find it at: http://clubuptime.wordpress.com
Posted by SSL-Nick, 03-06-2011, 07:12 PM
Not for long http://isitup.org/clubuptime.com . It's loading just really really slow :|.
Posted by SadisticAndroid, 03-06-2011, 07:28 PM
My server seems to keep going up and down Looks like they're still having issues...
Posted by FiberFy, 03-06-2011, 10:14 PM
If wordpress goes down, go to blogspot, they won't go down
Posted by Matt R, 03-07-2011, 05:37 AM
Unfortunately, it's probably someone from another country that we can't go after. However, we may go after Amazon as they *still* haven't responded to abuse reports sent days ago.
Posted by no69_2007, 03-08-2011, 03:47 AM
some amazon servers are attacting as well?

Add to Favourites Print this Article

Client Navigation

Useful Pages

Support Center

About 02link

02link provides enterprise hosting and server services at an amazing price. Don't see something that suits your needs? Get a free custom quote from one of our hosting experts.