Sunday, April 11, 2004
I know this is not about fantasy, but I did feel like living in a surreal world during last week, though it was more like a nightmare.
I would like to share what happened so that you may prevent it happening to you if you ever try to have your own site on the web. In one line: just beware of CWIHosting.com as a hosting company to keep your sites in.
I had been using their hosting services for just over a year, when last Sunday, March 28th, they just decided to kick me out!
I was checking on my site, just by chance, when it suddenly showed the following error message:
FORBIDDEN:
Site Owner, Please contact your support center for assistance.
I tried to access my control panel to see if there was anything wrong with my sites, like bandwith over quota, or similar, but my password was not valid any more and I could not access the control panel, nor my database.
I had not received a single warning about anything going wrong, so I went to Customer Support and opened an urgent ticket about this.
The reply I got from CWIHosting was:
[Tech: JT] 03/28/2004 12:30 PM
Your site has been suspended due to server abuse/overusage. An email was already sent to you at cwihosting@hypernovels.com
You should refer to that email and respond to it.
The thing is, hypernovels.com is one of the domains I was hosting with them, so the email had to be delivered through their servers. As I found out when I tried to check my e-mail, they had also cancelled my e-mail account and I could not retrieve my messages!, none of them, including, of course, the one they claimed they had sent me!!
So I opened another ticket (CWIHosting immediately closes support tickets even if their reply is not at all satisfactory):
[Opened] 03/28/2004 01:29 PM
---------------------------
In reference to ticket 43623
My hypernovels.com domain is now another pointer to my account, and it seems you have put down my email accounts too, so I can not receive any emails sent to hypernovels.com (you could have copied the email text in your reply).
Please send the email to
clemente_zamora@hotmail.com
with instructions on how to resolve this.
I haven't done anything special today!! what is the cause of the reported abuse?!
Their response was:
[Tech: Not Assigned] 03/28/2004 01:30 PM
We will forward your email address you wish the email to be sent to the individual who sent it.
After an hour I had not received the message:
03/28/2004 02:26 PM
---------------------------
In reference to ticket 43625
I haven't received the email yet.
My sites are still down!
Please tell me how to fix this!
Thanks.
Our Response [Tech: JT] 03/28/2004 02:28 PM
« Status: Final »
-----------------------------
Unfortunately submitting support tickets will not get this issue resolved. The email was resent to the email address you provided several minutes ago. You should make sure your email is wrking properly. If you have further questions please email cto@cwihosting.com
They insisted in sending an e-mail message even though it was obvious something wasn't working. I still had no idea what had happened and they just couldn't bother to enlighten me through the communication channel that did work!
And they still dared to complain about my opening tickets to ask for a solution! I am paying them to host my sites, they stop doing it, and I'm supposed not to even complain?! And, furthermore, there was no other way for me to contact them anyways.
03/28/2004 03:03 PM
---------------------------
In reference to ticket 43628
I have doubled checked and my hotmail account is working properly.
Your silence forces me to reopen the ticket as this is the only communication channel left.
Could you please reply with the text of the e-mail you say you are sending to me?
This is really urgent!
Thanks!
2 hours and a half later I still had no reply, so I insisted:
[Update] 03/28/2004 05:36 PM
-----------------------------
My sites have been down already for over 6 hours!
If you have really sent an email, why is it taking so long to reply to this ticket with that same information?
[Tech: Wade] 03/28/2004 07:02 PM
« Status: Final »
-----------------------------
Hello, your site has been suspended. You will need to contact cto@cwihosting.com regarding this matter. Please let me know if there is anything else that I can do for you. Regards.
Wade Lewis
They just insisted that my sites had been suspended (which was obvious, as they had been taken down by them), but not even a clue as to the reason why!
This is the message I sent to the e-mail address they indicated:
29 Mar:
Hi there,
I can not receive email in my other e-mail addresses thanks to CWI shutting down all my services, for which I pay!
So I'm forwarding you my message from my free e-mail account so that you can reply to it.
My sites are down now for several hours, and the only reply I get from the Support Center is that they are sending me an email with instructions. But they have put down my email accounts too so I never received any messages, and they claim they have sent the same message to my hotmail account, but I haven't received it (and I have double-checked that it's working). And they insist in not telling me what is wrong through the ticketing system! So I'm still waiting to know what happened and why you put my sites down.
The tickets so far are:
43623, 43625, 43628
Please put my sites up or tell me what I can do to resolve this!
I have recently upgraded my account and are already regretting it!
Thanks a lot for your help.
Clemente Zamora
http://www.tradecardsonline.com
Hi there,
I can not receive email in my other e-mail addresses thanks to CWI shutting down all my services, for which I pay!
So I'm forwarding you my message from my free e-mail account so that you can reply to it.
My sites are down now for several hours, and the only reply I get from the Support Center is that they are sending me an email with instructions. But they have put down my email accounts too so I never received any messages, and they claim they have sent the same message to my hotmail account, but I haven't received it (and I have double-checked that it's working). And they insist in not telling me what is wrong through the ticketing system! So I'm still waiting to know what happened and why you put my sites down.
The tickets so far are:
43623, 43625, 43628
Please put my sites up or tell me what I can do to resolve this!
I have recently upgraded my account and are already regretting it!
Thanks a lot for your help.
Clemente Zamora
http://www.tradecardsonline.com
29 Mar:
I'm told that my sites have been suspended, without any explanation, and that I should contact this email address. But I have already done so several hours ago and I still receive no reply!!
What is happening? My sites have been down for almost 12 hours now! My visitors are getting a horrible:
FORBIDDEN:
Site Owner, Please contact your support center for assistance.
Who is going to pay for all the discredit this is causing to my business?!
Please reply ASAP to this e-mail address.
Clemente Zamora
http://www.tradecardsonline.com
Over 10 hours later I still had no information and no way to get it. I tried to guide them and propose the obvious solutions to a problem of the kind.
03/29/2004 06:42 AM
---------------------------
You have suspended my sites and in the same process you have forbidden me any chances to resolve the problem, because I don't have access to the system any more, nor to my cPanel, nor even to my email accounts! So I have even been cut off from my customers!
Please give me ftp and ssh access to my account, and access to my cPanel so I can at least get a backup of my code and my databases.
I will also add an http redirect so that all traffic gets diverted to another server outside CWI until the DNSs get updated, and then I will need you to open http access to my account so that traffic can be diverted and my sites are up before the 24-72 hours DNSs need.
Thanks.
From : Jason Taylor
Reply-To : cto@cwihosting.com
Sent : Monday, March 29, 2004 7:28 AM
To : clemente_zamora@hotmail.com
Subject : Fw: URGENT - Your Site Has Been Suspended
Hello Clemente Zamora:
Your site www.tradecardsonline.com has been suspended today for
causing the server to become unstable and for causing consistent, high
load spikes within the past 24 hours. I have reviewed your account
and it seems that out of all of the requests to the server about 90%
of those are for your site alone. In addition, it was found that your
processes used consistently higher amounts of memor, cpu time and ran
longer than any other clients.
It would appear that by this and your traffic that your site is no
longer suitable for a shared environment as it is causing instability
and not allowing other users the ability to utilize the server but
instad it is causing lack of performance and sluggishness.
It is advisable at this point to move to a dedicated server in order
to accomodate your sites needs. At this time we cannot unsuspend your
site as it will begin again to cause sluggishness and slow responses.
However if you are able to disable part of your site and wish to get a
dedicated server we can reactivate the site so you can disable parts
of your site until you get moved to a dedicated server.
However if you are unable to get a dedicated server you will most
likely need to find another hosting provider as we do not have any
other packages which we could offer to meet the needs of your site.
Jason A. Taylor
Chief Technologies Officer
www.cwihosting.com
www.creative-webs.com
www.cwisupportcenter.com
So finally, the day after my sites had been put down unilaterally by CWIHosting.com, I got an alleged explanation of what had happened. But I was not given any chances to investigate and solve the problem. In fact, the only solution offered was to pay them 8 times more than what I was paying then, and rent a dedicated server with them. Otherwise, I should just leave! But they were holding my data away from me so I could not move my site to a new hosting company. Their strategy was perfect: they were forcing me to pay a lot more if I wanted my data and my sites back! CWIHosting.com was clearly black-mailing me!
29 Mar:
Hi there,
Could you please give me ftp and ssh access to the site so that I can at least transfer it temporarily to another server until I decide what to do?
I need them up immediately, but I would have to think carefully the only option you give me with CWI that is the dedicated server (a 1000% increase in price).
The only change that I have done recently has been adding a second pointer to a domain that hardly receives any visits (hypernovels.com). May this be causing any trouble?
Could you please give me more data about the kind of processes that are consuming resources, and when they appear or if they have any kind of periodicity or regularity? During this month I've noticed an important increase in bandwidth use that is not reflected in the number of users I have. Could it be that I'm being the target of some kind of attack?
Any help would be appreciated.
Thanks.
Clemente Zamora
From : Jason Taylor
Reply-To : cto@cwihosting.com
Sent : Monday, March 29, 2004 7:33 AM
To : "Clemente Zamora-Ramos"
Subject : Re: Sites still down!!
Hello Clemente:
Below you will find the original email which was sent to you when your
site was suspended. I have also reforwarded the original email that I
had sent and reforwarded earlier.
In addition to what was stated below in the original email, since your
site ssuspension the amount of traffic has seriously dropped. However
more importantly the load average on the server has remained low since
the time that your site was suspended. Prior to this the load would
vary as the server would become more and more sluggish. We have even
tried reopening your account for a short period of time and we see a
increase in traffic and load very quickly.
Jason A. Taylor
CWI
I had not made any changes to my server processes in a long time, and the number of visitors to my site had not increased so dramatically, so the only cause I could think of for what they claimed was happening (I had no way to verify it by myself) was that there was some kind of external attack to my account and someone else was using my resources. CWIHosting security would seem to be weak and I was suffering the consequences.
From : Chief Technologies Officer
Reply-To : "Chief Technologies Officer"
Sent : Monday, March 29, 2004 7:48 PM
To : "Clemente Zamora-Ramos"
Subject : Re: Sites still down!!
Hello Clemente:
Unfortunately as this is an administrative suspension, the staff of the
support center cannot assist you as they do not have the authority.
Therefore continuing to submit tickets there will not help out and will only
frustrate you more. Furthermore this email address is not unresponsive.
However as it was 1AM when you replied to my last response I was not in the
office waiting for your email.
We are not keeping your sites honest, it is more that you have an
unrealistic expectation of turnaround time on your emails that you have
sent, expecting an instantaneous reply. We do not have any need or want to
keep your data hostage. Our main concern is making sure your site does not
have a negative impact on the server and the service to other customers if
we do open it. The reason we can open your site if you get a dedicated
server is we can reroute your traffic internally to that server. However
without that server setup here we cannot open the site and reroute the
domain. The most we could do is put up a temporary maintenance page for
your customers and provide you a backup of your files.
Jason A. Taylor
CWI
My obvious reply was:
29 Mar:
Dear sir,
If you suspend an account without warning I do expect a 24/7 assistance to put it back up.
Also, I'm told the supension was based on technical reasons, so technical support should be able to help me find the problem and fix it (which nobody has even tried).
Ok, I will need a backup of all my files, of my current raw web log, and, specially and as fast as possible, of my databases.
For the file backup you can ignore the following directories:
/raw-logs
/db-backups
/public_html/cache
Please also put up a page rerouting my visitors to:
http://gold.he.net/~czamora
I would also be needing all the emails received in my account during these hours (which should be dozens).
Hope to hear from you soon now.
Please make sure the database backups are created and sent as soon as possible.
Clemente Zamora
I kept asking for information and offering ways that would allow me to minimize the damage, but neither was granted to me by CWIHosting.com
29 Mar:
I'm attaching my last support ticket and the corresponding unsatisfactory reply which directs me again to this unresponsive e-mail address.
I must obtain access to my account, as you are keeping my sites hostages, requesting that I rent a dedicated server as a ransom to return my data to me. I don't think that this is honest, or even legal, behaviour on your part.
Please fix this problem as soon as possible as my sites have been down now for 12 hours and my customers are presented with a message that destroys my business reputation.
My request and the reply from Support Center follow:
You have suspended my sites and in the same process you have forbidden me any chances to resolve the problem, because I don't have access to the system any more, nor to my cPanel, nor even to my email accounts! So I have even been cut off from my customers!
Please give me ftp and ssh access to my account, and access to my cPanel so I can at least get a backup of my code and my databases.
I will also add an http redirect so that all traffic gets diverted to another server outside CWI until the DNSs get updated, and then I will need you to open http access to my account so that traffic can be diverted and my sites are up before the 24-72 hours DNSs need.
Thanks.
Our Response [Tech: Not Assigned] 03/29/2004 10:32 AM
« Status: Final »
-----------------------------
Unfortunately we are unable to open the site for you. You will have to deal directly with the CTO via email at cto@cwihosting.com.
03/30/2004 01:40 PM
---------------------------
In reference to ticket 43645
I have tried to contact your TCO, but he doesn't reply to my emails.
I desperately need a dump of my database (tradecar_tradecardsonline). If you are a human being, please, please help me! My sites have been down for over 48 hours and I can't even contact my customers!
All I need is that you put a dump of my database in my anonymous ftp directory so I can download it, or give me access to my account for just an hour so I can take a backup of the database myself (the account won't affect the server as the domains are already redirected to other servers).
Please do something about this. I will appreciate it very much. I don't know what else I can do!
Please don't let me down!
Thanks a lot!
I was now begging for CWIHosting support staff to do something! I just needed a simple thing they could easily do for me to at least put my sites up in another server.
Our Response -----------------------------
[Tech: Not Assigned] 03/30/2004 02:15 PM
« Status: Final »
-----------------------------
Emails have been sent back to you in reply to your messages that you have sent. You have obviously received the messages since you have replied since then. You will again need to contact him directly at cto@cwihosting.com as you have already been informed that we cannot assist you.
Following is the reply that was sent to you on March 29, 2004 at 12:48 PM
Hello Clemente:
Unfortunately as this is an administrative suspension, the staff of the
support center cannot assist you as they do not have the authority.
Therefore continuing to submit tickets there will not help out and will only
frustrate you more. Furthermore this email address is not unresponsive.
However as it was 1AM when you replied to my last response I was not in the
office waiting for your email.
We are not keeping your sites honest, it is more that you have an
unrealistic expectation of turnaround time on your emails that you have
sent, expecting an instantaneous reply. We do not have any need or want to
keep your data hostage. Our main concern is making sure your site does not
have a negative impact on the server and the service to other customers if
we do open it. The reason we can open your site if you get a dedicated
server is we can reroute your traffic internally to that server. However
without that server setup here we cannot open the site and reroute the
domain. The most we could do is put up a temporary maintenance page for
your customers and provide you a backup of your files.
Jason A. Taylor
CWI
CWIHosting.com support staff was obviously not human, they just repeated once and again their learned response and ignored my begging and my desperation!
30 Mar:
Dear sir,
I wanted to indicate, just in case, that you should not send big files to this free e-mail address, as it does not have enough capacity.
And remind you that I'm still waiting for all my data, as indicated in previous mails.
Thanks,
Clemente Zamora
30 Mar:
Dear sir,
It's already 24 hours since you acknowledged my problem and I haven't received any of my data yet!
Can't you at least open a read-only access for me to retrieve my files and my database?
You're preventing me from opening my sites even in another server because I don't have the updated database. It's over 48 hours since you put my sites down without giving me a chance to explain or solve the alledged problem on the server, and you are not helping me in any way. What kind of customer support is this? Do I need to initiate other kind of actions to obtain my data?
If you intend to use email to send me the data, please use the following address:
cwihosting@hypernovels.com
Hope you at least reply soon.
Clemente Zamora
03/30/2004 05:53 PM
---------------------------
Following instructions from a phone call I just made to Customer Support, I'm opening this ticket to request that a complete database dump is created and made available to me for download.
Your CTO already did this but the dump created was not complete. I sent him an email about it, but as he is not replying and this is really urgent I called on the phone and was instructed to open a ticket about it and call again in about an hour. Please take into consideration that it is 2:48 AM here in Spain and I can not go to bed till this is resolved...
The database is tradecar_tradecardsonline and the public location for me to download it would be something like http://66.216.127.33/tradecar_tradecardsonline.sql
(please zip (compress) the resulting file so the download is more feasible, as the raw dump will have probably over 100Mb in size).
Please make sure the dump is complete so we don't have the same problem again (last table in the dump file should be 'cc_users_inactive').
Thanks a lot in advance.
[Tech: Not Assigned] 03/30/2004 05:59 PM
« Status: Open »
-----------------------------
Though certainly important to you and CWI, this is not an emergency case as the site/server is not down. This is the only time that status is to be used as it will page our emergency response team. Your request will be completed in a timely manner in the order received. Thank you for your understanding.
When opening a support ticket at CWIHosting they indicate that a ticket should be marked as urgent only if the site is down. They don't say anything about the whole server been down. My sites were obviously down, they were not online. Isn't that an emergency for a hosting company whose business it is to host web sites?!
03/30/2004 06:03 PM
-----------------------------
All my sites are certainly down!
They have been well over 48 hours now!
This is as much an emergency as it can be!
[Tech: Not Assigned] 03/30/2004 06:09 PM
« Status: Client Updated »
-----------------------------
Although this may be an emergency for you, it is not a server down emergency which is what the emergency check box is for. We are aware your site is down due to the fact that it is suspended. We are currently working on your issue and will have a response as soon as possible. However continuing to make these tickets emergency only will make your issue take longer to resolve.
Now they claim an emergency is only when the whole server is down. So if the server is working, even though the sites it is hosting are not, then CWIHosting has nothing to worry about?! They are not providing the service they are been paid for, but that's not an emergency for them, even if the situation has prolonged for 48 hours!
03/30/2004 06:15 PM
« Status: Final »
-----------------------------
Your database has been backed up by the CTO again and he has sent you an email. The specific URL in case you do not receive that email is http://66.216.127.33/tradecar.tar.gz
-----------------------------
They also sent an email message:
From : Chief Technologies Officer
Reply-To : "Chief Technologies Officer"
Sent : Tuesday, March 30, 2004 10:56 PM
To :
Subject : Your MySQL Dump
Hello Clemente:
I received your phone message and have dumped your database to a file for you to download. The URL is at http://66.216.127.33/tradecar_tradecardsonline.sql
Once you have downloaded it please let me know so I can close your account completely. In addition, I never received a response from you regarding my response on March 29.
Jason A. Taylor
CWI
Finally! It seemed they had finally listened to me and were returning to me what was mine in the first place, my data!
Unfortunately, when I downloaded the file, and checked it, I found out the database dump was not complete! It contained less than half the tables and content of my database!
CWIHosting support was proving to be the worst ever, but now even the technicians were proving to be unable to use a simple mysqldump command properly! This was unbelievable!
They had not bothered to compres the big file either (well over 100Mb in size!)
30 Mar:
Yes, thanks, I have retrieved the dump already.
I need now the email messages received during this time forwarded to my e-mail address, and download of my raw logs and the following files:
public_html/emails_sent*
home/tradecar/*.txt
Thanks a lot.
Clemente Zamora
31 Mar:
Dear Jason,
Can you update me on the status of the requests I made to you yesterday?
Thanks a lot.
Clemente Zamora
From : Chief Technologies Officer
Reply-To : "Chief Technologies Officer"
Sent : Wednesday, March 31, 2004 1:12 AM
To : "Clemente Zamora-Ramos"
Subject : Re: Your MySQL Dump
Hello Clemente:
It seems that you are receiving my emails properly. I have redumped your
database and it comes out to be 205 MB. I have zipped it and placed it at
http://66.216.127.33/tradecar.tar.gz
Jason A. Taylor
CWI
1 Apr:
Dear Jason,
Can you update me on the status of the requests I made to you yesterday?
Thanks a lot.
Clemente Zamora
From : Chief Technologies Officer
Reply-To : "Chief Technologies Officer"
Sent : Friday, April 2, 2004 5:38 AM
To : "Clemente Zamora-Ramos"
Subject : Re: Your MySQL Dump
Hello Clemente:
I have made a backup of the files you have requested and placed it at
http://66.216.127.33/tradecar.tar
Jason A. Taylor
CWI
6 Apr:
Hi Jason,
Thanks a lot for the files backup. Only it seems you forgot the weblogs I was keeping in my account for the previous months. Please make them available for me.
Thanks a lot again.
Clemente
From : Chief Technologies Officer
Reply-To : "Chief Technologies Officer"
Sent : Tuesday, April 6, 2004 9:20 PM
To : "Clemente Zamora-Ramos"
Subject : Re: Your MySQL Dump
Hello Clemente:
Where were your backups located in your account? As your account has been
closed already your account is gone, however we may be able to pull this
data off of the backup server.
Jason A. Taylor
CWI
So they had already completely removed my whole account, even though they had not sent me all my data yet!
