FOR THOSE OF YOU ALSO IMPACTED BY THIS OUTAGE, THAT HAVE BEEN CALLING/MESSAGING ME ABOUT WHERE TO GO, I HAVE BEEN RECOMMENDING HTTP://INTRINIUM.COM, THEY ARE RELIABLE, HONEST, RESPONSIVE, CAPABLE, ETC. THEY ARE A MORE STANDARD INDUSTRY APPROPRIATE RATE.
UPDATE: I NOW HAVE MY PHYSICAL SERVER BACK (doesn't help with the 6 virtual servers, those are a loss) AS OF LATE MARCH 14TH, AND I AM TRYING TO GET EVERYTHING WORKING AT THE NEW LOCATION WITH THE NEW SETUP. IT WILL LIKELY TAKE DAYS BEFORE THINGS ARE RELATIVELY BACK TO NORMAL AGAIN. APOLOGIES FOR THE INCONVENIENCE, AND THANK YOU FOR YOUR PATIENCE DURING THIS VERY TRYING TIME. THIS DOES NOT AFFECT ANY OF DEV 2 DEV PORTAL'S PAYING CUSTOMERS, ONLY THE (many) PRO BONO FREE COMMUNITIES.
See bottom of this page for the latest status updates.
Short summary about the outage:
Tierpoint.com in Liberty Lake, Washington, is denying access to everyone trying to get to their own physical servers.
No explanation as to why. Cyber-world.com (CW) / E-Insites systems administrator has also been denied access by Tierpoint to the CW servers and their customer's servers, according to the CW systems administrator, Stephanie, with no explanation as to why (according to Stephanie).
I had a single server, under my old company Dev 2 Dev Portal LLC our own property, in one of the many racks, in the cage with many other customers, as a customer of Cyber-World / E-Insites.
Cyber-world / E-insites (CW) was leasing a large caged area from Tierpoint.
According to the Cyber-world head SysAdmin,
Supposedly Tierpoint won't even tell Cyber-world's systems administrator why they have been locked out.
They have been completely offline since afternoon of Tuesday, March 7th, 2017.
Due to the ISP being down for so long, with no answer as to why or an ETA to get back online
and Tierpoint refusing to let anyone pick up their own physical servers
even though those servers NOT owned or leased from Cyberworld or Tierpoint, just the rackspace/colo used,
we are having to move all of our sites to a new hosting location.
This is going to take some time (a LOT of volunteer man-hours).
Have been hosting all of communities & sites for free at a very low-cost hosting facility.
We don't know if we can get our server back any time soon, Tierpoint is denying everyone access!
We have some backup servers that we will try to setup, but for now, we're temporarily using Godaddy
for these static pages. We will update as soon as possible
(as these begin to at least partially come back online, the plaintext will be converted to a link to the site. fortunately none of these are paying customers, they are all pro-bono communities I have been hosting/running over the decades. If I had actual paying customers, I would not have hosted them at such a place.)
We have noticed that all servers for all our websites went offline sometime during the afternoon.
Not only does this outage completely impair our 85 domains and many communities, it also froze the entire ZOE Z1 project!
Later investigation found that all of Cyber-World (CW) is entirely down. CW is the collocation facility our server is hosted at (we own the whole physical server, we just lease their 1u rack space and bandwidth, they lease an entire large cage from their provider Tierpoint).
No answers to our calls or texts about what is going.
Receive text message from Stephanie Bradley, the lead Systems Administrator and manager at Cyber-world / E-Insites:
"I'm headed out to data center now to get another client their servers if you would like to meet me out there and get yours as well. I'm not sure when we're going to be back up."
I spoke with Stephanie to ask what was going on. She said stated she didn't know.
She did state that Tierpoint was to blame for the outage, but she claimed she did not know why.
I asked to arrange to talk with her Wednesday at Noon to schedule my picking up the server, it was late, there was a snow storm, and I was tied up with a group at the time, she agreed to the time.
I called Kirt Runolfson of Intrinium (he is a friend and CEO of Intrinium) asking about emergency hosting services for the ZOE Z1 project (I knew I couldn't afford it for my 85 domains and related communities, but ZOE was able to budget for the higher quality hosting at Intrinium). Kirt indicated they would probably be able to make something work, even on such short notice, and said Kevin would contact me.
Later, I texted and called Stephanie of CW around noon, and received no answers.
Tried calling again at 12:23 pm, still no answer from Stephanie.
Spoke with Kevin of Intrinium, started invoicing request for some emergency virtual hosting for ZOE Z1 project. Ben approved pricing and invoice, and made payment and received confirmation invoice by 15:31.
Kevin contacted me for the setup process of the new virtual server for ZOE Z1.
There were a few glitches, but eventually, by 19:00-ish, we had the basic server up and running, and I began reinstalling everything (WikiMedia, NodeBB, etc.). We were originally supposed to be Launching this Thursday, but the Tierpoint + CW debacle completely killed that launch!
Now that we had an alternative host at least for the ZOE Z1 project, I called Tierpoint, hoping they would be reasonable and at least let us get access to all our content and code on the CW leased servers, and also hopefully my own physical server. They transferred me to Nicki Kopelson. She would not comment as to why the outage was happening, and that no one was allowed access to the servers, data, or anything else. Still would not explain at all why this was happening, and that it was now in the hands of the legal department. She suggested I speak with Alvin Slocombe of CW (I have never been able to get a hold of him, Stephanie has been good contact for a over a year, before that was the Irish fellow (Mark?)).
I asked if there were any options for at least retrieving my personal physical server, which is my property and not that of CW. She flatly said no. She spoke politely and sympathetically, and I was calm and respectful, but it was made clear others had been calling screaming and yelling about this issue (as Stephanie later would confirm).
We still can't get to all our configuration, content, and source code on the CW servers (those are leased from CW, while my server is my own separate property).
The way it is looking, due to what looks like a potential legal battle between CW & Tierpoint, it sounds like we can kiss all the ZOE Z1 work of the last several months (since November), and hundreds of man hours, goodbye, and we're having to start over from scratch. The Z1 data backups are also at the CW space.
Contacted KREM2 news that something was going on at the Liberty Lake facility.
Contacted KXLY news that something was going on at the Liberty Lake facility.
I have temporarily changed the DNS for all 85 domains to godaddy, and have paid for some hosting at godaddy to setup a static page to let people know what is going on. Next I am trying to back up and wipe an old system here at my house, and will try to setup dynamic DNS for all the 30 or so community sites, to try to get web services up, though it will be based on a much older back up (The more recent backups are at the colo alas). Then try to get email, IRC, streaming, and all the other services up as best I can from my home. Tierpoint still refusing to release my server. I told them I can prove ownership of the server, from my proof of purchase from Dell, proving it is not the property of CyberWorld. Niki at Tierpoint still said no, but that she would again consult with the rest of her legal team. I am not trying to contact Electronic Frontier Foundation Eff.org to see if they have any suggestions on how to get my data and my server back into my custody.
17:55 PST. The facility outage may be related to this:
March 8th, 2017: SpammerGate: The takeaway lessons and follow-ups on the River City Media data breach: http://www.csoonline.com/article/3178395/security/spammergate-the-takeaway-lessons-and-follow-ups-on-the-river-city-media-data-breach.html
And their rebuttal: March 9th, 2017: "River City Media Response to Defamatory and False Articles": http://finance.yahoo.com/news/river-city-media-response-defamatory-200900336.html
We will keep updating this page as we progress.
First we will get basic web pages up and running, then email, then IRC, then remaining services.
Thank you for your patience and understanding during this difficult time.
Spoke with Niki. I asked her if showing proof of my server ownership would help in releasing my server. She will ask legal. But still no for now. She let slip something about "Alvin has to pay for release".
Nicki Kopelson, 688-2566 called me on my cell phone, indicating that if I provided the proof of purchase information, she "I'm not promising anything" indicated it might be possible. I wasn't home, and was out running errands, so
quickly detoured home.
I emailed her proof of ownership. I also still have the original shipping box.
I called Nicki to let her know I had sent the email with the proof.
She reiterated she couldn't promise anything, but that she would forward that to legal and do what she could.
Fingers crossed. I have been having conversations with Tierpoint's legal department representative, trying to persuade them to release my server since it is my property and neither Cyber-world's nor Tierpoints, nor is my server involved in any of the issues those two companies are wrangling over. I have provided them with proof of purchase/ownwership on my part. We'll see how it goes. I also reached out to EFF's legal department to see if they had any suggestions, but likely will be next week at the earliest before hearing back, knowing how busy they are.
Meanwhile, at my house I finished backing up an old server so I could wipe it, and I am almost finished installing OpenBSD on that server. Then I will grab my backups from a few months ago from the colo server (the more recent backups are still at the colo alas). I am going to use the Godaddy virtual hosting for the front pages of all 85 domains, which will take a while to setup, but then have links to "go deeper" to my home server when it is online, especially all the research papers and such. This will hopefully be a survivable workaround. The nearest colo I spoke to is a minimum of $300/mo for very little bandwidth. That is typical pricing for hosting my server, that is why CW's $50/mo for the past 10 years was so appealing after being at Hurricane Electric for $500/mo for 6 years. Since I don't do this for a living anymore (tech), it is difficult to justify such high hosting expenses. And while all this is going on, been working daily on the Zombie Orpheus Entertainment platform (as a volunteer). While juggling Psych classes, and The RPG Trailer tour and The other RPG Research related efforts. And my boys' issues. Phew! What a week!
No word yet from EFF.org or Nicki at Tierpoint.
So far I have 20 of the 85 domains now with partial sites up for a few domains, barely any content but at least something to show up besides plain test, here is what is slightly up and running (again, with most of the content still missing currently, since mostly on server at colo). Basically just some photos, and contact information at best. All the research, videos, music, etc. still going to be many more man hours to restore, unless I can get that server back soon
I called Nicki to ask for an update on the server issue. She stated, more gruffly than our previous conversations, that something was being worked out with CW, and that I should hear back either from CW or Tierpoint in the next day or so about how to get my server back. Fingers crossed!
Received a Skype call (I was busy driving my son's Subaru with a stick shift, so I couldn't answer right away and had to call him back shortly after) from Chris Vickery, author of the article referenced in one of the earlier links in this log. Here is his original article related to "Spammergate" https://mackeeper.com/blog/post/339-spammergate-the-fall-of-an-empire
He stated via Skype his belief that this outage at CW is because of what he revealed in that article. He also apologized for the unfortunate effect of other innocent sites being taken down due to how Tierpoint may have decided to handle this, he didn't think they would handle it this way by pulling the plug to the entire cage.
Chris also stated he was an RPG fan, and was bummed that all my RPG related sites were inadvertently impacted. He seemed certain that the Spammergate issues were the cause of this outage and anything said otherwise was likely not correct.
He was friendly, and offered his help if he came across anything, such as affordable hosting, that could help me recover, which was very kind of him.
I have been receiving phones, IMs, and other messages from restaurant and pub owners and other small businesses upset about their site outage, asking what to do. I have referred them to Spokane-based Intrinium.com
While I can't afford the hosting, I know the owner, Kirt Runolfson to be a VERY ethical person and an all around good fellow. And the staff and attitude there is much more about quality, so they will be a much more reliable option, though not the cheapest.
On my end, I am going to use a poor man N-tier approach, with the front end web front on Godaddy's virtual hosting, but with all the links and content back on my OpenBSD servers with Plone (finally Plone works on OpenBSD 6!) I am building up back at my house, over my Comcast connection. It is by far less than optimal, but all I can afford any time soon. I still have a rack full of old servers that will suffice.
If I can get my server back in a day or two, then it will save me hundreds, if not thousands of man-hours recreating the sites, but I will still have to spend dozens of hours transitioning content to the new setup.
Have to run many errands, will resume restoration as soon as able.
More as it comes.
No further word from Tierpoint or CW.
Meanwhile, basic front page now up for the Eä Tolkien Society websites:
Need to go make dinner for the fam and hen schoolwork. Will resume website restorations later tonight.
Now basic version of another site up, deeper links using archive.org cache of older version for now:
Received email response from Electronic Frontier Foundation (EFF.org), with a list of questions for legal referral information.
I sent email with complete responses to all the questions in the email.
[11:25:55 AM] Dev2dev Portal: Morning. Yesterday Nicki said something was being worked out that I should be hearing something about how to get access to my server. Have you heard anything?
[11:37:53 AM] Stephanie: Sry missed call. Working in home care. Haven't heard anything.
[12:25:10 PM] Dev2dev Portal: Is there a number/Skype for me to try to get a hold of Alvin to find out? Nicki keeps claiming I need to speak with him directly.
[12:30:43 PM] Stephanie: He on skype under his name
Tried calling Nicki, received voicemail.
[1:42:42 PM] Dev2dev Portal: location New York? Colbert? something else? quite a few listed.
[1:46:47 PM] Dev2dev Portal: Btw, when you say you are working in home care, you mean you are no longer with Cyber World? Or just busy with something else?
[1:47:26 PM] Stephanie: No longer with cyber world. Got a job in health care
[1:48:22 PM] Dev2dev Portal: I see. Glad you have a back up so quickly. So sorry the CW stuff fell apart. Wish you the best
[1:52:09 PM] Stephanie: Ya. That sucks. And this is part time but pays some bills. I was told that they working in it and you should be able to get your server some time today hopefully.
[1:52:32 PM] Dev2dev Portal: <fingers crossed!>
[1:52:56 PM] Dev2dev Portal: Whom should I contact for that, since you aren't there any more?
[2:45:57 PM] Stephanie: Guess it going to be me going in since I'm familiar with the servers
[2:46:25 PM] Dev2dev Portal: ah. well thank you for that.
[2:48:18 PM] Dev2dev Portal: Are they giving you any timelines?
[2:49:32 PM] Stephanie: I'm making arrangements to go get everything right now. I can let you know when I have it so I can get it to you
[2:49:49 PM] Dev2dev Portal: thanks!
[3:10:39 PM] Dev2dev Portal: Would 5:00 pm be a good time for me to show up then?
[3:23:22 PM] Stephanie Bradley: I'll let u know when I have everything and we can go from there
[3:28:35 PM] Dev2dev Portal: understood. thanks.
Subject: RE: Dev 2 Dev Portal LLC's Proof of Server Purchase
Date: Tue, 14 Mar 2017 22:34:38 +0000
Please get in touch with Stephanie at E-Insites to get your equipment back. We have made arrangements with them to pick up some of the servers and return them to their customers. :)
Regional Office Manager | TierPoint
23403 E Mission Ave, Ste 121
Liberty Lake, WA 99019
Office: 509.688.2566 | Cell: 509.251.0360
email@example.com | www.tierpoint.com
Stephanie called to say she had the servers and could meet me somewhere for me to get mine. Agreed to meet her at a Maverick gas station on Highway 2, just north of North 40 supply. Feels very shady and odd.
Received server in the Maverick parking lot.
Stephanie still claimed not to know anything about what happened.
Just grateful to finally get the server back. Late for meeting with my volunteer staff to train them due to this (I informed them), meeting is for 4 hours.
Server is now plugged into my home server rack and basic network reconfigured so I can access it. Now have to change all the web, email, iRC, and other services to somehow work on the home NAT-ted setup.
Now have a few sites (rpgresearch.com and tolkienscholars.org) linking back to the lPlone instance on the server at the house, though not proxied, so for now port numbers and funky domain may give some users pause. Will need to setup some sort of proxying, or some other safer reference process to clean that up, but must sleep now.
I emailed Nicki at Tierpoint thanking her for the email to let me know I could get my server back. With meeting Stephanie on the roadside, without the email from Nicki, I would be very concerned about the circumstances of the server being released. Nicki's email let me know it was under relatively valid circumstances.
A bit of a long back story on why I took the (foolish) chance with E-insites / Cyber-World, despite all my misgivings and the warnings.
This is just a personal experience, and not very technical, so most people won't want to bother with reading this, and really won't care or want to read my opinions expressed hereing. But it is here for future warning to myself and others.
Not long after getting my GCIH, and later being published by the SANS Institute (pptp malware), I technically semi-retired from tech around 2003/2004 to focus on raising my kids. Planning for when they were adults, I decided to slowly train toward becoming a Recreation & Music therapist with added neuroscience and research psychology background.
As I incrementally handed clients off to others, by 2005 I really could no longer justify the $500+/mo at Hurricane Electric for hosting, so I moved the servers to my home.
This was problematic and unreliable, but most collocation facilities were just too expensive to justify the cost, and virtual hosting didn't allow the flexibility of what I was doing for communities (for free).
Eventually I found e-insites, while they were still in downtown Spokane in an old building. They were affordable, albeit I knew they wouldn't be the most reliable, and I wondered how they could afford to be so much cheaper than everyone else. My InfoSec senses were tingling, but I didn't want to delve deeper, the next cheapest options were not options short of just shutting most of the efforts/communities down.
Later they moved to the Tierpoint facility in Liberty Lake.
Some years later I did some part time tech work for Interlink, needing to shore up some of my income a little. Kirt, the owner, was great! The staff there were great, and I felt like I was back among my kindred that I had missed since leaving the Silicon Valley scene. Tech is addictive for me, it is hard to let it go, and hard for me to moderate it, I get very gung ho and focused when I work in tech. Which is great for the clients/projects, not so great for my relationships.
I later developed some increasingly debilitating health issues, and I needed to leave after some time due to the repeated illnesses that became nearly fatal. I later heard from one of E-Insites ex-employees that E-Insites might be a spam farm and that is why they were so cheap. I wanted to move, but again the costs were 5 to 10 times as much, so I held my nose, further tightened down my boxes, and hoped the rumors weren't true.
It would be 2 years before we tracked down the cause of my degenerating health, and even years later I am still recovering from the consequences of the intense medications. I only bring this up, the following politically sensitive topic, because it directly impacted my hosting decision making: All this happened as the AVA increasingly kicked in, making my catastrophic health insurance illegal for insurance companies to sell to anyone above 30 years old, and forcing me to buy health maintenance programs I did not want and have nearly bankrupted me. As the ACA/ObamaCare kicked in, due to my bracket, my insurance and health costs sky-rocketed (the opposite of the goals of the ACA). My premiums went from $160/mo before ACA to $900/mo in 2015, $1,380/mo in 2016, & $1,800+/mo for 2017, plus tens of thousands of dollars in loopholes not covered that I had to put on my credit cards, even though it was supposed to be the best insurance available for someone not working. This eventually lead to me going the past 2 years (including currently) without insurance or a doctor now (though keeping my kids insured ~$600/mo), So my financial strain is at a peak. I am hoping with the ACA removal I can recover, but we will see. Due to this, I had to shut down a lot of projects, especially many of the more expensive and time consuming community projects and websites. Over time I found some people to voluntarily step up and help keep them going for a few years while I tried to recover from the illnesses and expenses (still deep in hole on the medical expenses, and still uninsured and not seeing any doctors). I only bring all this up because it impacted my decision making regarding hosting costs, both from the illness and medication may contributing to impairing my judgement, and the expenses adding pressure to stick to a place that was increasingly concerning.
Over a year ago, once again, the entire IP block for E-Insites (now renamed Cyber-world) was listed in Spamhaus, and I saw that the COO Alvin Slocombe was listed in their ROKSO worst offenders list. Also I noticed tech support responsive all but died off. Getting a hold of someone by phone or email in the past had been fairly responsive, but the last 2-3 years it became increasingly difficult. And there were increasing errors and outages.
In November I volunteered (pro bono) to help Zombie Orpheus Entertainment to start building a new platform. First a rough prototype testing at my house, later we used CW (I gave repeated caveats not to use them for production, just use them for prototyping) to take the testing and development to the next stage.
When I confronted them about the blacklist going on for months still, they instead pointed me to a free Sendgrid account for sending email, since I don't send very much, with these community sites. This was supposed to be temporary, but alas it was only to get worse.
Last month the SSH connections were choking out, and CW then blocked port 22 for everyone. They claimed it was an ongoing DDoS. So they let me get temporary access so I could change my SSH ports. They said this was temporary, but it was not.
A few weeks later everything went down, hard. For several days. They kept saying they didn't know what it was, and that Tierpoint had multiple technicians working on it. Later they said they CW? Tierpoint? (they wouldn't elaborate) had hired a technician to update their core router security settings to fight off the ongoing DDoS attacks, but instead he hosed the router, and was going 3 days without sleep trying to fix it. Eventually it was back online, but SSH still blocked, and I noticed everything was running pretty slow, and they kept having intermittent packet loss on all ports. I pointed this out repeatedly, and there was minor improvement, but still not reliable.
Then all the events as logged earlier on this page took place.
I was paying $50/mo for my 1u at the colo, and $70/mo for leased dedicated servers they built, from around 2005/2006 to 2017. Not counting the ZOE project. They stopped billing me for the leased server years ago, and I really wasn't using it anymore. So just focusing on the single 1u server.
Hurricane Electric: $500/mo + bandwidth minimum cost from 2005-2017 (12 years) = $72,000.00+
Other less expensive colos: $250/mo + bandwidth (12 years) = $36,000.00+
E-insites/Cyber-world $50/mo (12 years): $7,200.00+
But, this doesn't take into account the cost to reliability, website reputation, and my professional and personal reputation, etc. And now I'm back at square one for 2005 hosting back in my house. However, I never would have been able to afford to do all that I did for the many communities and related sites if it had not been so inexpensive. And also, that would have been a lot of money that wouldn't be there for my kids (which I definitely couldn't justify spending)...
Tough decision and yet not when the kids and communities factored in, though painful in the end. And this past week's fiasco was indeed very painful.
I was already teetering on the brink from overload due to taking on too many projects and personal relationship (family, friends, etc.) time investments, severely overbooking my schedule. This outage pushed everything over the edge (temporarily).
I have fallen behind on multiple projects, deadlines, housework, schoolwork, and even screwed up important meetings with friends and other appointments during this time, not to mention very little sleep.
I am very sorry to the dozens of communities, and many other individuals and organizations, impacted by this. For all those people waiting for me to get you those research resources, I will be getting back to you in the next few days with links that will once again (hopefully) function. I hope to have things at least relatively stable (as much as a home setup can be) in the next few days/weeks.
Thank you for your support and patience during these challenges.
Wherever you may be, BE well.