Sorry, we are offline due to our upstream ISP being down indefinitely.

We are trying to move to a new location. The consequences of cheap hosting.

FOR THOSE OF YOU ALSO IMPACTED BY THIS OUTAGE, THAT HAVE BEEN CALLING/MESSAGING ME ABOUT WHERE TO GO, I HAVE BEEN RECOMMENDING HTTP://INTRINIUM.COM, THEY ARE RELIABLE, HONEST, RESPONSIVE, CAPABLE, ETC. THEY ARE A MORE STANDARD INDUSTRY APPROPRIATE RATE.

UPDATE: I NOW HAVE MY PHYSICAL SERVER BACK (doesn't help with the 6 virtual servers, those are a loss) AS OF LATE MARCH 14TH, AND I AM TRYING TO GET EVERYTHING WORKING AT THE NEW LOCATION WITH THE NEW SETUP. IT WILL LIKELY TAKE DAYS BEFORE THINGS ARE RELATIVELY BACK TO NORMAL AGAIN. APOLOGIES FOR THE INCONVENIENCE, AND THANK YOU FOR YOUR PATIENCE DURING THIS VERY TRYING TIME. THIS DOES NOT AFFECT ANY OF DEV 2 DEV PORTAL'S PAYING CUSTOMERS, ONLY THE (many) PRO BONO FREE COMMUNITIES.

See bottom of this page for the latest status updates.
 

Short summary about the outage:

Tierpoint.com in Liberty Lake, Washington, is denying access to everyone trying to get to their own physical servers.

No explanation as to why. Cyber-world.com (CW) / E-Insites systems administrator has also been denied access by Tierpoint to the CW servers and their customer's servers, according to the CW systems administrator, Stephanie, with no explanation as to why (according to Stephanie).

I had a single server, under my old company Dev 2 Dev Portal LLC our own property, in one of the many racks, in the cage with many other customers, as a customer of Cyber-World / E-Insites.

Cyber-world / E-insites (CW) was leasing a large caged area from Tierpoint.

According to the Cyber-world head SysAdmin,

Supposedly Tierpoint won't even tell Cyber-world's systems administrator why they have been locked out.

They have been completely offline since afternoon of Tuesday, March 7th, 2017.

------

Due to the ISP being down for so long, with no answer as to why or an ETA to get back online

and Tierpoint refusing to let anyone pick up their own physical servers

even though those servers NOT owned or leased from Cyberworld or Tierpoint, just the rackspace/colo used,

we are having to move all of our sites to a new hosting location.

This is going to take some time (a LOT of volunteer man-hours).

Have been hosting all of communities & sites for free at a very low-cost hosting facility.

We don't know if we can get our server back any time soon, Tierpoint is denying everyone access!

We have some backup servers that we will try to setup, but for now, we're temporarily using Godaddy

for these static pages. We will update as soon as possible

My Email: webmaster at dev2dev dot com

My Skype Phone: (509) 252-0800 (USA)

Skype: dev2devportal

Here is a list of all the 85 websites/domains impacted by this outage:

(as these begin to at least partially come back online, the plaintext will be converted to a link to the site. fortunately none of these are paying customers, they are all pro-bono communities I have been hosting/running over the decades. If I had actual paying customers, I would not have hosted them at such a place.)

  1. BRENCODE.COM
  2. DEV2DEV.BIZ
  3. DEV2DEV.COM
  4. DEV2DEVPORTAL.COM
  5. DRACONISDRAKE.COM
  6. EARPG.COM
  7. EATOLKIEN.COM
  8. HAWKENTERPRISING.COM
  9. HAWKES-HAVEN.COM
  10. HAWKETALKSHOW.COM
  11. HAWKTALKSHOW.COM
  12. Merp.com
  13. MERPCON.COM
  14. MIDDLE-EARTHRADIO.COM
  15. MIDDLE-EARTHTALK.COM
  16. MIDDLEEARTHRADIO.COM
  17. MIDDLEEARTHTALK.COM
  18. mtnorthwest.com
  19. MUSICTHERAPYNORTHWEST.COM
  20. MUSICTHERAPYNW.COM
  21. NORTHFIVEMILE.COM
  22. NORTHWESTLARP.COM
  23. NORTHWESTMUSICTHERAPY.COM
  24. northwestrpg.com
  25. NORTHWESTRT.COM
  26. NORTHWESTRT.ORG
  27. NWLARP.COM
  28. NWTOLKIEN.COM
  29. NWTOLKIEN.ORG
  30. NWTOLKIENSOCIETY.ORG
  31. OMZINE.ORG
  32. OTHER-MINDS.COM
  33. OTHERMINDS.NET
  34. OTHERMINDSMAGAZINE.COM
  35. rpg-bus.com
  36. RPG-RESEARCH.COM
  37. RPG-THERAPEUTICS.COM
  38. RPG-THERAPY.COM
  39. RPG-THERAPY.ORG
  40. rpg-trailer.com
  41. rpg-trailer.org
  42. rpgbus.com
  43. rpgfund.org
  44. RPGR.ORG
  45. RPGRESEARCH.COM
  46. rpgresearch.net
  47. RPGTHERAPEUTICS.COM
  48. RPGTHERAPY.COM
  49. RPGTHERAPY.NET
  50. RPGTHERAPY.ORG
  51. rpgtrailer.com
  52. rpgtrailer.org
  53. SPOKANE-RPG.COM
  54. SPOKANE2600.ORG
  55. SPOKANEASL.COM
  56. SPOKANELARP.COM
  57. SPOKANEMT.COM
  58. SPOKANEMUSICTHERAPY.COM
  59. SPOKANERPG.COM
  60. SPOKANETOLKIEN.COM
  61. SPOKANETOLKIEN.ORG
  62. SUITEGM.COM
  63. SYNTHETIC-ZEN.COM
  64. SYNTHETICZEN.COM
  65. SYNTHZEN.COM
  66. TECHTALKHAWK.COM
  67. TECHTALKHAWKE.COM
  68. THERAPEUTIC-RPG.COM
  69. THERAPEUTICRPG.COM
  70. THESYNTHETICZENSHOW.COM
  71. TOLKIEN-GAMING.COM
  72. TOLKIEN-MOOT.COM
  73. TOLKIEN-SOCIETY.COM
  74. TOLKIEN-SOCIETY.ORG
  75. TOLKIENCON.COM
  76. TOLKIENCONVENTION.COM
  77. TOLKIENGAMING.COM
  78. TOLKIENMOOT.COM
  79. TOLKIENMOOT.ORG
  80. TOLKIENRPG.COM
  81. TOLKIENSCHOLAR.COM
  82. TOLKIENSCHOLAR.ORG
  83. TOLKIENSCHOLARS.COM
  84. TOLKIENSCHOLARS.ORG
  85. will4real.org

BEGIN LOG

UPDATES

20170307 (Tuesday - day 1)

12:44

We have noticed that all servers for all our websites went offline sometime during the afternoon.

Not only does this outage completely impair our 85 domains and many communities, it also froze the entire ZOE Z1 project!

Later investigation found that all of Cyber-World (CW) is entirely down. CW is the collocation facility our server is hosted at (we own the whole physical server, we just lease their 1u rack space and bandwidth, they lease an entire large cage from their provider Tierpoint).

No answers to our calls or texts about what is going.

18:33

Receive text message from Stephanie Bradley, the lead Systems Administrator and manager at Cyber-world / E-Insites:

"I'm headed out to data center now to get another client their servers if you would like to meet me out there and get yours as well. I'm not sure when we're going to be back up."

19:38

I spoke with Stephanie to ask what was going on. She said stated she didn't know.

She did state that Tierpoint was to blame for the outage, but she claimed she did not know why.

I asked to arrange to talk with her Wednesday at Noon to schedule my picking up the server, it was late, there was a snow storm, and I was tied up with a group at the time, she agreed to the time.

20170308 (Wednesday - day 2)

10:21

I called Kirt Runolfson of Intrinium (he is a friend and CEO of Intrinium) asking about emergency hosting services for the ZOE Z1 project (I knew I couldn't afford it for my 85 domains and related communities, but ZOE was able to budget for the higher quality hosting at Intrinium). Kirt indicated they would probably be able to make something work, even on such short notice, and said Kevin would contact me.

Later, I texted and called Stephanie of CW around noon, and received no answers.

Tried calling again at 12:23 pm, still no answer from Stephanie.

2:27

Spoke with Kevin of Intrinium, started invoicing request for some emergency virtual hosting for ZOE Z1 project. Ben approved pricing and invoice, and made payment and received confirmation invoice by 15:31.

13:50

Kevin contacted me for the setup process of the new virtual server for ZOE Z1.

There were a few glitches, but eventually, by 19:00-ish, we had the basic server up and running, and I began reinstalling everything (WikiMedia, NodeBB, etc.). We were originally supposed to be Launching this Thursday, but the Tierpoint + CW debacle completely killed that launch!


 

20170309 (Thursday - day 3)

10:37

Now that we had an alternative host at least for the ZOE Z1 project, I called Tierpoint, hoping they would be reasonable and at least let us get access to all our content and code on the CW leased servers, and also hopefully my own physical server. They transferred me to Nicki Kopelson. She would not comment as to why the outage was happening, and that no one was allowed access to the servers, data, or anything else. Still would not explain at all why this was happening, and that it was now in the hands of the legal department. She suggested I speak with Alvin Slocombe of CW (I have never been able to get a hold of him, Stephanie has been good contact for a over a year, before that was the Irish fellow (Mark?)).

I asked if there were any options for at least retrieving my personal physical server, which is my property and not that of CW. She flatly said no. She spoke politely and sympathetically, and I was calm and respectful, but it was made clear others had been calling screaming and yelling about this issue (as Stephanie later would confirm).

We still can't get to all our configuration, content, and source code on the CW servers (those are leased from CW, while my server is my own separate property).

The way it is looking, due to what looks like a potential legal battle between CW & Tierpoint, it sounds like we can kiss all the ZOE Z1 work of the last several months (since November), and hundreds of man hours, goodbye, and we're having to start over from scratch. The Z1 data backups are also at the CW space.

11:16

Contacted KREM2 news that something was going on at the Liberty Lake facility.

11:55

Contacted KXLY news that something was going on at the Liberty Lake facility.

16:00

I have temporarily changed the DNS for all 85 domains to godaddy, and have paid for some hosting at godaddy to setup a static page to let people know what is going on. Next I am trying to back up and wipe an old system here at my house, and will try to setup dynamic DNS for all the 30 or so community sites, to try to get web services up, though it will be based on a much older back up (The more recent backups are at the colo alas). Then try to get email, IRC, streaming, and all the other services up as best I can from my home. Tierpoint still refusing to release my server. I told them I can prove ownership of the server, from my proof of purchase from Dell, proving it is not the property of CyberWorld. Niki at Tierpoint still said no, but that she would again consult with the rest of her legal team. I am not trying to contact Electronic Frontier Foundation Eff.org to see if they have any suggestions on how to get my data and my server back into my custody.

17:55 PST. The facility outage may be related to this:

March 8th, 2017: SpammerGate: The takeaway lessons and follow-ups on the River City Media data breach: http://www.csoonline.com/article/3178395/security/spammergate-the-takeaway-lessons-and-follow-ups-on-the-river-city-media-data-breach.html

And their rebuttal: March 9th, 2017: "River City Media Response to Defamatory and False Articles": http://finance.yahoo.com/news/river-city-media-response-defamatory-200900336.html

23:57 PST

We are now preparing some new servers to take over at a new location. We hope to begin having websites starting to come online sometime late Friday, March 10th, 2017.

We will keep updating this page as we progress.

First we will get basic web pages up and running, then email, then IRC, then remaining services.

Thank you for your patience and understanding during this difficult time.

20170310 (Friday - day 4)

09:30

Spoke with Niki. I asked her if showing proof of my server ownership would help in releasing my server. She will ask legal. But still no for now. She let slip something about "Alvin has to pay for release".

15:00

Nicki Kopelson, 688-2566 called me on my cell phone, indicating that if I provided the proof of purchase information, she "I'm not promising anything" indicated it might be possible. I wasn't home, and was out running errands, so
quickly detoured home.

15:27

I emailed her proof of ownership. I also still have the original shipping box.

15:28

I called Nicki to let her know I had sent the email with the proof.

She reiterated she couldn't promise anything, but that she would forward that to legal and do what she could.

15:40

Fingers crossed. I have been having conversations with Tierpoint's legal department representative, trying to persuade them to release my server since it is my property and neither Cyber-world's nor Tierpoints, nor is my server involved in any of the issues those two companies are wrangling over. I have provided them with proof of purchase/ownwership on my part. We'll see how it goes. I also reached out to EFF's legal department to see if they had any suggestions, but likely will be next week at the earliest before hearing back, knowing how busy they are.

15:46

Meanwhile, at my house I finished backing up an old server so I could wipe it, and I am almost finished installing OpenBSD on that server. Then I will grab my backups from a few months ago from the colo server (the more recent backups are still at the colo alas). I am going to use the Godaddy virtual hosting for the front pages of all 85 domains, which will take a while to setup, but then have links to "go deeper" to my home server when it is online, especially all the research papers and such. This will hopefully be a survivable workaround. The nearest colo I spoke to is a minimum of $300/mo for very little bandwidth. That is typical pricing for hosting my server, that is why CW's $50/mo for the past 10 years was so appealing after being at Hurricane Electric for $500/mo for 6 years. Since I don't do this for a living anymore (tech), it is difficult to justify such high hosting expenses. And while all this is going on, been working daily on the Zombie Orpheus Entertainment platform (as a volunteer). While juggling Psych classes, and The RPG Trailer tour and The other RPG Research related efforts. And my boys' issues. Phew! What a week!

20170312 (Sunday - day 6)

12:53

Slowly starting to recover from the outage listed here: http://dev2dev.com/all_offline/ I now have all the domains pointing to godaddy virtual hosting for now. Dev 2 Dev Portal LLC is my old (still alive) technology hosting and consulting company from before my retirement from tech. Once I stopped working regularly in the tech field, I found it difficult to justify the expense of a quality dedicated hosting facility, so eventually found the cheap hosting that we've been on since around ~2006 or so, but that recently "died" last week. The above link has more details at the top and bottom of the page.
I am now working to rebuild everything, while still not being allowed to get to my own server at the colo, or pull the data or the backups from the colo.
I now have a fresh install of OpenBSD 6.0 up and running on my older Asus 1u rackmount server running in my house over my Comcast 25Mbps (more like 5 Mbps to the public) connection (I will look into better options soon if I am going to have to keep this at my house).
I have started to get some of the front/landing pages for each of the 85 domains switched form just a plain-html "Sorry we're down" to either more info with links to our Youtube channels, twitter pages, Meetup groups, Patreon pages, etc. and some other sites putting in place WordPress, Joomla, or Drupal landing pages, that will then, for the deeper links, link to the server content in my house.
Other Minds Magazine is the furthest along for the front page (and soon the magazine archives will be back up too - http://omzine.org ). http://rpgresearch.com is going to take the longest to revive.
I am waiting on that and the other huge sites (merp.com has thousands and thousands of content items), to see if I can get my server back, rather than trying to recreate everything from scratch from my old backups.
If they let me get my server from the colo Monday, then I will be able to just pop that into place at my house (but will still need the front landing pages on godaddy first, and then link to that server), but at least then all the content and user accounts will be immediately back online. Fingers crossed!
Meanwhile, I will be adding a new goal to our Patreon, if we can get to around $300/mo I can afford to host at a colo once again. The location we were at for over 10 years, was only $50. Our previous location, Hurricane Electric, was $500/mo plus bandwidth overage. The new location would be Intrinium for $250/mo plus bandwidth overage. The current godaddy virtual hosting (which is very limited in features as all virtual hosts are, compared to what we have been able to do with our own dedicated servers), is $30/mo for unlimited sites/bandwidth. So, we'll manage, but it may be a while before things are stable again.
Soooo many man hours wasted. Not just the scores of hours I've lost myself, with all my projects now horribly backlogged, and many other people's projects. I had a number researchers recently accessing my sites, with their own deadlines at their universities, this is wreaking havoc!
Thank you for your support, patience, and understanding.
Happy Gaming!
-Hawke

20170313 (Monday - Day 7)

12:41

No word yet from EFF.org or Nicki at Tierpoint.

So far I have 20 of the 85 domains now with partial sites up for a few domains, barely any content but at least something to show up besides plain test, here is what is slightly up and running (again, with most of the content still missing currently, since mostly on server at colo). Basically just some photos, and contact information at best. All the research, videos, music, etc. still going to be many more man hours to restore, unless I can get that server back soon

  1. DEV2DEV.BIZ
  2. DEV2DEV.COM
  3. DEV2DEVPORTAL.COM
  4. OMZINE.ORG
  5. OTHER-MINDS.COM
  6. OTHERMINDS.NET
  7. OTHERMINDSMAGAZINE.COM
  8. RPG-RESEARCH.COM
  9. RPG-THERAPEUTICS.COM
  10. RPG-THERAPY.COM
  11. RPG-THERAPY.ORG
  12. RPGR.ORG
  13. RPGRESEARCH.COM
  14. rpgresearch.net
  15. RPGTHERAPEUTICS.COM
  16. RPGTHERAPY.COM
  17. RPGTHERAPY.NET
  18. RPGTHERAPY.ORG
  19. THERAPEUTIC-RPG.COM
  20. THERAPEUTICRPG.COM

13:14

I called Nicki to ask for an update on the server issue. She stated, more gruffly than our previous conversations, that something was being worked out with CW, and that I should hear back either from CW or Tierpoint in the next day or so about how to get my server back. Fingers crossed!

15:25

Received a Skype call (I was busy driving my son's Subaru with a stick shift, so I couldn't answer right away and had to call him back shortly after) from Chris Vickery, author of the article referenced in one of the earlier links in this log. Here is his original article related to "Spammergate" https://mackeeper.com/blog/post/339-spammergate-the-fall-of-an-empire

He stated via Skype his belief that this outage at CW is because of what he revealed in that article. He also apologized for the unfortunate effect of other innocent sites being taken down due to how Tierpoint may have decided to handle this, he didn't think they would handle it this way by pulling the plug to the entire cage.

Chris also stated he was an RPG fan, and was bummed that all my RPG related sites were inadvertently impacted. He seemed certain that the Spammergate issues were the cause of this outage and anything said otherwise was likely not correct.

He was friendly, and offered his help if he came across anything, such as affordable hosting, that could help me recover, which was very kind of him.

I have been receiving phones, IMs, and other messages from restaurant and pub owners and other small businesses upset about their site outage, asking what to do. I have referred them to Spokane-based Intrinium.com

While I can't afford the hosting, I know the owner, Kirt Runolfson to be a VERY ethical person and an all around good fellow. And the staff and attitude there is much more about quality, so they will be a much more reliable option, though not the cheapest.

On my end, I am going to use a poor man N-tier approach, with the front end web front on Godaddy's virtual hosting, but with all the links and content back on my OpenBSD servers with Plone (finally Plone works on OpenBSD 6!) I am building up back at my house, over my Comcast connection. It is by far less than optimal, but all I can afford any time soon. I still have a rack full of old servers that will suffice.

If I can get my server back in a day or two, then it will save me hundreds, if not thousands of man-hours recreating the sites, but I will still have to spend dozens of hours transitioning content to the new setup.

Have to run many errands, will resume restoration as soon as able.

More as it comes.

18:43

No further word from Tierpoint or CW.

Meanwhile, basic front page now up for the Eä Tolkien Society websites:

  1. EATOLKIEN.COM
  2. NWTOLKIEN.COM
  3. NWTOLKIEN.ORG
  4. NWTOLKIENSOCIETY.ORG
  5. SPOKANEASL.COM
  6. SPOKANETOLKIEN.COM
  7. SPOKANETOLKIEN.ORG
  8. TOLKIEN-SOCIETY.COM
  9. TOLKIEN-SOCIETY.ORG
  10. TOLKIENSCHOLAR.COM
  11. TOLKIENSCHOLAR.ORG
  12. TOLKIENSCHOLARS.COM
  13. TOLKIENSCHOLARS.ORG

Need to go make dinner for the fam and hen schoolwork. Will resume website restorations later tonight.

20:02

Now basic version of another site up, deeper links using archive.org cache of older version for now:

HAWKENTERPRISING.COM

22:51

Received email response from Electronic Frontier Foundation (EFF.org), with a list of questions for legal referral information.

23:25

I sent email with complete responses to all the questions in the email.

20170314 (Tuesday - day 8)

11:25

[11:25:55 AM] Dev2dev Portal: Morning. Yesterday Nicki said something was being worked out that I should be hearing something about how to get access to my server. Have you heard anything?
[11:37:53 AM] Stephanie: Sry missed call. Working in home care. Haven't heard anything.
[12:25:10 PM] Dev2dev Portal: Is there a number/Skype for me to try to get a hold of Alvin to find out? Nicki keeps claiming I need to speak with him directly.
[12:30:43 PM] Stephanie: He on skype under his name

13:00

Tried calling Nicki, received voicemail.

13:42
[1:42:42 PM] Dev2dev Portal: location New York? Colbert? something else? quite a few listed.
[1:46:47 PM] Dev2dev Portal: Btw, when you say you are working in home care, you mean you are no longer with Cyber World? Or just busy with something else?
[1:47:26 PM] Stephanie: No longer with cyber world. Got a job in health care
[1:48:22 PM] Dev2dev Portal: I see. Glad you have a back up so quickly. So sorry the CW stuff fell apart. Wish you the best
[1:52:09 PM] Stephanie: Ya. That sucks. And this is part time but pays some bills. I was told that they working in it and you should be able to get your server some time today hopefully.
[1:52:32 PM] Dev2dev Portal: <fingers crossed!>
[1:52:56 PM] Dev2dev Portal: Whom should I contact for that, since you aren't there any more?

14:45

[2:45:57 PM] Stephanie: Guess it going to be me going in since I'm familiar with the servers
[2:46:25 PM] Dev2dev Portal: ah. well thank you for that.
[2:48:18 PM] Dev2dev Portal: Are they giving you any timelines?
[2:49:32 PM] Stephanie: I'm making arrangements to go get everything right now. I can let you know when I have it so I can get it to you
[2:49:49 PM] Dev2dev Portal: thanks!

15:10

[3:10:39 PM] Dev2dev Portal: Would 5:00 pm be a good time for me to show up then?
[3:23:22 PM] Stephanie Bradley: I'll let u know when I have everything and we can go from there
[3:28:35 PM] Dev2dev Portal: understood. thanks.

15:34

Subject: RE: Dev 2 Dev Portal LLC's Proof of Server Purchase
Date: Tue, 14 Mar 2017 22:34:38 +0000

Hi Hawke,

Please get in touch with Stephanie at E-Insites to get your equipment back. We have made arrangements with them to pick up some of the servers and return them to their customers. :)

Thanks!

Nicki Kopelson
Regional Office Manager | TierPoint
23403 E Mission Ave, Ste 121
Liberty Lake, WA 99019
Office: 509.688.2566 | Cell: 509.251.0360
nicki.kopelson@tierpoint.com | www.tierpoint.com

17:35

Stephanie called to say she had the servers and could meet me somewhere for me to get mine. Agreed to meet her at a Maverick gas station on Highway 2, just north of North 40 supply. Feels very shady and odd.

18:09

Received server in the Maverick parking lot.

Stephanie still claimed not to know anything about what happened.

Just grateful to finally get the server back. Late for meeting with my volunteer staff to train them due to this (I informed them), meeting is for 4 hours.

20170315 (Wednesday - day 9)

01:57

Server is now plugged into my home server rack and basic network reconfigured so I can access it. Now have to change all the web, email, iRC, and other services to somehow work on the home NAT-ted setup.

05:45

Now have a few sites (rpgresearch.com and tolkienscholars.org) linking back to the lPlone instance on the server at the house, though not proxied, so for now port numbers and funky domain may give some users pause. Will need to setup some sort of proxying, or some other safer reference process to clean that up, but must sleep now.

14:36

I emailed Nicki at Tierpoint thanking her for the email to let me know I could get my server back. With meeting Stephanie on the roadside, without the email from Nicki, I would be very concerned about the circumstances of the server being released. Nicki's email let me know it was under relatively valid circumstances.

END LOG

Post Mortem - Infamous, "I knew better, but..."

A bit of a long back story on why I took the (foolish) chance with E-insites / Cyber-World, despite all my misgivings and the warnings.

This is just a personal experience, and not very technical, so most people won't want to bother with reading this, and really won't care or want to read my opinions expressed hereing. But it is here for future warning to myself and others.

Not long after getting my GCIH, and later being published by the SANS Institute (pptp malware), I technically semi-retired from tech around 2003/2004 to focus on raising my kids. Planning for when they were adults, I decided to slowly train toward becoming a Recreation & Music therapist with added neuroscience and research psychology background.

As I incrementally handed clients off to others, by 2005 I really could no longer justify the $500+/mo at Hurricane Electric for hosting, so I moved the servers to my home.

This was problematic and unreliable, but most collocation facilities were just too expensive to justify the cost, and virtual hosting didn't allow the flexibility of what I was doing for communities (for free).

Eventually I found e-insites, while they were still in downtown Spokane in an old building. They were affordable, albeit I knew they wouldn't be the most reliable, and I wondered how they could afford to be so much cheaper than everyone else. My InfoSec senses were tingling, but I didn't want to delve deeper, the next cheapest options were not options short of just shutting most of the efforts/communities down.

Later they moved to the Tierpoint facility in Liberty Lake.

Some years later I did some part time tech work for Interlink, needing to shore up some of my income a little. Kirt, the owner, was great! The staff there were great, and I felt like I was back among my kindred that I had missed since leaving the Silicon Valley scene. Tech is addictive for me, it is hard to let it go, and hard for me to moderate it, I get very gung ho and focused when I work in tech. Which is great for the clients/projects, not so great for my relationships.

I later developed some increasingly debilitating health issues, and I needed to leave after some time due to the repeated illnesses that became nearly fatal. I later heard from one of E-Insites ex-employees that E-Insites might be a spam farm and that is why they were so cheap. I wanted to move, but again the costs were 5 to 10 times as much, so I held my nose, further tightened down my boxes, and hoped the rumors weren't true.

It would be 2 years before we tracked down the cause of my degenerating health, and even years later I am still recovering from the consequences of the intense medications. I only bring this up, the following politically sensitive topic, because it directly impacted my hosting decision making: All this happened as the AVA increasingly kicked in, making my catastrophic health insurance illegal for insurance companies to sell to anyone above 30 years old, and forcing me to buy health maintenance programs I did not want and have nearly bankrupted me. As the ACA/ObamaCare kicked in, due to my bracket, my insurance and health costs sky-rocketed (the opposite of the goals of the ACA). My premiums went from $160/mo before ACA to $900/mo in 2015, $1,380/mo in 2016, & $1,800+/mo for 2017, plus tens of thousands of dollars in loopholes not covered that I had to put on my credit cards, even though it was supposed to be the best insurance available for someone not working. This eventually lead to me going the past 2 years (including currently)  without insurance or a doctor now (though keeping my kids insured ~$600/mo), So my financial strain is at a peak. I am hoping with the ACA removal I can recover, but we will see. Due to this, I had to shut down a lot of projects, especially many of the more expensive and time consuming community projects and websites. Over time I found some people to voluntarily step up and help keep them going for a few years while I tried to recover from the illnesses and expenses (still deep in hole on the medical expenses, and still uninsured and not seeing any doctors). I only bring all this up because it impacted my decision making regarding hosting costs, both from the illness and medication may contributing to impairing my judgement, and the expenses adding pressure to stick to a place that was increasingly concerning.

Over a year ago, once again, the entire IP block for E-Insites (now renamed Cyber-world) was listed in Spamhaus, and I saw that the COO Alvin Slocombe was listed in their ROKSO worst offenders list.  Also I noticed tech support responsive all but died off. Getting a hold of someone by phone or email in the past had been fairly responsive, but the last 2-3 years it became increasingly difficult. And there were increasing errors and outages.

In November I volunteered (pro bono) to help Zombie Orpheus Entertainment to start building a new platform. First a rough prototype testing at my house, later we used CW (I gave repeated caveats not to use them for production, just use them for prototyping) to take the testing and development to the next stage.

When I confronted them about the blacklist going on for months still, they instead pointed me to a free Sendgrid account for sending email, since I don't send very much, with these community sites. This was supposed to be temporary, but alas it was only to get worse.

Last month the SSH connections were choking out, and CW then blocked port 22 for everyone. They claimed it was an ongoing DDoS. So they let me get temporary access so I could change my SSH ports. They said this was temporary, but it was not.

A few weeks later everything went down, hard. For several days. They kept saying they didn't know what it was, and that Tierpoint had multiple technicians working on it. Later they said they CW? Tierpoint? (they wouldn't elaborate) had hired a technician to update their core router security settings to fight off the ongoing DDoS attacks, but instead he hosed the router, and was going 3 days without sleep trying to fix it. Eventually it was back online, but SSH still blocked, and I noticed everything was running pretty slow, and they kept having intermittent packet loss on all ports. I pointed this out repeatedly, and there was minor improvement, but still not reliable.

Then all the events as logged earlier on this page took place.

I was paying $50/mo for my 1u at the colo, and $70/mo for leased dedicated servers they built, from around 2005/2006 to 2017. Not counting the ZOE project. They stopped billing me for the leased server years ago, and I really wasn't using it anymore. So just focusing on the single 1u server.

Hurricane Electric: $500/mo + bandwidth minimum cost from 2005-2017 (12 years) = $72,000.00+

Other less expensive colos: $250/mo + bandwidth (12 years) = $36,000.00+

E-insites/Cyber-world $50/mo (12 years): $7,200.00+

But, this doesn't take into account the cost to reliability, website reputation, and my professional and personal reputation, etc. And now I'm back at square one for 2005 hosting back in my house. However, I never would have been able to afford to do all that I did for the many communities and related sites if it had not been so inexpensive. And also, that would have been a lot of money that wouldn't be there for my kids (which I definitely couldn't justify spending)...

Tough decision and yet not when the kids and communities factored in, though painful in the end. And this past week's fiasco was indeed very painful.

I was already teetering on the brink from overload due to taking on too many projects and personal relationship (family, friends, etc.) time investments, severely overbooking my schedule. This outage pushed everything over the edge (temporarily).

I have fallen behind on multiple projects, deadlines, housework, schoolwork, and even screwed up important meetings with friends and other appointments during this time, not to mention very little sleep.

I am very sorry to the dozens of communities, and many other individuals and organizations, impacted by this. For all those people waiting for me to get you those research resources, I will be getting back to you in the next few days with links that will once again (hopefully) function. I hope to have things at least relatively stable (as much as a home setup can be) in the next few days/weeks.

Thank you for your support and patience during these challenges.

Wherever you may be, BE well.