I've got the new refund application up.  You can find it here...

http://www.thirdrow.net/refunds.aspx

Thank you everyone for your patience in this trying situation.

Showdog

• Fixed breed prestige to update properly each day.
• Fixed an issue with events and missing dogs

Showcattle

• Rescheduled new show addition for the first Monday of each month.
• Added shows for October
• EPDs will update today and each Sunday from now on.
• AU fees are back online

Showdog

Fixed the SDWC entry process

Showcats

Got stud and sale listings back online.

Our web host realized that the HTTP wasn't pinging and checked the servers out this morning.  Evidently, there were some memory leaks causing the sites to time out.  They changed some configurations to fix that problem but that has started the issue that causes users to have to sign in every few minutes (in some cases seconds).

This problem has always existed it just wasn't something that came up because it just slowed the sites down before.

I've asked them to undo the configuration update and I'm going to spend the next day eliminating memory leaks from the code.  There won't be any other updates to the sites until I'm done with that.

Point schedule updates no longer go out to users who only own retired dogs in a certain breed. Updates will continue to go out to all owners of non-altered dogs.

Dogs that are listed for sale but have offers turned off will no longer be listed in the dogs for sale pages.

All session bids less than $5 have been deleted and the new minimum bid has been set at $5.

We've added an application to get a refund for the two lost days here...

http://www.thirdrow.net/refund.aspx

Please note, the prorated refunds will not be large due to the small size of our subscriptions.

Also, mail in subscribers will need to e-mail admin@showdog.com directly for information on recieving a refund if they wish.

It LOOKS like we're out of the forest now but you never know.  If anything happens, I'll keep you updated.

There have been a few momemts when the site has been a bit slow to respond but they were short periods of time and sporadic in nature.

I'm starting the process of sorting out what to do in order to get everyone going again. 

First, all session sales for the time the site was out have been refunded. 

Second, we will give all subscribers twice their normal amount of sessions for the next seven days. 

These two steps will likely have the side effect of decreasing both the salary and session prices for the next few days.

All users on all sites who might have had their subscriptions expire over the two day period the site was down likely got a notice that their subscription has expired.  This was because Paypal wasn't able to update our database with correct info.

So far, we haven't found anybody who had their account actually dropped or anything messed up.  However, if this did happen, please contact us ASAP.

...keep your fingers crossed.  It looks like the sites have been up for about an hour. 

I'm going to bed for a few hours but I'll be up early to see what we can do to move forward.  I would stay up and get things caught up but I want to see if this is going to stick first.

I have no clue when these fluctuations will stop.  I wish I could tell you something but I can't.

As I type this, all of the sites but Showdog are up.  For details on why Showdog isn't up, please see the following blog post...

http://blog.thirdrow.net/blog/default.aspx?id=78&t=Why-is-Showdogcom-the-only-site-that-is

We'll figure out how to move forward after the sites stay up for at least one hour.  Updates will be posted here and then on the relevant sites.

This will be the first place information is posted throughout the rest of the day.

Please let anybody you know who is wondering what's going on know about this blog and copy any posts they may want to know about.

I'm working on seeing if everything is good to go and potentially changing web hosts and won't be able to update everyone as well as I should.

Currently, all of the sites are up, something that hasn't been the case since early Sunday morning.  We'll see if this sticks or if they have another outage.

[It didn't stick for Showdog.  See the post below for an explanation of why Showdog is different]

No, the servers have been running this whole time.  Please see my blog titled “Why is Showdog.com the only site that isn’t back up yet?” for more details on why the servers are up but you cannot contact them.

Please see the schematic below...

Our problem has been with “Router I” which is the gateway between our original three servers and the rest of the world.  When that goes down, we cannot contact our web server which is why the sites don’t come up at all.

Even when the sites have come back up, the connections between “Router I” and “Router II” have been down.  That means, that you can see the Showdog website but not the Showdog database (notice the line from “Web Server” to “DB Server III”).

Since DB Server I and II are in the same ‘rack’ of servers, they are able to communicate with each other.  The database for this blog is also on DB Server I.  That’s why when “Router I” comes back up, you are able to access TH, Showcats, Showcattle, and the web portion of Showdog.  Unfortunately, the web portion of Showdog hasn’t been able to connect with the database.

"I know a place where you can get hosting for $20/month."

I've gotten a couple of these e-mails so I figued I should explain why.

We have four dedicated servers running the sites.  The database for Showdog is somewhere around 30GB in size.  The last time we moved just the Showdog database it took 9 hours just to transfer the file.

That doesn’t even begin to include the time it takes to get a new web host to setup the servers in the first place.  Typically, I have had to wait at least two weeks for an order to be processed.  Rush orders have been in the range of four days.

I just want to point this out to let people know why we don’t ‘just switch’ web hosts.  It’s a long process that takes a lot of planning, time, money, and effort to be done properly.  When we can’t even connect to our current web host, it’s literally impossible to do.

Our hosting bill is in the low four figures per month and we aren’t getting what we paid for either.  Trust me, nobody is as ticked about this as I am.  We’re shopping for new web hosts right now at prices that will increase our hosting costs by over $1,000 per month.  We realize up time is vital to the sims and are prepared to make the investment necessary to improve it.

About 6 AM on Sunday, all of our sites became unresponsive due to a network issue at our web host.  I immediately called them and they were already working on the issue.  They claim that their core router was compromised by “hackers from China”. 

Do I know if any of this is true?  Nope, it’s just what I was told and I’m passing it on to you.

In the past 28 hours, I have called them 14 times and all I’ve done is diverted resources they should have been devoting to getting the network back up.  I would apologize profusely if this were an issue I caused and while I’m sorry for the down time, I simply had no control over any of this.

Does this excuse the sites being down?  Heck no.  I was on the phone with another web host about moving when I got a page the sites were responding so I stopped to post this.

Please check the blog as the day goes on for more details.