Site outage 9/10-9/11

Announcements about the database, website, and plugins.
SpartanJer
Posts: 1
Joined: Sun Sep 11, 2011 3:02 pm

Re: Site outage 9/10-9/11

Postby SpartanJer » Sun Sep 11, 2011 3:06 pm

Dying to try 'sick beard'. Been wondering what type of config options I had set wrong. Whoops guess your just down. Thanks for the service. Never found IMDB to be great for tv. Can't wait to try your site!

Rufus
Posts: 1
Joined: Sun Sep 11, 2011 3:57 pm

Re: Site outage 9/10-9/11

Postby Rufus » Sun Sep 11, 2011 4:08 pm

devicenull wrote:
stubbers wrote:
Freddy wrote:So with POSIWID in mind, the system isn't doing what it's supposed to do. It's down.


I can't agree more. The site is "slow" and I've yet to actually see a request go through. For all intents and purposes, from the perspective of a user, the site is down.

Now, before hikaricore comes in and does some crowd control, I would also like to point out that I understand this service is provided free of charge, I've been lurking this thread in hopes that the site comes back up soon so I can get back to sitting back and letting Sickbeard do its thing. If the site truly is down because it is being hammered by backlogged requests, this isn't going to happen on its own. I work for a fairly high traffic website that does MUCH more traffic than this site does. Showing off? Nope, just providing some "credentials". I'm here to offer some help getting things back up and running smoothly, if you would like. Just send me a PM and we can find a better way to communicate. I have a few ideas that may help gradually bring the site to a usable state, instead of just hoping things will calm down (which is what it seems the admins are doing now).

mbr
Posts: 1
Joined: Sun Sep 11, 2011 9:23 am

Re: Site outage 9/10-9/11

Postby mbr » Sun Sep 11, 2011 4:45 pm

Apparently I am not the only one who signed up just to offer help. Kudos to all.

To the devs: thousands of XBMC users thanks you for providing a great service, keep up the good work :)

Coco
Site Admin
Posts: 2472
Joined: Tue Mar 13, 2007 10:16 pm
Location: Canada

Re: Site outage 9/10-9/11

Postby Coco » Sun Sep 11, 2011 4:50 pm

Freddy wrote:Doesn't sound like an issue. Depending on what kind of CPU and memory usage the site requires then i always think that something can be figured out. Who is the right admin to speak with?


If you're really serious you could talk to me or szsori(szori would be best) in a pm. Although we most likely wouldn't change hosts at this time. If you really have a pipe big enough once we sort out a proper mirroring solution we might be able to make use of such an offer.

As for the sites current status. I believe the last of the changes was made about 30 minutes ago. Things should start running more smoothly now.
Please don't PM me. Post in the forums instead unless there is a very good reason for a PM or I won't respond.

szsori
Site Admin
Posts: 2004
Joined: Fri Nov 03, 2006 5:23 pm

Re: Site outage 9/10-9/11

Postby szsori » Sun Sep 11, 2011 5:02 pm

Anyone with a suitable amount of bandwidth available (probably 15+TB/month) can contact me. The way we would configure it now and in the future is to set up a peering reverse proxy to handle as much static traffic as possible. If someone really wants to pay a CDN bill for the site, consider MaxCDN currently has a 25TB plan which should be enough for $1999/month. I'm sure slightly cheaper alternatives can be found, but I doubt any would be less than $1500/month.

Thanks to Matt S, crzykidd, and Coco for working hard to get the situation resolved, and thanks to everyone for being patient. Our downtime has been steadily decreasing with the Matt & Matt involved with the project.

Freddy
Posts: 12
Joined: Sat Sep 10, 2011 6:33 pm

Re: Site outage 9/10-9/11

Postby Freddy » Sun Sep 11, 2011 5:12 pm

What kind of resources does the website use? Thinking memory and CPU wise.

devicenull
Posts: 2
Joined: Sun Sep 11, 2011 12:28 pm

Re: Site outage 9/10-9/11

Postby devicenull » Sun Sep 11, 2011 7:07 pm

CDN services (especially if you're using some sort of gslb) can be pretty spendy. (For those uninitiated, that means serving content to the user from the geographically closest location)
The company I work for maintains their own global CDN services, and the amount they spend on networking, storage, datacenters, peering - not to mention corp-office networks around the world can most easily be described in scientific notation.

That said, I really doubt there are going to be many suggestions from the crowd that the admins haven't already thought of. My "average" infrastructure looks something like this: SSL terminated at the load balancer(s), edge servers packed full of memory with varnish over nginx for cacheable content, passthrough to nginx for uncachable content backed by <insert favorite storage solution> and a CDN that grabs up anything it can after an initial request is made. That seems to be industry standard right now.

It seems to me like they're doing everything right outside of having a continuity planning location. But I'd also venture that a second location is out of financial reach. They certainly don't PLASTER their site with advertisements. In fact, I'm at a loss for what their method of revenue generation is unless they're just really really chairitable. (In which case, we all should grovel at their kindness)

In my world, we build out an entire second location that mirrors our first. If we need to do any site maintenance, we kick all traffic over to the "cold" location, do our work, kick all traffic back to the original location then do the same maintenance to the "cold" location. Insanely expensive? You bet. Any customer downtime or lost ad revenue? Nope. This design is actually meant for emergencies like site1 getting nuked from orbit. The maintenance thing is just a handy biproduct. It's all about keeping the users unaware that there ever was a problem and keep the services and ad revenue rolling.

And hey, I know I'm a nobody on the forum here, but if anyone would like to reach out to me for any "well this is how we do it" sort of advice, please feel free.

szsori
Site Admin
Posts: 2004
Joined: Fri Nov 03, 2006 5:23 pm

Re: Site outage 9/10-9/11

Postby szsori » Sun Sep 11, 2011 7:36 pm

devicenull wrote:That said, I really doubt there are going to be many suggestions from the crowd that the admins haven't already thought of. My "average" infrastructure looks something like this: SSL terminated at the load balancer(s), edge servers packed full of memory with varnish over nginx for cacheable content, passthrough to nginx for uncachable content backed by <insert favorite storage solution> and a CDN that grabs up anything it can after an initial request is made. That seems to be industry standard right now.


That's pretty close what we're doing except we don't have SSL yet and no CDN. Plus we only have a single nginx server right now.

devicenull wrote:It seems to me like they're doing everything right outside of having a continuity planning location. But I'd also venture that a second location is out of financial reach. They certainly don't PLASTER their site with advertisements. In fact, I'm at a loss for what their method of revenue generation is unless they're just really really chairitable. (In which case, we all should grovel at their kindness)


Grovel away. Right now all time spent programming is donated by me, all time spent on admin stuff is donated by Matt A and Matt S, all time spent managing mods and addressing everyday issues with the site is donated by Coco, and all bandwidth and hardware is provided free of charge by Matt A. Without each of those, the site wouldn't exist right now. Mods also donate their time and deserve thanks. We make well under $100/month from ads and donations combined (mostly ads), most of which has gone toward forming our LLC. The new site has some additional revenue streams planned and some additional ad locations, but my goal is still for them to be unobtrusive.

devicenull wrote:In my world, we build out an entire second location that mirrors our first. If we need to do any site maintenance, we kick all traffic over to the "cold" location, do our work, kick all traffic back to the original location then do the same maintenance to the "cold" location. Insanely expensive? You bet. Any customer downtime or lost ad revenue? Nope. This design is actually meant for emergencies like site1 getting nuked from orbit. The maintenance thing is just a handy biproduct. It's all about keeping the users unaware that there ever was a problem and keep the services and ad revenue rolling.

And hey, I know I'm a nobody on the forum here, but if anyone would like to reach out to me for any "well this is how we do it" sort of advice, please feel free.


I value input from people like you, who have an understanding of the proven methods for doing things. It validates our decisions and also provides guidance for future decisions, and in some cases points out where we've screwed up.

Unless we get some corporate sponsorship CDN hosting will not be used, but I'm building out the new site's API to handle it if we eventually switch over to that route. We're also planning on allowing people to host a reverse proxy if they have the bandwidth to handle it and then load balancing, even if it's extremely simple DNS round-robin load balancing. Part of that will be to allow (and require for new commercial projects) that large projects host their own reverse proxy for their traffic or help out the project by having their reverse proxy sit in our group of load balanced servers. The only other options would be to go to a completely premium service, which I never want to do, or get a large corporation to sponsor the site.

jerbob92
Posts: 1
Joined: Fri Feb 04, 2011 6:53 am

Re: Site outage 9/10-9/11

Postby jerbob92 » Mon Sep 12, 2011 1:44 am

If it's all about hosting files, not generating with php, and it isn't too much data( < 1TB). I'm able to get you guys a mirror.
The bandwidth won't be a problem.

KeithLM
Posts: 2
Joined: Sun Sep 18, 2011 4:33 pm

Re: Site outage 9/10-9/11

Postby KeithLM » Sun Sep 18, 2011 4:35 pm

What is the status since this outage? I see the site is back up, but I can't get TV shows scraped with XBMC or Media Companion since the outage. Is the API still down? Is it dead or changed?