Restructure/Redesign

Announcements about the database, website, and plugins.
c4ho
Posts: 160
Joined: Wed Apr 18, 2007 3:23 am

Re: Restructure/Redesign

Postby c4ho » Sun Mar 01, 2009 2:46 am

szsori wrote:
click170 wrote:Premium Features:
...
Just for our mirror server, we're at 65GB/day. That's a TON. We can't depend on free hosting for that indefinitely. I have some possible solutions to help lighten the load using the donations and Google ads on the site (low cost mirrors), and free mirrors will help, but the main server will still have to handle considerable bandwidth per day. It's just not likely that it'll be there for the long run unless we can help pay for it.
...


In the context of a website thats alot but as soon as you step outside that narrow definition its not really alot. Thats about 2TB a month that could be delivered with a 29Euro a month Leaseweb rented server. I think there are a lot of ways to share this load. Number one should be to require that all apps that access the site cache the raw results and that should be enforced to some extent. For example XBMC doesn't meaning that people constantly blank and rehammer the website every time something goes wrong with their database. Please god dont have a premium service. If you need cash define what you need and when you need it to let the community provide as either cash or assets.

I guarantee bandwidth wont be a problem if you make it easy for users to help out with it.

emigrating
Site Admin
Posts: 278
Joined: Mon Aug 27, 2007 4:38 pm
Contact:

Re: Restructure/Redesign

Postby emigrating » Sun Mar 01, 2009 5:34 pm

c4ho wrote:Please god dont have a premium service. If you need cash define what you need and when you need it to let the community provide as either cash or assets.

Why not?

As long as the "core" site/service (i.e. everything it does today) remains free, why would it necessarily be a bad thing to require third-party developers to "licence" our data or ask end-users to pay a nominal monthly/yearly fee for access to premium services?

The vast majority of people are happy with what the site currently provides and would not notice a difference at all - those that do want premium services have the choice of actually getting them (for a small fee). Sounds like a great plan to me.

Also, I see no difference in actually requiring people pay up for these services rather than providing them for free on a "donation" basis. The same amount of cashflow would be generated from the sites perspective, the only difference would be that those using these extra services are all paying for them rather than relying on the [larger] donations of a select few.

c4ho
Posts: 160
Joined: Wed Apr 18, 2007 3:23 am

Re: Restructure/Redesign

Postby c4ho » Mon Mar 02, 2009 3:57 am

emigrating wrote:Why not?


Theres no easy answer to that but i will try to give a few examples.

emigrating wrote:As long as the "core" site/service (i.e. everything it does today) remains free, why would it necessarily be a bad thing to require third-party developers to "licence" our data or ask end-users to pay a nominal monthly/yearly fee for access to premium services?

Is it your data to sell? It is certainly your service to sell but if my understanding is correct you could not hide any data from non paying customers since that data is donated for free. I cant see a clear statement on the site about attribution so I amy be speaking out of turn but I certainly wouldn't expect you to have the right to sell the data I wrote so its fair to say this is probably the perception of many.


emigrating wrote:The vast majority of people are happy with what the site currently provides and would not notice a difference at all - those that do want premium services have the choice of actually getting them (for a small fee). Sounds like a great plan to me.


Thats sounds eprfectly ok but its a fine line between extra features and artificalliy reststicting access to user sub,itted data. This may be easily covered though with very regular and publiclly downloadable database dumps. This way you are saying "here is the data it is free" but "to access it in a slick way requires us to spend money so were charging ou for some premium services". Its a very fine line as this is what IMDB did and has... and whilst it is a superb source of data it is hardly in the same spirit as tvdb is this now.

emigrating wrote:Also, I see no difference in actually requiring people pay up for these services rather than providing them for free on a "donation" basis.


There is a fundamental difference between a donation based site and a subscription based site (even if some of it is still free).
emigrating wrote:The same amount of cashflow would be generated from the sites perspective, the only difference would be that those using these extra services are all paying for them rather than relying on the [larger] donations of a select few.


The same amount of cash flow is never generated. At some point you wont be making enough but eventually you will be making a profit. What are you plans to deal with this profit? Will there be transparancy of outgoing and incomings.?



In general though the sites built by volunteers, populated by volunteers and hosted by volunteers along with volunteer donations of kit and money. From what has been posted previously there are very little expenses and whilst this will certainly increase theres needs to be a clear statement about licensing, attribution and the future of tvdb. After all you will be deciding what will be free and what will be paid for and if it wasnt for the community tvdb wouldn't really exist.

Please dont read things into this that i dont intend. You just have to realize that whilst a handful of devs hold all the power, that power came from a myriad of users that cannot be represented properly in your design meetings/plans. Any decisions that effect all users should be taken very carefully.

szsori
Site Admin
Posts: 1911
Joined: Fri Nov 03, 2006 5:23 pm

Re: Restructure/Redesign

Postby szsori » Tue Mar 03, 2009 2:29 am

None of us on the design/implementation side are expecting to make money off the site. We're looking to cover the costs that it incurs. As emigrating said, access to data will remain free. What we'd be charging for are services that manipulate the data above and beyond what the current interfaces do. A good example would be the complete database backups. While all of the site data would remain available as XML/ZIP files, we'd give access to complete database backups as a premium service, which would allow commercial developers the ability to do more flexible things than just the XML/ZIP files. Since those backups would require more bandwidth, we'd have to charge for them or they wouldn't be possible.

Frankly, while donations and ads on the site have sustained it to this point, it's been completely out of the good will of our current hosts that the site is still up. Despite having faith in the community to provide some support, I don't have faith in them to completely sustain the site. The site's history has shown that.

As I said, we're not looking to change currently functionality for anyone, but rather extend functionality in creative ways for people.

c4ho
Posts: 160
Joined: Wed Apr 18, 2007 3:23 am

Re: Restructure/Redesign

Postby c4ho » Tue Mar 03, 2009 3:42 am

Perhaps this would be easier to discuss if you could give a couple more examples of features non paying users wont be getting?

Im not adverse to the idea and I accept that you need resources to do things but as I said its a fine line.

I also humbly point out that at some point you will be making a profit, whats your plans to deal with this.?

Also playing devils advocate. Dbase dumps will be a paid for feature because bandwidth costs money. Thats fair enough until someone says I will host a torrent for it so you have virtually no bandwidth costs. Im am not being anal I am just predicting the reaction of some to hopefully make the transition easier.

I have deablt with something similar to this in the past and there was a huge backlash. In the end all was good but it was handled incorrectly, what should have happened was full disclosure before anything really changed.

emigrating
Site Admin
Posts: 278
Joined: Mon Aug 27, 2007 4:38 pm
Contact:

Re: Restructure/Redesign

Postby emigrating » Tue Mar 03, 2009 7:18 am

Nothing has been decided as to what level of features would be considered "premium", so actual examples can't be given. I'll throw some ideas up in the air however (just remember, they are nothing more than just that, ideas).

  • Non licenced applications (I'm talking about the actual developer rather than the 'end user' here) would receive a [generous] bandwidth limit per day. (This would actually serve more as an incentive to cache data locally rather than hitting up our APIs for every single call to their application - also, this is/was mainly aimed at those developers running their own websites simply pointing to our images rather than those creating free [OS] utilities for working with TheTVDB data.)
  • Licensed users would receive their own "my collection" page here at TheTVDB (see my sig for a rudimentary implementation of what I'm talking about). This would track what episodes they have from which series, which episodes they have seen (when and how many times) as well as offer access to a backup/restore API (which their applications developers would have to program for obviously) which would give them the opportunity to easily backup/restore their local database. Some people do like the "Lookie here, this is all the TV I have stored on my harddrive" concept and others could benefit from the restore/backup when/if they change computers, have to reinstall the OS etc etc.
  • Licensed users could have access to personalized RSS feeds with "their" new episodes aired yesterday, today, this week etc.
  • Licensed users could have access to a personalized premium "recommendation" service which will recommend other shows based on their current collection - as opposed to non-licensed usage which would generate "preset" recommendations.
  • Basically anything that goes beyond what you would expect from a site like this could be considered "premium" and would levy a fee of some sort. As previously said, this would obviously not include the core services which is metadata retrieval of tvseries/episodes.

As for DB dumps being considered a premium service - sure, we could do monthly torrents of the database, but what's so wrong with actually charging a few bucks for it and be done with it? At least then the data contained would be fully up to date (and likely include tools to resync the DB on a regular basis). Also, please note that a handful of people requested a complete dump of the DB last year. So it's really dumb to allocate manpower and bandwidth on something that may not be downloaded for weeks at a time.

I would also like to address your concerns about the site making a profit. Like szsori said, the intention is not to make a profit, but rather make the site self-sufficient (something it is not, and has not been, for a very long time). Firstly, the LeaseWeb dedicated server for €29.00 would not be much of an alternative for this site. Yes, the actual bandwidth will work (just) but a single AMD Sempron 3100 and only 512MB RAM? The site is currently running on 4x dual core Intel Xeon 3GHz platform and CPU usage is maxed out at times. Obviously, CPU usage will be somewhat tamed with the redesign of the database, but even so trying to move from an 8-core to a single-core setup is madness.

Once all of this is paid for, should there be any money left over it can be put to good use in plenty of ways.
  • Lower the licensing costs.
  • Run competitions with "valuable" prizes (DVD/BD box-sets spring to mind as an obvious choice). The winner could be the top contributors of [good] data.
  • Considering the site is/will be run as a business rather than just a "spare-time project" there will also be [small] administrative costs.
  • Should we, after all of the above, still not manage to "burn" through the pile of money you expect us to be sitting on how about actually dividing it to the companies owners as dividends? (Of course, I don't foresee this happening anytime soon, if at all).

Will people get mad at the site moving from completely-free to semi-free? Sure, some folks will. There are plenty of die-hard, opensource the world, free4all zealots out there that have never, nor will never pay for cable, satellite, broadband etc but rather "steal" their on-the-air TV or mooch their internet of neighbors with unsecured wifi access points. The majority of people however understand the way the world works. It is not free to stick a server on the internet (especially one that used 2.1875 TB of bandwidth in February), and if they can pay us a few bucks now and then they will.

So, predicting your next question ;) "Why not just do it donation-based, if so many people are happy to pay a few bucks?"
Simple, my experience in this "field" tells me that people are happier "donating" if they get something above the norm in return. So, whilst a user may not necessarily need nor utilize our premium services they know they are getting them for their $9.99 a year (bear in mind, I just made this figure up) and are more likely to purchase this service just to help us out keeping the site online. An alternative is to give [any] donors access to the same services but then we're really being unfair as user A may donate $1.00 once and utilize as much of these services as he possibly can whereas user B may donate $10 a few times a year and only utilize the backup/restore service.

I would also like point out that people who contribute to the site could earn "points" which would go towards their licensing cost. For example, someone taking "ownership" of Heroes and maintaining that data could be considered excempt from any costs and would still have full access to all premium services. Someone providing a high-quality FanArt image for a series could receive, say, 100 points which would allow them access to these services for a week (or month/year).

DieBagger
Posts: 84
Joined: Sat Oct 27, 2007 9:17 pm

Re: Restructure/Redesign

Postby DieBagger » Tue Mar 03, 2009 8:42 am

emigrating wrote:Nothing has been decided as to what level of features would be considered "premium", so actual examples can't be given. I'll throw some ideas up in the air however (just remember, they are nothing more than just that, ideas).
  • Non licenced applications (I'm talking about the actual developer rather than the 'end user' here) would receive a [generous] bandwidth limit per day. (This would actually serve more as an incentive to cache data locally rather than hitting up our APIs for every single call to their application - also, this is/was mainly aimed at those developers running their own websites simply pointing to our images rather than those creating free [OS] utilities for working with TheTVDB data.)
  • Licensed users would receive their own "my collection" page here at TheTVDB (see my sig for a rudimentary implementation of what I'm talking about). This would track what episodes they have from which series, which episodes they have seen (when and how many times) as well as offer access to a backup/restore API (which their applications developers would have to program for obviously) which would give them the opportunity to easily backup/restore their local database. Some people do like the "Lookie here, this is all the TV I have stored on my harddrive" concept and others could benefit from the restore/backup when/if they change computers, have to reinstall the OS etc etc.
  • Licensed users could have access to personalized RSS feeds with "their" new episodes aired yesterday, today, this week etc.
  • Licensed users could have access to a personalized premium "recommendation" service which will recommend other shows based on their current collection - as opposed to non-licensed usage which would generate "preset" recommendations.
  • Basically anything that goes beyond what you would expect from a site like this could be considered "premium" and would levy a fee of some sort. As previously said, this would obviously not include the core services which is metadata retrieval of tvseries/episodes.

See, here's the problem, those are all new features to the api I'd consider regular site improvements. Btw what you're suggesting sounds oddly similar to the concept of private torrent trackers (so I've heard ^^).

emigrating wrote:Will people get mad at the site moving from completely-free to semi-free? Sure, some folks will. There are plenty of die-hard, opensource the world, free4all zealots out there that have never, nor will never pay for cable, satellite, broadband etc but rather "steal" their on-the-air TV or mooch their internet of neighbors with unsecured wifi access points. The majority of people however understand the way the world works. It is not free to stick a server on the internet (especially one that used 2.1875 TB of bandwidth in February), and if they can pay us a few bucks now and then they will.

Sorry, but you have obviously not understood the concept of open source.

What I don't understand is that applications that used the api were at some point supposed to offer a mirroring server in return (iirc?). So if there are large (and/or commercial) applications using the api, why not ask them to provide (or let their community provide) mirroring solutions?

emigrating
Site Admin
Posts: 278
Joined: Mon Aug 27, 2007 4:38 pm
Contact:

Re: Restructure/Redesign

Postby emigrating » Tue Mar 03, 2009 9:41 am

DieBagger wrote:See, here's the problem, those are all new features to the api I'd consider regular site improvements. Btw what you're suggesting sounds oddly similar to the concept of private torrent trackers (so I've heard ^^).


What, you think it's entirely acceptable that a handful of guys should provide hardware, support and bandwidth worth thousands of dollars just so you can get personalized RSS feeds of what episodes airs today? Those queries are, when you take into the account the number of users this site has, fairly expensive (in terms of CPU time). I'm sorry, I don't [think that's acceptable at all].

Running the site without any kind of guaranteed income will eventually mean it goes down [again]. Yes, it's fairly easy finding someone willing to mirror the actual data stored, it's slightly more difficult finding someone willing to sacrifice a dedicated 4- or 8- core 12GB RAM server for the project - for free.

Now, I'm not going to compare this site to TPB, as there are obviously more bandwidth used and more users registered at TPB, but their monthly running costs are in the region of $90000.00. But using this as a rough guide, one can argue that TheTVDB incurrs costs of ~$1125.00/month (based on figures from February). At the moment the revenue stream from the site wouldn't even manage to cover a tenth of that - and doing the math, the electricity cost alone for running the site is in the region of $53.33 (based on the national average of 11.43 per kWh). Add ongoing hardware costs, bandwidth costs, actual on-site support etc and it quickly mounts up. Still way below the $1125 above, but in any case it's a lot more than the site is currently generating in revenue.

DieBagger wrote:
emigrating wrote:Will people get mad at the site moving from completely-free to semi-free? Sure, some folks will. There are plenty of die-hard, opensource the world, free4all zealots out there that have never, nor will never pay for cable, satellite, broadband etc but rather "steal" their on-the-air TV or mooch their internet of neighbors with unsecured wifi access points. The majority of people however understand the way the world works. It is not free to stick a server on the internet (especially one that used 2.1875 TB of bandwidth in February), and if they can pay us a few bucks now and then they will.

Sorry, but you have obviously not understood the concept of open source.

What I don't understand is that applications that used the api were at some point supposed to offer a mirroring server in return (iirc?). So if there are large (and/or commercial) applications using the api, why not ask them to provide (or let their community provide) mirroring solutions?
[/quote]
How do you figure? No-one is saying the site will not remain open source, it most definitely will. There is however a big difference between open source and being free. You are welcome to take our code and database, modify it to your needs and setup your own TheTVDB; that's what open source is all about - providing services for free [or not] is not part of that equation.

My point above was simply that the open-source morons (note: I'm not saying all pro open-source people are morons - they are a minority) see open-source as being equal to free (as in no charge, and if you try to levy a fee for use, support or continued development of your open-source application you should go die).

As for commerical applications providing a mirror in return - considering there hasn't been a mirroring system available ... ever ... that's obviously not been possible.


So what other alternatives do you have? Would you like to offer up a server as a full mirror for free? Would you like to donate some money on a regular basis (if the answer is yes, why don't you - there's a donate button on the site)? Would you like to see the site be slower for end-users but remain free for all?

To be honest, I don't see what the fuzz is about - the services provided by the site will not change from before, it will simply provide more [expensive (in terms of CPU/RAM/BW)] services for those willing to pay for it. Can you honestly say you, personally, would rather pay $50.00 to allow everyone access to these services, rather than paying $10.00 and only allowing access to those that have paid for them? Also, like I pointed out above - the list I presented was in no way or form a definite list of what may or may not incur a charge in the future.

DieBagger
Posts: 84
Joined: Sat Oct 27, 2007 9:17 pm

Re: Restructure/Redesign

Postby DieBagger » Tue Mar 03, 2009 10:47 am

emigrating wrote:How do you figure? No-one is saying the site will not remain open source, it most definitely will. There is however a big difference between open source and being free. You are welcome to take our code and database, modify it to your needs and setup your own TheTVDB; that's what open source is all about - providing services for free [or not] is not part of that equation.

My point above was simply that the open-source morons (note: I'm not saying all pro open-source people are morons - they are a minority) see open-source as being equal to free (as in no charge, and if you try to levy a fee for use, support or continued development of your open-source application you should go die).

you're completely right, open source != free, but that's not what you implied in your last post!

emigrating wrote:As for commerical applications providing a mirror in return - considering there hasn't been a mirroring system available ... ever ... that's obviously not been possible.

Is this true? I thought mirroring was already in place for images? If not, why not implement this if the site is undergoing a complete redesign anyways?

emigrating wrote:So what other alternatives do you have? Would you like to offer up a server as a full mirror for free? Would you like to donate some money on a regular basis (if the answer is yes, why don't you - there's a donate button on the site)? Would you like to see the site be slower for end-users but remain free for all?

The decision of an individual to donate or not is non of your business quite frankly. Could any of the Mods give an estimated numbers on how much the costs of maintaining the site are in reality? I'm just surprised that the income through the adds is so little that it can't even support a single root server (I'm guessing 50 - 100 €).

And finally, could we please calm down the discussion a bit before this drifts into a flame war? No reason to get all aggressive over this (at least not for me)!

emigrating
Site Admin
Posts: 278
Joined: Mon Aug 27, 2007 4:38 pm
Contact:

Re: Restructure/Redesign

Postby emigrating » Tue Mar 03, 2009 11:07 am

DieBagger wrote:you're completely right, open source != free, but that's not what you implied in your last post!

No, what I implied was that the people who would complain or get "mad" that parts of the site was now considered a premium service you had to pay for was those that had completely misunderstood the concept of open-source.

DieBagger wrote:Is this true? I thought mirroring was already in place for images? If not, why not implement this if the site is undergoing a complete redesign anyways?

Correct. We quickly setup a mirror for images (and zip files) late last year as the site's traffic was congesting the 6Mbps pipe it was sitting on - the site in turn became unavailable for extended periods of time. Obviously, an actual system for mirrors will be implemented as part of the restructure - but as it stands today, there is no platform to build on for this. The current implementation also eats CPU cycles like hell, so it's not viable to add more mirrors at the moment.

DieBagger wrote:
emigrating wrote:So what other alternatives do you have? Would you like to offer up a server as a full mirror for free? Would you like to donate some money on a regular basis (if the answer is yes, why don't you - there's a donate button on the site)? Would you like to see the site be slower for end-users but remain free for all?

The decision of an individual to donate or not is non of your business quite frankly. Could any of the Mods give an estimated numbers on how much the costs of maintaining the site are in reality? I'm just surprised that the income through the adds is so little that it can't even support a single root server (I'm guessing 50 - 100 €).

And finally, could we please calm down the discussion a bit before this drifts into a flame war? No reason to get all aggressive over this (at least not for me)!

Ok, so as a site admin it's non of my business if a user has donated or not? That makes a lot of sense. Geez.

As for the costs running the site; dare I say that I (or crzykidd) am probably in a better situation to answer this than szsori or coco? Since TheTVDB was given the boot from their previous shared host more than a year ago it has been hosted for free, on free hardware running of free electricity - so "the costs of maintaining the site are in reality" $0.00. That's really not the point though, as continuing to provide free hardware, bandwidth and electricity is unviable in the long run.

You are of course correct, there is some revenue from adverts and donations, but not nearly enough to have purchased the hardware the site is currently sitting on, nor paying for rackspace/bandwidth costs per month. The point is; if I power off the server today it won't be back online tomorrow or the next day. Of course it would eventually come back up, but I'm fairly certain it would take some time to find a host providing a [powerful enough] dedicated server and over 2TB monthly bandwidth for what's currently in TheTVDB's coffers. Just look at themoviedb that was started a year or two ago, when the plug was pulled on that site it was down and unavailable for a very long time.

And finally, I'm not flaming - I was asking for further suggestions as to how to make the site pay for itself considering you are dead set against some minor features of the site becoming "subscription" based. Sure I could have asked in a nicer manner, but considering your initial response I saw no need for niceties.