New site development update - Jan 2010

Announcements about the database, website, and plugins.
ipartola
Posts: 1
Joined: Sat Oct 17, 2009 10:57 pm

Re: New site development update - Jan 2010

Postby ipartola » Fri Jan 29, 2010 1:37 pm

Changing the episode ids will be a problem for me also. I'd be happy to convert them using some [temporary] interface, but changing them outright will break my application which links episode ids with DVD hashes.

kevint11
Posts: 4
Joined: Sun Jan 03, 2010 12:28 am

Re: New site development update - Jan 2010

Postby kevint11 » Tue Feb 02, 2010 6:18 pm

I totally understand your guys points with changing the episodeID, it would really make conversion a lot easier for you.

This change tho would break my application as well, all of my users have user data that is linked to the episodeID. The best would be to keep episodeID the same ... if this isn't possible some other way of linking the old ID to the new one would be fine for the transition.

Thanks for all your guys great work!

adama
Posts: 4
Joined: Sat Feb 06, 2010 11:33 pm
Location: Northumberland, UK
Contact:

Re: New site development update - Jan 2010

Postby adama » Sun Feb 07, 2010 1:32 am

Changing the episode_id breaks the number one rule of the web, locations and identifiers must never change.

For a lot of people (i'd guess most?) it'd mean dumping almost all of the currently used data and repolling it all, not to mention all those people who don't know about the ID change, and get stuck with a lot of incorrect and/or incomplete data, possible even ruining files if things are being auto-renamed?

Consider this practice for the real world, where you'd get fired if you tried to change a whole load of active data just to make your life a bit easier! ;)

hikaricore
Gaius Baltar
Posts: 5970
Joined: Tue Apr 28, 2009 11:28 am

Re: New site development update - Jan 2010

Postby hikaricore » Sun Feb 07, 2010 2:37 am

adama wrote:Changing the episode_id breaks the number one rule of the web, locations and identifiers must never change.

For a lot of people (i'd guess most?) it'd mean dumping almost all of the currently used data and repolling it all, not to mention all those people who don't know about the ID change, and get stuck with a lot of incorrect and/or incomplete data, possible even ruining files if things are being auto-renamed?

Consider this practice for the real world, where you'd get fired if you tried to change a whole load of active data just to make your life a bit easier! ;)


Could you please provide a link to these "rules of the web"? Because it kinda sounds like you're talking out of your ass.

adama
Posts: 4
Joined: Sat Feb 06, 2010 11:33 pm
Location: Northumberland, UK
Contact:

Re: New site development update - Jan 2010

Postby adama » Sun Feb 07, 2010 7:28 am

http://www.w3.org/Provider/Style/URI

HAND.

Incidentally, having read some of your other thread replies here, you seem to be to be a bit of a dick. I do hope you're not representative of thetvdb people in general, because that truly would be sad.

szsori
Site Admin
Posts: 1911
Joined: Fri Nov 03, 2006 5:23 pm

Re: New site development update - Jan 2010

Postby szsori » Sun Feb 07, 2010 10:37 am

adama wrote:Changing the episode_id breaks the number one rule of the web, locations and identifiers must never change.

For a lot of people (i'd guess most?) it'd mean dumping almost all of the currently used data and repolling it all, not to mention all those people who don't know about the ID change, and get stuck with a lot of incorrect and/or incomplete data, possible even ruining files if things are being auto-renamed?

Consider this practice for the real world, where you'd get fired if you tried to change a whole load of active data just to make your life a bit easier! ;)


1. My boss is nice. If it would make the update cost effective he'd say to go ahead with it.
2. Locations and identifiers never changing is a "rule" for a completely different reason. We're not talking about normal web pages here. We're talking about a dynamic API with dynamic information.
3. I will be making it possible for devs to do a lookup using the old id and convert it to the new one. However, maintaining the old id as the pk throughout the transfer means it takes up to 20 times longer for the transfer. It's not just about convenience for me at that point... it's about making the transfer itself go smoothly with little downtime.
4. Don't insult the moderators. If you do it again, you won't be welcome here. They've earned a right to be cranky after sifting through countless threads created by people with absolutely no clue and are expected to work magic for those people. This site is free and the moderators (and developers) donate their time for free.

adama
Posts: 4
Joined: Sat Feb 06, 2010 11:33 pm
Location: Northumberland, UK
Contact:

Re: New site development update - Jan 2010

Postby adama » Sun Feb 07, 2010 3:17 pm

No, they haven't. You shouldn't have people who respond the way I've seen him respond to people as moderators. It just makes you *all* look incompetent. Are you really saying the correct way for you to treat your contributors (not just users, but the people who *create* your content) is to tell them that "basically you can suck it"?

I really don't understand /why/ you need to change the episode id, surely if you're just migrating it all to another database you just insert the records with the id, even if it is auto increment.

szsori
Site Admin
Posts: 1911
Joined: Fri Nov 03, 2006 5:23 pm

Re: New site development update - Jan 2010

Postby szsori » Sun Feb 07, 2010 3:40 pm

adama wrote:No, they haven't. You shouldn't have people who respond the way I've seen him respond to people as moderators. It just makes you *all* look incompetent. Are you really saying the correct way for you to treat your contributors (not just users, but the people who *create* your content) is to tell them that "basically you can suck it"?


Each of our moderators contributes an hour or two daily keeping things in order. Due to our tools being absolute crap right now, they have to jump through hoops to get anything done. They deal with people that get confrontational, people that are ignorant, and people that make the same mistakes over and over again. While I hope they're polite with our users, everyone still has bad days and forum posts will reflect this. In any case, I'll ALWAYS take the side of someone who has put hundreds of hours into this site over someone who has put one or two. That's just the way it goes. No matter what, it's not YOUR place to call them out on inappropriate responses, that's our place.

adama wrote:I really don't understand /why/ you need to change the episode id, surely if you're just migrating it all to another database you just insert the records with the id, even if it is auto increment.


Exactly. You don't understand. Which is why you're continuing to argue this despite me explaining that I'm going to at least have functionality in place to allow devs to translate from the old ids to the new. Yet you won't take me at my word for it, so I'm forced to waste time explaining it. And people wonder why our mods are cranky.

The new database structure is significantly different than the current one. We've normalized a lot of the data and are actually doing things the right way this time through. One of the issues with normalizing the data is that certain steps rely on data from others to be in place first. Another part of it is that maintaining ids requires that we iterate through the records instead of just being able to use batch processes. If you've ever used cursors in SQL, you know how damn slow they are. These two issues combined means that we end up having to iterate records AND read records into temporary tables and data structures in order to preserve the ids. That means that we exceed the total amount of memory on my local system (at that point I can't do it on the server since a 20+ hour proc would crush the site) and ends up hitting my virtual memory (ie even slower). That causes the process to take upwards of 20 hours to run, and that was before I even had the entire process done. By not maintaining the ids I was able to rearrange the steps and do it all without temporary tables/structs. That takes the entire process below 20 minutes total. The benefit of that is when we're starting to switch the site we can have the old site update the new and move all API traffic before moving the main site over. Realistically that could happen as soon as we get our Windows server running and the API rewritten on the new site. That also allows us to test the load before flipping the switch.

So, in a nutshell, there are a number of factors that go into the decision that I've considered. I'm not just trying to make things more complex for devs. In fact, I'm specifically trying to make it easier by at least making a conversion API for them. I think you'd find from talking with most devs that we work hard to try making things easier on them. Things aren't perfect now, but we're trying to improve them.

This is all that will be said on the matter. I have better things to do with my time than argue on our forums.

joeserg
Posts: 6
Joined: Fri Jan 08, 2010 7:26 am

Re: New site development update - Jan 2010

Postby joeserg » Mon Feb 08, 2010 7:15 am

Just my two cents, if you need to change the episode IDs, so be it. Yes it will be a pain for some people, but hey, we're getting a free service. What's the alternative, the TVRage API? Been there, done that. Never again!

Keep up the good work guys, I can't wait to see the new site and API features.

Omertron
Posts: 9
Joined: Fri Mar 27, 2009 1:31 am

Re: New site development update - Jan 2010

Postby Omertron » Fri Feb 19, 2010 10:00 am

joeserg wrote:Just my two cents, if you need to change the episode IDs, so be it. Yes it will be a pain for some people, but hey, we're getting a free service. What's the alternative, the TVRage API? Been there, done that. Never again!

Keep up the good work guys, I can't wait to see the new site and API features.

agreed
And if there's an api to convert IDs how difficult is it to write an app to convert that user data over?