Regenerating XML... Again

Announcements about the database, website, and plugins.
szsori
Site Admin
Posts: 1911
Joined: Fri Nov 03, 2006 5:23 pm

Regenerating XML... Again

Postby szsori » Wed Nov 18, 2009 1:44 pm

After reading some of the complaints about the XML data, I decided to look into it more in depth. It appeared people were right... parts were missing from the regen. I looked into it a bit and saw the episode info wasn't being generated. Unfortunately I forgot that I had made episodes fully dynamic this past summer. Anyway, I rewrote that part of code (sigh) and restarted the regen before realizing what really needed to be done.

It seems like during our server move we lost a .htaccess file with rewrite information for episode data. I rewrote that and it seems to be working fine. Unfortunately, the script needed a bit of modification to get it to match the rewrite, so I would have had to regen all over anyway.

In any case, I've tested a couple of series and they've worked perfectly this time. The number on the reports screen is accurate and once it reaches 0 people should be able to fully use the API. In the meantime series will gradually become available on the API, with the series being updated most recently getting generated first.

Sorry for all the issues we've faced on this. On the new site the API will be fully dynamic and just cached for a few minutes, which should avoid this entire problem set as well as a bunch of others.

slords
Posts: 1
Joined: Wed Nov 18, 2009 2:54 pm

Re: Regenerating XML... Again

Postby slords » Wed Nov 18, 2009 2:56 pm

I was just trying to scrape some information and was getting an error on House S06E05. Looking deeper I noticed that S06E06 was receiving the description for E05.

http://thetvdb.com/api/<apikey>/series/73255/default/6/6/en.xml

actually is returning the information for episode 5 and passing /6/5/ returns 404.

Jaerin
Posts: 2
Joined: Mon Nov 16, 2009 7:26 pm

Re: Regenerating XML... Again

Postby Jaerin » Wed Nov 18, 2009 4:15 pm

slords wrote:I was just trying to scrape some information and was getting an error on House S06E05. Looking deeper I noticed that S06E06 was receiving the description for E05.

http://thetvdb.com/api/<apikey>/series/73255/default/6/6/en.xml

actually is returning the information for episode 5 and passing /6/5/ returns 404.



Take a look at this thread:

viewtopic.php?f=7&t=1893&start=20&hilit=house

I'm guessing since they had to regenerate the files they may need to "fix" it again.

crzykidd
Site Admin
Posts: 418
Joined: Sat Feb 16, 2008 4:48 pm

Re: Regenerating XML... Again

Postby crzykidd » Wed Nov 18, 2009 6:26 pm

All the static files are generated from the data in the DB. When an episode changes the information gets updated in the DB, and then within 5 minutes the static xml files for that episode/series get updated. The problem you are encountoring is most likely related to a corrupt xml file. Doing the entire regen should fix it unless the DB is incorrect. It will take some time for the regen to happen. Check again tomorrow and see if the problem still exists.

Thanks,
Crzy
thetvdb.com Site Admin

liquidox
Posts: 15
Joined: Fri Nov 06, 2009 5:24 am

Re: Regenerating XML... Again

Postby liquidox » Thu Nov 19, 2009 4:35 am

Currently getting this error from the Jamu scraper:

Code: Select all

Traceback (most recent call last):
  File "/usr/share/mythtv/mythvideo/scripts/jamu.py", line 6344, in <module>
    main()
  File "/usr/share/mythtv/mythvideo/scripts/jamu.py", line 6329, in main
    process.processMythTvMetaData()
  File "/usr/share/mythtv/mythvideo/scripts/jamu.py", line 6045, in processMythTvMetaData
    if not len(self.config['localpaths']['posters']):
KeyError: 'posters'

morphiousonline
Posts: 4
Joined: Thu Nov 19, 2009 4:22 am

Re: Regenerating XML... Again

Postby morphiousonline » Thu Nov 19, 2009 4:37 am

Hi, I've gotta thank you for the full on work being done to get the system back up, I'd just like to give an update on the success from the user end. I'm using XBMC on an old xbox, Since the last fix the scaper is now working, loading episode series/name/plot correctly, however it is not bringing through the banner picture or episode images. Even when a manual selection is attempted the banner pictures are not listed. I know for almost of the series i'm scraping that banner pictures are available (I recently wiped my library) and i'm really hoping its part of the ongoing repairs you are doing and not my xbox on the fritz.

szsori
Site Admin
Posts: 1911
Joined: Fri Nov 03, 2006 5:23 pm

Re: Regenerating XML... Again

Postby szsori » Thu Nov 19, 2009 9:40 am

morphiousonline wrote:Hi, I've gotta thank you for the full on work being done to get the system back up, I'd just like to give an update on the success from the user end. I'm using XBMC on an old xbox, Since the last fix the scaper is now working, loading episode series/name/plot correctly, however it is not bringing through the banner picture or episode images. Even when a manual selection is attempted the banner pictures are not listed. I know for almost of the series i'm scraping that banner pictures are available (I recently wiped my library) and i'm really hoping its part of the ongoing repairs you are doing and not my xbox on the fritz.


Do you have the latest scraper for XBMC? The one in their stable release is outdated... it lists images.thetvdb.com. It should list www.thetvdb.com instead. There are threads here and on the XBMC forums about it.

ChaosCharza
Posts: 3
Joined: Tue Nov 17, 2009 2:58 pm

Re: Regenerating XML... Again

Postby ChaosCharza » Thu Nov 19, 2009 10:05 am

Uh, is a fluctuation of 7 shows within the update signifigant? About a half hour ago it was @ 153. As of this writing it's... 160.

Does it update itself if extra things are added when it's regenerating? I'm just confirming if there's a problem with the script.


Oh, and iDentify's been working like a dream w/ all my shows since yesterday. Just thought people would like to know.

szsori
Site Admin
Posts: 1911
Joined: Fri Nov 03, 2006 5:23 pm

Re: Regenerating XML... Again

Postby szsori » Thu Nov 19, 2009 10:34 am

There's not a problem. It completed and now I've added it back into crontab so it'll run every 10 minutes. It'll get through those remaining 170 and then stay up to date every 10 minutes thereafter.

morphiousonline
Posts: 4
Joined: Thu Nov 19, 2009 4:22 am

Re: Regenerating XML... Again

Postby morphiousonline » Thu Nov 19, 2009 10:47 am

szsori wrote:Do you have the latest scraper for XBMC? The one in their stable release is outdated... it lists images.thetvdb.com. It should list http://www.thetvdb.com instead. There are threads here and on the XBMC forums about it.


the scraper i've got in XMBC was working perfectly last week, automatically downloading banners and episode thumbnails. But since the down time it only receives the plot and nameing information, no images and non of the wide banner images are available when trying to manually select them.