Scraping issues

Discussion about the website and interfaces.
Report general bugs here.
Forum rules
DO NOT POST SERIES REQUESTS HERE.
Use the appropriate forum.
Requests posted here are subject to immediate deletion.
bluesquirrel
Posts: 307
Joined: Thu Jan 19, 2017 5:44 pm
Location: Australia

Thu Jan 23, 2020 4:58 pm

User avatar
mmhere
Posts: 57
Joined: Sun May 29, 2016 5:47 pm

Thu Jan 23, 2020 6:41 pm

The problem seems to be that something change at thetvdb again with older APIs so now Kodi (16.1) cannot parse at all. Some are reporting JSON is returned when XML is expected but I cannot confirm this yet. The trouble with the Kodi forum link presented at https://forum.kodi.tv/showthread.php?ti ... pid2918674 (and linked in the previous post by bluesquirrel here) is that this only works with newer versions of Kodi.

Is thetvdb API supporting legacy APIs or not? Why does the info returned keep changing?
User avatar
mmhere
Posts: 57
Joined: Sun May 29, 2016 5:47 pm

Thu Jan 23, 2020 6:42 pm

P.S.: "Update to later Kodi versions" is not an easy option. The media box has had the same version of Kodi for quite some time and would need to be replaced with a new box.
User avatar
ExTechOp
Posts: 26
Joined: Tue Oct 30, 2018 1:07 am
Location: Helsinki, Finland

Thu Jan 23, 2020 10:14 pm

bluesquirrel wrote:
Thu Jan 23, 2020 4:58 pm
Please read here for Kodi fix... https://forum.kodi.tv/showthread.php?ti ... pid2918674
Updating to TheTVDB 3.2.4 scraper (on OSMC running Kodi 18.5) fixed the problem with finding "Star Trek: Picard", but my original issue with "The Life-Sized City" (see the start of this chain) unfortunately still seems to persist :(
User avatar
mmhere
Posts: 57
Joined: Sun May 29, 2016 5:47 pm

Thu Jan 23, 2020 11:42 pm

Another one that fails to load for me is "The Dead Lands (2020)". No issues there with unusual characters such as ":" in the series name (which some have suspected as being problematic for Picard).

The data simply isn't getting through to Kodi like it used to yesterday.

Are the legacy APIs and their data formats supported or not?
bluesquirrel
Posts: 307
Joined: Thu Jan 19, 2017 5:44 pm
Location: Australia

Fri Jan 24, 2020 12:16 am

There is no use reporting shows that don't scrape if you are still using the bugged version of the scraper.

A scraper update for v16 and earlier has not yet been released. Hopefully it will soon.

As you are posting on this forum, it is safe to assume you have a computer like Windows. Why don't you install Kodi on your computer, scrape using the updated version of the scrapers, export to separate files to create nfo files then scan them into your Android box.
bluesquirrel
Posts: 307
Joined: Thu Jan 19, 2017 5:44 pm
Location: Australia

Fri Jan 24, 2020 2:31 am

A scraper update for Kodi v13-v16 has been merged and will be available shortly.... https://github.com/xbmc/repo-scrapers/pull/173

This is untested as none of the developers are using old versions anymore. Please report any scraper issues at Kodi forum.
User avatar
mmhere
Posts: 57
Joined: Sun May 29, 2016 5:47 pm

Fri Jan 24, 2020 11:10 am

bluesquirrel wrote:
Fri Jan 24, 2020 2:31 am
A scraper update for Kodi v13-v16 has been merged and will be available shortly.... https://github.com/xbmc/repo-scrapers/pull/173
Thanks very much for the scraper update even though it is for older versions. I'll also post many powerful ThankYous in the Kodi forum. Will test once it propagates.
User avatar
mmhere
Posts: 57
Joined: Sun May 29, 2016 5:47 pm

Fri Jan 24, 2020 12:14 pm

mmhere wrote:
Thu Jan 23, 2020 11:42 pm
... "The Dead Lands (2020)". No issues there with unusual characters such as ":" in the series name. ...

The data simply isn't getting through to Kodi like it used to yesterday.

Are the legacy APIs and their data formats supported or not?
I'd like to underscore this point, aimed at thetvdb API maintainers (not the Kodi scraper maintainers, who have graciously continued to provide new code to adapt to the ever changing thetvdb-API situation).

Why would the legacy APIs suddenly switch without notice to returning incompatible data? Either the JSON they are presently providing changed so as to break unchanged parsers, or as others have suggested, the data used to be XML and is now JSON? Why did this change if this is a legacy API?

Is the legacy API supported or not?

Either way, Kodi does get a data response from the API then records to its log that it could not parse. To wit--

from ~/.kodi/temp/kodi.log:

Code: Select all

09:43:03 T:546013442272   ERROR: Run: Unable to parse web site
Data that failed to parse can be found here:

Code: Select all

# cd ~/.kodi/temp/scrapers/metadata.tvdb.com

# ls -l search*
-rw-r--r--  1  root  root  520  Jan 24 09:43  search-The%20Dead%20Lands-en.json

# cat 'search-The%20Dead%20Lands-en.json' | python -m json.tool
and looks like this:

Code: Select all

{ "data": [ {
    "aliases": [],
    "banner": "/banners/62060651.jpg",
    "firstAired": "2020-1-23",
    "id": 373425,
    "image": "/banners/series/373425/posters/62019129.jpg",
    "network": "Shudder",
    "overview": "Waka, a murdered Maori warrior returned from the Afterlife, and Mehe, a determined young woman, embark on a quest to find who 'broke the world' and how to close the breach between the living and the dead. ",
    "poster": "/banners/series/373425/posters/62019129.jpg",
    "seriesName": "The Dead Lands",
    "slug": "the-dead-lands",
    "status": "Upcoming"
    }
  ]
}
I want to try to understand whether the intent is to support legacy APIs or deprecate them and eventually disappear them. If supporting legacy formats is intended, then existing appliances will [should be able to] operate as they used to. Otherwise such appliances require being replaced.

Replacing such boxen is certainly possible but it would be useful to know ahead of time that this is necessary before the legacy APIs change and/or go away. It is not an insigificant project to replace appliances that otherwise just sit there unchanged doing their thing because the network data format continues to be supported. If it isn't going to be supported, that's fine. Just let us know and give us some lead time.

I'm glad my toaster firmware does not stop the toaster making toast just because its network data connection changes. :D
User avatar
mmhere
Posts: 57
Joined: Sun May 29, 2016 5:47 pm

Fri Jan 24, 2020 4:20 pm

bluesquirrel wrote:
Fri Jan 24, 2020 2:31 am
A scraper update for Kodi v13-v16 has been merged and will be available shortly.... https://github.com/xbmc/repo-scrapers/pull/173

This is untested as none of the developers are using old versions anymore. Please report any scraper issues at Kodi forum.
Thanks to the Kodi scraper maintainers. I was able to grab the metadata.tvdb.com-2.1.4.zip package and update using that; I got it here (Kodi "Jarvis" install, version 16.1): http://mirrors.kodi.tv/addons/jarvis/metadata.tvdb.com/

I can confirm that once again new shows can be added and episodes scraped.

It hadn't yet propagated through the automatic updates/channels, so I made the zip file available and did "Install from zip file" within the Kodi GUI.

Thank You!

For various versions of Kodi, these URLs should have it:

v13: http://mirrors.kodi.tv/addons/gotham/metadata.tvdb.com/
v14: http://mirrors.kodi.tv/addons/helix/metadata.tvdb.com/
v15: http://mirrors.kodi.tv/addons/krypton/m ... .tvdb.com/
v16: http://mirrors.kodi.tv/addons/jarvis/metadata.tvdb.com/

This overall post at forums.kodi.tv may also help:
https://forum.kodi.tv/showthread.php?ti ... id=2918674
Post Reply