NOINDEX html meta tag prevents search engines indexing the pages

Discussion about the website and interfaces.
Report general bugs here.
Forum rules
DO NOT POST SERIES REQUESTS HERE.
Use the appropriate forum.
Requests posted here are subject to immediate deletion.
facan
Posts: 3
Joined: Fri Jun 29, 2018 11:20 pm

NOINDEX html meta tag prevents search engines indexing the pages

Postby facan » Mon Jul 02, 2018 2:08 pm

https://www.thetvdb.com/series/gravity-falls includes <meta name="robots" content="noindex"> preventing search engines from indexing your TV Pages. Expected?

DarklightIndigo
Just zis guy, you know?
Posts: 7645
Joined: Sun Apr 27, 2014 9:16 am
Location: San Francisco
Contact:

Re: NOINDEX html meta tag prevents search engines indexing the pages

Postby DarklightIndigo » Mon Jul 02, 2018 4:13 pm

I don't know that it's intended to block everyone, it's likely a first step to get people to use the API. I think maybe we could allow Google, Yahoo, Bing a periodic scan, but the staff is really trying to discourage that kind of screen scraping.
Strangers are just the friends we haven't met yet.

facan
Posts: 3
Joined: Fri Jun 29, 2018 11:20 pm

Re: NOINDEX html meta tag prevents search engines indexing the pages

Postby facan » Fri Jul 13, 2018 2:30 am

You can allow Bing (and so Yahoo) and Google by adding
<meta name="bingbot" content="index"/>
<meta name="gogolebot" content="index"/>
<meta name="robots" content="noindex"/>

Thanks,
Fabrice

facan
Posts: 3
Joined: Fri Jun 29, 2018 11:20 pm

Re: NOINDEX html meta tag prevents search engines indexing the pages

Postby facan » Wed Jul 25, 2018 12:56 am

Using the API, won't allow search engines to make your good quality content discoverable / linked in search results.
Can you allow?