[googlealchemist]

Included the full robots file below for reference and in case something else stuck out that I'm not specifically asking about. Trying to figure out if this robots file is a good one currently or if it should be tweaked, or if i should even have one at all!

The bits and pieces that stuck out I'll comment on here:

/redirect.php*
why would we want something like this? i had this on two sites that i use for some 301 for some test projects. i checked the 301 with a header addon for ff and they do say they seem to work fine and are indeed a 301. i guess robots only look for urls with that string of text specifically in the url? my redirects are more like domain.com/blahblah.php so /redirect.php doesnt actually come up...

Disallow: /category/

I'm torn on cats as well as tags, on the one hand i worry about the various pages duplicating content across my site, on the other hand i worry about nav issues for seo reasons, at least with the cats...and worry about internal linking missed opportunities with the tags

This isnt just a q for a robots file, ive also been using the duplicate content cure plugin for wp on my sites which has a main description saying it stops onsite problems by preventing spiders from going to your cat/tag/etc pages. I also use a plugin that works via siloing the cats, BUT it also places a cat link at the bottom of the pages to nav thru, so i dunno if that is contradicting or not....

Disallow: /page/

Similar q as for the redirects...robots works by what is in the actual url right? would this be to cover internal tag pages or what?

Disallow: /blog-/

wtf?

I've had a few similar posts elsewhere, especially at blf, and might have even made one here, and even though its archived I cant seem to find the right posts as to check the answers so apologies for redundancies

robots file aside,the more clients i pick up the more funky issues im seeing with internal structure. im having current headaches with an opencart based site and i look an internal look at another site that is the general sort of site with the 40+ nav links on the bottom going to different models etc as its a physical product being shipped type of site...bout started to cry haha! what a mess!

most sites im getting great results with regardless but the few that are more of a mess are causing me ranking grief. and either way, i have a strong pet peeve about wasting time/resources so anything causing me to swim against the current and spend more of my time and money to rank for the current goals, and move on to bigger/better ones....really pisses me off.

appreciate any insight on any of the above issues

thanks all

User-agent:  *
# disallow all files in these directories
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/
Disallow: /go/
Disallow: /archives/
disallow: /*?*
Disallow: /wp-*
Disallow: /author
Disallow: /comments/feed/
Disallow: /page/
Disallow: /trackback/
Disallow: /wp-/
Disallow: /blog-/
Disallow: /?s=
Disallow: /cdn-/
Disallow: /xmlrpc.php
Disallow: /comment/reply/
Disallow: /comment/
Disallow: /useronline/
Disallow: /redirect.php*
Disallow: /wp-content/themes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /category/
Disallow: /date/
Disallow: /tag/
Disallow: /author/
User-agent: Mediapartners-Google*
Allow: /
User-agent: Googlebot-Image
Allow: /wp-content/uploads/
User-agent: Adsbot-Google
Allow: /
User-agent: Googlebot-Mobile
Allow: /
Sitemap: http://www.domain.org/sitemap.xml

[Fishingman1]

Seems awfully redundant to me.
My robots text is very simple.
When I need to find robots.text I just look at my competitors.
Make changes necessary , if it works for them ?