Results 1 to 6 of 6

Thread: what is robots. txt file in site?

  1. #1
    shahid1 is offline Advance Member
    Last Online
    6th October 2023 @ 11:04 PM
    Join Date
    25 Jul 2007
    Age
    39
    Gender
    Male
    Posts
    2,218
    Threads
    386
    Credits
    186
    Thanked
    28

    Default what is robots. txt file in site?



    Kisi website main robots.txt file kya hoti hai, Aur ye kaise banai jati hai, aur iss ka kya role hota hai.
    Kindly agar kisi ko pata hai, to mujhe guide kar dain, mujhe is bare main urgent info chayie.

    Thanks in Advanced

  2. #2
    rajput21 is offline Senior Member+
    Last Online
    24th January 2016 @ 02:34 PM
    Join Date
    07 Jul 2009
    Age
    40
    Posts
    111
    Threads
    2
    Credits
    975
    Thanked
    3

    Default

    Robot.txt
    Robots are programs that automatically crawl the Web and retrieve documents. Web browsers like Internet Explorer or FireFox are operated by humans and don’t automatically retrieve text from referenced documents. Robots are are most often referred to as crawlers, bots, or spiders. These robots visit sites by requesting documents from them. Search engines like Google, Yahoo! and MSN Search employ robots to crawl web documents for the purposes of being indexed and provided as search engine results.

    Robots decide to visit a site based on a historical list of URLs, especially of documents with many links elsewhere. A directory or any web page that lists external links is a candidate for a robot visit. Most search engines allow you to submit URLs manually, which will then be queued and visited by the robot. Robots select URLs to visit and to parse as a source for new URLs. Most robots–benevolent robots–routinely check for a special file called “robots.txt” which can be installed by the server administrator of any web site. There may be reasons which a webmaster would want to exclude a robot from visiting his site. One very common reason is for exclusion is due to the large amount of bandwidth that robots eat up. A webmaster may also want the robot to exclude sensitive information or images or other files.

    Robots.txt Exclusion

    To prevent robots visiting your site put these two lines into the /robots.txt file that lives in the root directory of the server:

    User-agent: *
    Disallow: /


    But rarely does a webmaster want to exclude robots from visiting an entire site. Webmasters can write a structured text file instructing robots to stay away from certain areas of the server. Webmasters can even choose which robots to allow or disallow. Below is an example of how an exclusion may be written inside a robots.txt file:

    # /robots.txt file for http://www.google.com

    User-agent: Googlebot
    Disallow:

    User-agent: sillycrawler
    Disallow: /

    User-agent: *
    Disallow: /tmp
    Disallow: /cgi-bin


    The first two lines, starting with ‘#’, specify a comment

    The first example specifies that the robot called “Googlebot” is allowed to go anywhere.

    The second example indicates that the robot called “sillycrawler” has all relative URLs starting with ‘/’ disallowed. Because all relative URL’s on a server start with ‘/’, this means the entire site is disallowed.

    The third example indicates that all robots should not visit URLs starting with /tmp or /cgi-bin. The “*” is a special token that refers to “any other User-agent”; wildcard patterns or regular expressions cannot be used in either User-agent or Disallow lines.

    There are scripts which helps creating robot.txt called robots.txt generator
    Here's one http://tools.seobook.com/robots-txt/generator/


    [Source: clickfire.com]

  3. #3
    shahid1 is offline Advance Member
    Last Online
    6th October 2023 @ 11:04 PM
    Join Date
    25 Jul 2007
    Age
    39
    Gender
    Male
    Posts
    2,218
    Threads
    386
    Credits
    186
    Thanked
    28

    Default

    thanxx
    kya koi urdu main bata sakta hai???

  4. #4
    Join Date
    08 Feb 2009
    Location
    Abbottabad
    Posts
    291
    Threads
    23
    Thanked
    4

    Default

    simple hai ager aap aap apni web site k kuch content means kuch file

    e.g. page4.htm , page 10.php etc

    search engine mai display na kerna chahin tou aap

    robots.txt mai woh add ker lain

    lakin ager aap apnay saaray site k contents search engine mai add kerna chahin tou

    aap sitemap generate kerwa lain

    us mai saaray aap k contents search engine mai submit hoo jain gay.

  5. #5
    shahid1 is offline Advance Member
    Last Online
    6th October 2023 @ 11:04 PM
    Join Date
    25 Jul 2007
    Age
    39
    Gender
    Male
    Posts
    2,218
    Threads
    386
    Credits
    186
    Thanked
    28

    Default

    Quote hallianonline said: View Post
    simple hai ager aap aap apni web site k kuch content means kuch file

    e.g. page4.htm , page 10.php etc

    search engine mai display na kerna chahin tou aap

    robots.txt mai woh add ker lain

    lakin ager aap apnay saaray site k contents search engine mai add kerna chahin tou

    aap sitemap generate kerwa lain

    us mai saaray aap k contents search engine mai submit hoo jain gay.
    Thanks for guide.
    Brother aap example batao ge, ke robot.txt file bana kar us main kiss tarha wo pages rakhain, jo ke ham google main show nahi karwana chate?
    kya un pages ka link rakhna hota hai?

  6. #6
    Join Date
    08 Feb 2009
    Location
    Abbottabad
    Posts
    291
    Threads
    23
    Credits
    0
    Thanked
    4

    Default

    aap google pay online robots.txt generator search karian

    kisi bhi web site say aap robots.txt generate ker saktay hain

    aur jin pages ko aap search engine mai display nai kerna chatay

    unhain aap robots.txt mai rehnay dain aur baki remove ker dain

Similar Threads

  1. robots marathon. . .
    By vakas89 in forum General Knowledge
    Replies: 1
    Last Post: 4th March 2011, 03:35 PM
  2. Pix-22-10 Dus Mush-hoor Robots
    By hafizbond in forum Photo Gallery
    Replies: 8
    Last Post: 18th August 2009, 02:13 AM
  3. Military Robots
    By AliDaMalang in forum Photo Gallery
    Replies: 6
    Last Post: 11th July 2009, 09:06 PM

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •