Table of Contents - robotx-0.1.1 Documentation
Classes and Modules
Methods
- ::new — Robotx
- #allow_regex — Robotx
- #allowed — Robotx
- #allowed? — Robotx
- #check_permission — Robotx
- #crawl_delay — Robotx
- #disallow_regex — Robotx
- #disallowed — Robotx
- #load_robots_txt — Robotx
- #parse_robots_txt — Robotx
- #regex_value — Robotx
- #sitemap — Robotx
- #strip_slashes — Robotx
- #user_agents — Robotx