Package org.apache.manifoldcf.crawler.connectors.rss

Class Robots.Record

  • Enclosing class:
    Robots
    protected static class Robots.Record
extends java.lang.Object
    This class represents a record in a robots.txt file. It contains one or more user-agents, and one or more disallows.

    • Field Summary

      Fields 
      Modifier and Type Field Description
      protected java.util.ArrayList allows  
      protected java.util.ArrayList disallows  
      protected java.util.ArrayList userAgents  

    • Constructor Summary

      Constructors 
      Constructor Description
      Record()
      Constructor.

    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      void addAgent​(java.lang.String agentName)
      Add a user-agent.
      void addAllow​(java.lang.String allowPath)
      Add an allow.
      void addDisallow​(java.lang.String disallowPath)
      Add a disallow.
      boolean isAgentMatch​(java.lang.String agentNameUpper, boolean exactMatch)
      See if user-agent matches.
      boolean isAllowed​(java.lang.String path)
      See if path is allowed.
      boolean isDisallowed​(java.lang.String path)
      See if path is disallowed.

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    • Field Detail

      • userAgents

        protected java.util.ArrayList userAgents

      • disallows

        protected java.util.ArrayList disallows

      • allows

        protected java.util.ArrayList allows

    • Constructor Detail

      • Record

        public Record()
        Constructor.

    • Method Detail

      • addAgent

        public void addAgent​(java.lang.String agentName)
        Add a user-agent.

      • addDisallow

        public void addDisallow​(java.lang.String disallowPath)
        Add a disallow.

      • addAllow

        public void addAllow​(java.lang.String allowPath)
        Add an allow.

      • isAgentMatch

        public boolean isAgentMatch​(java.lang.String agentNameUpper,
                            boolean exactMatch)
        See if user-agent matches.

      • isDisallowed

        public boolean isDisallowed​(java.lang.String path)
        See if path is disallowed. Only called if user-agent has already matched. (This checks if there's an explicit match with one of the Disallows clauses.)

      • isAllowed

        public boolean isAllowed​(java.lang.String path)
        See if path is allowed. Only called if user-agent has already matched. (This checks if there's an explicit match with one of the Allows clauses).