Class QuorumCnxManager


  • public class QuorumCnxManager
    extends java.lang.Object
    This class implements a connection manager for leader election using TCP. It maintains one connection for every pair of servers. The tricky part is to guarantee that there is exactly one connection for every pair of servers that are operating correctly and that can communicate over the network. If two servers try to start a connection concurrently, then the connection manager uses a very simple tie-breaking mechanism to decide which connection to drop based on the IP addressed of the two parties. For every peer, the manager maintains a queue of messages to send. If the connection to any particular peer drops, then the sender thread puts the message back on the list. As this implementation currently uses a queue implementation to maintain messages to send to another peer, we add the message to the tail of the queue, thus changing the order of messages. Although this is not a problem for the leader election, it could be a problem when consolidating peer communication. This is to be verified, though.
    • Constructor Summary

      Constructors 
      Constructor Description
      QuorumCnxManager​(long mySid, java.util.Map<java.lang.Long,​QuorumPeer.QuorumServer> view, QuorumAuthServer authServer, QuorumAuthLearner authLearner, int socketTimeout, boolean listenOnAllIPs, int quorumCnxnThreadsSize, boolean quorumSaslAuthEnabled)  
      QuorumCnxManager​(long mySid, java.util.Map<java.lang.Long,​QuorumPeer.QuorumServer> view, QuorumAuthServer authServer, QuorumAuthLearner authLearner, int socketTimeout, boolean listenOnAllIPs, int quorumCnxnThreadsSize, boolean quorumSaslAuthEnabled, java.util.concurrent.ConcurrentHashMap<java.lang.Long,​org.apache.zookeeper.server.quorum.QuorumCnxManager.SendWorker> senderWorkerMap)  
    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      void addToRecvQueue​(QuorumCnxManager.Message msg)
      Inserts an element in the recvQueue.
      void connectAll()
      Try to establish a connection with each server if one doesn't exist.
      boolean connectedToPeer​(long peerSid)  
      void connectOne​(long sid)
      Try to establish a connection to server with id sid.
      long getConnectionThreadCount()
      Return number of connection processing threads.
      long getThreadCount()
      Return number of worker threads
      void halt()
      Flag that it is time to wrap up all activities and interrupt the listener.
      void initiateConnection​(java.net.Socket sock, java.lang.Long sid)
      If this server has initiated the connection, then it gives up on the connection if it loses challenge.
      void initiateConnectionAsync​(java.net.Socket sock, java.lang.Long sid)
      Server will initiate the connection request to its peer server asynchronously via separate connection thread.
      QuorumCnxManager.Message pollRecvQueue​(long timeout, java.util.concurrent.TimeUnit unit)
      Retrieves and removes a message at the head of this queue, waiting up to the specified wait time if necessary for an element to become available.
      void receiveConnection​(java.net.Socket sock)
      If this server receives a connection request, then it gives up on the new connection if it wins.
      void receiveConnectionAsync​(java.net.Socket sock)
      Server receives a connection request and handles it asynchronously via separate thread.
      void softHalt()
      A soft halt simply finishes workers.
      void testInitiateConnection​(long sid)
      Invokes initiateConnection for testing purposes
      void toSend​(java.lang.Long sid, java.nio.ByteBuffer b)
      Processes invoke this message to queue a message to send.
      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
    • Constructor Detail

      • QuorumCnxManager

        public QuorumCnxManager​(long mySid,
                                java.util.Map<java.lang.Long,​QuorumPeer.QuorumServer> view,
                                QuorumAuthServer authServer,
                                QuorumAuthLearner authLearner,
                                int socketTimeout,
                                boolean listenOnAllIPs,
                                int quorumCnxnThreadsSize,
                                boolean quorumSaslAuthEnabled)
      • QuorumCnxManager

        public QuorumCnxManager​(long mySid,
                                java.util.Map<java.lang.Long,​QuorumPeer.QuorumServer> view,
                                QuorumAuthServer authServer,
                                QuorumAuthLearner authLearner,
                                int socketTimeout,
                                boolean listenOnAllIPs,
                                int quorumCnxnThreadsSize,
                                boolean quorumSaslAuthEnabled,
                                java.util.concurrent.ConcurrentHashMap<java.lang.Long,​org.apache.zookeeper.server.quorum.QuorumCnxManager.SendWorker> senderWorkerMap)
    • Method Detail

      • testInitiateConnection

        public void testInitiateConnection​(long sid)
                                    throws java.lang.Exception
        Invokes initiateConnection for testing purposes
        Parameters:
        sid -
        Throws:
        java.lang.Exception
      • initiateConnection

        public void initiateConnection​(java.net.Socket sock,
                                       java.lang.Long sid)
        If this server has initiated the connection, then it gives up on the connection if it loses challenge. Otherwise, it keeps the connection.
      • initiateConnectionAsync

        public void initiateConnectionAsync​(java.net.Socket sock,
                                            java.lang.Long sid)
        Server will initiate the connection request to its peer server asynchronously via separate connection thread.
      • receiveConnection

        public void receiveConnection​(java.net.Socket sock)
        If this server receives a connection request, then it gives up on the new connection if it wins. Notice that it checks whether it has a connection to this server already or not. If it does, then it sends the smallest possible long value to lose the challenge.
      • receiveConnectionAsync

        public void receiveConnectionAsync​(java.net.Socket sock)
        Server receives a connection request and handles it asynchronously via separate thread.
      • toSend

        public void toSend​(java.lang.Long sid,
                           java.nio.ByteBuffer b)
        Processes invoke this message to queue a message to send. Currently, only leader election uses it.
      • connectOne

        public void connectOne​(long sid)
        Try to establish a connection to server with id sid.
        Parameters:
        sid - server id
      • connectAll

        public void connectAll()
        Try to establish a connection with each server if one doesn't exist.
      • halt

        public void halt()
        Flag that it is time to wrap up all activities and interrupt the listener.
      • softHalt

        public void softHalt()
        A soft halt simply finishes workers.
      • getThreadCount

        public long getThreadCount()
        Return number of worker threads
      • getConnectionThreadCount

        public long getConnectionThreadCount()
        Return number of connection processing threads.
      • addToRecvQueue

        public void addToRecvQueue​(QuorumCnxManager.Message msg)
        Inserts an element in the recvQueue. If the Queue is full, this methods removes an element from the head of the Queue and then inserts the element at the tail of the queue. This method is synchronized to achieve fairness between two threads that are trying to insert an element in the queue. Each thread checks if the queue is full, then removes the element at the head of the queue, and then inserts an element at the tail. This three-step process is done to prevent a thread from blocking while inserting an element in the queue. If we do not synchronize the call to this method, then a thread can grab a slot in the queue created by the second thread. This can cause the call to insert by the second thread to fail. Note that synchronizing this method does not block another thread from polling the queue since that synchronization is provided by the queue itself.
        Parameters:
        msg - Reference to the message to be inserted in the queue
      • pollRecvQueue

        public QuorumCnxManager.Message pollRecvQueue​(long timeout,
                                                      java.util.concurrent.TimeUnit unit)
                                               throws java.lang.InterruptedException
        Retrieves and removes a message at the head of this queue, waiting up to the specified wait time if necessary for an element to become available. ArrayBlockingQueue.poll(long, java.util.concurrent.TimeUnit)
        Throws:
        java.lang.InterruptedException
      • connectedToPeer

        public boolean connectedToPeer​(long peerSid)