Fixing DatabaseConnection to Prevent the "Too Many Connections" Problem

Results 1 to 12 of 12
  1. #1
    Apprentice BlackRabbit is offline
    MemberRank
    Jul 2014 Join Date
    Little GardenLocation
    22Posts

    Fixing DatabaseConnection to Prevent the "Too Many Connections" Problem

    Hello. This may not be super useful to those of you using an Odin-based source to just mess around; however, if you run a server, then I know for sure you have ran into this problem at least once. The longer your server is online, the more problems you have. This is especially true with the amount of connections your server has made to your database. Left completely untouched, even 10 players online can cause these problems over the course of time.

    What causes this problems is very simple and can be traced down to two key reasons.
    1) All connections made to the database are stored forever in memory. There have been numerous attempts to fix this, some work temporarily as a bandage and some just do not work at all.
    2) Compounding on this, connections that are about to close will be forced to be reopened due to the autoReconnect parameter in the URL even if the server no longer requires that connection since the thread that spawned it died ages ago.

    Why these are contributing factors lies in the technique used for connection management. All connections are initialized to be thread local (i.e. one connection per thread created) and are further stored in live memory until the thread ends. While this is what the design intends, it does not happen since a hard reference to the connection exists outside of the ThreadLocal object and the GC has no reason to reclaim the memory as well as close the connection if it is still open.

    So now we will compare base OdinMS' implementation versus something I drafted with something else that is Odin-based.

    Original Odin:
    Code:
    /** 
     * @author Frz
     */
    public class DatabaseConnection {
    
    
        private static ThreadLocal<Connection> con = new ThreadLocalConnection();
        private final static Logger log = LoggerFactory.getLogger(DatabaseConnection.class);
        private static Properties props = null;
    
    
        public static Connection getConnection() {
            if (props == null) throw new RuntimeException("DatabaseConnection not initialized");
            return con.get();
        }
        
        public static boolean isInitialized() {
            return props != null;
        }
        
        public static void setProps(Properties aProps) {
            props = aProps;
        }
        
        public static void closeAll() throws SQLException {
            for (Connection con : ThreadLocalConnection.allConnections) {
                con.close();
            }
        }
    
    
        private static class ThreadLocalConnection extends ThreadLocal<Connection> {
            public static Collection<Connection> allConnections = new LinkedList<Connection>();
            
            @Override
            protected Connection initialValue() {
                String driver = props.getProperty("driver");
                String url = props.getProperty("url");
                String user = props.getProperty("user");
                String password = props.getProperty("password");
                try {
                    Class.forName(driver); // touch the mysql driver
                } catch (ClassNotFoundException e) {
                    log.error("ERROR", e);
                }
                try {
                    Connection con = DriverManager.getConnection(url, user, password);
                    allConnections.add(con);
                    return con;
                } catch (SQLException e) {
                    log.error("ERROR", e);
                    return null;
                }
            }
        }
    }
    Modifications:
    Code:
    /**
     * @author Frz
     * @author BlackRabbit
     */
    public class DatabaseConnection {
    
    
        private static ThreadLocal<Connection> con = new ThreadLocalConnection();
    
    
        static {
            try {
                Class.forName("com.mysql.jdbc.Driver"); // touch the mysql driver
            } catch (ClassNotFoundException e) {
                System.out.println("[SEVERE] SQL Driver Not Found. Consider death by clams.");
                e.printStackTrace();
            }
        }
    
    
        public static Connection getConnection() {
            Connection c = con.get();
            try {
                c.getMetaData();
            } catch (SQLException e) { // connection is dead, therefore discard old object
                con.remove();
                c = con.get();
            }
            return c;
        }
    
    
        private static class ThreadLocalConnection extends ThreadLocal<Connection> {
    
    
            @Override
            protected Connection initialValue() {
                try {
                    return DriverManager.getConnection(ServerConstants.DB_URL, ServerConstants.DB_USER, ServerConstants.DB_PASS);
                } catch (SQLException e) {
                    System.out.println("[SEVERE] Unable to make database connection.");
                    e.printStackTrace();
                    return null;
                }
            }
        }
    }
    The latter can function regardless of whether or not the autoReconnect parameter is given in the URL or not (since there is no hard reference to the connection after the thread dies). This ensures that you will scale linearly with the amount of threads you actually have versus having connections that are just taking up memory and destroying your ability to run your server in the long run.

    This is not the be-all end-all with making servers stay online longer. If you want to do that, then you should look at TimerManager since a lot of issues spawn in the long run due to the hard reliance the source has on using it to schedule tasks.

    Hope this helps someone. Have a nice day ~
    Last edited by BlackRabbit; 22-04-15 at 09:13 PM.


  2. #2
    :l Cygnus is offline
    MemberRank
    Mar 2015 Join Date
    f425Location
    237Posts

    Re: Fixing DatabaseConnection to Prevent the "Too Many Connections" Problem

    Thanks!
    It's greatly appreciated that you go into detail as to what this does and how it works, so that newcomers 'cough' me 'cough', actually learn rather than just copy-paste your work! :)
    Regards, Cygnus

  3. #3
    Enthusiast Snakeday is offline
    MemberRank
    Oct 2010 Join Date
    35Posts

    Re: Fixing DatabaseConnection to Prevent the "Too Many Connections" Problem

    Nice catch!
    It's not such a great idea to rely on the garbage collector to close your sql connections though (but it's ok). What should be done is configure the threadpool(s) to close the connections when they destroy a thread.
    But then again - it's much better to do it this way than the way it was before (but I'm unsure if it was like this in clean OdinMS)

  4. #4
    Apprentice BlackRabbit is offline
    MemberRank
    Jul 2014 Join Date
    Little GardenLocation
    22Posts

    Re: Fixing DatabaseConnection to Prevent the "Too Many Connections" Problem

    Quote Originally Posted by Snakeday View Post
    Nice catch!
    It's not such a great idea to rely on the garbage collector to close your sql connections though (but it's ok). What should be done is configure the threadpool(s) to close the connections when they destroy a thread.
    But then again - it's much better to do it this way than the way it was before (but I'm unsure if it was like this in clean OdinMS)
    I agree; however, if the autoReconnect parameter is not included in the URL, the connection would timeout regardless of the object being collected sooner versus later (this was the main goal). This is especially true since ThreadLocal may or may not do anything directly when the thread dies from what I've found at least. This is still not the greatest implementation ever; and, I would personally lean towards just having a set pool for connections, but that would not really fit well in a release thread considering one modification needing to be made for everywhere a Connection object is required can get really bad for the purposes of a small release.

    Default OdinMS did not necessarily have the "Too Many Connections" problems since they did not use autoReconnect in the URL by default. My guess is that the one issue it resolved is that the change was made later down the line to combat connections timing out. Also, I think I may be wrong about ThreadLocal removing the object when the thread dies. Not very helpful when online resources completely vary for what exactly constitutes thread death (making this a gray area and I would rather not try figuring out how exactly the JVM handles ending threads for something as trivial as this).

    I guess at the end of the day this release is meant to dissuade people from using autoReconnect versus just refreshing the connection in the source when you need it since it causes a major problem in the long run. The root of the problem was primarily people having bad ideas on how to fix connections timing out before thread death (though I kind of want to blame Frz for using ThreadLocal to begin with).

    Here's an example of something using pooled connections for anyone interested. This requires more work to manage since you have to release your connection after each usage (pulled from Invictus by Zygon, I just made javadocs for it):
    Code:
    public final class Database {
    
        /**
         * The connection URL for the database. Connections will be attempted to be
         * made to this URL.
         */
        private String URL;
        /**
         * Manages the amount of sessions that can exist within a service/process
         * rather than using something like ThreadLocal which can leak.
         */
        private final Semaphore s;
        /**
         * Connection properties that are used for the database credentials.
         */
        private final Properties cProp;
        /**
         * Safety variable for preventing new connections after closing.
         */
        private boolean closed = false;
        /**
         * Queue that contains all the connections (references to this collection
         * are thread-safe).
         */
        private final ConcurrentLinkedQueue<Connection> queue;
    
    
        static {
            try {
                Class.forName("com.mysql.jdbc.Driver"); // touch the driver first
            } catch (Exception e) {
                System.err.println("SEVERE : SQL Driver has not been located during runtime. Please make sure it is in the correct path.");
                System.exit(0);
            }
        }
    
    
        /**
         * Creates a new Database object using specific credentials and information
         * for maintaining the server.
         *
         * @Param url the URL for the database.
         * @Param user the user credential for the database.
         * @Param pass the password credential for the database.
         * @Param connections the amount of connections that can exist in this
         * service at one time.
         * @return a new Database object for usage in a service.
         */
        public static Database newDatabase(String url, String user, String pass, int connections) {
            Database db = new Database(connections);
            db.URL = url;
            db.cProp.setProperty("user", user);
            db.cProp.setProperty("password", pass);
            return db;
        }
    
    
        /**
         * @See Database#newDatabase(java.lang.String, java.lang.String,
         * java.lang.String, int)
         */
        private Database(int connections) {
            cProp = new Properties();
            s = new Semaphore(connections, true);
            queue = new ConcurrentLinkedQueue();
        }
        
        /**
         * Closes this database and closes all connections.
         */
        public final void close() {
            try {
                s.tryAcquire(10, 1, TimeUnit.HOURS);
            } catch (Exception e) {
                return;
            }
            for (Connection c : queue) {
                try {
                    c.close();
                } catch (Exception e) {
                    e.printStackTrace();
                }
            }
            closed = true;
        }
    
    
        /**
         * Acquires a new connection using a default timeout for the Semaphore to
         * acquire a connection. If the time has passed, an exception will be thrown
         * and no connection will be given.
         *
         * @return ideally a connection to the desired database; if nothing pans out
         * and the Semaphore has to wait too long, no connection will be returned.
         */
        public final Connection getConnection() {
            return getConnection(60000);
        }
    
    
        /**
         *
         * @Param timeout time-out for the Semaphore.
         * @return a connection to the database.
         * @See Database#getConnection()
         */
        public final Connection getConnection(long timeout) {
            if (URL == null || cProp.isEmpty() || closed) {
                return null;
            }
            try {
                s.tryAcquire(timeout, TimeUnit.MILLISECONDS);
            } catch (Exception e) {
                return null;
            }
            Connection c = queue.poll();
            if (c == null) {
                try {
                    c = DriverManager.getConnection(URL, cProp);
                } catch (Exception e) {
                    e.printStackTrace();
                    s.release();
                    return null;
                }
            } else { // validates connection is still active
                try {
                    c.getMetaData();
                } catch (Exception e) {
                    try {
                        c = DriverManager.getConnection(URL, cProp);
                    } catch (Exception ex) {
                        ex.printStackTrace();
                        s.release();
                        return null;
                    }
                }
            }
            return c;
        }
    
    
        /**
         * Releases a connection back into the queue for usage later. This is a
         * critical step in this model since if the semaphore is blocked forever,
         * then the server cannot possibly function properly.
         *
         * @Param c the connection to be released back into the queue.
         */
        public final void release(Connection c) {
            try {
                if (!c.isClosed()) {
                    queue.add(c);
                }
            } catch (Exception e) {
            } finally {
                s.release();
            }
        }
    }

  5. #5
    .. .-- - .-.. .. ...- . iwtLive is offline
    MemberRank
    Sep 2012 Join Date
    260Posts

    Re: Fixing DatabaseConnection to Prevent the "Too Many Connections" Problem

    The only problem I got with this was an error with
    PHP Code:
        static {
            try {
                Class.
    forName("com.mysql.jdbc.Driver"); // touch the mysql driver
            
    } catch (ClassNotFoundException e) {
                
    System.out.println("[SEVERE] SQL Driver Not Found. Consider death by clams.");
                
    e.printStackTrace();
                
    //return null;
            
    }
        } 
    the "return null" line so I just disabled it, it shouldn't give me any future problems right?

  6. #6
    Apprentice BlackRabbit is offline
    MemberRank
    Jul 2014 Join Date
    Little GardenLocation
    22Posts

    Re: Fixing DatabaseConnection to Prevent the "Too Many Connections" Problem

    Quote Originally Posted by dorkie4ever View Post
    The only problem I got with this was an error with
    PHP Code:
        static {
            try {
                Class.
    forName("com.mysql.jdbc.Driver"); // touch the mysql driver
            
    } catch (ClassNotFoundException e) {
                
    System.out.println("[SEVERE] SQL Driver Not Found. Consider death by clams.");
                
    e.printStackTrace();
                
    //return null;
            
    }
        } 
    the "return null" line so I just disabled it, it shouldn't give me any future problems right?
    Yeah, my bad. I moved that code outside of the initialValue method since it was pretty silly to have it try to find the driver every time a connection was going to be made. I'll fix that in the original post.

  7. #7
    (O_o(o_O(O_O)o_O)O_o) Novak is offline
    MemberRank
    Apr 2009 Join Date
    The NetherlandsLocation
    1,120Posts

    Re: Fixing DatabaseConnection to Prevent the "Too Many Connections" Problem

    Ahh, I was wondering why this happened. Thanks for thoroughly explaining it! Best release I've seen in a while! :D

  8. #8
    Enthusiast Snakeday is offline
    MemberRank
    Oct 2010 Join Date
    35Posts

    Re: Fixing DatabaseConnection to Prevent the "Too Many Connections" Problem

    Quote Originally Posted by BlackRabbit View Post
    I guess at the end of the day this release is meant to dissuade people from using autoReconnect versus just refreshing the connection in the source when you need it since it causes a major problem in the long run. The root of the problem was primarily people having bad ideas on how to fix connections timing out before thread death (though I kind of want to blame Frz for using ThreadLocal to begin with).
    I agree. That probably wasn't the smartest choice. The idea was probably to work around the problems that connection pooling brings (forgetting to return connections to the pool is disastrous) - also it is required/useful to not block on acquiring db connections -> there should be as many connections as threads accessing the database - that is ensured by holding the connections threadlocal in the executor threads.
    But - you are completely right - a smart connection pool (one that grows on demand - like the ThreadPoolExecutor does) would have been a good idea.

  9. #9
    Account Upgraded | Title Enabled! Syre is offline
    MemberRank
    Jan 2013 Join Date
    700Posts

    Re: Fixing DatabaseConnection to Prevent the "Too Many Connections" Problem

    You finally did something about it sir. Very fitting....Thank you for telling me what the original issue was over skype, but I never did end up working at fixing it

  10. #10
    while(true) spam(); kevintjuh93 is offline
    MemberRank
    Jun 2008 Join Date
    The NetherlandsLocation
    4,119Posts

    Re: Fixing DatabaseConnection to Prevent the "Too Many Connections" Problem

    Nice work, I've always hated working with Databases.... because I never really tried to make a good one... :P

  11. #11
    Account Upgraded | Title Enabled! lockingkz is offline
    MemberRank
    Dec 2012 Join Date
    475Posts

    Re: Fixing DatabaseConnection to Prevent the "Too Many Connections" Problem

    So what happened if the source has no serverconstant?
    Make 1 ? O_O

  12. #12
    :l Cygnus is offline
    MemberRank
    Mar 2015 Join Date
    f425Location
    237Posts

    Re: Fixing DatabaseConnection to Prevent the "Too Many Connections" Problem

    Quote Originally Posted by lockingkz View Post
    So what happened if the source has no serverconstant?
    Make 1 ? O_O
    Make it read db configs the same way your current databaseconnection does.



Advertisement