2013-02-14 16:20:16,626 [main] INFO com.terracottatech.general - New logging session started. 2013-02-14 16:20:16,680 [main] INFO com.terracottatech.console - Terracotta 3.7.4, as of 20130209-172320 (Revision unknown-22216 by cruise@su10vmo104 from 3.7.4) 2013-02-14 16:20:16,855 [main] INFO com.tc.config.schema.setup.StandardXMLFileConfigurationCreator - Attempting to load configuration from the file at '/usr/local/terracotta/conf/tc-config-staging.xml'... 2013-02-14 16:20:17,979 [main] INFO com.terracottatech.console - Successfully loaded base configuration from file at '/usr/local/terracotta/conf/tc-config-staging.xml'. 2013-02-14 16:20:18,004 [main] INFO com.tc.config.schema.setup.StandardXMLFileConfigurationCreator - The configuration specified by 'base configuration from file at '/usr/local/terracotta/conf/tc-config-staging.xml'': production /usr/local/terracotta/data /usr/local/terracotta/logs /usr/local/terracotta/index /usr/local/terracotta/statistics 9510 65531 9530 120 true false 3600 /usr/local/terracotta/data /usr/local/terracotta/logs /usr/local/terracotta/index /usr/local/terracotta/statistics 9510 65532 9530 120 true false 3600 networked-active-passive 5 %(catalina.home)/logs 2013-02-14 16:20:18,008 [main] INFO com.tc.properties.TCProperties - Loading default properties from tc.properties 2013-02-14 16:20:18,008 [main] INFO com.tc.properties.TCProperties - The property "l2.dump.on.exception.timeout" was read before initialization completed. "l2.dump.on.exception.timeout" = 30 2013-02-14 16:20:18,008 [main] INFO com.tc.properties.TCProperties - The property "tc.config.total.timeout" was read before initialization completed. "tc.config.total.timeout" = 300000 2013-02-14 16:20:18,008 [main] INFO com.tc.properties.TCProperties - The property "tc.config.getFromSource.timeout" was read before initialization completed. "tc.config.getFromSource.timeout" = 30000 2013-02-14 16:20:18,008 [main] INFO com.tc.properties.TCProperties - The property "logging.maxLogFileSize" was overridden to 10 from 512 by the tc-config file 2013-02-14 16:20:18,010 [main] INFO com.tc.properties.TCProperties - Loaded TCProperties : TCProperties = { appgroups.debug = false, aw.asmclassinfo.ignore.errors = com.jinspired., cvt.buffer.randomsuffix.enabled = false, cvt.client.fail.buffer.open = false, cvt.rest.interface.enabled = false, cvt.retriever.notification.interval = 60, cvt.statistics.logging.interval = 900, cvt.store.randomsuffix.enabled = false, dev.console.max.operator.events = 5000, ehcache.clusterAllCacheManagers = true, ehcache.concurrency = 128, ehcache.evictor.logging.enabled = false, ehcache.evictor.pool.size = 1, ehcache.global.eviction.enable = true, ehcache.global.eviction.frequency = 10, ehcache.global.eviction.rest.timeMillis = 10, ehcache.global.eviction.segments = 2, ehcache.lock.readLevel = READ, ehcache.lock.writeLevel = WRITE, ehcache.logging.enabled = false, ehcache.storageStrategy.dcv2.evictUnexpiredEntries.enabled = true, ehcache.storageStrategy.dcv2.eviction.overshoot = 15, ehcache.storageStrategy.dcv2.localcache.enabled = true, ehcache.storageStrategy.dcv2.perElementTTITTL.enabled = false, ehcache.storageStrategy.dcv2.periodicEviction.enabled = true, ehcache.storageStrategy.dcv2.pinSegments.enabled = true, http.defaultservlet.attribute.aliases = false;, http.defaultservlet.attribute.dirallowed = false;, http.defaultservlet.enabled = false;, instrumentation.finalField.fastRead = true, l1.cachemanager.criticalObjectThreshold = -1, l1.cachemanager.criticalThreshold = 70, l1.cachemanager.enabled = true, l1.cachemanager.leastCount = 2, l1.cachemanager.logging.enabled = false, l1.cachemanager.monitorOldGenOnly = true, l1.cachemanager.percentageToEvict = 10, l1.cachemanager.sleepInterval = 3000, l1.cachemanager.threshold = 50, l1.clusterevents.outofbandnotifier.jointime.millis = 100, l1.connect.versionMatchCheck.enabled = true, l1.healthcheck.l2.bindAddress = 0.0.0.0, l1.healthcheck.l2.bindPort = 0, l1.healthcheck.l2.ping.enabled = true, l1.healthcheck.l2.ping.idletime = 5000, l1.healthcheck.l2.ping.interval = 1000, l1.healthcheck.l2.ping.probes = 3, l1.healthcheck.l2.socketConnect = true, l1.healthcheck.l2.socketConnectCount = 13, l1.healthcheck.l2.socketConnectTimeout = 5, l1.jvm.check.compatibility = true, l1.l2.config.validation.enabled = true, l1.lock.statistics.gatherInterval = 1, l1.lock.statistics.traceDepth = 0, l1.lockmanager.pinning.enabled = true, l1.lockmanager.striped.count = 128, l1.lockmanager.timeout.interval = 60000, l1.max.connect.retries = -1, l1.modules.additional = , l1.modules.default = , l1.modules.repositories = , l1.modules.tc-version-check = , l1.modules.toolkitSearchRange = 100, l1.objectmanager.fault.logging.enabled = false, l1.objectmanager.flush.logging.enabled = false, l1.objectmanager.objectid.request.size = 50000, l1.objectmanager.remote.batchLookupTimePeriod = 1, l1.objectmanager.remote.logging.enabled = false, l1.objectmanager.remote.maxDNALRUSize = 60, l1.objectmanager.remote.maxRequestSentImmediately = 4, l1.seda.pinned.entry.fault.stage.threads = 8, l1.seda.stage.sink.capacity = -1, l1.serverarray.objectCreationStrategy = round-robin, l1.serverarray.objectCreationStrategy.groupAffinity.groupName = mirror-group1, l1.serverarray.objectCreationStrategy.roundRobin.startIndex = sequential, l1.servermapmanager.remote.batchLookupTimePeriod = 1, l1.servermapmanager.remote.maxRequestSentImmediately = 4, l1.shutdown.threadgroup.gracetime = 30000, l1.socket.connect.timeout = 10000, l1.socket.reconnect.waitInterval = 1000, l1.transactionmanager.completedAckFlushTimeout = 5000, l1.transactionmanager.folding.debug = false, l1.transactionmanager.folding.enabled = true, l1.transactionmanager.folding.lock.limit = 0, l1.transactionmanager.folding.object.limit = 0, l1.transactionmanager.logging.enabled = false, l1.transactionmanager.maxBatchSizeInKiloBytes = 128, l1.transactionmanager.maxOutstandingBatchSize = 4, l1.transactionmanager.maxPendingBatches = 88, l1.transactionmanager.maxSleepTimeBeforeHalt = 1024, l1.transactionmanager.strings.compress.enabled = true, l1.transactionmanager.strings.compress.logging.enabled = false, l1.transactionmanager.strings.compress.minSize = 512, l1.transactionmanager.timeoutForAckOnExit = 300, l2.beanshell.enabled = false, l2.beanshell.port = 9929, l2.berkeleydb.je.checkpointer.bytesInterval = 100000000, l2.berkeleydb.je.cleaner.bytesInterval = 20000000, l2.berkeleydb.je.cleaner.detailMaxMemoryPercentage = 5, l2.berkeleydb.je.cleaner.lookAheadCacheSize = 32768, l2.berkeleydb.je.cleaner.minAge = 5, l2.berkeleydb.je.lock.nLockTables = 7, l2.berkeleydb.je.lock.timeout = 180000000, l2.berkeleydb.je.maxMemoryPercent = 25, l2.cachemanager.criticalObjectThreshold = -1, l2.cachemanager.enabled = true, l2.cachemanager.leastCount = 2, l2.cachemanager.logging.enabled = false, l2.cachemanager.monitorOldGenOnly = true, l2.cachemanager.percentageToEvict = 10, l2.cachemanager.sleepInterval = 3000, l2.data.backup.throttle.timeInMillis = 0, l2.db.factory.name = com.tc.objectserver.storage.berkeleydb.BerkeleyDBFactory, l2.derbydb.derby.locks.deadlockTimeout = 170, l2.derbydb.derby.locks.deadlockTrace = true, l2.derbydb.derby.locks.escalationThreshold = 100000, l2.derbydb.derby.locks.waitTimeout = 180, l2.derbydb.derby.maxMemoryPercent = 25, l2.derbydb.derby.storage.checkpointInterval = 134217728, l2.derbydb.derby.storage.logSwitchInterval = 134217728, l2.derbydb.derby.storage.pageSize = 32768, l2.dump.on.exception.timeout = 30, l2.healthcheck.l1.ping.enabled = true, l2.healthcheck.l1.ping.idletime = 5000, l2.healthcheck.l1.ping.interval = 1000, l2.healthcheck.l1.ping.probes = 3, l2.healthcheck.l1.socketConnect = true, l2.healthcheck.l1.socketConnectCount = 10, l2.healthcheck.l1.socketConnectTimeout = 5, l2.healthcheck.l2.ping.enabled = true, l2.healthcheck.l2.ping.idletime = 5000, l2.healthcheck.l2.ping.interval = 1000, l2.healthcheck.l2.ping.probes = 3, l2.healthcheck.l2.socketConnect = true, l2.healthcheck.l2.socketConnectCount = 10, l2.healthcheck.l2.socketConnectTimeout = 5, l2.l1reconnect.enabled = true, l2.l1reconnect.maxDelayedAcks = 16, l2.l1reconnect.sendWindow = 32, l2.l1reconnect.sendqueue.cap = 5000, l2.l1reconnect.timeout.millis = 5000, l2.lfu.agingFactor = 1, l2.lfu.recentlyAccessedIgnorePercentage = 20, l2.lockmanager.greedy.lease.enabled = true, l2.lockmanager.greedy.lease.leaseTimeInMillis = 50, l2.lockmanager.greedy.locks.enabled = true, l2.nha.autoRestart = true, l2.nha.dirtydb.autoDelete = true, l2.nha.dirtydb.backup.enabled = true, l2.nha.dirtydb.rolling = 0, l2.nha.send.timeout.millis = 16000, l2.nha.tcgroupcomm.discovery.interval = 1000, l2.nha.tcgroupcomm.handshake.timeout = 5000, l2.nha.tcgroupcomm.reconnect.enabled = true, l2.nha.tcgroupcomm.reconnect.maxDelayedAcks = 16, l2.nha.tcgroupcomm.reconnect.sendWindow = 32, l2.nha.tcgroupcomm.reconnect.sendqueue.cap = 5000, l2.nha.tcgroupcomm.reconnect.timeout = 5000, l2.objectmanager.cachePolicy = lfu, l2.objectmanager.deleteBatchSize = 5000, l2.objectmanager.dgc.enterpriseMarkStageInterval = 1000, l2.objectmanager.dgc.faulting.optimization = true, l2.objectmanager.dgc.inline.intervalInSeconds = 10, l2.objectmanager.dgc.inline.maxObjects = 10000, l2.objectmanager.dgc.throttle.requestsPerThrottle = 1000, l2.objectmanager.dgc.throttle.timeInMillis = 0, l2.objectmanager.dgc.young.enabled = false, l2.objectmanager.dgc.young.frequencyInMillis = 180000, l2.objectmanager.fault.logging.enabled = false, l2.objectmanager.flush.logging.enabled = false, l2.objectmanager.loadObjectID.checkpoint.maxlimit = -1, l2.objectmanager.loadObjectID.checkpoint.maxsleep = 10000, l2.objectmanager.loadObjectID.longsPerDiskEntry = 8, l2.objectmanager.loadObjectID.mapsdatabase.longsPerDiskEntry = 1, l2.objectmanager.maxObjectsInTxnObjGrouping = 5000, l2.objectmanager.maxObjectsToCommit = 5000, l2.objectmanager.maxTxnsInTxnObjectGrouping = 500, l2.objectmanager.objectrequest.logging.enabled = false, l2.objectmanager.objectrequest.split.size = 500, l2.objectmanager.passive.sync.batch.size = 500, l2.objectmanager.passive.sync.message.maxSizeInMegaBytes = 10, l2.objectmanager.passive.sync.throttle.maxPendingMessages = 10, l2.objectmanager.passive.sync.throttle.timeInMillis = 0, l2.objectmanager.persistor.logging.enabled = false, l2.objectmanager.request.logging.enabled = false, l2.offHeapCache.allocation.critical = 15000, l2.offHeapCache.allocation.critical.halt = true, l2.offHeapCache.allocation.slow = 1500, l2.offHeapCache.map.concurrency = 1, l2.offHeapCache.map.max.page.size = 8m, l2.offHeapCache.map.min.page.size = 4k, l2.offHeapCache.map.tableSize = 128, l2.offHeapCache.operator.event.generator.sleepInterval = 60000, l2.offHeapCache.operator.event.generator.threshold = 80, l2.offHeapCache.temp.swap.flush.to.disk.count = 5000, l2.offHeapCache.temp.swap.throttle.megaBytes = 100, l2.operator.events.store = 1500, l2.remotejmx.idletime = 5, l2.remotejmx.maxthreads = 50, l2.seda.commitstage.threads = 4, l2.seda.evictionprocessorstage.sink.capacity = 1000, l2.seda.gcdeletestage.threads = 1, l2.seda.local.cache.invalidations.sink.capacity = 5000, l2.seda.local.cache.transaction.complete.sink.capacity = -1, l2.seda.local.cache.transaction.complete.threads = 8, l2.seda.query.threads = 4, l2.seda.search.threads = 16, l2.seda.stage.sink.capacity = -1, l2.serverarray.2pc.enabled = true, l2.servermap.eviction.broadcast.maxkeys = 10000, l2.servermap.eviction.clientObjectReferences.refresh.interval = 60000, l2.startuplock.retries.enabled = false, l2.transactionmanager.logging.enabled = false, l2.transactionmanager.logging.printBroadcastStats = false, l2.transactionmanager.logging.printCommits = false, l2.transactionmanager.logging.printStats = true, l2.transactionmanager.logging.verbose = false, l2.transactionmanager.passive.throttle.enabled = true, l2.transactionmanager.passive.throttle.maxSleepSeconds = 5, l2.transactionmanager.passive.throttle.threshold = 20000, lock.statistics.enabled = false, logging.longgc.threshold = 8000, logging.maxBackups = 20, logging.maxLogFileSize = 10, memory.monitor.forcebasic = false, net.core.keepalive = false, net.core.tcpnodelay = true, offHeapFileSystem.file.maxDataPageSize = 262144, offHeapFilesystem.chm.segments = 4, offHeapFilesystem.file.blockSize = 8192, search.lucene.indexes.per.cache = 4, search.lucene.max.boolean.clauses = 1024, search.lucene.max.buffer = 16.0, search.lucene.use.offHeap.directory = false, search.lucene.use.ram.directory = false, search.passive.max.chunk = 2097152, search.passive.max.pending = 8, session.debug.hops = false, session.debug.hops.interval = 100, session.debug.invalidate = false, session.debug.sessions = false, session.invalidator.bench.enabled = true, session.invalidator.sleep = 300, session.request.bench.enabled = false, session.request.tracking = false, session.request.tracking.dump = false, session.request.tracking.interval = 2500, session.request.tracking.threshold = 15000, session.statistics.enabled = false, session.verify.set.attribute = false, session.vhosts.excluded = , sigar.enabled = false, stats.printer.intervalInMillis = 5000, tc.bytebuffer.common.pool.maxcount = 3000, tc.bytebuffer.pooling.enabled = true, tc.bytebuffer.threadlocal.pool.maxcount = 2000, tc.config.getFromSource.timeout = 30000, tc.config.total.timeout = 300000, tc.management.mbeans.enabled = true, tc.management.test.mbeans.enabled = false, tc.messages.grouping.enabled = true, tc.messages.grouping.maxSizeKiloBytes = 1024, tc.messages.packup.enabled = true, tc.stage.monitor.delay = 5000, tc.stage.monitor.enabled = false, tc.time.sync.threshold = 30, tc.transport.handshake.timeout = 10000, tcm.monitor.delay = 5, tcm.monitor.enabled = false } 2013-02-14 16:20:18,070 [main] INFO com.terracottatech.console - Log file: '/usr/local/terracotta/logs/terracotta-server.log'. 2013-02-14 16:20:18,070 [main] INFO com.tc.logging.TCLogging - All Java System Properties for this Terracotta instance: ======================================================================== JVM arguments: [-XX:MaxDirectMemorySize=9223372036854775807, -XX:+HeapDumpOnOutOfMemoryError, -Xms7669832k, -Xmx7669832k, -XX:NewSize=2556610k, -XX:MaxNewSize=2556610k, -Dcom.sun.management.jmxremote, -Dtc.install-root=/usr/local/terracotta/bin/.., -Dsun.rmi.dgc.server.gcInterval=31536000000, -Dcom.sun.management.jmxremote, -Dcom.sun.management.jmxremote.port=24034, -Dcom.sun.management.jmxremote.ssl=false, -Dcom.sun.management.jmxremote.authenticate=false, -Djava.rmi.server.hostname=localhost, -Dnewrelic.config.app_name=dorothy-terracotta-1-staging, -javaagent:/usr/local/newrelic/newrelic.jar] awt.toolkit : sun.awt.X11.XToolkit com.sun.management.jmxremote : com.sun.management.jmxremote.authenticate: false com.sun.management.jmxremote.port : 24034 com.sun.management.jmxremote.ssl : false file.encoding : UTF-8 file.encoding.pkg : sun.io file.separator : / java.awt.graphicsenv : sun.awt.X11GraphicsEnvironment java.awt.printerjob : sun.print.PSPrinterJob java.class.path : /usr/local/terracotta/bin/../lib/tc.jar:/usr/local/newrelic/newrelic.jar java.class.version : 51.0 java.endorsed.dirs : /opt/jdk1.7.0_09/jre/lib/endorsed java.ext.dirs : /opt/jdk1.7.0_09/jre/lib/ext:/usr/java/packages/lib/ext java.home : /opt/jdk1.7.0_09/jre java.io.tmpdir : /tmp java.library.path : /usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib java.rmi.server.hostname : localhost java.rmi.server.randomIDs : true java.runtime.name : Java(TM) SE Runtime Environment java.runtime.version : 1.7.0_09-b05 java.specification.name : Java Platform API Specification java.specification.vendor : Oracle Corporation java.specification.version : 1.7 java.vendor : Oracle Corporation java.vendor.url : http://java.oracle.com/ java.vendor.url.bug : http://bugreport.sun.com/bugreport/ java.version : 1.7.0_09 java.vm.info : mixed mode java.vm.name : Java HotSpot(TM) 64-Bit Server VM java.vm.specification.name : Java Virtual Machine Specification java.vm.specification.vendor : Oracle Corporation java.vm.specification.version : 1.7 java.vm.vendor : Oracle Corporation java.vm.version : 23.5-b02 line.separator : newrelic.config.app_name : dorothy-terracotta-1-staging os.arch : amd64 os.name : Linux os.version : 3.2.0-32-virtual path.separator : : sun.arch.data.model : 64 sun.boot.class.path : /opt/jdk1.7.0_09/jre/lib/resources.jar:/opt/jdk1.7.0_09/jre/lib/rt.jar:/opt/jdk1.7.0_09/jre/lib/sunrsasign.jar:/opt/jdk1.7.0_09/jre/lib/jsse.jar:/opt/jdk1.7.0_09/jre/lib/jce.jar:/opt/jdk1.7.0_09/jre/lib/charsets.jar:/opt/jdk1.7.0_09/jre/lib/jfr.jar:/opt/jdk1.7.0_09/jre/classes sun.boot.library.path : /opt/jdk1.7.0_09/jre/lib/amd64 sun.cpu.endian : little sun.cpu.isalist : sun.io.unicode.encoding : UnicodeLittle sun.java.command : com.tc.server.TCServerMain -f /usr/local/terracotta/conf/tc-config-staging.xml -n terracotta-1 sun.java.launcher : SUN_STANDARD sun.jnu.encoding : UTF-8 sun.management.compiler : HotSpot 64-Bit Tiered Compilers sun.os.patch.level : unknown sun.rmi.dgc.server.gcInterval : 31536000000 tc.install-root : /usr/local/terracotta/bin/.. user.country : US user.dir : /home/ubuntu user.home : /home/ubuntu user.language : en user.name : ubuntu user.timezone : America/Toronto ======================================================================== 2013-02-14 16:20:20,876 [main] INFO com.terracottatech.dso - Statistics store: '/usr/local/terracotta/statistics'. 2013-02-14 16:20:20,900 [main] INFO com.terracottatech.console - Available Max Runtime Memory: 7178MB 2013-02-14 16:20:21,245 [main] INFO com.terracottatech.dso - Standard DSO Server created 2013-02-14 16:20:21,248 [main] INFO com.terracottatech.dso - Creating server nodeID: NodeID[stg.tc1:9510] 2013-02-14 16:20:21,298 [Statistics Logger] INFO com.terracottatech.dso - memory free : 6428.975960 MB 2013-02-14 16:20:21,298 [Statistics Logger] INFO com.terracottatech.dso - memory used : 749.086540 MB 2013-02-14 16:20:21,299 [Statistics Logger] INFO com.terracottatech.dso - memory max : 7178.062500 MB 2013-02-14 16:20:23,359 [main] INFO com.terracottatech.dso - Statistics buffer: '/usr/local/terracotta/statistics'. 2013-02-14 16:20:23,397 [main] INFO com.tc.objectserver.managedobject.ConcurrentDistributedMapManagedObjectState - Eviction overshoot threshold is 15.0 2013-02-14 16:20:23,468 [main] INFO com.terracottatech.console - JMX Server started. Available at URL[service:jmx:jmxmp://0.0.0.0:65531] 2013-02-14 16:20:23,485 [main] INFO com.terracottatech.dso - deleting persistence database: /usr/local/terracotta/data/objectdb 2013-02-14 16:20:23,506 [main] INFO com.tc.objectserver.storage.berkeleydb.BerkeleyDBEnvironment - Sleepy cat version being used 5.0.55 2013-02-14 16:20:23,569 [main] INFO com.tc.objectserver.storage.berkeleydb.BerkeleyDBEnvironment - Env config = allowCreate=true cacheSize=0 txnNoSync=false txnWriteNoSync=false {je.cleaner.minAge=5, je.cleaner.bytesInterval=20000000, je.env.isTransactional=false, je.cleaner.lookAheadCacheSize=32768, je.env.isReadOnly=false, je.maxMemoryPercent=25, je.lock.nLockTables=7, je.checkpointer.bytesInterval=100000000, je.cleaner.detailMaxMemoryPercentage=5, je.txn.durability=WRITE_NO_SYNC,NO_SYNC,SIMPLE_MAJORITY, je.lock.timeout=180000000} DB Config = allowCreate=true exclusiveCreate=false transactional=false readOnly=false sortedDuplicates=false deferredWrite=false temporary=false keyPrefixing=false JE Properties = {je.cleaner.detailMaxMemoryPercentage=5, je.cleaner.bytesInterval=20000000, je.cleaner.lookAheadCacheSize=32768, je.checkpointer.bytesInterval=100000000, je.maxMemoryPercent=25, je.lock.nLockTables=7, je.cleaner.minAge=5, je.lock.timeout=180000000} 2013-02-14 16:20:24,026 [main] INFO com.terracottatech.dso - Terracotta persistence version is 2.2 2013-02-14 16:20:24,250 [main] INFO com.tc.net.core.TCWorkerCommManager - Creating 8 worker comm threads for L2_L1 2013-02-14 16:20:24,295 [main] INFO com.tc.net.protocol.transport.ConnectionHealthCheckerImpl: DSO Server - HealthChecker Started 2013-02-14 16:20:24,295 [main] INFO com.tc.net.protocol.tcm.CommunicationsManager - HealthCheck CallbackPort Listener not requested 2013-02-14 16:20:24,308 [main] INFO com.tc.objectserver.l1.impl.ClientObjectReferenceSet - ServerMapEviction Client Object References refresh interval 60 seconds. 2013-02-14 16:20:24,364 [main] INFO com.tc.runtime.TCMemoryManagerImpl - GarbageCollector: PS Scavenge 2013-02-14 16:20:24,364 [main] INFO com.tc.runtime.TCMemoryManagerImpl - GarbageCollector: PS MarkSweep 2013-02-14 16:20:24,644 [main] INFO com.tc.net.core.TCWorkerCommManager - Creating 8 worker comm threads for L2_L2 2013-02-14 16:20:24,651 [main] INFO com.tc.net.protocol.transport.ConnectionHealthCheckerImpl: TCGroupManager - HealthChecker Started 2013-02-14 16:20:24,651 [main] INFO com.tc.net.protocol.tcm.CommunicationsManager - HealthCheck CallbackPort Listener not requested 2013-02-14 16:20:24,679 [main] INFO com.terracottatech.dso - L2 Networked HA Enabled 2013-02-14 16:20:24,820 [main] INFO com.tc.statistics.StatisticRetrievalAction - "message monitor" statistic is not enabled. Please enable the property "tcm.monitor.delay" to collect this statistics. 2013-02-14 16:20:24,821 [main] INFO com.tc.statistics.retrieval.actions.SRAL2FaultsFromDisk - "l2 faults from disk" statistic is not enabled. Please enable the property "l2.objectmanager.fault.logging.enabled" to collect this statistic. 2013-02-14 16:20:24,832 [main] INFO com.tc.l2.L2DebugLogging - L2 debug logging: DISABLED 2013-02-14 16:20:24,867 [main] INFO com.terracottatech.dso - This L2 Node ID = NodeID[stg.tc1:9510] 2013-02-14 16:20:24,869 [main] INFO com.tc.l2.state.ElectionManagerImpl - Election Started : Enrollment [ NodeID[stg.tc1:9510], isNew = true, weights = 4050112562012978763,-1940222907801143155 ] 2013-02-14 16:20:24,960 [WorkerThread(group_discovery_stage, 0, 0)] INFO com.tc.net.core.TCConnection - Comms Message Batching enabled 2013-02-14 16:20:24,968 [WorkerThread(group_discovery_stage, 0, 0)] WARN com.tc.net.groups.TCGroupMemberDiscoveryStatic - Node:Node{host=stg.tc2:9510} not up. IOException occured:Connection refused 2013-02-14 16:20:29,870 [main] INFO com.tc.l2.state.ElectionManagerImpl - Election Complete : [Enrollment [ NodeID[stg.tc1:9510], isNew = true, weights = 4050112562012978763,-1940222907801143155 ]] : State[ Election-Complete ] 2013-02-14 16:20:29,872 [main] INFO com.tc.l2.state.StateManagerImpl - Becoming State[ ACTIVE-COORDINATOR ] 2013-02-14 16:20:29,872 [main] INFO com.terracottatech.console - Becoming State[ ACTIVE-COORDINATOR ] 2013-02-14 16:20:29,881 [main] INFO tc.operator.event - NODE : terracotta-1 Subsystem: CLUSTER_TOPOLOGY Message: Moved to ACTIVE-COORDINATOR 2013-02-14 16:20:29,881 [main] INFO com.tc.l2.state.ElectionManagerImpl - Declared as Winner: Winner is : Enrollment [ NodeID[stg.tc1:9510], isNew = true, weights = 4050112562012978763,-1940222907801143155 ] 2013-02-14 16:20:29,882 [WorkerThread(l2_state_change_stage, 0)] INFO com.tc.l2.ha.StripeIDStateManagerImpl - putToStore GroupID[0] StripeID[07aea39fd7ec429db62661d00dd9cca8] 2013-02-14 16:20:29,882 [WorkerThread(l2_state_change_stage, 0)] INFO com.tc.l2.ha.StripeIDStateManagerImpl - Notify local StripeID ready 2013-02-14 16:20:29,882 [WorkerThread(l2_state_change_stage, 0)] INFO com.tc.l2.ha.StripeIDStateManagerImpl - Notify StripeIDMap ready 2013-02-14 16:20:29,883 [WorkerThread(l2_state_change_stage, 0)] INFO com.tc.objectserver.tx.ServerTransactionManager - Waiting for txns to complete 2013-02-14 16:20:29,886 [WorkerThread(l2_state_change_stage, 0)] INFO com.tc.objectserver.tx.ServerTransactionManager - No more txns in the system. 2013-02-14 16:20:29,886 [WorkerThread(l2_state_change_stage, 0)] INFO com.tc.objectserver.tx.ResentTransactionSequencer - Making callback com.tc.objectserver.gtx.GlobalTransactionIDLowWaterMarkProvider$2@1934ba58 pending since in State[ ADD_RESENT ] resent txns size : 0 2013-02-14 16:20:29,887 [WorkerThread(l2_state_change_stage, 0)] INFO com.tc.objectserver.handshakemanager.ServerClientHandshakeManager - Starting DSO services... 2013-02-14 16:20:29,887 [WorkerThread(l2_state_change_stage, 0)] INFO com.tc.objectserver.persistence.db.TransactionStoreImpl - shutdownAllClientsExcept() : Removing txns from DB : 0 2013-02-14 16:20:29,890 [WorkerThread(l2_state_change_stage, 0)] INFO com.tc.objectserver.impl.GarbageCollectionManagerImpl$PassiveGarbageCollectionManager - In passive mode, not scheduling inline cleanup. 2013-02-14 16:20:29,891 [WorkerThread(l2_state_change_stage, 0)] INFO com.tc.objectserver.impl.ObjectRequestManagerRestartImpl - Processing Pending Lookups = 0 2013-02-14 16:20:29,891 [WorkerThread(l2_state_change_stage, 0)] INFO com.tc.objectserver.tx.ResentTransactionSequencer - Unregistering ResentTransactionSequencer since no more resent Transactions : 0 2013-02-14 16:20:29,892 [WorkerThread(l2_state_change_stage, 0)] INFO com.tc.objectserver.tx.ResentTransactionSequencer - Adding Pending resent CallBacks to TxnMgr : com.tc.objectserver.gtx.GlobalTransactionIDLowWaterMarkProvider$2@1934ba58 2013-02-14 16:20:29,892 [WorkerThread(l2_state_change_stage, 0)] INFO com.tc.objectserver.gtx.GlobalTransactionIDLowWaterMarkProvider - Switching GlobalTransactionID Low Water mark provider since all resent transactions are applied 2013-02-14 16:20:29,892 [WorkerThread(l2_state_change_stage, 0)] INFO com.tc.objectserver.impl.ServerMapEvictionManagerImpl - Server Map Eviction : Evictor will run every 1800000 ms 2013-02-14 16:20:29,892 [WorkerThread(l2_state_change_stage, 0)] INFO com.tc.objectserver.impl.ServerMapEvictionManagerImpl - ehcache.evictor.logging.enabled : false 2013-02-14 16:20:29,892 [WorkerThread(l2_state_change_stage, 0)] INFO com.tc.objectserver.impl.ServerMapEvictionManagerImpl - ehcache.storageStrategy.dcv2.periodicEviction.enabled : true 2013-02-14 16:20:29,892 [WorkerThread(l2_state_change_stage, 0)] INFO com.tc.objectserver.impl.ServerMapEvictionManagerImpl - ehcache.storageStrategy.dcv2.perElementTTITTL.enabled : false 2013-02-14 16:20:29,893 [WorkerThread(l2_state_change_stage, 0)] INFO com.tc.objectserver.l1.impl.InvalidateObjectManagerImpl - Restart txn processing complete : Adding validation of Objects for 0 Clients 2013-02-14 16:20:29,894 [WorkerThread(l2_state_change_stage, 0)] INFO com.terracottatech.console - Terracotta Server instance has started up as ACTIVE node on 0:0:0:0:0:0:0:0:9510 successfully, and is now ready for work. 2013-02-14 16:20:29,926 [main] INFO com.tc.server.TCServer - Server started as terracotta-1 2013-02-14 16:20:29,926 [Update Checker] INFO com.tc.server.UpdateCheckAction - Update Checker: Checking... 2013-02-14 16:20:30,404 [Update Checker] INFO com.tc.server.UpdateCheckAction - Update Checker: 3.7.4 is the latest GA release from Terracotta 2013-02-14 16:20:30,405 [Update Checker] INFO com.tc.server.UpdateCheckAction - Update Checker: No updates found 2013-02-14 16:20:30,405 [Update Checker] INFO com.tc.server.UpdateCheckAction - Update Checker: Next check at Thu Feb 21 16:20:30 EST 2013 2013-02-14 16:21:19,711 [L2_L2:TCComm Main Selector Thread_R (listen 0:0:0:0:0:0:0:0:9530)] INFO com.tc.net.protocol.transport.ServerMessageTransport - ConnectionID(0.5045b78ad2ed46f3bebec2c31255c2a5.ebdd0a47-171e-4aa2-8bcf-221832808fec-13cda950c80)[]: Attaching new connection: com.tc.net.core.TCConnectionImpl@1816583573: connected: true, closed: false local=10.149.9.6:9530 remote=10.149.20.157:42594 connect=[Thu Feb 14 16:21:19 EST 2013] idle=8ms [0 read, 0 write] 2013-02-14 16:21:19,717 [L2_L2:TCComm Main Selector Thread_R (listen 0:0:0:0:0:0:0:0:9530)] INFO com.tc.net.protocol.transport.ServerStackProvider - User null successfully authenticated 2013-02-14 16:21:19,717 [L2_L2:TCComm Main Selector Thread_R (listen 0:0:0:0:0:0:0:0:9530)] INFO com.tc.net.core.TCWorkerCommManager - Selecting [L2_L2:TCWorkerComm # 0, FD, wt:0] from [[L2_L2:TCWorkerComm # 0, FD, wt:0], [L2_L2:TCWorkerComm # 1, FD, wt:0], [L2_L2:TCWorkerComm # 2, FD, wt:0], [L2_L2:TCWorkerComm # 3, FD, wt:0], [L2_L2:TCWorkerComm # 4, FD, wt:0], [L2_L2:TCWorkerComm # 5, FD, wt:0], [L2_L2:TCWorkerComm # 6, FD, wt:0], [L2_L2:TCWorkerComm # 7, FD, wt:0]] 2013-02-14 16:21:19,782 [L2_L2:TCComm Main Selector Thread_R (listen 0:0:0:0:0:0:0:0:9530)] INFO com.tc.net.core.CoreNIOServices - selection key cancelled key@545102012 2013-02-14 16:21:19,797 [L2_L2:TCWorkerComm # 0_R] INFO com.tc.net.protocol.transport.ConnectionHealthCheckerImpl. TCGroupManager - Health monitoring agent started for ip-10-149-20-157.ec2.internal:42594 2013-02-14 16:21:19,807 [L2_L2:TCComm Main Selector Thread_R (listen 0:0:0:0:0:0:0:0:9530)] INFO com.tc.net.protocol.transport.ConnectionHealthCheckerImpl. TCGroupManager - HealthCheckCallbackPort verification PASSED for ip-10-149-20-157.ec2.internal:42594(callbackport: 9530) 2013-02-14 16:21:19,849 [WorkerThread(group_handshake_message_stage, 0)] INFO tc.operator.event - NODE : terracotta-1 Subsystem: CLUSTER_TOPOLOGY Message: Node terracotta-2 joined the cluster 2013-02-14 16:21:19,849 [WorkerThread(group_events_dispatch_stage, 0)] INFO com.tc.l2.ha.L2HACoordinator - NodeID[stg.tc2:9510] joined the cluster 2013-02-14 16:21:19,849 [WorkerThread(group_events_dispatch_stage, 0)] INFO com.terracottatech.console - NodeID[stg.tc2:9510] joined the cluster 2013-02-14 16:21:19,877 [WorkerThread(receive_group_message_stage, 0)] INFO com.tc.l2.objectserver.ReplicatedObjectManagerImpl - DGC is disabled. 2013-02-14 16:21:19,881 [WorkerThread(receive_group_message_stage, 0)] INFO com.tc.l2.objectserver.L2ObjectStateManagerImpl - NodeID[stg.tc2:9510] : is missing 0 out of 0 objects of which missing roots = 0 2013-02-14 16:21:19,882 [WorkerThread(receive_group_message_stage, 0)] INFO com.tc.l2.objectserver.ReplicatedObjectManagerImpl - Reenabling DGC as all passive are synced up 2013-02-14 16:21:19,882 [WorkerThread(receive_group_message_stage, 0)] INFO com.tc.l2.state.StateManagerImpl - Requesting node NodeID[stg.tc2:9510] to move to State[ PASSIVE-STANDBY ] 2013-02-14 16:22:31,832 [L2_L1:TCComm Main Selector Thread_R (listen 0:0:0:0:0:0:0:0:9510)] INFO com.tc.net.protocol.transport.ServerMessageTransport - ConnectionID(0.07aea39fd7ec429db62661d00dd9cca8.a8a7734d-1c60-4009-a38f-d5d662591e1f-13cda95ab48)[]: Attaching new connection: com.tc.net.core.TCConnectionImpl@1018101143: connected: true, closed: false local=10.149.9.6:9510 remote=10.190.151.166:56143 connect=[Thu Feb 14 16:22:31 EST 2013] idle=5ms [0 read, 0 write] 2013-02-14 16:22:31,833 [L2_L1:TCComm Main Selector Thread_R (listen 0:0:0:0:0:0:0:0:9510)] INFO com.tc.net.protocol.transport.ConnectionPolicyImpl - Allocated connection license for jvm a8a7734d-1c60-4009-a38f-d5d662591e1f-13cda95ab48; ConnectionPolicy[maxConnections=2147483647, connectedJvmCount=1] 2013-02-14 16:22:31,833 [L2_L1:TCComm Main Selector Thread_R (listen 0:0:0:0:0:0:0:0:9510)] INFO com.tc.net.protocol.transport.ConnectionPolicyImpl - New connection [0] from jvm a8a7734d-1c60-4009-a38f-d5d662591e1f-13cda95ab48 2013-02-14 16:22:31,833 [L2_L1:TCComm Main Selector Thread_R (listen 0:0:0:0:0:0:0:0:9510)] INFO com.tc.net.protocol.transport.ServerStackProvider - User null successfully authenticated 2013-02-14 16:22:31,833 [L2_L1:TCComm Main Selector Thread_R (listen 0:0:0:0:0:0:0:0:9510)] INFO com.tc.net.core.CoreNIOServices - selection key cancelled key@76211105 2013-02-14 16:22:31,842 [L2_L1:TCWorkerComm # 0_R] INFO com.tc.net.protocol.transport.ConnectionHealthCheckerImpl. DSO Server - Health monitoring agent started for ip-10-190-151-166.ec2.internal:56143 2013-02-14 16:22:31,844 [L2_L1:TCComm Main Selector Thread_R (listen 0:0:0:0:0:0:0:0:9510)] INFO com.tc.net.protocol.transport.ConnectionHealthCheckerImpl. DSO Server - HealthCheckCallbackPort verification PASSED for ip-10-190-151-166.ec2.internal:56143(callbackport: 41924) 2013-02-14 16:22:31,901 [WorkerThread(client_handshake_stage, 0)] INFO com.tc.objectserver.handshakemanager.ServerClientHandshakeManager - Client connected ClientID[0] 2013-02-14 16:22:31,906 [WorkerThread(client_handshake_stage, 0)] INFO tc.operator.event - NODE : terracotta-1 Subsystem: CLUSTER_TOPOLOGY Message: Node ClientID[0] joined the cluster 2013-02-14 16:22:31,926 [WorkerThread(jmxremote_tunnel_stage, 0)] INFO com.tc.management.remote.protocol.terracotta.ClientTunnelingEventHandler - L1[ChannelID=[0]] notified us that their JMX server is now available 2013-02-14 16:22:37,749 [Thread-58] WARN com.tc.management.remote.connect.ClientBeanBag - Unable to register DSO client bean[net.sf.ehcache:type=SampledCache,SampledCacheManager=imageproxy,name=LongTermImageCache,node=80531e6d1a084389b39f343fda555b81] due to net.sf.ehcache:type=SampledCache,SampledCacheManager=imageproxy,name=LongTermImageCache,node=80531e6d1a084389b39f343fda555b81 2013-02-14 16:22:37,923 [Thread-58] WARN com.tc.management.remote.connect.ClientBeanBag - Unable to register DSO client bean[net.sf.ehcache:type=SampledCache,SampledCacheManager=imageproxy,name=ShortTermImageCache,node=80531e6d1a084389b39f343fda555b81] due to net.sf.ehcache:type=SampledCache,SampledCacheManager=imageproxy,name=ShortTermImageCache,node=80531e6d1a084389b39f343fda555b81 2013-02-14 16:22:37,923 [Thread-58] INFO com.tc.management.remote.connect.ClientBeanBag - Ignoring bean for unregistration: net.sf.ehcache:type=SampledCache,SampledCacheManager=imageproxy,name=ShortTermImageCache,node=80531e6d1a084389b39f343fda555b81 2013-02-14 16:22:37,923 [Thread-58] INFO com.tc.management.remote.connect.ClientBeanBag - Ignoring bean for unregistration: net.sf.ehcache:type=SampledCache,SampledCacheManager=imageproxy,name=LongTermImageCache,node=80531e6d1a084389b39f343fda555b81 2013-02-14 16:23:04,511 [L2_L1:TCComm Main Selector Thread_W (listen 0:0:0:0:0:0:0:0:9510)] WARN com.tc.net.core.CoreNIOServices - Skipping modifyInterest - WRITE on com.tc.net.core.TCConnectionImpl@1948031726: connected: true, closed: false local=10.149.9.6:9510 remote=10.190.151.166:56197 connect=[Thu Feb 14 16:23:04 EST 2013] idle=5ms [476 read, 256 write] 2013-02-14 16:23:15,566 [L2_L1:TCComm Main Selector Thread_W (listen 0:0:0:0:0:0:0:0:9510)] WARN com.tc.net.core.CoreNIOServices - Skipping modifyInterest - WRITE on com.tc.net.core.TCConnectionImpl@28057270: connected: true, closed: false local=10.149.9.6:9510 remote=10.190.151.166:56221 connect=[Thu Feb 14 16:23:15 EST 2013] idle=3ms [476 read, 259 write] 2013-02-14 16:23:16,571 [L2_L1:TCComm Main Selector Thread_W (listen 0:0:0:0:0:0:0:0:9510)] WARN com.tc.net.core.CoreNIOServices - Skipping modifyInterest - WRITE on com.tc.net.core.TCConnectionImpl@845101515: connected: true, closed: false local=10.149.9.6:9510 remote=10.190.151.166:56223 connect=[Thu Feb 14 16:23:16 EST 2013] idle=3ms [476 read, 259 write] 2013-02-14 16:23:18,584 [L2_L1:TCComm Main Selector Thread_W (listen 0:0:0:0:0:0:0:0:9510)] WARN com.tc.net.core.CoreNIOServices - Skipping modifyInterest - WRITE on com.tc.net.core.TCConnectionImpl@703671491: connected: true, closed: false local=10.149.9.6:9510 remote=10.190.151.166:56227 connect=[Thu Feb 14 16:23:18 EST 2013] idle=4ms [476 read, 259 write] 2013-02-14 16:23:23,611 [L2_L1:TCComm Main Selector Thread_W (listen 0:0:0:0:0:0:0:0:9510)] INFO com.tc.net.core.CoreNIOServices - selection key cancelled key@640313454 2013-02-14 16:24:28,741 [L2_L1:TCComm Main Selector Thread_R (listen 0:0:0:0:0:0:0:0:9510)] INFO com.tc.net.protocol.transport.ServerMessageTransport - ConnectionID(1.07aea39fd7ec429db62661d00dd9cca8.a8a7734d-1c60-4009-a38f-d5d662591e1f-13cda95ab48)[]: Attaching new connection: com.tc.net.core.TCConnectionImpl@2139569354: connected: true, closed: false local=10.149.9.6:9510 remote=10.190.151.166:56380 connect=[Thu Feb 14 16:24:28 EST 2013] idle=2ms [0 read, 0 write] 2013-02-14 16:24:28,742 [L2_L1:TCComm Main Selector Thread_R (listen 0:0:0:0:0:0:0:0:9510)] INFO com.tc.net.protocol.transport.ConnectionPolicyImpl - New connection [1] from jvm a8a7734d-1c60-4009-a38f-d5d662591e1f-13cda95ab48 2013-02-14 16:24:28,742 [L2_L1:TCComm Main Selector Thread_R (listen 0:0:0:0:0:0:0:0:9510)] INFO com.tc.net.protocol.transport.ServerStackProvider - User null successfully authenticated 2013-02-14 16:24:28,742 [L2_L1:TCComm Main Selector Thread_R (listen 0:0:0:0:0:0:0:0:9510)] INFO com.tc.net.core.CoreNIOServices - selection key cancelled key@128463663 2013-02-14 16:24:28,748 [L2_L1:TCWorkerComm # 2_R] INFO com.tc.net.protocol.transport.ConnectionHealthCheckerImpl. DSO Server - Health monitoring agent started for ip-10-190-151-166.ec2.internal:56380 2013-02-14 16:24:28,750 [L2_L1:TCComm Main Selector Thread_R (listen 0:0:0:0:0:0:0:0:9510)] INFO com.tc.net.protocol.transport.ConnectionHealthCheckerImpl. DSO Server - HealthCheckCallbackPort verification PASSED for ip-10-190-151-166.ec2.internal:56380(callbackport: 58220) 2013-02-14 16:24:28,790 [WorkerThread(client_handshake_stage, 0)] INFO com.tc.objectserver.handshakemanager.ServerClientHandshakeManager - Client connected ClientID[1] 2013-02-14 16:24:28,790 [WorkerThread(client_handshake_stage, 0)] INFO tc.operator.event - NODE : terracotta-1 Subsystem: CLUSTER_TOPOLOGY Message: Node ClientID[1] joined the cluster 2013-02-14 16:24:28,823 [WorkerThread(jmxremote_tunnel_stage, 0)] INFO com.tc.management.remote.protocol.terracotta.ClientTunnelingEventHandler - L1[ChannelID=[1]] notified us that their JMX server is now available