Tomcat 9.0.37 Clustered DeltaManager Duplicates Session And Loses Session Attributes

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Tomcat 9.0.37 Clustered DeltaManager Duplicates Session And Loses Session Attributes

Tim N
Hi,

I'm in the early stages of analysing this problem:

   - Tomcat Embedded 9.0.37 with clustering enabled
   - SpringBoot application, 2.1.16
   - Login with no existing JSESSIONID fails

I can see the following code is executed twice within one request, the
login attempt:

*~/.m2/repository/org/apache/tomcat/tomcat-catalina/9.0.37/tomcat-catalina-9.0.37-sources.jar!/org/apache/catalina/session/ManagerBase.java:677*

    public void add(Session session) {
        sessions.put(session.getIdInternal(), session);

The first time it's executed, the session with the Spring context is added.
The second time it's executed, a second session with the same ID, but
without the Spring context, or any other session attribute I add for that
matter, overwrites the existing session, and login fails. If I debug and
prevent this by renaming the second session ID, login works because the
original session is preserved.

The stack-trace for the first call is shown below:

add:678, ManagerBase (org.apache.catalina.session)
setId:358, StandardSession (org.apache.catalina.session)
setId:327, DeltaSession (org.apache.catalina.ha.session)
setId:345, DeltaSession (org.apache.catalina.ha.session)
createSession:719, ManagerBase (org.apache.catalina.session)
createSession:422, DeltaManager (org.apache.catalina.ha.session)
createSession:410, DeltaManager (org.apache.catalina.ha.session)
doGetSession:3043, Request (org.apache.catalina.connector)
getSession:2441, Request (org.apache.catalina.connector)
getSession:908, RequestFacade (org.apache.catalina.connector)
getSession:920, RequestFacade (org.apache.catalina.connector)
getSession:253, HttpServletRequestWrapper (javax.servlet.http)
getSession:253, HttpServletRequestWrapper (javax.servlet.http)
getSession:253, HttpServletRequestWrapper (javax.servlet.http)
onAuthentication:66, RegisterSessionAuthenticationStrategy
(org.springframework.security.web.authentication.session)
doFilter:218, AbstractAuthenticationProcessingFilter
(org.springframework.security.web.authentication)
doFilter:334, FilterChainProxy$VirtualFilterChain
(org.springframework.security.web)
doFilter:200, AbstractAuthenticationProcessingFilter
(org.springframework.security.web.authentication)
doFilter:334, FilterChainProxy$VirtualFilterChain
(org.springframework.security.web)
doFilter:116, LogoutFilter
(org.springframework.security.web.authentication.logout)
doFilter:334, FilterChainProxy$VirtualFilterChain
(org.springframework.security.web)
doFilterInternal:74, HeaderWriterFilter
(org.springframework.security.web.header)
doFilter:119, OncePerRequestFilter (org.springframework.web.filter)
doFilter:334, FilterChainProxy$VirtualFilterChain
(org.springframework.security.web)
doFilter:74, CSRFSameOriginHeaderCheckFilter (ourcompany.common.web)
doFilter:334, FilterChainProxy$VirtualFilterChain
(org.springframework.security.web)
doFilter:36, SessionTrackingFilter (ourcompany.common.web)
doFilter:334, FilterChainProxy$VirtualFilterChain
(org.springframework.security.web)
doFilter:105, SecurityContextPersistenceFilter
(org.springframework.security.web.context)
doFilter:334, FilterChainProxy$VirtualFilterChain
(org.springframework.security.web)
doFilterInternal:56, WebAsyncManagerIntegrationFilter
(org.springframework.security.web.context.request.async)
doFilter:119, OncePerRequestFilter (org.springframework.web.filter)
doFilter:334, FilterChainProxy$VirtualFilterChain
(org.springframework.security.web)
doFilter:157, ChannelProcessingFilter
(org.springframework.security.web.access.channel)
doFilter:334, FilterChainProxy$VirtualFilterChain
(org.springframework.security.web)
doFilter:87, MetadataGeneratorFilter
(org.springframework.security.saml.metadata)
doFilter:334, FilterChainProxy$VirtualFilterChain
(org.springframework.security.web)
doFilterInternal:215, FilterChainProxy (org.springframework.security.web)
doFilter:178, FilterChainProxy (org.springframework.security.web)
invokeDelegate:358, DelegatingFilterProxy (org.springframework.web.filter)
doFilter:271, DelegatingFilterProxy (org.springframework.web.filter)
internalDoFilter:193, ApplicationFilterChain (org.apache.catalina.core)
doFilter:166, ApplicationFilterChain (org.apache.catalina.core)
invoke:202, StandardWrapperValve (org.apache.catalina.core)
__invoke:96, StandardContextValve (org.apache.catalina.core)
invoke:41002, StandardContextValve (org.apache.catalina.core)
invoke:541, AuthenticatorBase (org.apache.catalina.authenticator)
invoke:139, StandardHostValve (org.apache.catalina.core)
invoke:182, JvmRouteBinderValve (org.apache.catalina.ha.session)
invoke:330, ReplicationValve (org.apache.catalina.ha.tcp)
invoke:92, ErrorReportValve (org.apache.catalina.valves)
invoke:74, StandardEngineValve (org.apache.catalina.core)
invoke:747, RemoteIpValve (org.apache.catalina.valves)
service:343, CoyoteAdapter (org.apache.catalina.connector)
service:373, Http11Processor (org.apache.coyote.http11)
process:65, AbstractProcessorLight (org.apache.coyote)
process:868, AbstractProtocol$ConnectionHandler (org.apache.coyote)
doRun:1589, NioEndpoint$SocketProcessor (org.apache.tomcat.util.net)
run:49, SocketProcessorBase (org.apache.tomcat.util.net)
runWorker:1149, ThreadPoolExecutor (java.util.concurrent)
run:624, ThreadPoolExecutor$Worker (java.util.concurrent)
run:61, TaskThread$WrappingRunnable (org.apache.tomcat.util.threads)
run:748, Thread (java.lang)


The stack-trace for the second call is shown below:

add:678, ManagerBase (org.apache.catalina.session)
setId:358, StandardSession (org.apache.catalina.session)
setId:327, DeltaSession (org.apache.catalina.ha.session)
handleSESSION_CREATED:1322, DeltaManager (org.apache.catalina.ha.session)
messageReceived:1192, DeltaManager (org.apache.catalina.ha.session)
messageDataReceived:949, DeltaManager (org.apache.catalina.ha.session)
messageReceived:77, ClusterSessionListener (org.apache.catalina.ha.session)
messageReceived:788, SimpleTcpCluster (org.apache.catalina.ha.tcp)
messageReceived:771, SimpleTcpCluster (org.apache.catalina.ha.tcp)
messageReceived:335, GroupChannel (org.apache.catalina.tribes.group)
messageReceived:91, ChannelInterceptorBase (org.apache.catalina.tribes.group)
messageReceived:97, StaticMembershipInterceptor
(org.apache.catalina.tribes.group.interceptors)
messageReceived:91, ChannelInterceptorBase (org.apache.catalina.tribes.group)
messageReceived:175, TcpPingInterceptor
(org.apache.catalina.tribes.group.interceptors)
messageReceived:91, ChannelInterceptorBase (org.apache.catalina.tribes.group)
messageReceived:117, TcpFailureDetector
(org.apache.catalina.tribes.group.interceptors)
messageReceived:91, ChannelInterceptorBase (org.apache.catalina.tribes.group)
messageReceived:91, ChannelInterceptorBase (org.apache.catalina.tribes.group)
messageReceived:274, ChannelCoordinator (org.apache.catalina.tribes.group)
messageDataReceived:261, ReceiverBase (org.apache.catalina.tribes.transport)
drainChannel:216, NioReplicationTask (org.apache.catalina.tribes.transport.nio)
run:101, NioReplicationTask (org.apache.catalina.tribes.transport.nio)
runWorker:1149, ThreadPoolExecutor (java.util.concurrent)
run:624, ThreadPoolExecutor$Worker (java.util.concurrent)
run:748, Thread (java.lang)


Any help would be appreciated. I can replicate this every time and spend
some time investigating this.

Cheers,
Tim
Reply | Threaded
Open this post in threaded view
|

Re: Tomcat 9.0.37 Clustered DeltaManager Duplicates Session And Loses Session Attributes

markt
On 08/10/2020 10:04, Tim N wrote:

> Hi,
>
> I'm in the early stages of analysing this problem:
>
>    - Tomcat Embedded 9.0.37 with clustering enabled
>    - SpringBoot application, 2.1.16
>    - Login with no existing JSESSIONID fails
>
> I can see the following code is executed twice within one request, the
> login attempt:
>
> *~/.m2/repository/org/apache/tomcat/tomcat-catalina/9.0.37/tomcat-catalina-9.0.37-sources.jar!/org/apache/catalina/session/ManagerBase.java:677*
>
>     public void add(Session session) {
>         sessions.put(session.getIdInternal(), session);
>
> The first time it's executed, the session with the Spring context is added.
> The second time it's executed, a second session with the same ID, but
> without the Spring context, or any other session attribute I add for that
> matter, overwrites the existing session, and login fails. If I debug and
> prevent this by renaming the second session ID, login works because the
> original session is preserved.
>
> The stack-trace for the first call is shown below:
>
> add:678, ManagerBase (org.apache.catalina.session)
> setId:358, StandardSession (org.apache.catalina.session)
> setId:327, DeltaSession (org.apache.catalina.ha.session)
> setId:345, DeltaSession (org.apache.catalina.ha.session)
> createSession:719, ManagerBase (org.apache.catalina.session)
> createSession:422, DeltaManager (org.apache.catalina.ha.session)
> createSession:410, DeltaManager (org.apache.catalina.ha.session)
> doGetSession:3043, Request (org.apache.catalina.connector)
> getSession:2441, Request (org.apache.catalina.connector)

<snip/>

This is the app triggering the creation of the session.

> The stack-trace for the second call is shown below:
>
> add:678, ManagerBase (org.apache.catalina.session)
> setId:358, StandardSession (org.apache.catalina.session)
> setId:327, DeltaSession (org.apache.catalina.ha.session)
> handleSESSION_CREATED:1322, DeltaManager (org.apache.catalina.ha.session)
> messageReceived:1192, DeltaManager (org.apache.catalina.ha.session)

<snip/>

This is the DeltaManager receiving notification that a new session has
been created.

> Any help would be appreciated. I can replicate this every time and spend
> some time investigating this.

The new session created message should be send to (and then processed
on) every node *except* the node on which the session was originally
created.

Can you show us how you configured this cluster please?

Mark

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Tomcat 9.0.37 Clustered DeltaManager Duplicates Session And Loses Session Attributes

Tim N
> Can you show us how you configured this cluster please?

Sure.

Tomcat tomcat = new Tomcat();
tomcat.setBaseDir(baseDir);
tomcat.getServer().setAddress("127.0.0.1");
tomcat.getServer().setPort(shutDownPort);

tomcat.getServer().setShutdown("SHUTDOWN");
tomcat.getServer().addLifecycleListener(new AprLifecycleListener());
tomcat.getServer().addLifecycleListener(new JreMemoryLeakPreventionListener());
tomcat.getServer().addLifecycleListener(new GlobalResourcesLifecycleListener());
tomcat.getServer().addLifecycleListener(new
ThreadLocalLeakPreventionListener());

tomcat.setPort(httpPort);
tomcat.getConnector().getPort();//The default connector will only be
created if getConnector is called

LaunchServerHelper.addRemoteIpValve(tomcat);

//...

StandardHost hostManage = getStandardHost(tomcat, "manage." +
sgDomain, "sg-manage", serverProperties);
tomcat.getEngine().addChild(hostManage);

tomcat.start();
tomcat.getServer().await();
tomcat.stop();


private static StandardHost getStandardHost(Tomcat tomcat, String
name, String module, Properties serverProperties) {
  StandardHost host = new StandardHost();

  modifyHost(tomcat, name, module, host, serverProperties);
  return host;
}


private static void modifyHost(Tomcat tomcat, String name, String
module, StandardHost host, Properties serverProperties) {
  String webappDirLocation = module + "/src/main/webapp/";
  File webAppDir = new File(webappDirLocation);

  host.setName(name);
  String[] aliases = serverProperties.getProperty("tomcat-" + module +
"-aliases", "").split(",");
  for (String alias : aliases) {
    if (alias.length() > 0) {
      logger.log(Level.INFO, "Adding alias {0} for {1}", new Object[]
{alias, module});
      host.addAlias(alias);
    }
  }
  host.setParent(tomcat.getEngine());
  host.setAppBase(webAppDir.getAbsolutePath());

  StandardContext ctx = (StandardContext) tomcat.addWebapp(host, "/",
webAppDir.getAbsolutePath());
  ((StandardJarScanner) ctx.getJarScanner()).setScanManifest(false);
  ctx.setCookieProcessor(new LegacyCookieProcessor());

  // Declare an alternative location for your "WEB-INF/classes" dir
  // Servlet 3.0 annotation will work
  File additionWebInfClasses = new File(module + "/target/classes");
  WebResourceRoot resources = new StandardRoot(ctx);
  resources.addPreResources(new DirResourceSet(resources,
"/WEB-INF/classes", additionWebInfClasses.getAbsolutePath(), "/"));
  ctx.setResources(resources);

  LaunchServerHelper.addSimpleTcpCluster(host, 4000, serverProperties);
}


public static void addSimpleTcpCluster(StandardHost host, int port,
Properties serverProperties) {
  if (!serverProperties.containsKey("tomcat-clusterMemberCount")) {
    return;
  }

  SimpleTcpCluster cluster = new SimpleTcpCluster();
  cluster.setClusterName("SG-cluster");
  cluster.setChannelStartOptions(3);
  cluster.setChannelSendOptions(8);

  {
    DeltaManager clusterManager = new DeltaManager();
    clusterManager.setName("SG-cluster-manager");
    clusterManager.setExpireSessionsOnShutdown(false);
    clusterManager.setNotifyListenersOnReplication(true);
    cluster.registerManager(clusterManager);
  }

  {
    GroupChannel channel = new GroupChannel();
    channel.setName("SG-channel");
    {
      NioReceiver channelReceiver = new NioReceiver();
      channelReceiver.setAddress(serverProperties.getProperty("tomcat-clusterAddress"));
      channelReceiver.setPort(port);
      channelReceiver.setAutoBind(100);
      channelReceiver.setSelectorTimeout(5000);
      channelReceiver.setMaxThreads(6);
      channel.setChannelReceiver(channelReceiver);
    }
    cluster.setChannel(channel);

    {
      StaticMembershipInterceptor interceptor = new
StaticMembershipInterceptor();
      int clusterMemberCount =
Integer.parseInt(serverProperties.getProperty("tomcat-clusterMemberCount"));
      {
        StaticMember localMember = new StaticMember();
        localMember.setPort(port);
        localMember.setSecurePort(-1);
        localMember.setHost(serverProperties.getProperty("tomcat-clusterAddress"));
        localMember.setDomain("publish-cluster");
        localMember.setUniqueId(serverProperties.getProperty("tomcat-clusterMemberUniqueId"));
        interceptor.setLocalMember(localMember);
        interceptor.addStaticMember(localMember);
      }
      for (int i = 1; i <= clusterMemberCount; i++) {
        StaticMember member = new StaticMember();
        member.setLocal(false);
        member.setPort(port);
        member.setSecurePort(-1);
        member.setHost(serverProperties.getProperty("tomcat-clusterMemberAddress"
+ i));
        member.setDomain("publish-cluster");
        member.setUniqueId(serverProperties.getProperty("tomcat-clusterMemberUniqueId"
+ i));
        interceptor.addStaticMember(member);
      }
      channel.addInterceptor(interceptor);
    }
    channel.addInterceptor(new TcpPingInterceptor());
    channel.addInterceptor(new TcpFailureDetector());
    channel.addInterceptor(new MessageDispatchInterceptor());
  }


  ReplicationTransmitter replicationTransmitter = new ReplicationTransmitter();
  PooledParallelSender sender = new PooledParallelSender();
  sender.setTimeout(60000);
  sender.setMaxRetryAttempts(5);
  replicationTransmitter.setTransport(sender);

  ReplicationValve replicationValve = new ReplicationValve();
  replicationValve.setFilter("");
  cluster.addValve(replicationValve);
  cluster.addValve(new JvmRouteBinderValve());

  FarmWarDeployer farmDeployer = new FarmWarDeployer();
  farmDeployer.setTempDir("/tmp/war-temp/");
  farmDeployer.setDeployDir("/tmp/war-deploy/");
  farmDeployer.setWatchDir("/tmp/war-listen/");
  farmDeployer.setWatchEnabled(false);
  cluster.setClusterDeployer(farmDeployer);

  //cluster.addClusterListener(new
JvmRouteSessionIDBinderListener());//Removed in Tomcat 8.
https://tomcat.apache.org/migration-8.html#Clustering
  cluster.addClusterListener(new ClusterSessionListener());

  host.setCluster(cluster);
}


tomcat-clusterAddress=192.168.0.2
tomcat-clusterMemberUniqueId={0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1}
tomcat-clusterMemberCount=1
tomcat-clusterMemberAddress1=192.168.0.3
tomcat-clusterMemberUniqueId1={0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2}


This was initially converted from a server.xml configuration a year ago on
Tomcat 7 with minor modifications as we've upgraded to Tomcat 8 then 9.

Cheers,

Tim


On Thu, Oct 8, 2020 at 10:17 PM Mark Thomas <[hidden email]> wrote:

> On 08/10/2020 10:04, Tim N wrote:
> > Hi,
> >
> > I'm in the early stages of analysing this problem:
> >
> >    - Tomcat Embedded 9.0.37 with clustering enabled
> >    - SpringBoot application, 2.1.16
> >    - Login with no existing JSESSIONID fails
> >
> > I can see the following code is executed twice within one request, the
> > login attempt:
> >
> >
> *~/.m2/repository/org/apache/tomcat/tomcat-catalina/9.0.37/tomcat-catalina-9.0.37-sources.jar!/org/apache/catalina/session/ManagerBase.java:677*
> >
> >     public void add(Session session) {
> >         sessions.put(session.getIdInternal(), session);
> >
> > The first time it's executed, the session with the Spring context is
> added.
> > The second time it's executed, a second session with the same ID, but
> > without the Spring context, or any other session attribute I add for that
> > matter, overwrites the existing session, and login fails. If I debug and
> > prevent this by renaming the second session ID, login works because the
> > original session is preserved.
> >
> > The stack-trace for the first call is shown below:
> >
> > add:678, ManagerBase (org.apache.catalina.session)
> > setId:358, StandardSession (org.apache.catalina.session)
> > setId:327, DeltaSession (org.apache.catalina.ha.session)
> > setId:345, DeltaSession (org.apache.catalina.ha.session)
> > createSession:719, ManagerBase (org.apache.catalina.session)
> > createSession:422, DeltaManager (org.apache.catalina.ha.session)
> > createSession:410, DeltaManager (org.apache.catalina.ha.session)
> > doGetSession:3043, Request (org.apache.catalina.connector)
> > getSession:2441, Request (org.apache.catalina.connector)
>
> <snip/>
>
> This is the app triggering the creation of the session.
>
> > The stack-trace for the second call is shown below:
> >
> > add:678, ManagerBase (org.apache.catalina.session)
> > setId:358, StandardSession (org.apache.catalina.session)
> > setId:327, DeltaSession (org.apache.catalina.ha.session)
> > handleSESSION_CREATED:1322, DeltaManager (org.apache.catalina.ha.session)
> > messageReceived:1192, DeltaManager (org.apache.catalina.ha.session)
>
> <snip/>
>
> This is the DeltaManager receiving notification that a new session has
> been created.
>
> > Any help would be appreciated. I can replicate this every time and spend
> > some time investigating this.
>
> The new session created message should be send to (and then processed
> on) every node *except* the node on which the session was originally
> created.
>
> Can you show us how you configured this cluster please?
>
> Mark
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Tomcat 9.0.37 Clustered DeltaManager Duplicates Session And Loses Session Attributes

Tim N
Here's the two stack traces, in order of occurrence, for the DeltaSession
creation. The second seems to the result of a cluster message received
which seems odd on the machine where the session is being created:

<init>:109, DeltaSession (org.apache.catalina.ha.session)
createEmptySession:471, DeltaManager (org.apache.catalina.ha.session)
createSession:708, ManagerBase (org.apache.catalina.session)
createSession:422, DeltaManager (org.apache.catalina.ha.session)
createSession:410, DeltaManager (org.apache.catalina.ha.session)
doGetSession:3043, Request (org.apache.catalina.connector)
getSession:2441, Request (org.apache.catalina.connector)
getSession:908, RequestFacade (org.apache.catalina.connector)
getSession:920, RequestFacade (org.apache.catalina.connector)
getSession:253, HttpServletRequestWrapper (javax.servlet.http)
getSession:253, HttpServletRequestWrapper (javax.servlet.http)
getSession:253, HttpServletRequestWrapper (javax.servlet.http)
onAuthentication:66, RegisterSessionAuthenticationStrategy
(org.springframework.security.web.authentication.session)
doFilter:218, AbstractAuthenticationProcessingFilter
(org.springframework.security.web.authentication)
doFilter:334, FilterChainProxy$VirtualFilterChain
(org.springframework.security.web)
doFilter:200, AbstractAuthenticationProcessingFilter
(org.springframework.security.web.authentication)
doFilter:334, FilterChainProxy$VirtualFilterChain
(org.springframework.security.web)
doFilter:116, LogoutFilter
(org.springframework.security.web.authentication.logout)
doFilter:334, FilterChainProxy$VirtualFilterChain
(org.springframework.security.web)
doFilterInternal:74, HeaderWriterFilter
(org.springframework.security.web.header)
doFilter:119, OncePerRequestFilter (org.springframework.web.filter)
doFilter:334, FilterChainProxy$VirtualFilterChain
(org.springframework.security.web)
doFilter:74, CSRFSameOriginHeaderCheckFilter (ourcompany.common.web)
doFilter:334, FilterChainProxy$VirtualFilterChain
(org.springframework.security.web)
doFilter:36, SessionTrackingFilter (ourcompany.common.web)
doFilter:334, FilterChainProxy$VirtualFilterChain
(org.springframework.security.web)
doFilter:105, SecurityContextPersistenceFilter
(org.springframework.security.web.context)
doFilter:334, FilterChainProxy$VirtualFilterChain
(org.springframework.security.web)
doFilterInternal:56, WebAsyncManagerIntegrationFilter
(org.springframework.security.web.context.request.async)
doFilter:119, OncePerRequestFilter (org.springframework.web.filter)
doFilter:334, FilterChainProxy$VirtualFilterChain
(org.springframework.security.web)
doFilter:157, ChannelProcessingFilter
(org.springframework.security.web.access.channel)
doFilter:334, FilterChainProxy$VirtualFilterChain
(org.springframework.security.web)
doFilter:87, MetadataGeneratorFilter
(org.springframework.security.saml.metadata)
doFilter:334, FilterChainProxy$VirtualFilterChain
(org.springframework.security.web)
doFilterInternal:215, FilterChainProxy (org.springframework.security.web)
doFilter:178, FilterChainProxy (org.springframework.security.web)
invokeDelegate:358, DelegatingFilterProxy (org.springframework.web.filter)
doFilter:271, DelegatingFilterProxy (org.springframework.web.filter)
internalDoFilter:193, ApplicationFilterChain (org.apache.catalina.core)
doFilter:166, ApplicationFilterChain (org.apache.catalina.core)
invoke:202, StandardWrapperValve (org.apache.catalina.core)
__invoke:96, StandardContextValve (org.apache.catalina.core)
invoke:41002, StandardContextValve (org.apache.catalina.core)
invoke:541, AuthenticatorBase (org.apache.catalina.authenticator)
invoke:139, StandardHostValve (org.apache.catalina.core)
invoke:182, JvmRouteBinderValve (org.apache.catalina.ha.session)
invoke:330, ReplicationValve (org.apache.catalina.ha.tcp)
invoke:92, ErrorReportValve (org.apache.catalina.valves)
invoke:74, StandardEngineValve (org.apache.catalina.core)
invoke:747, RemoteIpValve (org.apache.catalina.valves)
service:343, CoyoteAdapter (org.apache.catalina.connector)
service:373, Http11Processor (org.apache.coyote.http11)
process:65, AbstractProcessorLight (org.apache.coyote)
process:868, AbstractProtocol$ConnectionHandler (org.apache.coyote)
doRun:1589, NioEndpoint$SocketProcessor (org.apache.tomcat.util.net)
run:49, SocketProcessorBase (org.apache.tomcat.util.net)
runWorker:1149, ThreadPoolExecutor (java.util.concurrent)
run:624, ThreadPoolExecutor$Worker (java.util.concurrent)
run:61, TaskThread$WrappingRunnable (org.apache.tomcat.util.threads)
run:748, Thread (java.lang)

...and the second session creation:

<init>:109, DeltaSession (org.apache.catalina.ha.session)
createEmptySession:471, DeltaManager (org.apache.catalina.ha.session)
handleSESSION_CREATED:1314, DeltaManager (org.apache.catalina.ha.session)
messageReceived:1192, DeltaManager (org.apache.catalina.ha.session)
messageDataReceived:949, DeltaManager (org.apache.catalina.ha.session)
messageReceived:77, ClusterSessionListener (org.apache.catalina.ha.session)
messageReceived:788, SimpleTcpCluster (org.apache.catalina.ha.tcp)
messageReceived:771, SimpleTcpCluster (org.apache.catalina.ha.tcp)
messageReceived:335, GroupChannel (org.apache.catalina.tribes.group)
messageReceived:91, ChannelInterceptorBase (org.apache.catalina.tribes.group)
messageReceived:97, StaticMembershipInterceptor
(org.apache.catalina.tribes.group.interceptors)
messageReceived:91, ChannelInterceptorBase (org.apache.catalina.tribes.group)
messageReceived:175, TcpPingInterceptor
(org.apache.catalina.tribes.group.interceptors)
messageReceived:91, ChannelInterceptorBase (org.apache.catalina.tribes.group)
messageReceived:117, TcpFailureDetector
(org.apache.catalina.tribes.group.interceptors)
messageReceived:91, ChannelInterceptorBase (org.apache.catalina.tribes.group)
messageReceived:91, ChannelInterceptorBase (org.apache.catalina.tribes.group)
messageReceived:274, ChannelCoordinator (org.apache.catalina.tribes.group)
messageDataReceived:261, ReceiverBase (org.apache.catalina.tribes.transport)
drainChannel:216, NioReplicationTask (org.apache.catalina.tribes.transport.nio)
run:101, NioReplicationTask (org.apache.catalina.tribes.transport.nio)
runWorker:1149, ThreadPoolExecutor (java.util.concurrent)
run:624, ThreadPoolExecutor$Worker (java.util.concurrent)
run:748, Thread (java.lang)



On Fri, Oct 9, 2020 at 11:40 AM Tim N <[hidden email]> wrote:

> > Can you show us how you configured this cluster please?
>
> Sure.
>
> Tomcat tomcat = new Tomcat();
> tomcat.setBaseDir(baseDir);
> tomcat.getServer().setAddress("127.0.0.1");
> tomcat.getServer().setPort(shutDownPort);
>
> tomcat.getServer().setShutdown("SHUTDOWN");
> tomcat.getServer().addLifecycleListener(new AprLifecycleListener());
> tomcat.getServer().addLifecycleListener(new JreMemoryLeakPreventionListener());
> tomcat.getServer().addLifecycleListener(new GlobalResourcesLifecycleListener());
> tomcat.getServer().addLifecycleListener(new ThreadLocalLeakPreventionListener());
>
> tomcat.setPort(httpPort);
> tomcat.getConnector().getPort();//The default connector will only be created if getConnector is called
>
> LaunchServerHelper.addRemoteIpValve(tomcat);
>
> //...
>
> StandardHost hostManage = getStandardHost(tomcat, "manage." + sgDomain, "sg-manage", serverProperties);
> tomcat.getEngine().addChild(hostManage);
>
> tomcat.start();
> tomcat.getServer().await();
> tomcat.stop();
>
>
> private static StandardHost getStandardHost(Tomcat tomcat, String name, String module, Properties serverProperties) {
>   StandardHost host = new StandardHost();
>
>   modifyHost(tomcat, name, module, host, serverProperties);
>   return host;
> }
>
>
> private static void modifyHost(Tomcat tomcat, String name, String module, StandardHost host, Properties serverProperties) {
>   String webappDirLocation = module + "/src/main/webapp/";
>   File webAppDir = new File(webappDirLocation);
>
>   host.setName(name);
>   String[] aliases = serverProperties.getProperty("tomcat-" + module + "-aliases", "").split(",");
>   for (String alias : aliases) {
>     if (alias.length() > 0) {
>       logger.log(Level.INFO, "Adding alias {0} for {1}", new Object[] {alias, module});
>       host.addAlias(alias);
>     }
>   }
>   host.setParent(tomcat.getEngine());
>   host.setAppBase(webAppDir.getAbsolutePath());
>
>   StandardContext ctx = (StandardContext) tomcat.addWebapp(host, "/", webAppDir.getAbsolutePath());
>   ((StandardJarScanner) ctx.getJarScanner()).setScanManifest(false);
>   ctx.setCookieProcessor(new LegacyCookieProcessor());
>
>   // Declare an alternative location for your "WEB-INF/classes" dir
>   // Servlet 3.0 annotation will work
>   File additionWebInfClasses = new File(module + "/target/classes");
>   WebResourceRoot resources = new StandardRoot(ctx);
>   resources.addPreResources(new DirResourceSet(resources, "/WEB-INF/classes", additionWebInfClasses.getAbsolutePath(), "/"));
>   ctx.setResources(resources);
>
>   LaunchServerHelper.addSimpleTcpCluster(host, 4000, serverProperties);
> }
>
>
> public static void addSimpleTcpCluster(StandardHost host, int port, Properties serverProperties) {
>   if (!serverProperties.containsKey("tomcat-clusterMemberCount")) {
>     return;
>   }
>
>   SimpleTcpCluster cluster = new SimpleTcpCluster();
>   cluster.setClusterName("SG-cluster");
>   cluster.setChannelStartOptions(3);
>   cluster.setChannelSendOptions(8);
>
>   {
>     DeltaManager clusterManager = new DeltaManager();
>     clusterManager.setName("SG-cluster-manager");
>     clusterManager.setExpireSessionsOnShutdown(false);
>     clusterManager.setNotifyListenersOnReplication(true);
>     cluster.registerManager(clusterManager);
>   }
>
>   {
>     GroupChannel channel = new GroupChannel();
>     channel.setName("SG-channel");
>     {
>       NioReceiver channelReceiver = new NioReceiver();
>       channelReceiver.setAddress(serverProperties.getProperty("tomcat-clusterAddress"));
>       channelReceiver.setPort(port);
>       channelReceiver.setAutoBind(100);
>       channelReceiver.setSelectorTimeout(5000);
>       channelReceiver.setMaxThreads(6);
>       channel.setChannelReceiver(channelReceiver);
>     }
>     cluster.setChannel(channel);
>
>     {
>       StaticMembershipInterceptor interceptor = new StaticMembershipInterceptor();
>       int clusterMemberCount = Integer.parseInt(serverProperties.getProperty("tomcat-clusterMemberCount"));
>       {
>         StaticMember localMember = new StaticMember();
>         localMember.setPort(port);
>         localMember.setSecurePort(-1);
>         localMember.setHost(serverProperties.getProperty("tomcat-clusterAddress"));
>         localMember.setDomain("publish-cluster");
>         localMember.setUniqueId(serverProperties.getProperty("tomcat-clusterMemberUniqueId"));
>         interceptor.setLocalMember(localMember);
>         interceptor.addStaticMember(localMember);
>       }
>       for (int i = 1; i <= clusterMemberCount; i++) {
>         StaticMember member = new StaticMember();
>         member.setLocal(false);
>         member.setPort(port);
>         member.setSecurePort(-1);
>         member.setHost(serverProperties.getProperty("tomcat-clusterMemberAddress" + i));
>         member.setDomain("publish-cluster");
>         member.setUniqueId(serverProperties.getProperty("tomcat-clusterMemberUniqueId" + i));
>         interceptor.addStaticMember(member);
>       }
>       channel.addInterceptor(interceptor);
>     }
>     channel.addInterceptor(new TcpPingInterceptor());
>     channel.addInterceptor(new TcpFailureDetector());
>     channel.addInterceptor(new MessageDispatchInterceptor());
>   }
>
>
>   ReplicationTransmitter replicationTransmitter = new ReplicationTransmitter();
>   PooledParallelSender sender = new PooledParallelSender();
>   sender.setTimeout(60000);
>   sender.setMaxRetryAttempts(5);
>   replicationTransmitter.setTransport(sender);
>
>   ReplicationValve replicationValve = new ReplicationValve();
>   replicationValve.setFilter("");
>   cluster.addValve(replicationValve);
>   cluster.addValve(new JvmRouteBinderValve());
>
>   FarmWarDeployer farmDeployer = new FarmWarDeployer();
>   farmDeployer.setTempDir("/tmp/war-temp/");
>   farmDeployer.setDeployDir("/tmp/war-deploy/");
>   farmDeployer.setWatchDir("/tmp/war-listen/");
>   farmDeployer.setWatchEnabled(false);
>   cluster.setClusterDeployer(farmDeployer);
>
>   //cluster.addClusterListener(new JvmRouteSessionIDBinderListener());//Removed in Tomcat 8. https://tomcat.apache.org/migration-8.html#Clustering
>   cluster.addClusterListener(new ClusterSessionListener());
>
>   host.setCluster(cluster);
> }
>
>
> tomcat-clusterAddress=192.168.0.2
> tomcat-clusterMemberUniqueId={0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1}
> tomcat-clusterMemberCount=1
> tomcat-clusterMemberAddress1=192.168.0.3
> tomcat-clusterMemberUniqueId1={0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2}
>
>
> This was initially converted from a server.xml configuration a year ago on
> Tomcat 7 with minor modifications as we've upgraded to Tomcat 8 then 9.
>
> Cheers,
>
> Tim
>
>
> On Thu, Oct 8, 2020 at 10:17 PM Mark Thomas <[hidden email]> wrote:
>
>> On 08/10/2020 10:04, Tim N wrote:
>> > Hi,
>> >
>> > I'm in the early stages of analysing this problem:
>> >
>> >    - Tomcat Embedded 9.0.37 with clustering enabled
>> >    - SpringBoot application, 2.1.16
>> >    - Login with no existing JSESSIONID fails
>> >
>> > I can see the following code is executed twice within one request, the
>> > login attempt:
>> >
>> >
>> *~/.m2/repository/org/apache/tomcat/tomcat-catalina/9.0.37/tomcat-catalina-9.0.37-sources.jar!/org/apache/catalina/session/ManagerBase.java:677*
>> >
>> >     public void add(Session session) {
>> >         sessions.put(session.getIdInternal(), session);
>> >
>> > The first time it's executed, the session with the Spring context is
>> added.
>> > The second time it's executed, a second session with the same ID, but
>> > without the Spring context, or any other session attribute I add for
>> that
>> > matter, overwrites the existing session, and login fails. If I debug and
>> > prevent this by renaming the second session ID, login works because the
>> > original session is preserved.
>> >
>> > The stack-trace for the first call is shown below:
>> >
>> > add:678, ManagerBase (org.apache.catalina.session)
>> > setId:358, StandardSession (org.apache.catalina.session)
>> > setId:327, DeltaSession (org.apache.catalina.ha.session)
>> > setId:345, DeltaSession (org.apache.catalina.ha.session)
>> > createSession:719, ManagerBase (org.apache.catalina.session)
>> > createSession:422, DeltaManager (org.apache.catalina.ha.session)
>> > createSession:410, DeltaManager (org.apache.catalina.ha.session)
>> > doGetSession:3043, Request (org.apache.catalina.connector)
>> > getSession:2441, Request (org.apache.catalina.connector)
>>
>> <snip/>
>>
>> This is the app triggering the creation of the session.
>>
>> > The stack-trace for the second call is shown below:
>> >
>> > add:678, ManagerBase (org.apache.catalina.session)
>> > setId:358, StandardSession (org.apache.catalina.session)
>> > setId:327, DeltaSession (org.apache.catalina.ha.session)
>> > handleSESSION_CREATED:1322, DeltaManager
>> (org.apache.catalina.ha.session)
>> > messageReceived:1192, DeltaManager (org.apache.catalina.ha.session)
>>
>> <snip/>
>>
>> This is the DeltaManager receiving notification that a new session has
>> been created.
>>
>> > Any help would be appreciated. I can replicate this every time and spend
>> > some time investigating this.
>>
>> The new session created message should be send to (and then processed
>> on) every node *except* the node on which the session was originally
>> created.
>>
>> Can you show us how you configured this cluster please?
>>
>> Mark
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>
Reply | Threaded
Open this post in threaded view
|

Re: Tomcat 9.0.37 Clustered DeltaManager Duplicates Session And Loses Session Attributes

Tim N
> The second seems to the result of a cluster message received which seems
odd on the machine where the session is being created

I think this was the issue. I've changed:

StaticMembershipInterceptor interceptor = new StaticMembershipInterceptor();
int clusterMemberCount =
Integer.parseInt(serverProperties.getProperty("tomcat-clusterMemberCount"));
{
  StaticMember localMember = new StaticMember();
  localMember.setPort(port);
  localMember.setSecurePort(-1);
  localMember.setHost(serverProperties.getProperty("tomcat-clusterAddress"));
  localMember.setDomain("publish-cluster");
  localMember.setUniqueId(serverProperties.getProperty("tomcat-clusterMemberUniqueId"));
  interceptor.setLocalMember(localMember);
  interceptor.addStaticMember(localMember);//Removed
}

...to...

StaticMembershipInterceptor interceptor = new StaticMembershipInterceptor();
int clusterMemberCount =
Integer.parseInt(serverProperties.getProperty("tomcat-clusterMemberCount"));
{
  StaticMember localMember = new StaticMember();
  localMember.setLocal(true);//Added
  localMember.setPort(port);
  localMember.setSecurePort(-1);
  localMember.setHost(serverProperties.getProperty("tomcat-clusterAddress"));
  localMember.setDomain("publish-cluster");
  localMember.setUniqueId(serverProperties.getProperty("tomcat-clusterMemberUniqueId"));
  interceptor.setLocalMember(localMember);
}

...and it seems to be fine now.

On Fri, Oct 9, 2020 at 12:26 PM Tim N <[hidden email]> wrote:

> Here's the two stack traces, in order of occurrence, for the DeltaSession
> creation. The second seems to the result of a cluster message received
> which seems odd on the machine where the session is being created:
>
> <init>:109, DeltaSession (org.apache.catalina.ha.session)
> createEmptySession:471, DeltaManager (org.apache.catalina.ha.session)
> createSession:708, ManagerBase (org.apache.catalina.session)
> createSession:422, DeltaManager (org.apache.catalina.ha.session)
> createSession:410, DeltaManager (org.apache.catalina.ha.session)
> doGetSession:3043, Request (org.apache.catalina.connector)
> getSession:2441, Request (org.apache.catalina.connector)
> getSession:908, RequestFacade (org.apache.catalina.connector)
> getSession:920, RequestFacade (org.apache.catalina.connector)
> getSession:253, HttpServletRequestWrapper (javax.servlet.http)
> getSession:253, HttpServletRequestWrapper (javax.servlet.http)
> getSession:253, HttpServletRequestWrapper (javax.servlet.http)
> onAuthentication:66, RegisterSessionAuthenticationStrategy (org.springframework.security.web.authentication.session)
> doFilter:218, AbstractAuthenticationProcessingFilter (org.springframework.security.web.authentication)
> doFilter:334, FilterChainProxy$VirtualFilterChain (org.springframework.security.web)
> doFilter:200, AbstractAuthenticationProcessingFilter (org.springframework.security.web.authentication)
> doFilter:334, FilterChainProxy$VirtualFilterChain (org.springframework.security.web)
> doFilter:116, LogoutFilter (org.springframework.security.web.authentication.logout)
> doFilter:334, FilterChainProxy$VirtualFilterChain (org.springframework.security.web)
> doFilterInternal:74, HeaderWriterFilter (org.springframework.security.web.header)
> doFilter:119, OncePerRequestFilter (org.springframework.web.filter)
> doFilter:334, FilterChainProxy$VirtualFilterChain (org.springframework.security.web)
> doFilter:74, CSRFSameOriginHeaderCheckFilter (ourcompany.common.web)
> doFilter:334, FilterChainProxy$VirtualFilterChain (org.springframework.security.web)
> doFilter:36, SessionTrackingFilter (ourcompany.common.web)
> doFilter:334, FilterChainProxy$VirtualFilterChain (org.springframework.security.web)
> doFilter:105, SecurityContextPersistenceFilter (org.springframework.security.web.context)
> doFilter:334, FilterChainProxy$VirtualFilterChain (org.springframework.security.web)
> doFilterInternal:56, WebAsyncManagerIntegrationFilter (org.springframework.security.web.context.request.async)
> doFilter:119, OncePerRequestFilter (org.springframework.web.filter)
> doFilter:334, FilterChainProxy$VirtualFilterChain (org.springframework.security.web)
> doFilter:157, ChannelProcessingFilter (org.springframework.security.web.access.channel)
> doFilter:334, FilterChainProxy$VirtualFilterChain (org.springframework.security.web)
> doFilter:87, MetadataGeneratorFilter (org.springframework.security.saml.metadata)
> doFilter:334, FilterChainProxy$VirtualFilterChain (org.springframework.security.web)
> doFilterInternal:215, FilterChainProxy (org.springframework.security.web)
> doFilter:178, FilterChainProxy (org.springframework.security.web)
> invokeDelegate:358, DelegatingFilterProxy (org.springframework.web.filter)
> doFilter:271, DelegatingFilterProxy (org.springframework.web.filter)
> internalDoFilter:193, ApplicationFilterChain (org.apache.catalina.core)
> doFilter:166, ApplicationFilterChain (org.apache.catalina.core)
> invoke:202, StandardWrapperValve (org.apache.catalina.core)
> __invoke:96, StandardContextValve (org.apache.catalina.core)
> invoke:41002, StandardContextValve (org.apache.catalina.core)
> invoke:541, AuthenticatorBase (org.apache.catalina.authenticator)
> invoke:139, StandardHostValve (org.apache.catalina.core)
> invoke:182, JvmRouteBinderValve (org.apache.catalina.ha.session)
> invoke:330, ReplicationValve (org.apache.catalina.ha.tcp)
> invoke:92, ErrorReportValve (org.apache.catalina.valves)
> invoke:74, StandardEngineValve (org.apache.catalina.core)
> invoke:747, RemoteIpValve (org.apache.catalina.valves)
> service:343, CoyoteAdapter (org.apache.catalina.connector)
> service:373, Http11Processor (org.apache.coyote.http11)
> process:65, AbstractProcessorLight (org.apache.coyote)
> process:868, AbstractProtocol$ConnectionHandler (org.apache.coyote)
> doRun:1589, NioEndpoint$SocketProcessor (org.apache.tomcat.util.net)
> run:49, SocketProcessorBase (org.apache.tomcat.util.net)
> runWorker:1149, ThreadPoolExecutor (java.util.concurrent)
> run:624, ThreadPoolExecutor$Worker (java.util.concurrent)
> run:61, TaskThread$WrappingRunnable (org.apache.tomcat.util.threads)
> run:748, Thread (java.lang)
>
> ...and the second session creation:
>
> <init>:109, DeltaSession (org.apache.catalina.ha.session)
> createEmptySession:471, DeltaManager (org.apache.catalina.ha.session)
> handleSESSION_CREATED:1314, DeltaManager (org.apache.catalina.ha.session)
> messageReceived:1192, DeltaManager (org.apache.catalina.ha.session)
> messageDataReceived:949, DeltaManager (org.apache.catalina.ha.session)
> messageReceived:77, ClusterSessionListener (org.apache.catalina.ha.session)
> messageReceived:788, SimpleTcpCluster (org.apache.catalina.ha.tcp)
> messageReceived:771, SimpleTcpCluster (org.apache.catalina.ha.tcp)
> messageReceived:335, GroupChannel (org.apache.catalina.tribes.group)
> messageReceived:91, ChannelInterceptorBase (org.apache.catalina.tribes.group)
> messageReceived:97, StaticMembershipInterceptor (org.apache.catalina.tribes.group.interceptors)
> messageReceived:91, ChannelInterceptorBase (org.apache.catalina.tribes.group)
> messageReceived:175, TcpPingInterceptor (org.apache.catalina.tribes.group.interceptors)
> messageReceived:91, ChannelInterceptorBase (org.apache.catalina.tribes.group)
> messageReceived:117, TcpFailureDetector (org.apache.catalina.tribes.group.interceptors)
> messageReceived:91, ChannelInterceptorBase (org.apache.catalina.tribes.group)
> messageReceived:91, ChannelInterceptorBase (org.apache.catalina.tribes.group)
> messageReceived:274, ChannelCoordinator (org.apache.catalina.tribes.group)
> messageDataReceived:261, ReceiverBase (org.apache.catalina.tribes.transport)
> drainChannel:216, NioReplicationTask (org.apache.catalina.tribes.transport.nio)
> run:101, NioReplicationTask (org.apache.catalina.tribes.transport.nio)
> runWorker:1149, ThreadPoolExecutor (java.util.concurrent)
> run:624, ThreadPoolExecutor$Worker (java.util.concurrent)
> run:748, Thread (java.lang)
>
>
>
> On Fri, Oct 9, 2020 at 11:40 AM Tim N <[hidden email]> wrote:
>
>> > Can you show us how you configured this cluster please?
>>
>> Sure.
>>
>> Tomcat tomcat = new Tomcat();
>> tomcat.setBaseDir(baseDir);
>> tomcat.getServer().setAddress("127.0.0.1");
>> tomcat.getServer().setPort(shutDownPort);
>>
>> tomcat.getServer().setShutdown("SHUTDOWN");
>> tomcat.getServer().addLifecycleListener(new AprLifecycleListener());
>> tomcat.getServer().addLifecycleListener(new JreMemoryLeakPreventionListener());
>> tomcat.getServer().addLifecycleListener(new GlobalResourcesLifecycleListener());
>> tomcat.getServer().addLifecycleListener(new ThreadLocalLeakPreventionListener());
>>
>> tomcat.setPort(httpPort);
>> tomcat.getConnector().getPort();//The default connector will only be created if getConnector is called
>>
>> LaunchServerHelper.addRemoteIpValve(tomcat);
>>
>> //...
>>
>> StandardHost hostManage = getStandardHost(tomcat, "manage." + sgDomain, "sg-manage", serverProperties);
>> tomcat.getEngine().addChild(hostManage);
>>
>> tomcat.start();
>> tomcat.getServer().await();
>> tomcat.stop();
>>
>>
>> private static StandardHost getStandardHost(Tomcat tomcat, String name, String module, Properties serverProperties) {
>>   StandardHost host = new StandardHost();
>>
>>   modifyHost(tomcat, name, module, host, serverProperties);
>>   return host;
>> }
>>
>>
>> private static void modifyHost(Tomcat tomcat, String name, String module, StandardHost host, Properties serverProperties) {
>>   String webappDirLocation = module + "/src/main/webapp/";
>>   File webAppDir = new File(webappDirLocation);
>>
>>   host.setName(name);
>>   String[] aliases = serverProperties.getProperty("tomcat-" + module + "-aliases", "").split(",");
>>   for (String alias : aliases) {
>>     if (alias.length() > 0) {
>>       logger.log(Level.INFO, "Adding alias {0} for {1}", new Object[] {alias, module});
>>       host.addAlias(alias);
>>     }
>>   }
>>   host.setParent(tomcat.getEngine());
>>   host.setAppBase(webAppDir.getAbsolutePath());
>>
>>   StandardContext ctx = (StandardContext) tomcat.addWebapp(host, "/", webAppDir.getAbsolutePath());
>>   ((StandardJarScanner) ctx.getJarScanner()).setScanManifest(false);
>>   ctx.setCookieProcessor(new LegacyCookieProcessor());
>>
>>   // Declare an alternative location for your "WEB-INF/classes" dir
>>   // Servlet 3.0 annotation will work
>>   File additionWebInfClasses = new File(module + "/target/classes");
>>   WebResourceRoot resources = new StandardRoot(ctx);
>>   resources.addPreResources(new DirResourceSet(resources, "/WEB-INF/classes", additionWebInfClasses.getAbsolutePath(), "/"));
>>   ctx.setResources(resources);
>>
>>   LaunchServerHelper.addSimpleTcpCluster(host, 4000, serverProperties);
>> }
>>
>>
>> public static void addSimpleTcpCluster(StandardHost host, int port, Properties serverProperties) {
>>   if (!serverProperties.containsKey("tomcat-clusterMemberCount")) {
>>     return;
>>   }
>>
>>   SimpleTcpCluster cluster = new SimpleTcpCluster();
>>   cluster.setClusterName("SG-cluster");
>>   cluster.setChannelStartOptions(3);
>>   cluster.setChannelSendOptions(8);
>>
>>   {
>>     DeltaManager clusterManager = new DeltaManager();
>>     clusterManager.setName("SG-cluster-manager");
>>     clusterManager.setExpireSessionsOnShutdown(false);
>>     clusterManager.setNotifyListenersOnReplication(true);
>>     cluster.registerManager(clusterManager);
>>   }
>>
>>   {
>>     GroupChannel channel = new GroupChannel();
>>     channel.setName("SG-channel");
>>     {
>>       NioReceiver channelReceiver = new NioReceiver();
>>       channelReceiver.setAddress(serverProperties.getProperty("tomcat-clusterAddress"));
>>       channelReceiver.setPort(port);
>>       channelReceiver.setAutoBind(100);
>>       channelReceiver.setSelectorTimeout(5000);
>>       channelReceiver.setMaxThreads(6);
>>       channel.setChannelReceiver(channelReceiver);
>>     }
>>     cluster.setChannel(channel);
>>
>>     {
>>       StaticMembershipInterceptor interceptor = new StaticMembershipInterceptor();
>>       int clusterMemberCount = Integer.parseInt(serverProperties.getProperty("tomcat-clusterMemberCount"));
>>       {
>>         StaticMember localMember = new StaticMember();
>>         localMember.setPort(port);
>>         localMember.setSecurePort(-1);
>>         localMember.setHost(serverProperties.getProperty("tomcat-clusterAddress"));
>>         localMember.setDomain("publish-cluster");
>>         localMember.setUniqueId(serverProperties.getProperty("tomcat-clusterMemberUniqueId"));
>>         interceptor.setLocalMember(localMember);
>>         interceptor.addStaticMember(localMember);
>>       }
>>       for (int i = 1; i <= clusterMemberCount; i++) {
>>         StaticMember member = new StaticMember();
>>         member.setLocal(false);
>>         member.setPort(port);
>>         member.setSecurePort(-1);
>>         member.setHost(serverProperties.getProperty("tomcat-clusterMemberAddress" + i));
>>         member.setDomain("publish-cluster");
>>         member.setUniqueId(serverProperties.getProperty("tomcat-clusterMemberUniqueId" + i));
>>         interceptor.addStaticMember(member);
>>       }
>>       channel.addInterceptor(interceptor);
>>     }
>>     channel.addInterceptor(new TcpPingInterceptor());
>>     channel.addInterceptor(new TcpFailureDetector());
>>     channel.addInterceptor(new MessageDispatchInterceptor());
>>   }
>>
>>
>>   ReplicationTransmitter replicationTransmitter = new ReplicationTransmitter();
>>   PooledParallelSender sender = new PooledParallelSender();
>>   sender.setTimeout(60000);
>>   sender.setMaxRetryAttempts(5);
>>   replicationTransmitter.setTransport(sender);
>>
>>   ReplicationValve replicationValve = new ReplicationValve();
>>   replicationValve.setFilter("");
>>   cluster.addValve(replicationValve);
>>   cluster.addValve(new JvmRouteBinderValve());
>>
>>   FarmWarDeployer farmDeployer = new FarmWarDeployer();
>>   farmDeployer.setTempDir("/tmp/war-temp/");
>>   farmDeployer.setDeployDir("/tmp/war-deploy/");
>>   farmDeployer.setWatchDir("/tmp/war-listen/");
>>   farmDeployer.setWatchEnabled(false);
>>   cluster.setClusterDeployer(farmDeployer);
>>
>>   //cluster.addClusterListener(new JvmRouteSessionIDBinderListener());//Removed in Tomcat 8. https://tomcat.apache.org/migration-8.html#Clustering
>>   cluster.addClusterListener(new ClusterSessionListener());
>>
>>   host.setCluster(cluster);
>> }
>>
>>
>> tomcat-clusterAddress=192.168.0.2
>> tomcat-clusterMemberUniqueId={0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1}
>> tomcat-clusterMemberCount=1
>> tomcat-clusterMemberAddress1=192.168.0.3
>> tomcat-clusterMemberUniqueId1={0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2}
>>
>>
>> This was initially converted from a server.xml configuration a year ago
>> on Tomcat 7 with minor modifications as we've upgraded to Tomcat 8 then 9.
>>
>> Cheers,
>>
>> Tim
>>
>>
>> On Thu, Oct 8, 2020 at 10:17 PM Mark Thomas <[hidden email]> wrote:
>>
>>> On 08/10/2020 10:04, Tim N wrote:
>>> > Hi,
>>> >
>>> > I'm in the early stages of analysing this problem:
>>> >
>>> >    - Tomcat Embedded 9.0.37 with clustering enabled
>>> >    - SpringBoot application, 2.1.16
>>> >    - Login with no existing JSESSIONID fails
>>> >
>>> > I can see the following code is executed twice within one request, the
>>> > login attempt:
>>> >
>>> >
>>> *~/.m2/repository/org/apache/tomcat/tomcat-catalina/9.0.37/tomcat-catalina-9.0.37-sources.jar!/org/apache/catalina/session/ManagerBase.java:677*
>>> >
>>> >     public void add(Session session) {
>>> >         sessions.put(session.getIdInternal(), session);
>>> >
>>> > The first time it's executed, the session with the Spring context is
>>> added.
>>> > The second time it's executed, a second session with the same ID, but
>>> > without the Spring context, or any other session attribute I add for
>>> that
>>> > matter, overwrites the existing session, and login fails. If I debug
>>> and
>>> > prevent this by renaming the second session ID, login works because the
>>> > original session is preserved.
>>> >
>>> > The stack-trace for the first call is shown below:
>>> >
>>> > add:678, ManagerBase (org.apache.catalina.session)
>>> > setId:358, StandardSession (org.apache.catalina.session)
>>> > setId:327, DeltaSession (org.apache.catalina.ha.session)
>>> > setId:345, DeltaSession (org.apache.catalina.ha.session)
>>> > createSession:719, ManagerBase (org.apache.catalina.session)
>>> > createSession:422, DeltaManager (org.apache.catalina.ha.session)
>>> > createSession:410, DeltaManager (org.apache.catalina.ha.session)
>>> > doGetSession:3043, Request (org.apache.catalina.connector)
>>> > getSession:2441, Request (org.apache.catalina.connector)
>>>
>>> <snip/>
>>>
>>> This is the app triggering the creation of the session.
>>>
>>> > The stack-trace for the second call is shown below:
>>> >
>>> > add:678, ManagerBase (org.apache.catalina.session)
>>> > setId:358, StandardSession (org.apache.catalina.session)
>>> > setId:327, DeltaSession (org.apache.catalina.ha.session)
>>> > handleSESSION_CREATED:1322, DeltaManager
>>> (org.apache.catalina.ha.session)
>>> > messageReceived:1192, DeltaManager (org.apache.catalina.ha.session)
>>>
>>> <snip/>
>>>
>>> This is the DeltaManager receiving notification that a new session has
>>> been created.
>>>
>>> > Any help would be appreciated. I can replicate this every time and
>>> spend
>>> > some time investigating this.
>>>
>>> The new session created message should be send to (and then processed
>>> on) every node *except* the node on which the session was originally
>>> created.
>>>
>>> Can you show us how you configured this cluster please?
>>>
>>> Mark
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: [hidden email]
>>> For additional commands, e-mail: [hidden email]
>>>
>>>
Reply | Threaded
Open this post in threaded view
|

Re: Tomcat 9.0.37 Clustered DeltaManager Duplicates Session And Loses Session Attributes

Christopher Schultz-2
Tim,

On 10/9/20 02:18, Tim N wrote:
>> The second seems to the result of a cluster message received which seems
>> odd on the machine where the session is being created

I was going to ask about that registration process. It looks like each
machine on the cluster registers every machine in the cluster *including
itself*.

If the current member thinks that there is another static member in the
cluster which is also itself, then it will send updates to itself which
... is not what you want.

> I think this was the issue. I've changed:
>
> StaticMembershipInterceptor interceptor = new StaticMembershipInterceptor();
> int clusterMemberCount =
> Integer.parseInt(serverProperties.getProperty("tomcat-clusterMemberCount"));
> {
>   StaticMember localMember = new StaticMember();
>   localMember.setPort(port);
>   localMember.setSecurePort(-1);
>   localMember.setHost(serverProperties.getProperty("tomcat-clusterAddress"));
>   localMember.setDomain("publish-cluster");
>   localMember.setUniqueId(serverProperties.getProperty("tomcat-clusterMemberUniqueId"));
>   interceptor.setLocalMember(localMember);
>   interceptor.addStaticMember(localMember);//Removed
> }
>
> ...to...
>
> StaticMembershipInterceptor interceptor = new StaticMembershipInterceptor();
> int clusterMemberCount =
> Integer.parseInt(serverProperties.getProperty("tomcat-clusterMemberCount"));
> {
>   StaticMember localMember = new StaticMember();
>   localMember.setLocal(true);//Added
>   localMember.setPort(port);
>   localMember.setSecurePort(-1);
>   localMember.setHost(serverProperties.getProperty("tomcat-clusterAddress"));
>   localMember.setDomain("publish-cluster");
>   localMember.setUniqueId(serverProperties.getProperty("tomcat-clusterMemberUniqueId"));
>   interceptor.setLocalMember(localMember);
> }
>
> ...and it seems to be fine now.

That would do it.

-chris

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]