Whitelabel Error Page

[ror6ax]

With following config:

{
  "id": "elasticsearch",
  "container": {
    "docker": {
      "image": "mesos/elasticsearch-scheduler",
      "network": "BRIDGE",
	  "parameters":[ 
       {
        "key": "publish",
        "value": "31100:31100"
      },
       {
        "key": "env",
        "value": "JAVA_OPTS=-Xms1024m -Xmx2048m"
      },
       {
        "key": "env",
        "value": "ES_JAVA_OPTS=-Xms1024m -Xmx2048m"
      }
      ],
      "privileged": true
    }
  },
  "args": ["--zookeeperMesosUrl", "zk://master.mesos:2181/mesos"],
  "cpus": 1,
  "mem": 2048.0,
  "env": {
	"JAVA_OPTS":"-Xms1024m -Xmx2048m",
	"ES_JAVA_OPTS":"-Xms1024m -Xmx2048m"
	},
  "instances": 1
}

There is lots of following errors:

http://hostname-slave-1:31100/v1/es/_cluster/stats

Whitelabel Error Page
This application has no explicit mapping for /error, so you are seeing this as a fallback.
Fri Mar 10 12:14:40 UTC 2017
There was an unexpected error (type=Internal Server Error, status=500).
No value present

How do I debug this?

[ror6ax]

Also, a side question. What do you mean by this:

That the slaves have routable IP addresses. The ES ports are exposed on the slaves, so that the ES cluster can discover each other. Please double check that your slaves are routable.

My cluster uses mesos-dns. Nodes therefore have unique hostnames and are aware of each other. Is this considered routable?

[ror6ax]

[ERROR] 2017-03-10 12:32:03,960 org.apache.catalina.core.ContainerBase.[Tomcat].[localhost].[/].[dispatcherServlet] log - Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is java.util.NoSuchElementException: No value present] with root cause
java.util.NoSuchElementException: No value present
	at java.util.Optional.get(Optional.java:135)
	at org.apache.mesos.elasticsearch.scheduler.controllers.SearchProxyController.stats(SearchProxyController.java:41)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221)
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:137)
	at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:110)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:776)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:705)
	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:959)
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:893)
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:967)
	at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:858)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:622)
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:843)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:291)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:77)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:85)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:142)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:518)
	at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1091)
	at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:668)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1521)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1478)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:745)

[ror6ax]

I noticed that tasks endpoint returns []. I'm not a Java expert, but it seems this error results from tasks being empty.

[philwinder]

@ror6ax Hi there.
I'm reasonably confident that the issues you are seeing are due to the fact that you are running the scheduler in bridge mode. I think that the scheduler is on its own network. Both you and the other ES containers won't be able to route to the scheduler (without some sort of network overlay).

Try changing to run in HOST mode to see if that works?

[ror6ax]

I've changed to HOST and provided --elasticsearchSettingsLocation, otherwise there is lookup error on it.,
Now, executors still fail with:

Starting task elasticsearch_some-slave-6_20170313T102134.658Z
Error: encountered environment variables that are no longer supported
Use jvm.options or ES_JAVA_OPTS to configure the JVM
ES_HEAP_SIZE=192m: set -Xms192m and -Xmx192m in jvm.options or add "-Xms192m -Xmx192m" to ES_JAVA_OPTS

Why is this coming up? I've tried all possible combinations of JVM settings, but it's still there...

[philwinder]

Probably because ES have changed their environmental variable API.
Take a look at the code here
And see if any of the env vars are no longer compatible.