ERROR [HashedWheelTimer.reportTooManyInstances] - You are creating too many HashedWheelTimer instances. HashedWheelTimer is a shared resource that must be reused across the JVM,so that only a few instances are created.

[mcopjan]

🐛 Bug Report

4.0.0-beta-1-prerelease-20210114
I have a grid4 with 20 chrome nodes (SE_NODE_MAX_CONCURRENT_SESSIONS=5), when running 50 (max) headless tests in parallel I can see hub dying with the error above. When I stopped execution I could see that the hub recovered after some time.

selenium_hub_logs.txt.zip

version: '3.7'

services:

  hub:
    image: selenium/hub:4.0.0-beta-1-prerelease-20210114
    ports:
      - "4442:4442"
      - "4443:4443"
      - "4444:4444"
    deploy:
      placement:
        constraints:
          - node.role == manager
      restart_policy:
        condition: on-failure

  chrome:
    image: selenium/node-chrome:4.0.0-beta-1-prerelease-20210114
    volumes:
      - /dev/shm:/dev/shm
    depends_on:
      - hub
    environment:
      - SE_EVENT_BUS_HOST=hub
      - SE_EVENT_BUS_PUBLISH_PORT=4442
      - SE_EVENT_BUS_SUBSCRIBE_PORT=4443
      - LANG=en_GB.UTF-8
      - SE_NODE_MAX_CONCURRENT_SESSIONS=5
      - START_XVFB=false
      - SCREEN_WIDTH=1920
      - SCREEN_HEIGHT=1080
      - SCREEN_DEPTH=24
    entrypoint: bash -c 'SE_OPTS="--host $$HOSTNAME" /opt/bin/entry_point.sh'
    deploy:
      replicas: 20
      placement:
        constraints:
          - node.role == worker
      restart_policy:
        condition: on-failure

[mcopjan]

I would like to point out that when are using selenium 3 containers running 80-100 tests in parallel on 20 chrome nodes with maxSessions=5 (hence 100 browser instances), there are no problems.

[mcopjan]

One more thing I noticed, when using the compose file above I would expect 100 instances being registered with the hub (20 chrome replicas, 5 chrome instances each) but graphql shows me totalSlots:80 yet all chrome containers seem to be healthy.

[mcopjan]

SeleniumHQ/docker-selenium#1183 (comment)

[dylanlive]

Replicating on selenium-server-4.0.0-prerelease-beta-1-02d5e641d5.jar as a Hub. Didn't even run any sessions. Just had 1 node registered.

00:58:19.779 INFO [LogManager$RootLogger.log] - Using the system default encoding
00:58:19.786 INFO [LoggingOptions.createTracer] - Using OpenTelemetry for tracing
00:58:20.030 INFO [BoundZmqEventBus.<init>] - XPUB binding to [binding to tcp://*:4442, advertising as tcp://172.17.0.3:4442], XSUB binding to [binding to tcp://*:4443, advertising as tcp://172.17.0.3:4443]
00:58:20.212 INFO [UnboundZmqEventBus.<init>] - Connecting to tcp://172.17.0.3:4442 and tcp://172.17.0.3:4443
00:58:20.284 INFO [UnboundZmqEventBus.<init>] - Sockets created
00:58:20.290 INFO [UnboundZmqEventBus.lambda$new$7] - Bus started
00:58:20.490 INFO [BoundZmqEventBus.<init>] - Event bus ready
00:58:22.430 INFO [Hub.execute] - Started Selenium hub 4.0.0-beta-1 (revision 02d5e641d5): http://172.17.0.3:4444
01:00:03.699 INFO [Node.<init>] - Binding additional locator mechanisms: name, id
01:00:04.416 INFO [LocalDistributor.add] - Added node 9dbf56e3-2d42-4f41-9277-1653beac329e at http://1.2.3.4:5555.
01:00:04.735 INFO [GridModel.setAvailability] - Switching node 9dbf56e3-2d42-4f41-9277-1653beac329e (uri: http://1.2.3.4:5555) from DOWN to UP
01:02:41.748 ERROR [HashedWheelTimer.reportTooManyInstances] - You are creating too many HashedWheelTimer instances. HashedWheelTimer is a sharedresource that must be reused across the JVM,so that only a few instances are created.
01:05:14.752 INFO [GridModel.setAvailability] - Switching node 9dbf56e3-2d42-4f41-9277-1653beac329e (uri: http://1.2.3.4:5555) from UP to DOWN
Exception in thread "AsyncHttpClient-58-1" java.lang.OutOfMemoryError: Java heap space
Exception in thread "AsyncHttpClient-592-1" java.lang.OutOfMemoryError: Java heap space
03:35:57.062 WARN [HashedWheelTimer$HashedWheelTimeout.expire] - An exception was thrown by TimerTask.
java.lang.OutOfMemoryError: Java heap space

[mcopjan]

Also reported here SeleniumHQ/docker-selenium#1201

[diemol]

@dylanlive what hardware/OS combination were you using to get that?

[dylanlive]

@diemol Within docker (not docker-selenium), running Ubuntu Bionic & openjdk-8. I didn't think it was dependent to Docker, but I surprisingly couldn't repo this with just the Jar running on Amazon Linux 2 & Corretto.

I'm going to experiment more with this in the coming 1-2 weeks, starting with pulling the latest pre-release. I will share updates. If this looks dependent to Docker, perhaps we should move to SeleniumHQ/docker-selenium#1201

[dylanlive]

Happy to share I've been able to come up with a solid replication on pre-release aec4c8c2c5, and this is replicable outside of Docker.

The issue is occurring for me when:

1. I have at least 1 node connected to the hub
2. I make frequent http calls to /status

The reason it took me awhile to figure this out is because the instance had an ELB attached to it, which was calling /status as its health check. @mcopjan I'm curious if you also have health checks occurring on /status

To replicate:

1. Run Hub: java -jar selenium-server-4.0.0-prerelease-beta-1-aec4c8c2c5.jar hub --log-level info
2. Connect a node to the hub
3. Run this bash script (replacing the IP with your hub's ip) to make repeated http calls

#!/bin/bash

for i in {1..200}
do
   curl "REPLACE_IP:4444/status"
   date +"%T"
   sleep 1
done