[Bug] Scan stops when request to metadata provider fails

[bonny1992]

RomM version
4.0.1 / Docker

Describe the bug
When scanning, if a request to Hasheous and/or RetroAchievement fails (not sure about others), the entire scan will stop. At least, in my instance.

To Reproduce
Steps to reproduce the behavior:

1. Scan ("Quick scan" but shouldn't matter)

Expected behavior
I'd expect the software to try some times (maybe an environment variable setting?) before interrupting the scan.

Screenshots
No screenshot but I have the part of the logs:

INFO:     [RomM][hasheous_handler][2025-08-08 16:56:37] No IGDB ID provided for Hasheous IGDB game lookup.
CRITICAL: [RomM][retroachievements][2025-08-08 16:56:42] Connection error: can't connect to RetroAchievements
Traceback (most recent call last):
  File "/src/.venv/lib/python3.13/site-packages/aiohttp/connector.py", line 1532, in _create_direct_connection
    hosts = await self._resolve_host(host, port, traces=traces)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/src/.venv/lib/python3.13/site-packages/aiohttp/connector.py", line 1148, in _resolve_host
    return await asyncio.shield(resolved_host_task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/src/.venv/lib/python3.13/site-packages/aiohttp/connector.py", line 1179, in _resolve_host_with_throttle
    addrs = await self._resolver.resolve(host, port, family=self._family)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/src/.venv/lib/python3.13/site-packages/aiohttp/resolver.py", line 40, in resolve
    infos = await self._loop.getaddrinfo(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<5 lines>...
    )
    ^
  File "/usr/local/lib/python3.13/asyncio/base_events.py", line 939, in getaddrinfo
    return await self.run_in_executor(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
        None, getaddr_func, host, port, family, type, proto, flags)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/concurrent/futures/thread.py", line 59, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.13/socket.py", line 977, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
               ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
socket.gaierror: [Errno -3] Try again

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/backend/adapters/services/retroachievements.py", line 54, in _request
    res = await aiohttp_session.get(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<3 lines>...
    )
    ^
  File "/src/.venv/lib/python3.13/site-packages/aiohttp/client.py", line 770, in _request
    resp = await handler(req)
           ^^^^^^^^^^^^^^^^^^
  File "/src/.venv/lib/python3.13/site-packages/aiohttp/client_middlewares.py", line 36, in single_middleware_handler
    return await middleware(req, handler)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/backend/adapters/services/retroachievements.py", line 31, in auth_middleware
    return await handler(req)
           ^^^^^^^^^^^^^^^^^^
  File "/src/.venv/lib/python3.13/site-packages/aiohttp/client.py", line 725, in _connect_and_send_request
    conn = await self._connector.connect(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        req, traces=traces, timeout=real_timeout
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/src/.venv/lib/python3.13/site-packages/aiohttp/connector.py", line 642, in connect
    proto = await self._create_connection(req, traces, timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/src/.venv/lib/python3.13/site-packages/aiohttp/connector.py", line 1209, in _create_connection
    _, proto = await self._create_direct_connection(req, traces, timeout)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/src/.venv/lib/python3.13/site-packages/aiohttp/connector.py", line 1538, in _create_direct_connection
    raise ClientConnectorDNSError(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorDNSError: Cannot connect to host retroachievements.org:443 ssl:default [Try again]
ERROR:    [RomM][scan][2025-08-08 16:56:42] Error in scan_platform: 503: Can't connect to RetroAchievements, check your internet connection
INFO:     [RomM][nginx][2025-08-08 16:56:42] 192.168.178.2 | - | GET /ws/socket.io/?EIO=4&transport=websocket 101 | 182704 | Chrome Windows | 2266.684
ERROR:    [RomM][worker][2025-08-08 16:56:42] [Job d2c257c1-85b4-4bf8-b60b-6197f9833eef]: exception raised while executing (endpoints.sockets.scan.scan_platforms)
Traceback (most recent call last):
  File "/src/.venv/lib/python3.13/site-packages/aiohttp/connector.py", line 1532, in _create_direct_connection
    hosts = await self._resolve_host(host, port, traces=traces)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/src/.venv/lib/python3.13/site-packages/aiohttp/connector.py", line 1148, in _resolve_host
    return await asyncio.shield(resolved_host_task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/src/.venv/lib/python3.13/site-packages/aiohttp/connector.py", line 1179, in _resolve_host_with_throttle
    addrs = await self._resolver.resolve(host, port, family=self._family)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/src/.venv/lib/python3.13/site-packages/aiohttp/resolver.py", line 40, in resolve
    infos = await self._loop.getaddrinfo(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<5 lines>...
    )
    ^
  File "/usr/local/lib/python3.13/asyncio/base_events.py", line 939, in getaddrinfo
    return await self.run_in_executor(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
        None, getaddr_func, host, port, family, type, proto, flags)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/concurrent/futures/thread.py", line 59, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.13/socket.py", line 977, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
               ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
socket.gaierror: [Errno -3] Try again

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/backend/adapters/services/retroachievements.py", line 54, in _request
    res = await aiohttp_session.get(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<3 lines>...
    )
    ^
  File "/src/.venv/lib/python3.13/site-packages/aiohttp/client.py", line 770, in _request
    resp = await handler(req)
           ^^^^^^^^^^^^^^^^^^
  File "/src/.venv/lib/python3.13/site-packages/aiohttp/client_middlewares.py", line 36, in single_middleware_handler
    return await middleware(req, handler)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/backend/adapters/services/retroachievements.py", line 31, in auth_middleware
    return await handler(req)
           ^^^^^^^^^^^^^^^^^^
  File "/src/.venv/lib/python3.13/site-packages/aiohttp/client.py", line 725, in _connect_and_send_request
    conn = await self._connector.connect(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        req, traces=traces, timeout=real_timeout
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/src/.venv/lib/python3.13/site-packages/aiohttp/connector.py", line 642, in connect
    proto = await self._create_connection(req, traces, timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/src/.venv/lib/python3.13/site-packages/aiohttp/connector.py", line 1209, in _create_connection
    _, proto = await self._create_direct_connection(req, traces, timeout)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/src/.venv/lib/python3.13/site-packages/aiohttp/connector.py", line 1538, in _create_direct_connection
    raise ClientConnectorDNSError(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorDNSError: Cannot connect to host retroachievements.org:443 ssl:default [Try again]

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/src/.venv/lib/python3.13/site-packages/rq/worker.py", line 1643, in perform_job
    return_value = job.perform()
  File "/src/.venv/lib/python3.13/site-packages/rq/job.py", line 1359, in perform
    self._result = self._execute()
                   ~~~~~~~~~~~~~^^
  File "/src/.venv/lib/python3.13/site-packages/rq/job.py", line 1396, in _execute
    coro_result = loop.run_until_complete(result)
  File "/usr/local/lib/python3.13/asyncio/base_events.py", line 725, in run_until_complete
    return future.result()
           ~~~~~~~~~~~~~^^
  File "/usr/local/lib/python3.13/contextlib.py", line 101, in inner
    return await func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/backend/endpoints/sockets/scan.py", line 517, in scan_platforms
    raise e
  File "/backend/endpoints/sockets/scan.py", line 492, in scan_platforms
    scan_stats += await _identify_platform(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<6 lines>...
    )
    ^
  File "/backend/endpoints/sockets/scan.py", line 402, in _identify_platform
    scan_stats += await _identify_rom(
                  ^^^^^^^^^^^^^^^^^^^^
    ...<7 lines>...
    )
    ^
  File "/backend/endpoints/sockets/scan.py", line 216, in _identify_rom
    scanned_rom = await scan_rom(
                  ^^^^^^^^^^^^^^^
    ...<6 lines>...
    )
    ^
  File "/backend/handler/scan_handler.py", line 501, in scan_rom
    ) = await asyncio.gather(
        ^^^^^^^^^^^^^^^^^^^^^
    ...<6 lines>...
    )
    ^
  File "/backend/handler/scan_handler.py", line 458, in fetch_ra_rom
    return await meta_ra_handler.get_rom_by_id(rom=rom, ra_id=h_ra_id)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/backend/handler/metadata/ra_handler.py", line 240, in get_rom_by_id
    rom_details = await self.ra_service.get_game_extended_details(ra_id)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/backend/adapters/services/retroachievements.py", line 118, in get_game_extended_details
    response = await self._request(str(url))
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/backend/adapters/services/retroachievements.py", line 68, in _request
    raise HTTPException(
    ...<2 lines>...
    ) from exc
fastapi.exceptions.HTTPException: 503: Can't connect to RetroAchievements, check your internet connection

INFO:     [RomM][server][2025-08-08 16:56:42] connection closed

Desktop (please complete the following information):

Not needed

Smartphone (please complete the following information):

Not needed

Server (I've added this)
It's Docker v28.3.3 + Docker Compose v2.39.1 on Ubuntu 24.04.2 LTS virtualized in Proxmox.

Additional context
Maybe it scans 400 roms and then stops, or scans 2000 then stops, or 10 then stops.

[bonny1992]

Happens with SteamGridDB too.

ERROR:    [RomM][scan][2025-08-08 22:48:45] Error in scan_platform: Cannot connect to host steamgriddb.com:443 ssl:default [Try again]
INFO:     [RomM][nginx][2025-08-08 22:48:45] 192.168.178.2 | - | GET /ws/socket.io/?EIO=4&transport=websocket 101 | 287154 | Chrome Windows | 4881.225
INFO:     [RomM][server][2025-08-08 22:48:45] connection closed
ERROR:    [RomM][worker][2025-08-08 22:48:45] [Job e74b3c79-8c1d-4342-8b55-62968b8f2e00]: exception raised while executing (endpoints.sockets.scan.scan_platforms)
Traceback (most recent call last):
  File "/src/.venv/lib/python3.13/site-packages/aiohttp/connector.py", line 1532, in _create_direct_connection
    hosts = await self._resolve_host(host, port, traces=traces)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/src/.venv/lib/python3.13/site-packages/aiohttp/connector.py", line 1148, in _resolve_host
    return await asyncio.shield(resolved_host_task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/src/.venv/lib/python3.13/site-packages/aiohttp/connector.py", line 1179, in _resolve_host_with_throttle
    addrs = await self._resolver.resolve(host, port, family=self._family)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/src/.venv/lib/python3.13/site-packages/aiohttp/resolver.py", line 40, in resolve
    infos = await self._loop.getaddrinfo(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<5 lines>...
    )
    ^
  File "/usr/local/lib/python3.13/asyncio/base_events.py", line 939, in getaddrinfo
    return await self.run_in_executor(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
        None, getaddr_func, host, port, family, type, proto, flags)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/concurrent/futures/thread.py", line 59, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.13/socket.py", line 977, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
               ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
socket.gaierror: [Errno -3] Try again

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/src/.venv/lib/python3.13/site-packages/rq/worker.py", line 1643, in perform_job
    return_value = job.perform()
  File "/src/.venv/lib/python3.13/site-packages/rq/job.py", line 1359, in perform
    self._result = self._execute()
                   ~~~~~~~~~~~~~^^
  File "/src/.venv/lib/python3.13/site-packages/rq/job.py", line 1396, in _execute
    coro_result = loop.run_until_complete(result)
  File "/usr/local/lib/python3.13/asyncio/base_events.py", line 725, in run_until_complete
    return future.result()
           ~~~~~~~~~~~~~^^
  File "/usr/local/lib/python3.13/contextlib.py", line 101, in inner
    return await func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/backend/endpoints/sockets/scan.py", line 517, in scan_platforms
    raise e
  File "/backend/endpoints/sockets/scan.py", line 492, in scan_platforms
    scan_stats += await _identify_platform(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<6 lines>...
    )
    ^
  File "/backend/endpoints/sockets/scan.py", line 402, in _identify_platform
    scan_stats += await _identify_rom(
                  ^^^^^^^^^^^^^^^^^^^^
    ...<7 lines>...
    )
    ^
  File "/backend/endpoints/sockets/scan.py", line 216, in _identify_rom
    scanned_rom = await scan_rom(
                  ^^^^^^^^^^^^^^^
    ...<6 lines>...
    )
    ^
  File "/backend/handler/scan_handler.py", line 602, in scan_rom
    sgdb_hander_rom = await fetch_sgdb_details()
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/backend/handler/scan_handler.py", line 598, in fetch_sgdb_details
    return await meta_sgdb_handler.get_details_by_names(game_names)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/backend/handler/metadata/sgdb_handler.py", line 60, in get_details_by_names
    games = await self.sgdb_service.search_games(term=search_term)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/backend/adapters/services/steamgriddb.py", line 178, in search_games
    response = await self._request(str(url))
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/backend/adapters/services/steamgriddb.py", line 55, in _request
    res = await aiohttp_session.get(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<3 lines>...
    )
    ^
  File "/src/.venv/lib/python3.13/site-packages/aiohttp/client.py", line 770, in _request
    resp = await handler(req)
           ^^^^^^^^^^^^^^^^^^
  File "/src/.venv/lib/python3.13/site-packages/aiohttp/client_middlewares.py", line 36, in single_middleware_handler
    return await middleware(req, handler)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/backend/adapters/services/steamgriddb.py", line 32, in auth_middleware
    return await handler(req)
           ^^^^^^^^^^^^^^^^^^
  File "/src/.venv/lib/python3.13/site-packages/aiohttp/client.py", line 725, in _connect_and_send_request
    conn = await self._connector.connect(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        req, traces=traces, timeout=real_timeout
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/src/.venv/lib/python3.13/site-packages/aiohttp/connector.py", line 642, in connect
    proto = await self._create_connection(req, traces, timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/src/.venv/lib/python3.13/site-packages/aiohttp/connector.py", line 1209, in _create_connection
    _, proto = await self._create_direct_connection(req, traces, timeout)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/src/.venv/lib/python3.13/site-packages/aiohttp/connector.py", line 1538, in _create_direct_connection
    raise ClientConnectorDNSError(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorDNSError: Cannot connect to host steamgriddb.com:443 ssl:default [Try again]

INFO:     [RomM][worker][2025-08-08 22:48:45] Cleaning registries for queue: high
...

[gantoine]

Yeah it's expected and happens anytime we can't reach a metadata provider. We're planning to improve this soon, will backlog it in the meantime.

[bonny1992]

Yeah it's expected and happens anytime we can't reach a metadata provider. We're planning to improve this soon, will backlog it in the meantime.

Hey thanks for the fast reply!

I understand of course, it's just a bit annoying when there's thousands of entries to scan.

Just as a workaround, do we have some API available?
We could check with a script if the last scan ended with an error and restart it.

[gantoine]

No API as it's a websocket, but can craft something that connects to the websocket, fires a scan and listens for failures.

[bonny1992]

No API as it's a websocket, but can craft something that connects to the websocket, fires a scan and listens for failures.

That would be awesome!
We could also add some "checks" (for example: is my connection really down? Pings google.com or whatever - so it won't just fire scans ad nauseam) too!

[gantoine]

Oh sorry I typo'd, we're gonna fix this internally eventually, but I'm saying for now you can probably vibe code something that connects to the same websocket endpoint and reacts on scan failure events.

[bonny1992]

Oh sorry I typo'd, we're gonna fix this internally eventually, but I'm saying for now you can probably vibe code something that connects to the same websocket endpoint and reacts on scan failure events.

Oh sure I'll see what I can do :)
Ty in the meanwhile!

[bonny1992]

Oh sorry I typo'd, we're gonna fix this internally eventually, but I'm saying for now you can probably vibe code something that connects to the same websocket endpoint and reacts on scan failure events.

You know what? I had nothing to do yesterday so I actually vibe coded it haha.

Here's the result:

ERROR:    [RomM][scan-monitor][2025-08-10 15:28:06] Scan failed: 503: Can't connect to RetroAchievements, check your internet connection
WARNING:  [RomM][scan-monitor][2025-08-10 15:28:06] Connection error detected (1/1)
INFO:     [RomM][scan-monitor][2025-08-10 15:28:06] Internet check skipped, restarting scan immediately...
INFO:     [RomM][scan-monitor][2025-08-10 15:28:06] Restarting scan with 9 platforms and 5 metadata sources
INFO:     [RomM][scan-monitor][2025-08-10 15:28:06] Scan restart command sent via websocket

https://gist.github.com/bonny1992/0ec33191233783bd4585946b89dadc16

I might package it into a docker image so it can be deployed alongside Romm for the time being, until the issue is solved internally.

[slimshizn]

Been seeing this between Streamgrid and Hasheous. Hopefully a fix can be deployed soon as full scans are impossible currently. Especially with a larger library.

[bonny1992]

Been seeing this between Streamgrid and Hasheous. Hopefully a fix can be deployed soon as full scans are impossible currently. Especially with a larger library.

So far I managed to finish a full scan (well, it was multiple but with no interaction from my end, more than 18k entries) with my script.

One can start it in a tmux or screen session and be somewhat fine for the time being.