Idom backend crashes, UI still responsive

[brentbaum]

Symptom: I can still load the idom page, but any events on the page are not registered. Unclear what the trigger for this error is but it happens the most on pages that are open for 24+ hours. Restarting the python app is required to fix.

Stack trace is below.

 2021-06-11T16:59:01+0000 | INFO | Failed to receive incoming event
 asrs-rdk  Traceback (most recent call last):
 asrs-rdk    File "/root/.cache/pypoetry/virtualenvs/rdk-host-9TtSrW0h-py3.8/lib/python3.8/site-packages/websockets/protocol.py", line 827, in transfer_data
 asrs-rdk      message = await self.read_message()
 asrs-rdk    File "/root/.cache/pypoetry/virtualenvs/rdk-host-9TtSrW0h-py3.8/lib/python3.8/site-packages/websockets/protocol.py", line 895, in read_message
 asrs-rdk      frame = await self.read_data_frame(max_size=self.max_size)
 asrs-rdk    File "/root/.cache/pypoetry/virtualenvs/rdk-host-9TtSrW0h-py3.8/lib/python3.8/site-packages/websockets/protocol.py", line 971, in read_data_frame
 asrs-rdk      frame = await self.read_frame(max_size)
 asrs-rdk    File "/root/.cache/pypoetry/virtualenvs/rdk-host-9TtSrW0h-py3.8/lib/python3.8/site-packages/websockets/protocol.py", line 1047, in read_frame
 asrs-rdk      frame = await Frame.read(
 asrs-rdk    File "/root/.cache/pypoetry/virtualenvs/rdk-host-9TtSrW0h-py3.8/lib/python3.8/site-packages/websockets/framing.py", line 105, in read
 asrs-rdk      data = await reader(2)
 asrs-rdk    File "/usr/local/lib/python3.8/asyncio/streams.py", line 723, in readexactly
 asrs-rdk      await self._wait_for_data('readexactly')
 asrs-rdk    File "/usr/local/lib/python3.8/asyncio/streams.py", line 517, in _wait_for_data
 asrs-rdk      await self._waiter
 asrs-rdk  asyncio.exceptions.CancelledError

asrs-rdk  The above exception was the direct cause of the following exception:
 asrs-rdk  
 asrs-rdk  Traceback (most recent call last):
 asrs-rdk    File "/root/.cache/pypoetry/virtualenvs/rdk-host-9TtSrW0h-py3.8/lib/python3.8/site-packages/idom/core/dispatcher.py", line 66, in _incoming_loop
 asrs-rdk      await self._incoming(self.layout, context, await recv())
 asrs-rdk    File "/root/.cache/pypoetry/virtualenvs/rdk-host-9TtSrW0h-py3.8/lib/python3.8/site-packages/idom/server/sanic.py", line 96, in sock_recv
 asrs-rdk      return LayoutEvent(**json.loads(await socket.recv()))
 asrs-rdk    File "/root/.cache/pypoetry/virtualenvs/rdk-host-9TtSrW0h-py3.8/lib/python3.8/site-packages/websockets/protocol.py", line 509, in recv
 asrs-rdk      await self.ensure_open()
 asrs-rdk    File "/root/.cache/pypoetry/virtualenvs/rdk-host-9TtSrW0h-py3.8/lib/python3.8/site-packages/websockets/protocol.py", line 812, in ensure_open
 asrs-rdk      raise self.connection_closed_exc()
 asrs-rdk  websockets.exceptions.ConnectionClosedError: code = 1006 (connection closed abnormally [internal]), no reason

[rmorshea]

It seems like this error is related to the client unexpectedly closing the connection for any of the reasons described in the websockets docs. I suppose this disconnect event is tied to your infrastructure and client connectivity issues.

The problem with IDOM then, is the fact that you have to restart the service to get it working again. Maybe the way to fix this is by putting a try/except for ConnectionClosedError around these two lines:

• https://github.com/idom-team/idom/blob/main/src/idom/server/sanic.py#L197
• https://github.com/idom-team/idom/blob/main/src/idom/server/sanic.py#L230

[brentbaum]

I've verified that this issue happened due to an exception that originated in my code and added exception handling appropriately so it didn't bubble up into idom land. That said, this was a confusing experience for a new user and I think adding some friendly error messages in the two places you recommended would go a long way.

[rmorshea]

At what point in the code-flow did the error arise? In responding to an event or during a render?

I'm guessing/hoping it's the former because then it'd be really easy to put a try/except here where we call the user's event handler and log the error:

• https://github.com/idom-team/idom/blob/main/src/idom/core/layout.py#L108

If the offending error happened during a render, we might want to dig a bit deeper since I'd expect that kind of error to be caught by this try/except here:

• https://github.com/idom-team/idom/blob/main/src/idom/core/layout.py#L160-L169

[brentbaum]

The flow:

1. Event handler onClick called
2. Event handler dumps an event (dict) into an anyio memory_stream's send_stream
3. Event handler done
4. Separate anyio-spawned thread picks up the event (inside an async for)
5. Event is executed, exception hit
6. Stack trace posted above happens

The rest of the application (the event queue) is responsive when this happens and I'm able to dispatch/process events via non-idom means.

My fix was to add a try/catch around step 5. The problem is resolved but the stack trace was getting swallowed somehow from the exception in event execution so it took a while to realize it was my code at fault.

Hopefully that all makes sense.

[rmorshea]

Based on that description, I think a try/except around this line should fix the problem: https://github.com/idom-team/idom/blob/main/src/idom/core/layout.py#L108

[rmorshea]

Resolved by: f2bf589#diff-aef3c056c129e76bbfbeb913e6d82b68e23f19e0bcf634aa75341c4b73887313