watcher silently fails when receiving huge data

Bug #2072492 reported by Gregory Thiemonge
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
etcd3gw
Fix Released
Medium
Gregory Thiemonge
python-etcd3gw (Ubuntu)
Fix Released
Undecided
Unassigned
Jammy
Deferred
Medium
Unassigned
Noble
Deferred
Medium
Unassigned
Oracular
Deferred
Medium
Unassigned

Bug Description

Bug detected while testing octavia/taskflow with etcd3gw.

When using a watcher, if a huge payload is inserted in etcd, the watcher reads it and then fails silently, it no longer receives any data after that.

Reproducer:

======
from concurrent.futures import ThreadPoolExecutor
import time

from etcd3gw.client import Etcd3Client

def watcher_func(client):
    watcher, watcher_cancel = client.watch_prefix("/myprefix")
    for event in watcher:
        print(event)

with ThreadPoolExecutor(max_workers=1) as executor:
    client = Etcd3Client(host='192.168.1.15', port=2379)
    future = executor.submit(watcher_func, client)

    client.put("/myprefix/foo", 'a' * 100)
    time.sleep(1)
    client.put("/myprefix/foo", 'a' * 10000)
    time.sleep(1)
    client.put("/myprefix/foo", 'b')
======

when executing the reproducer print(event) displays only the 2 first events.

The issue is in the _watch function (https://github.com/openstack/etcd3gw/blob/master/etcd3gw/watch.py#L22-L24), in case of big payload, the data of the event is returned in 2 chunks: the first one contains the json doc, and the second only a '\n'

then json.loads("\n") triggers an exception

When adding a try/except block:

Traceback (most recent call last):
  File "/home/gthiemon/venv/lib64/python3.9/site-packages/etcd3gw/watch.py", line 25, in _watch
    payload = json.loads(decoded_line)
  File "/usr/lib64/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib64/python3.9/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 2 column 1 (char 1)

Revision history for this message
Gregory Thiemonge (gthiemonge) wrote :

$ /opt/stack/bin/etcd --version
WARNING: Package "github.com/golang/protobuf/protoc-gen-go/generator" is deprecated.
        A future release of golang/protobuf will delete this package,
        which has long been excluded from the compatibility promise.

etcd Version: 3.4.27
Git SHA: c92fb80f3
Go Version: go1.19.10
Go OS/Arch: linux/amd64

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to etcd3gw (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/etcd3gw/+/923636

Changed in python-etcd3gw:
status: New → In Progress
Changed in python-etcd3gw:
importance: Undecided → High
assignee: nobody → Gregory Thiemonge (gthiemonge)
importance: High → Medium
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to etcd3gw (master)

Reviewed: https://review.opendev.org/c/openstack/etcd3gw/+/923636
Committed: https://opendev.org/openstack/etcd3gw/commit/b5ff7f1625e04af8211ba963bd0cabd0dd2ab0a2
Submitter: "Zuul (22348)"
Branch: master

commit b5ff7f1625e04af8211ba963bd0cabd0dd2ab0a2
Author: Gregory Thiemonge <email address hidden>
Date: Mon Jul 8 13:50:40 2024 +0200

    Fix watcher failing with huge payloads

    When the watcher receives a huge payload from the Etcd server, the
    payload may be split into 2 chunks: the first chunk contains the json
    data, and the second contains only a newline (\n).
    Skip these empty lines because json.loads raises an uncaught exception
    and the thread fails silently.

    Closes-Bug: #2072492
    Change-Id: Iced7f057cbe928033b7a8bb8fe7086a8a1d3c4c5

Changed in python-etcd3gw:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/etcd3gw 2.4.1

This issue was fixed in the openstack/etcd3gw 2.4.1 release.

Changed in python-etcd3gw (Ubuntu):
status: New → Fix Released
Changed in python-etcd3gw (Ubuntu Jammy):
importance: Undecided → Medium
Changed in python-etcd3gw (Ubuntu Oracular):
importance: Undecided → Medium
Changed in python-etcd3gw (Ubuntu Jammy):
assignee: nobody → Wesley Hershberger (whershberger)
Changed in python-etcd3gw (Ubuntu Noble):
importance: Undecided → Medium
assignee: nobody → Wesley Hershberger (whershberger)
Changed in python-etcd3gw (Ubuntu Oracular):
assignee: nobody → Wesley Hershberger (whershberger)
Changed in python-etcd3gw (Ubuntu Noble):
status: New → In Progress
Changed in python-etcd3gw (Ubuntu Jammy):
status: New → In Progress
Changed in python-etcd3gw (Ubuntu Oracular):
status: New → In Progress
Changed in python-etcd3gw (Ubuntu Jammy):
assignee: Wesley Hershberger (whershberger) → nobody
Changed in python-etcd3gw (Ubuntu Oracular):
assignee: Wesley Hershberger (whershberger) → nobody
status: In Progress → Deferred
Changed in python-etcd3gw (Ubuntu Noble):
status: In Progress → Deferred
Changed in python-etcd3gw (Ubuntu Jammy):
status: In Progress → Deferred
Changed in python-etcd3gw (Ubuntu Noble):
assignee: Wesley Hershberger (whershberger) → nobody
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.