read_attribute_asynch sometimes takes 3 seconds to return

# 3 weeks ago
FloSchweiger	Hello everyone, I'm quite new to using the asynch commands of PyTango. From my understanding, the "read_attribute_asynch()" command should always return more or less immediately. Most of the time this also works out. However, if the device I want to read an attribute from is in a somehow unresponsive state this command takes three seconds to timeout before returning. Is there a possibility to set a timeout for that? I tried to put the timeout in the attribute/device proxy but this seems to have no influence. The timeout of the command itself from my understanding only defines the timeout for the "read_attribute_reply". The respective code in the Tango cpp implementation also seems to try to reconnect to the device: long DeviceProxy::read_attributes_asynch(const std::vector<std::string> &attr_names) { TANGO_TELEMETRY_TRACE_BEGIN(({{"tango.operation.target", dev_name()}})); // // Reconnect to device in case it is needed // try { check_and_reconnect(); } catch(Tango::ConnectionFailed &e) { TangoSys_OMemStream desc; desc << "Failed to execute read_attributes_asynch on device " << dev_name() << std::ends; TANGO_RETHROW_DETAILED_EXCEPTION(ApiConnExcept, e, API_CommandFailed, desc.str()); } Thanks for any tip! :)

# 3 weeks ago

Hello everyone,

I'm quite new to using the asynch commands of PyTango.

From my understanding, the "read_attribute_asynch()" command should always return more or less immediately. Most of the time this also works out. However, if the device I want to read an attribute from is in a somehow unresponsive state this command takes three seconds to timeout before returning.

Is there a possibility to set a timeout for that? I tried to put the timeout in the attribute/device proxy but this seems to have no influence. The timeout of the command itself from my understanding only defines the timeout for the "read_attribute_reply".

The respective code in the Tango cpp implementation also seems to try to reconnect to the device:

long DeviceProxy::read_attributes_asynch(const std::vector<std::string> &attr_names)
{
TANGO_TELEMETRY_TRACE_BEGIN(({{"tango.operation.target", dev_name()}}));

//
// Reconnect to device in case it is needed
//

try
{
check_and_reconnect();
}
catch(Tango::ConnectionFailed &e)
{
TangoSys_OMemStream desc;
desc << "Failed to execute read_attributes_asynch on device " << dev_name() << std::ends;
TANGO_RETHROW_DETAILED_EXCEPTION(ApiConnExcept, e, API_CommandFailed, desc.str());
}

Thanks for any tip! :)

# 3 weeks ago
AlexK	Hello Florian, If I understand you correctly, you want your device server to trigger a timeout if the hardware does not respond. You can do this in the following way: import asyncio from tango import GreenMode from tango.server import Device, attribute class DevWithTimeOut(Device): green_mode = GreenMode.Asyncio async def init_device(self): await super().init_device() async def problematic_hw_read(self): await asyncio.sleep(3) # Simulate long hw response time @attribute( dtype = str ) async def attr_with_timeout(self): try: await asyncio.wait_for(self.problematic_hw_read(), timeout = 1) #define how long you want wait return "Everything ok, no timeout" except asyncio.TimeoutError: return "TimeoutError!!!" if __name__ == "__main__": DevWithTimeOut.run_server() I hope this helps.

# 3 weeks ago

AlexK

Hello Florian,

If I understand you correctly, you want your device server to trigger a timeout if the hardware does not respond. You can do this in the following way:


import asyncio
from tango import  GreenMode
from tango.server import Device, attribute

class DevWithTimeOut(Device):
    green_mode = GreenMode.Asyncio

    async def init_device(self):
        await super().init_device()

    async def problematic_hw_read(self):
        await asyncio.sleep(3)   # Simulate long hw response time

    @attribute( dtype = str )
    async def attr_with_timeout(self):
        try:
            await asyncio.wait_for(self.problematic_hw_read(), timeout = 1) #define how long you want wait
            return "Everything ok, no timeout"
        except asyncio.TimeoutError:
            return "TimeoutError!!!"

if __name__ == "__main__":
    DevWithTimeOut.run_server()

I hope this helps.

# 2 weeks ago
Anton	Hi Florian You can set a global timeout for each DeviceProxy instance, using the method set_timeout_millis. The default is 3000 ms. This affects all remote calls the device proxy makes to the device. See: https://pytango.readthedocs.io/en/v9.5.1/client_api/device_proxy.html#tango.DeviceProxy.set_timeout_millis The timeout argument for the read_attribute_asynch method only applies when using a DeviceProxy with non-synchronous green mode. More about that here: https://pytango.readthedocs.io/en/v9.5.1/green_modes/green.html#client-green-modes As some background, you see from the cppTango code that a read_attribute_asynch() call from the client still needs to communicate with the device to "register" its request for the attribute read before the client code returns. If the device is busy with something else (blocked, or busy with a slow command or attribute access), then the device has to complete that work before it can accept and respond to the read_attribute_asynch() call. The client just keeps waiting. Part of that is due to the device synchronisation model (serialisation by device, is the default). Read more here: https://tango-controls.readthedocs.io/en/latest/development/advanced/threading.html#serialization-model-within-a-device-server /Anton

# 2 weeks ago

Anton

Hi Florian

You can set a global timeout for each DeviceProxy instance, using the method set_timeout_millis. The default is 3000 ms. This affects all remote calls the device proxy makes to the device.

See: https://pytango.readthedocs.io/en/v9.5.1/client_api/device_proxy.html#tango.DeviceProxy.set_timeout_millis

The timeout argument for the read_attribute_asynch method only applies when using a DeviceProxy with non-synchronous green mode.
More about that here: https://pytango.readthedocs.io/en/v9.5.1/green_modes/green.html#client-green-modes

As some background, you see from the cppTango code that a read_attribute_asynch() call from the client still needs to communicate with the device to "register" its request for the attribute read before the client code returns. If the device is busy with something else (blocked, or busy with a slow command or attribute access), then the device has to complete that work before it can accept and respond to the read_attribute_asynch() call. The client just keeps waiting.

Part of that is due to the device synchronisation model (serialisation by device, is the default).
Read more here: https://tango-controls.readthedocs.io/en/latest/development/advanced/threading.html#serialization-model-within-a-device-server

/Anton

# 2 weeks ago
FloSchweiger	Hello, We checked out code again and in fact we do something like: `dev = DeviceProxy("…") dev.set_timeout_millis(600) dev.read_attribute_asynch("…")` However, if our device is in a somehow intermediate state between exported and not exported, this last line takes 3s to return. From what Anton explained it seems that there is nothing to do about that without modifying our green mode and do something like AlexK showed. Since, our developers are not used to asynchio yet we will probably solve the problem with simple threading for now. Thanks a lot for both of your responses!!! Cheers, Flo

# 2 weeks ago

FloSchweiger

Hello,

We checked out code again and in fact we do something like:

dev = DeviceProxy("…")
dev.set_timeout_millis(600)
dev.read_attribute_asynch("…")

However, if our device is in a somehow intermediate state between exported and not exported, this last line takes 3s to return. From what Anton explained it seems that there is nothing to do about that without modifying our green mode and do something like AlexK showed.

Since, our developers are not used to asynchio yet we will probably solve the problem with simple threading for now.

Thanks a lot for both of your responses!!!

Cheers,
Flo