How to increase Tango serialization monitor timeout

# 2 years ago
zreszela	Hi Tango dancers! First, do you think there may be consequences of increasing the Tango Serialization monitor timeout? Second, is there a better way to do it than changing the hardcoded 3200 ms value in tango_const.h and recompiling cppTango? Third, are you aware on anyone already doing that? I tell you our case: In Sardana we have a MeasurementGroup device which starts multiple hardware controllers in the Start() Tango command. As soon as the first controller successfully starts we change the MeasurementGroup device state to MOVING and we push the change event. This approach does not scale up well, since the time necessary to start multiple hardware controllers sums up and exceeds 3 s CORBA transient timeout. Here, we wanted to hide this problem by extending this timeout for this particular Measurement Group device (usually 6 s is enough for the applications we run at ALBA). But then, sometimes, we hit the Tango Serialization Monitor timeout on the call to the push_change_event() - it can not be handled since the monitor is still busy with the Start() command. We are aware that we need to refactor this code and execute the whole starting process asynchronously, but this will require a major effort. We wonder if this simple workaround could allow us to postpone this refactoring.

# 2 years ago

Hi Tango dancers!

First, do you think there may be consequences of increasing the Tango Serialization monitor timeout?
Second, is there a better way to do it than changing the hardcoded 3200 ms value in tango_const.h and recompiling cppTango?
Third, are you aware on anyone already doing that?

I tell you our case:
In Sardana we have a MeasurementGroup device which starts multiple hardware controllers in the Start() Tango command. As soon as the first controller successfully starts we change the MeasurementGroup device state to MOVING and we push the change event. This approach does not scale up well, since the time necessary to start multiple hardware controllers sums up and exceeds 3 s CORBA transient timeout. Here, we wanted to hide this problem by extending this timeout for this particular Measurement Group device (usually 6 s is enough for the applications we run at ALBA). But then, sometimes, we hit the Tango Serialization Monitor timeout on the call to the push_change_event() - it can not be handled since the monitor is still busy with the Start() command.

We are aware that we need to refactor this code and execute the whole starting process asynchronously, but this will require a major effort. We wonder if this simple workaround could allow us to postpone this refactoring.

# 2 years ago
Reynald	Hi Zibi, zreszela First, do you think there may be consequences of increasing the Tango Serialization monitor timeout? I don't know. zreszela Second, is there a better way to do it than changing the hardcoded 3200 ms value in tango_const.h and recompiling cppTango? At first sight, from the cppTango perspective, it seems like there is a timeout(long new_timeout) method in TangoMonitor class which allows changing the timeout of a TangoMonitor (See https://gitlab.com/tango-controls/cppTango/-/blob/9.3.5/cppapi/server/tango_monitor.h#L63). And there are some public and undocumented (so you're not supposed to use them) methods to get the different TangoMonitor references from DeviceImpl class: TangoMonitor &get_dev_monitor(); TangoMonitor &get_poll_monitor(); TangoMonitor &get_att_conf_monitor(); TangoMonitor &get_pipe_conf_monitor(); See https://gitlab.com/tango-controls/cppTango/-/blob/9.3.5/cppapi/server/device.h#L3345-3348 And a get_heartbeat_monitor() method (still undocumented, so you're still not supposed to use it ) in Tango::Util class too to get the heartbeat monitor: https://gitlab.com/tango-controls/cppTango/-/blob/9.3.5/cppapi/server/utils.h#L808 If you're using a Tango Serialization by class, I think you won't be able to get the device class monitor because it is declared private in DeviceClass class (See https://gitlab.com/tango-controls/cppTango/-/blob/9.3.5/cppapi/server/deviceclass.h#L519) and it seems there is no method to retrieve a reference to this TangoMonitor object in DeviceClass class which is friend only with AutoTangoMonitor class which also does not provide any method to retrieve the reference to the TangoMonitor object (private here as well, see https://gitlab.com/tango-controls/cppTango/-/blob/9.3.5/cppapi/server/auto_tango_monitor.h#L118). zreszela Third, are you aware on anyone already doing that? I am not. It depends on where you're doing the push_change_event calls. If it's in a separate thread, did you try a simple try/catch around the push_change_event() calls and a retry in case of failure? Because if you increase the TangoMonitor(s) timeout(s), the push_change_event call might not fail but will take anyway more time to execute. Hoping this helps, Reynald Rosenberg's Law: Software is easy to make, except when you want it to do something new. Corollary: The only software that's worth making is software that does something new.

# 2 years ago

Reynald

Hi Zibi,

zreszela
First, do you think there may be consequences of increasing the Tango Serialization monitor timeout?

I don't know.

zreszela
Second, is there a better way to do it than changing the hardcoded 3200 ms value in tango_const.h and recompiling cppTango?

At first sight, from the cppTango perspective, it seems like there is a timeout(long new_timeout) method in TangoMonitor class which allows changing the timeout of a TangoMonitor (See https://gitlab.com/tango-controls/cppTango/-/blob/9.3.5/cppapi/server/tango_monitor.h#L63).
And there are some public and undocumented (so you're not supposed to use them) methods to get the different TangoMonitor references from DeviceImpl class:

TangoMonitor &get_dev_monitor();
TangoMonitor &get_poll_monitor();
TangoMonitor &get_att_conf_monitor();
TangoMonitor &get_pipe_conf_monitor();

See https://gitlab.com/tango-controls/cppTango/-/blob/9.3.5/cppapi/server/device.h#L3345-3348

And a get_heartbeat_monitor() method (still undocumented, so you're still not supposed to use it smile

) in Tango::Util class too to get the heartbeat monitor: https://gitlab.com/tango-controls/cppTango/-/blob/9.3.5/cppapi/server/utils.h#L808

If you're using a Tango Serialization by class, I think you won't be able to get the device class monitor because it is declared private in DeviceClass class
(See https://gitlab.com/tango-controls/cppTango/-/blob/9.3.5/cppapi/server/deviceclass.h#L519) and it seems there is no method to retrieve a reference to this TangoMonitor object in DeviceClass class which is friend only with AutoTangoMonitor class which also does not provide any method to retrieve the reference to the TangoMonitor object (private here as well, see https://gitlab.com/tango-controls/cppTango/-/blob/9.3.5/cppapi/server/auto_tango_monitor.h#L118).

zreszela
Third, are you aware on anyone already doing that?

I am not.

It depends on where you're doing the push_change_event calls.
If it's in a separate thread, did you try a simple try/catch around the push_change_event() calls and a retry in case of failure?
Because if you increase the TangoMonitor(s) timeout(s), the push_change_event call might not fail but will take anyway more time to execute.

Hoping this helps,
Reynald

Rosenberg's Law: Software is easy to make, except when you want it to do something new.
Corollary: The only software that's worth making is software that does something new.

# 2 years ago
zreszela	Reynald If it's in a separate thread, did you try a simple try/catch around the push_change_event() calls and a retry in case of failure? Because if you increase the TangoMonitor(s) timeout(s), the push_change_event call might not fail but will take anyway more time to execute. Yes, we are using a spearate therad to push change events. No, I've not tried it, but it looks like a solution to this problem. I will check it next week and will let you know. Thanks for this hint!

# 2 years ago

zreszela

Reynald
If it's in a separate thread, did you try a simple try/catch around the push_change_event() calls and a retry in case of failure?
Because if you increase the TangoMonitor(s) timeout(s), the push_change_event call might not fail but will take anyway more time to execute.

Yes, we are using a spearate therad to push change events. No, I've not tried it, but it looks like a solution to this problem. I will check it next week and will let you know. Thanks for this hint!