Attribute Events gets missed

# 6 years ago
TCS_GMRT	Hi All, We have subscribed an attribute change event. The polling period of attribute is kept 200. We have observed that some of the events are getting missed. If the polling period is reduced too much like 10, then it results in low performance. Whether anyone observed the same problem? How to get it resolved? Thanks & Regards TCS-GMRT Regards, TCS_GMRT

# 6 years ago
Reynald	Hi, Is your problem the same as this one which was reported by Andy: https://github.com/tango-controls/cppTango/issues/424 ? If not, please give us more details on your device server and ideally the source code of a device server to reproduce your problem… Knowing the configuration of your device would be very interesting for us too. How many attributes in the server? Are they all polled? Are they all polled at the same frequency? How many clients are connected? Are all the clients using events? Are the clients sending some commands from time to time? Are you pushing events by code? Do you have some "polling thread is late" errors? Kind regards, Reynald Rosenberg's Law: Software is easy to make, except when you want it to do something new. Corollary: The only software that's worth making is software that does something new.

# 6 years ago

Reynald

Hi,

Is your problem the same as this one which was reported by Andy: https://github.com/tango-controls/cppTango/issues/424 ?
If not, please give us more details on your device server and ideally the source code of a device server to reproduce your problem…
Knowing the configuration of your device would be very interesting for us too.
How many attributes in the server?
Are they all polled?
Are they all polled at the same frequency?
How many clients are connected? Are all the clients using events?
Are the clients sending some commands from time to time?
Are you pushing events by code?
Do you have some "polling thread is late" errors?

Kind regards,
Reynald

Rosenberg's Law: Software is easy to make, except when you want it to do something new.
Corollary: The only software that's worth making is software that does something new.

# 6 years ago
lorenzo	Hi, moreover, are those "missed" events not sent by the device or not received by the client? Cheers, Lorenzo

# 6 years ago
TCS_GMRT	Hi Reynald, Reynald Is your problem the same as this one which was reported by Andy: https://github.com/tango-controls/cppTango/issues/424 ? We are facing this problem also. As well missing events problem which is not mentioned here. Reynald If not, please give us more details on your device server and ideally the source code of a device server to reproduce your problem… Knowing the configuration of your device would be very interesting for us too. The details of configuration are as follows: Reynald How many attributes in the server? The server has 40 static attributes and 40-50 dynamic attributes. Reynald Are they all polled? Yes Reynald Are they all polled at the same frequency? No. Some attributes have 1000ms polling and some have 200ms. The polling frequency is kept 200ms for the attributes for which we expect frequent events. Reynald How many clients are connected? Are all the clients using events? Upto 10 clients are connected. All clients use events Reynald Are the clients sending some commands from time to time? Yes Reynald Are you pushing events by code? No Reynald Do you have some "polling thread is late" errors? No this error has not occurred Regards, TCS_GMRT

# 6 years ago

TCS_GMRT

Hi Reynald,

Reynald
Is your problem the same as this one which was reported by Andy: https://github.com/tango-controls/cppTango/issues/424 ?

We are facing this problem also. As well missing events problem which is not mentioned here.

Reynald
If not, please give us more details on your device server and ideally the source code of a device server to reproduce your problem…
Knowing the configuration of your device would be very interesting for us too.

The details of configuration are as follows:

Reynald
How many attributes in the server?

The server has 40 static attributes and 40-50 dynamic attributes.

Reynald
Are they all polled?

Yes

Reynald
Are they all polled at the same frequency?

No. Some attributes have 1000ms polling and some have 200ms. The polling frequency is kept 200ms for the attributes for which we expect frequent events.

Reynald
How many clients are connected? Are all the clients using events?

Upto 10 clients are connected. All clients use events

Reynald
Are the clients sending some commands from time to time?

Yes

Reynald
Are you pushing events by code?

Reynald
Do you have some "polling thread is late" errors?

No this error has not occurred

Regards,
TCS_GMRT

# 6 years ago
TCS_GMRT	lorenzo Hi, moreover, are those "missed" events not sent by the device or not received by the client? Cheers, Lorenzo Hi Lorenzo, The attributes are sent by device i.e updated at the polled attribute by device but not received by client. Regards, TCS_GMRT

# 6 years ago
Reynald	You have many attributes, so you might be hitting the issue created by Andy (https://github.com/tango-controls/cppTango/issues/424). We are currently working on it. Are the commands sent by the clients taking a long time to execute? It looks like you didn't receive any Missed Event errors on the client side… Can you confirm that? Tango has a mechanism on the client side to detect missed ZMQ events because there is a counter field associated with each event received. If an event is received with a counter value unexpectedly too high, an API_MissedEvents exception should be received on the client side with an error description like "Missed some events! Zmq queue has reached HWM?". Did you see anything like that on the clients side? If not, the reason why you are not seeing an event sent for every update of the polling value could be because of your event configuration… Did you specify any Absolute or Relative threshold as event configuration for the attributes which are missing events? Maybe the server is not sending events because the conditions are not met (the attribute value didn't change enough to send an event)? Could you please verify that? Cheers, Reynald Rosenberg's Law: Software is easy to make, except when you want it to do something new. Corollary: The only software that's worth making is software that does something new.

# 6 years ago

Reynald

You have many attributes, so you might be hitting the issue created by Andy (https://github.com/tango-controls/cppTango/issues/424).
We are currently working on it.

Are the commands sent by the clients taking a long time to execute?

It looks like you didn't receive any Missed Event errors on the client side… Can you confirm that?
Tango has a mechanism on the client side to detect missed ZMQ events because there is a counter field associated with each event received. If an event is received with a counter value unexpectedly too high, an API_MissedEvents exception should be received on the client side with an error description like "Missed some events! Zmq queue has reached HWM?". Did you see anything like that on the clients side?

If not, the reason why you are not seeing an event sent for every update of the polling value could be because of your event configuration… Did you specify any Absolute or Relative threshold as event configuration for the attributes which are missing events? Maybe the server is not sending events because the conditions are not met (the attribute value didn't change enough to send an event)?
Could you please verify that?

Cheers,
Reynald

Rosenberg's Law: Software is easy to make, except when you want it to do something new.
Corollary: The only software that's worth making is software that does something new.

# 6 years ago
lorenzo	One additional point you may want to consider is that almost 100 attributes at 200ms polling period means 2ms execution time slice per method with no spare CPU time for the device… which may be on the low side. I know your setup uses larger polling periods as well, but you may want to check the real execution time slice of each method (e.g. attribute/command) as, as per Reynald suggestion, slow ones may impact the fast ones as well (delaying, or even dropping, execution depending on TANGO version). Lorenzo Edited 6 years ago

# 6 years ago

lorenzo

One additional point you may want to consider is that almost 100 attributes at 200ms polling period means 2ms execution time slice per method with no spare CPU time for the device… which may be on the low side. I know your setup uses larger polling periods as well, but you may want to check the real execution time slice of each method (e.g. attribute/command) as, as per Reynald suggestion, slow ones may impact the fast ones as well (delaying, or even dropping, execution depending on TANGO version).
Lorenzo

Edited 6 years ago

# 6 years ago
Andy	Hi TCS_GMRT, are you using Java to implement your device server? In this case the issue #424 Reynald is working on is not going to help because it is only for C++. When does you start losing events? In order to quantify the event loss can you try with only a few attributes and then increase the number until you start losing events. Have you tried splitting the attributes across multiple device servers? Does it still happen? Is there anything in the logs? Andy

# 6 years ago

Andy

Hi TCS_GMRT,

are you using Java to implement your device server? In this case the issue #424 Reynald is working on is not going to help because it is only for C++. When does you start losing events? In order to quantify the event loss can you try with only a few attributes and then increase the number until you start losing events. Have you tried splitting the attributes across multiple device servers? Does it still happen? Is there anything in the logs?

Andy

# 6 years ago
TCS_GMRT	Hi Raynald, Thanks for the reply. Reynald Are the commands sent by the clients taking a long time to execute? No. commands are getting executed within 3 sec. Reynald It looks like you didn't receive any Missed Event errors on the client side… Can you confirm that? Tango has a mechanism on the client side to detect missed ZMQ events because there is a counter field associated with each event received. If an event is received with a counter value unexpectedly too high, an API_MissedEvents exception should be received on the client side with an error description like "Missed some events! Zmq queue has reached HWM?". Did you see anything like that on the clients side? We are not getting any such error. Reynald If not, the reason why you are not seeing an event sent for every update of the polling value could be because of your event configuration… Did you specify any Absolute or Relative threshold as event configuration for the attributes which are missing events? No Absolute and Relative threshold is specified. It is None as by default. Whether we need to specify any such configurations? Reynald Maybe the server is not sending events because the conditions are not met (the attribute value didn't change enough to send an event)? Could you please verify that? Attribute values are getting changed surely. (any change in spectrum attribute value should send an event) Regards, TCS_GMRT

# 6 years ago

TCS_GMRT

Hi Raynald,

Thanks for the reply.

Reynald
Are the commands sent by the clients taking a long time to execute?

No. commands are getting executed within 3 sec.

Reynald
It looks like you didn't receive any Missed Event errors on the client side… Can you confirm that?
Tango has a mechanism on the client side to detect missed ZMQ events because there is a counter field associated with each event received. If an event is received with a counter value unexpectedly too high, an API_MissedEvents exception should be received on the client side with an error description like "Missed some events! Zmq queue has reached HWM?". Did you see anything like that on the clients side?

We are not getting any such error.

Reynald
If not, the reason why you are not seeing an event sent for every update of the polling value could be because of your event configuration… Did you specify any Absolute or Relative threshold as event configuration for the attributes which are missing events?

No Absolute and Relative threshold is specified. It is None as by default. Whether we need to specify any such configurations?

Reynald
Maybe the server is not sending events because the conditions are not met (the attribute value didn't change enough to send an event)?
Could you please verify that?

Attribute values are getting changed surely. (any change in spectrum attribute value should send an event)

Regards,
TCS_GMRT

# 6 years ago
TCS_GMRT	Hi Lorenzo, lorenzo One additional point you may want to consider is that almost 100 attributes at 200ms polling period means 2ms execution time slice per method with no spare CPU time for the device… which may be on the low side. I know your setup uses larger polling periods as well, but you may want to check the real execution time slice of each method (e.g. attribute/command) as, as per Reynald suggestion, slow ones may impact the fast ones as well (delaying, or even dropping, execution depending on TANGO version). Lorenzo Agree with your points. We will check it and let you know. Regards, TCS_GMRT

# 6 years ago

TCS_GMRT

Hi Lorenzo,

lorenzo
One additional point you may want to consider is that almost 100 attributes at 200ms polling period means 2ms execution time slice per method with no spare CPU time for the device… which may be on the low side. I know your setup uses larger polling periods as well, but you may want to check the real execution time slice of each method (e.g. attribute/command) as, as per Reynald suggestion, slow ones may impact the fast ones as well (delaying, or even dropping, execution depending on TANGO version).
Lorenzo

Agree with your points.
We will check it and let you know.

Regards,
TCS_GMRT