Attribute Events gets missed

# 6 years ago
TCS_GMRT	Hi Andy, Andy are you using Java to implement your device server? In this case the issue #424 Reynald is working on is not going to help because it is only for C++. Yes we are using Java to implement device server. Andy n order to quantify the event loss can you try with only a few attributes and then increase the number until you start losing events. Have you tried splitting the attributes across multiple device servers? Does it still happen? Is there anything in the logs? We will try this and let you know the results/observations Regards, TCS_GMRT

# 6 years ago

Hi Andy,

Andy
are you using Java to implement your device server? In this case the issue #424 Reynald is working on is not going to help because it is only for C++.

Yes we are using Java to implement device server.

Andy
n order to quantify the event loss can you try with only a few attributes and then increase the number until you start losing events. Have you tried splitting the attributes across multiple device servers? Does it still happen? Is there anything in the logs?

We will try this and let you know the results/observations

Regards,
TCS_GMRT

# 6 years ago
Reynald	Hi, I think we still need more details. What is the data type of the attributes for which you are missing events? Spectrum attributes? DevDouble, DevFloat, DevState, DevBoolean…? What are your clients? Atkpanels? Java clients? C++ clients? Are the clients subscribing to change events only or also to periodic events? You were writing your commands are not taking a long time to execute, less than 3 seconds. If a command is taking 2 seconds to execute, this means that during all this time, no attribute can be read because there is a locking mechanism in Tango which by default makes it impossible to read an attribute in parallel of executing a command or reading another attribute. This is to simplify the device server programmer's life and make thread-safe device servers. So 2 seconds command execution for instance would be a very long execution time in your case if you want to be able to poll your attributes at 200 ms, or even 10 ms! So please check your commands execution times and ensure they are not the cause for event losses. If your attribute is a number scalar/spectrum or image attribute (types DevDouble, DevFloat, Dev(U)Short, Dev(U)Long(64), DevUChar), you need to specify an absolute change or relative change threshold configuration, unless you manually push events by code without checking the thresholds, otherwise you should get errors during the subscriptions (Example for a C++ device server): Error reason = API_EventPropertiesNotSet Desc : Event properties (abs_change or rel_change) for attribute devfloatspectrumro are not set Origin : DServer::event_subscription If you are using a generic client like atkpanel, I think it tries to subscribe to change events by default and if this fails, it will subscribe to periodic events. If you didn't change the period configuration parameter of the periodic event for the attribute you are subscribing to, you will get the default period value, which is 1 second. So, if you subscribe to a number attribute with a generic client like atkpanel and did not specify the absolute change or relative threshold, you will end up with an event received every 1 second, even if you specified a polling period below 1 second. In atkpanel, you can identify that easily by clicking on "View->Diagnostics…" in the menu, and then choosing the Attribute tab. You will see if ATK is receiving events for your attribute, and also what kind of event (Change event, Periodic event). You can use the "Update fields" button at the bottom of this ATK diagnostic window to refresh the values. Hoping this helps, Kind regards, Reynald Rosenberg's Law: Software is easy to make, except when you want it to do something new. Corollary: The only software that's worth making is software that does something new. Edited 6 years ago

# 6 years ago

Reynald

Hi,

I think we still need more details.
What is the data type of the attributes for which you are missing events? Spectrum attributes? DevDouble, DevFloat, DevState, DevBoolean…?
What are your clients? Atkpanels? Java clients? C++ clients?

Are the clients subscribing to change events only or also to periodic events?

You were writing your commands are not taking a long time to execute, less than 3 seconds.
If a command is taking 2 seconds to execute, this means that during all this time, no attribute can be read because there is a locking mechanism in Tango which by default makes it impossible to read an attribute in parallel of executing a command or reading another attribute. This is to simplify the device server programmer's life and make thread-safe device servers.
So 2 seconds command execution for instance would be a very long execution time in your case if you want to be able to poll your attributes at 200 ms, or even 10 ms!
So please check your commands execution times and ensure they are not the cause for event losses.

If your attribute is a number scalar/spectrum or image attribute (types DevDouble, DevFloat, Dev(U)Short, Dev(U)Long(64), DevUChar), you need to specify an absolute change or relative change threshold configuration, unless you manually push events by code without checking the thresholds, otherwise you should get errors during the subscriptions (Example for a C++ device server):

Error reason = API_EventPropertiesNotSet
Desc : Event properties (abs_change or rel_change) for attribute devfloatspectrumro are not set
Origin : DServer::event_subscription

If you are using a generic client like atkpanel, I think it tries to subscribe to change events by default and if this fails, it will subscribe to periodic events. If you didn't change the period configuration parameter of the periodic event for the attribute you are subscribing to, you will get the default period value, which is 1 second.
So, if you subscribe to a number attribute with a generic client like atkpanel and did not specify the absolute change or relative threshold, you will end up with an event received every 1 second, even if you specified a polling period below 1 second.
In atkpanel, you can identify that easily by clicking on "View->Diagnostics…" in the menu, and then choosing the Attribute tab. You will see if ATK is receiving events for your attribute, and also what kind of event (Change event, Periodic event).
You can use the "Update fields" button at the bottom of this ATK diagnostic window to refresh the values.

Hoping this helps,

Kind regards,
Reynald

Rosenberg's Law: Software is easy to make, except when you want it to do something new.
Corollary: The only software that's worth making is software that does something new.

Edited 6 years ago

# 6 years ago
lorenzo	TCS_GMRT Reynald Maybe the server is not sending events because the conditions are not met (the attribute value didn't change enough to send an event)? Could you please verify that? Attribute values are getting changed surely. (any change in spectrum attribute value should send an event) Just for clarity, in addition to Reynald reply, any change in spectrum attribute value should send an event provided you specified the event (change, archive) thresholds in the Attribute event configuration. If not, no event will be sent, and you should get an exception when subscribing… Edited 6 years ago

# 6 years ago

lorenzo

TCS_GMRT
Reynald
Maybe the server is not sending events because the conditions are not met (the attribute value didn't change enough to send an event)?
Could you please verify that?

Attribute values are getting changed surely. (any change in spectrum attribute value should send an event)

Just for clarity, in addition to Reynald reply, any change in spectrum attribute value should send an event *provided* you specified the event (change, archive) thresholds in the Attribute event configuration. If not, no event will be sent, and you should get an exception when subscribing…

Edited 6 years ago

# 6 years ago
TCS_GMRT	lorenzo TCS_GMRT Reynald Maybe the server is not sending events because the conditions are not met (the attribute value didn't change enough to send an event)? Could you please verify that? Attribute values are getting changed surely. (any change in spectrum attribute value should send an event) Just for clarity, in addition to Reynald reply, any change in spectrum attribute value should send an event provided you specified the event (change, archive) thresholds in the Attribute event configuration. If not, no event will be sent, and you should get an exception when subscribing… Sorry for the late reply. What exactly do you mean by Attribute event configuration? While subscribing client subscribes the attribute as change event. No attribute event configurations are done at server side. Actually the problem is not that we are not receiving the event any time, we do get changed events most of the time. But some times it gets missed. This problem occurs more frequently when one client subscribes the attribute events of more than 5 device servers. And the attributes of all these device servers changes frequently. So in this scenario out of 5 we miss events of at least 2 device servers. Any idea why this might have happened? and how can we resolve this Regards, TCS_GMRT

# 6 years ago

TCS_GMRT

lorenzo
TCS_GMRT
Reynald
Maybe the server is not sending events because the conditions are not met (the attribute value didn't change enough to send an event)?
Could you please verify that?

Attribute values are getting changed surely. (any change in spectrum attribute value should send an event)

Just for clarity, in addition to Reynald reply, any change in spectrum attribute value should send an event *provided* you specified the event (change, archive) thresholds in the Attribute event configuration. If not, no event will be sent, and you should get an exception when subscribing…

Sorry for the late reply.
What exactly do you mean by Attribute event configuration? While subscribing client subscribes the attribute as change event.
No attribute event configurations are done at server side.
Actually the problem is not that we are not receiving the event any time, we do get changed events most of the time. But some times it gets missed.
This problem occurs more frequently when one client subscribes the attribute events of more than 5 device servers. And the attributes of all these device servers changes frequently. So in this scenario out of 5 we miss events of at least 2 device servers.
Any idea why this might have happened? and how can we resolve this

Regards,
TCS_GMRT

# 6 years ago
TCS_GMRT	Reynald You were writing your commands are not taking a long time to execute, less than 3 seconds. If a command is taking 2 seconds to execute, this means that during all this time, no attribute can be read because there is a locking mechanism in Tango which by default makes it impossible to read an attribute in parallel of executing a command or reading another attribute. This is to simplify the device server programmer's life and make thread-safe device servers. So 2 seconds command execution for instance would be a very long execution time in your case if you want to be able to poll your attributes at 200 ms, or even 10 ms! So please check your commands execution times and ensure they are not the cause for event losses. Hi Reynald, The command execution time is less. I dont think this is the problem because whenever we have only 1 or 2 device servers, and a client subscribes the change event of spectrum attribute of both device servers, client gets all the events. No event is missed. The problem is when we increase the number of device servers and a single client. The spectrum attribute of all the device servers changes simultaneously, so client gets some events and some events are getting missed. Any suggestion regarding this will be helpful Regards, TCS_GMRT

# 6 years ago

TCS_GMRT

Reynald
You were writing your commands are not taking a long time to execute, less than 3 seconds.
If a command is taking 2 seconds to execute, this means that during all this time, no attribute can be read because there is a locking mechanism in Tango which by default makes it impossible to read an attribute in parallel of executing a command or reading another attribute. This is to simplify the device server programmer's life and make thread-safe device servers.
So 2 seconds command execution for instance would be a very long execution time in your case if you want to be able to poll your attributes at 200 ms, or even 10 ms!
So please check your commands execution times and ensure they are not the cause for event losses.

Hi Reynald,
The command execution time is less. I dont think this is the problem because whenever we have only 1 or 2 device servers, and a client subscribes the change event of spectrum attribute of both device servers, client gets all the events. No event is missed.
The problem is when we increase the number of device servers and a single client.
The spectrum attribute of all the device servers changes simultaneously, so client gets some events and some events are getting missed.
Any suggestion regarding this will be helpful

Regards,
TCS_GMRT

# 6 years ago
Reynald	TCS_GMRT What exactly do you mean by Attribute event configuration? I mean the event related configuration parameters described in the following section of the documentation: http://tango-controls.readthedocs.io/en/latest/development/advanced/reference.html?highlight=change%20event#the-event-related-configuration-parameters. You can configure them in Jive and/or in the code using POGO (See the attached pictures) These parameters are important because Tango will use to decide whether it should send an event or not. To help you, we need more details about your current configuration. You didn't answer to some of my previous questions. What is your client? Is it a generic client (ATKPanel, TaurusGUI, custom one), written in C++, Java, Python? What is the excact type of the all attributes you are subscribing to and their size in case of spectrum of images? What is the polling period of all the attributes you are subscribing to? Are all these attributes exported by JAVA device servers? How do you see that you missed some events? Do you have some kind of counter in the attributes you subscribed to? What are these attributes monitoring (hardware related values)? It would be great if you could find a way to reproduce your issue with simple device servers and clients we could execute from anywhere. Kind regards Reynald Rosenberg's Law: Software is easy to make, except when you want it to do something new. Corollary: The only software that's worth making is software that does something new. Attachments:

# 6 years ago

Reynald

TCS_GMRT
What exactly do you mean by Attribute event configuration?

I mean the event related configuration parameters described in the following section of the documentation: http://tango-controls.readthedocs.io/en/latest/development/advanced/reference.html?highlight=change%20event#the-event-related-configuration-parameters.
You can configure them in Jive and/or in the code using POGO (See the attached pictures)

These parameters are important because Tango will use to decide whether it should send an event or not.

To help you, we need more details about your current configuration.
You didn't answer to some of my previous questions.
What is your client?
Is it a generic client (ATKPanel, TaurusGUI, custom one), written in C++, Java, Python?

What is the excact type of the all attributes you are subscribing to and their size in case of spectrum of images?
What is the polling period of all the attributes you are subscribing to?
Are all these attributes exported by JAVA device servers?

How do you see that you missed some events? Do you have some kind of counter in the attributes you subscribed to?
What are these attributes monitoring (hardware related values)?

It would be great if you could find a way to reproduce your issue with simple device servers and clients we could execute from anywhere.

Kind regards
Reynald

Rosenberg's Law: Software is easy to make, except when you want it to do something new.
Corollary: The only software that's worth making is software that does something new.

# 6 years ago
TCS_GMRT	Hi Reynald, Thanks for the quick response. Reynald I mean the event related configuration parameters described in the following section of the documentation: http://tango-controls.readthedocs.io/en/latest/development/advanced/reference.html?highlight=change%20event#the-event-related-configuration-parameters. You can configure them in Jive and/or in the code using POGO (See the attached pictures) These parameters are important because Tango will use to decide whether it should send an event or not. Okay. In our case I think abs_change needs to be set(as any change in attribute should raise an event). Whether I am correct? What should be ideal value of this? Reynald o help you, we need more details about your current configuration. You didn't answer to some of my previous questions. What is your client? Is it a generic client (ATKPanel, TaurusGUI, custom one), written in C++, Java, Python? The issue has occurred with TaurusGUI as well as Java as a client. Reynald What is the excact type of the all attributes you are subscribing to and their size in case of spectrum of images? What is the polling period of all the attributes you are subscribing to? Are all these attributes exported by JAVA device servers? The attributes we are subscribing are of spectrum of String of length 6. The polling period of these attributes is set to 100. All the attributes are exported by JAVA device servers Reynald How do you see that you missed some events? Do you have some kind of counter in the attributes you subscribed to? We could see this with logs at device servers and client. We see the changed attribute value at device servers logs but that changed events has not occurred at clients log. Reynald What are these attributes monitoring (hardware related values)? These attributes represent some asynchronous messages specific to application. Reynald It would be great if you could find a way to reproduce your issue with simple device servers and clients we could execute from anywhere. I am working on separating the patch of code to reproduce this issue. I will post it once it is ready. Regards, TCS_GMRT

# 6 years ago

TCS_GMRT

Hi Reynald,

Thanks for the quick response.

Reynald
I mean the event related configuration parameters described in the following section of the documentation: http://tango-controls.readthedocs.io/en/latest/development/advanced/reference.html?highlight=change%20event#the-event-related-configuration-parameters.
You can configure them in Jive and/or in the code using POGO (See the attached pictures)

These parameters are important because Tango will use to decide whether it should send an event or not.

Okay. In our case I think abs_change needs to be set(as any change in attribute should raise an event). Whether I am correct? What should be ideal value of this?

Reynald
o help you, we need more details about your current configuration.
You didn't answer to some of my previous questions.
What is your client?
Is it a generic client (ATKPanel, TaurusGUI, custom one), written in C++, Java, Python?

The issue has occurred with TaurusGUI as well as Java as a client.

Reynald
What is the excact type of the all attributes you are subscribing to and their size in case of spectrum of images?
What is the polling period of all the attributes you are subscribing to?
Are all these attributes exported by JAVA device servers?

The attributes we are subscribing are of spectrum of String of length 6.
The polling period of these attributes is set to 100. All the attributes are exported by JAVA device servers

Reynald
How do you see that you missed some events? Do you have some kind of counter in the attributes you subscribed to?

We could see this with logs at device servers and client. We see the changed attribute value at device servers logs but that changed events has not occurred at clients log.

Reynald
What are these attributes monitoring (hardware related values)?

These attributes represent some asynchronous messages specific to application.

Reynald
It would be great if you could find a way to reproduce your issue with simple device servers and clients we could execute from anywhere.

I am working on separating the patch of code to reproduce this issue. I will post it once it is ready.

Regards,
TCS_GMRT

# 6 years ago
TCS_GMRT	Hi, I had one more doubt. While subscribing event we need to specify callback. In our case we have set same callback for all subscription. Whether callback should be different for all subscription? Regards, TCS_GMRT

# 6 years ago
Reynald	TCS_GMRT The attributes we are subscribing are of spectrum of String of length 6. The polling period of these attributes is set to 100. All the attributes are exported by JAVA device servers Can you confirm all the attributes you are subscribing to are spectrum of String of length 6? For String, State and Boolean attributes, the relative change and absolute change thresholds do not make sense. An event should be sent as soon as the attribute value changes in this case. How big are the strings contained in the spectrum? Rosenberg's Law: Software is easy to make, except when you want it to do something new. Corollary: The only software that's worth making is software that does something new.

# 6 years ago

Reynald

TCS_GMRT
The attributes we are subscribing are of spectrum of String of length 6.
The polling period of these attributes is set to 100. All the attributes are exported by JAVA device servers

Can you confirm all the attributes you are subscribing to are spectrum of String of length 6?

For String, State and Boolean attributes, the relative change and absolute change thresholds do not make sense.
An event should be sent as soon as the attribute value changes in this case.

How big are the strings contained in the spectrum?

Rosenberg's Law: Software is easy to make, except when you want it to do something new.
Corollary: The only software that's worth making is software that does something new.

# 6 years ago
Reynald	TCS_GMRT Hi, I had one more doubt. While subscribing event we need to specify callback. In our case we have set same callback for all subscription. Whether callback should be different for all subscription? It depends on your use case. You can use the same callback or a different callback. Both are correct. In some cases, it is more convenient to have a separate callback and in some cases, a generic callback is better. It all depends on what you need to do. Rosenberg's Law: Software is easy to make, except when you want it to do something new. Corollary: The only software that's worth making is software that does something new.

# 6 years ago

Reynald

TCS_GMRT
Hi,

I had one more doubt. While subscribing event we need to specify callback. In our case we have set same callback for all subscription. Whether callback should be different for all subscription?

It depends on your use case. You can use the same callback or a different callback. Both are correct.
In some cases, it is more convenient to have a separate callback and in some cases, a generic callback is better.
It all depends on what you need to do.

Rosenberg's Law: Software is easy to make, except when you want it to do something new.
Corollary: The only software that's worth making is software that does something new.