HDB++ NULL records - TANGO Controls

# 4 years ago
grizzli	Hi all, Project Context Tango 9 is up and running since last july within our small astronomy observatory named ExTra (approx. 50 DS, based in Chili, lead by IPAG CNRS Grenoble). Recently we focussed on archiving thanks to HDB++. Tango 8.1.2 / 9.2.2 HDB++ Configurator 3.12 Our issue We did setup 27 attributes from different DS to be recorded every 10 sec. For some of them we got a kind of double recording : 1 success + 1 fail every 10 sec period. Please see detail into screenshot hdbppRecords.png In parallel we got this log lines from hdbpp-es-srv (INFO-level) : logviewer.png Do you know if is correlated ? By the way, the config in Jive is as follow (same for all 27 attributes) : 1. Events => Periodic event / Period = 1000 2. Events => Archive event / Period = 10000 3. Polling => Attribute / Period = 10000 Actually I didn't catch the difference between 1 and 3. To me both are related to the polling thread behaviour… am I wrong ? Any help would be greatly appreciated :) Thank you Stephane Attachments:

# 4 years ago

Hi all,

Project Context
Tango 9 is up and running since last july within our small astronomy observatory named ExTra (approx. 50 DS, based in Chili, lead by IPAG CNRS Grenoble). Recently we focussed on archiving thanks to HDB++.
Tango 8.1.2 / 9.2.2
HDB++ Configurator 3.12

Our issue
We did setup 27 attributes from different DS to be recorded every 10 sec. For some of them we got a kind of double recording : 1 success + 1 fail every 10 sec period. Please see detail into screenshot hdbppRecords.png

In parallel we got this log lines from hdbpp-es-srv (INFO-level) : logviewer.png

Do you know if is correlated ?

By the way, the config in Jive is as follow (same for all 27 attributes) :
1. Events => Periodic event / Period = 1000
2. Events => Archive event / Period = 10000
3. Polling => Attribute / Period = 10000

Actually I didn't catch the difference between 1 and 3. To me both are related to the polling thread behaviour… am I wrong ?

Any help would be greatly appreciated :)
Thank you
Stephane

# 4 years ago
Reynald	Hi, Thanks for the update on the status of your project. It's always nice to hear some news from the different projects using Tango. The errors you are seeing look like heartbeat events issues. Tango device servers are sending special events every ~9 seconds to the event clients. If the client does not get this heartbeat event after 10 seconds, it considers the connection is broken between the event client (in your case the HDB++ Event subscriber) and the device server, so it tries to re-subscribe to the events. During this subscription phase, a read_attribute synchronous call is executed and an event is generated with the result of the synchronous read_attribute call, this is the valid data you are seeing every 10 seconds in HDB. An error event is generated as well before the re-subscription, theses are the error events you're seeing every 10 seconds too in your HDB and you're right, this is linked to the errors you are seeing in the LogViewer. My recommendation is to update to cppTango (Tango C++ library) >= 9.3.3 at least because many issues related to the heartbeat events have been solved since this release. These problems are usually seen when using network aliases and when having a network configuration giving different names for the same computer. The event client might resolve an IP address with a certain name and the device server might resolve it with another name. And since the names seen on the client side and on the server side are different, the low level ZeroMQ subscription does not work as expected. These issues should be solved in cppTango >= 9.3.3 By the way, the config in Jive is as follow (same for all 27 attributes) : 1. Events => Periodic event / Period = 1000 2. Events => Archive event / Period = 10000 3. Polling => Attribute / Period = 10000 Actually I didn't catch the difference between 1 and 3. To me both are related to the polling thread behaviour… am I wrong ? 1. is the PERIODIC event configuration. By default it is set to 1000 (ms). This means that if a client subscribes to Tango PERIODIC event on this attribute, it will receive an event every second. In your case it won't be the case because as 3. is stating, your attribute polling period is set to 10 seconds, which means, your attribute is read automatically by the Tango polling thread every 10 seconds only. So Tango will be able to send PERIODIC events only at 0.1Hz at the fastest in your case. 2. is the configuration for ARCHIVE events (the ones used by HDB++ event subscribers). An ARCHIVE event should be sent every 10 seconds in your current configuration. There are different kinds of events in Tango and a client usually subscribes to 1 type of events: PERIODIC events, CHANGE events, ARCHIVE events, DATA_READY events, USER events, PIPE events. For HDB++, the relevant events are the ARCHIVE events. Rosenberg's Law: Software is easy to make, except when you want it to do something new. Corollary: The only software that's worth making is software that does something new.

# 4 years ago

Reynald

Hi,

Thanks for the update on the status of your project. It's always nice to hear some news from the different projects using Tango.

The errors you are seeing look like heartbeat events issues.
Tango device servers are sending special events every ~9 seconds to the event clients.
If the client does not get this heartbeat event after 10 seconds, it considers the connection is broken between the event client (in your case the HDB++ Event subscriber) and the device server, so it tries to re-subscribe to the events.
During this subscription phase, a read_attribute synchronous call is executed and an event is generated with the result of the synchronous read_attribute call, this is the valid data you are seeing every 10 seconds in HDB.
An error event is generated as well before the re-subscription, theses are the error events you're seeing every 10 seconds too in your HDB and you're right, this is linked to the errors you are seeing in the LogViewer.

My recommendation is to update to cppTango (Tango C++ library) >= 9.3.3 at least because many issues related to the heartbeat events have been solved since this release.

These problems are usually seen when using network aliases and when having a network configuration giving different names for the same computer. The event client might resolve an IP address with a certain name and the device server might resolve it with another name. And since the names seen on the client side and on the server side are different, the low level ZeroMQ subscription does not work as expected.
These issues should be solved in cppTango >= 9.3.3

By the way, the config in Jive is as follow (same for all 27 attributes) :
1. Events => Periodic event / Period = 1000
2. Events => Archive event / Period = 10000
3. Polling => Attribute / Period = 10000

Actually I didn't catch the difference between 1 and 3. To me both are related to the polling thread behaviour… am I wrong ?

1. is the PERIODIC event configuration. By default it is set to 1000 (ms).
This means that if a client subscribes to Tango PERIODIC event on this attribute, it will receive an event every second.
In your case it won't be the case because as 3. is stating, your attribute polling period is set to 10 seconds, which means, your attribute is read automatically by the Tango polling thread every 10 seconds only. So Tango will be able to send PERIODIC events only at 0.1Hz at the fastest in your case.

2. is the configuration for ARCHIVE events (the ones used by HDB++ event subscribers). An ARCHIVE event should be sent every 10 seconds in your current configuration.

There are different kinds of events in Tango and a client usually subscribes to 1 type of events:
PERIODIC events, CHANGE events, ARCHIVE events, DATA_READY events, USER events, PIPE events.

For HDB++, the relevant events are the ARCHIVE events.

Rosenberg's Law: Software is easy to make, except when you want it to do something new.
Corollary: The only software that's worth making is software that does something new.

# 4 years ago
grizzli	Hi Reynald, thanks for your answer which is very enlightening. I will try to move on version 9.3.3. One more question though : Does it mean that PERIODIC events (and probably CHANGE events) are dependant of the polling thread ? Thus, if an attribute is not polled at all, a client cannot apply for PERIODIC event ? Thank you Stephane

# 4 years ago
Reynald	grizzli Does it mean that PERIODIC events (and probably CHANGE events) are dependant of the polling thread ? Thus, if an attribute is not polled at all, a client cannot apply for PERIODIC event ? I think this is true for PERIODIC events. For change events, your server can decide to push events manually by code, without your attribute to be polled by the polling thread. When a device server is able to send events by code, without relying on the polling thread, there is a special option in POGO that needs to be activated, saying that the device server is able to push events for this attribute. This is required so event clients which are subscribing do not get an error (something like "attribute is not polled") when they subscribe to events on this attribute. You can refer to the doxygen documentation to get more information on the methods used to push change or archive events by code: https://tango-controls.github.io/cppTango-docs/classTango_1_1DeviceImpl.html#a25b61671395cd833817d073449d2a240 https://tango-controls.github.io/cppTango-docs/classTango_1_1DeviceImpl.html#a49d509f24520272f94c3b62f05f0f2f4 Rosenberg's Law: Software is easy to make, except when you want it to do something new. Corollary: The only software that's worth making is software that does something new. Edited 4 years ago

# 4 years ago

Reynald

grizzli
Does it mean that PERIODIC events (and probably CHANGE events) are dependant of the polling thread ? Thus, if an attribute is not polled at all, a client cannot apply for PERIODIC event ?

I think this is true for PERIODIC events.
For change events, your server can decide to push events manually by code, without your attribute to be polled by the polling thread.
When a device server is able to send events by code, without relying on the polling thread, there is a special option in POGO that needs to be activated, saying that the device server is able to push events for this attribute. This is required so event clients which are subscribing do not get an error (something like "attribute is not polled") when they subscribe to events on this attribute.

You can refer to the doxygen documentation to get more information on the methods used to push change or archive events by code:
https://tango-controls.github.io/cppTango-docs/classTango_1_1DeviceImpl.html#a25b61671395cd833817d073449d2a240
https://tango-controls.github.io/cppTango-docs/classTango_1_1DeviceImpl.html#a49d509f24520272f94c3b62f05f0f2f4

Rosenberg's Law: Software is easy to make, except when you want it to do something new.
Corollary: The only software that's worth making is software that does something new.

Edited 4 years ago

# 4 years ago
lorenzo	In Tango, the default behavior is to rely on the polling thread for sending events. When the polling thread executes, the device checks the current attribute value against the previous one and, if the threshold conditions are satisfied, send events to the registered clients. The same applies to the PERIODiC event. This is the reason why specifying event periods smaller than the polling period makes no sense. Configuring and enabling polling, anyhow, has additional benefits. So, unless you really need to do some special requirement pushing events in the code, properly configuring and enabling polling is the recommended approach. Edited 4 years ago

# 4 years ago

lorenzo

In Tango, the default behavior is to rely on the polling thread for sending events. When the polling thread executes, the device checks the current attribute value against the previous one and, if the threshold conditions are satisfied, send events to the registered clients. The same applies to the PERIODiC event. This is the reason why specifying event periods smaller than the polling period makes no sense.
Configuring and enabling polling, anyhow, has additional benefits. So, unless you really need to do some special requirement pushing events in the code, properly configuring and enabling polling is the recommended approach.

Edited 4 years ago

# 4 years ago
grizzli	Great ! It is clear for me now. Many thanks to you guys. Stephane

# 4 years ago
grizzli	Sorry me again. It seems that it's impossible to find out windows x64 binaries/msi for version 9.3.3… do you confirm that using a compiler is the only way ? Thank you

# 4 years ago
Reynald	grizzli It seems that it's impossible to find out windows x64 binaries/msi for version 9.3.3… do you confirm that using a compiler is the only way ? You can get them at the end of this page: https://github.com/tango-controls/cppTango/releases/tag/9.3.3 In the Assets section Rosenberg's Law: Software is easy to make, except when you want it to do something new. Corollary: The only software that's worth making is software that does something new.

# 4 years ago

Reynald

grizzli
It seems that it's impossible to find out windows x64 binaries/msi for version 9.3.3… do you confirm that using a compiler is the only way ?

You can get them at the end of this page:
https://github.com/tango-controls/cppTango/releases/tag/9.3.3

In the Assets section

Rosenberg's Law: Software is easy to make, except when you want it to do something new.
Corollary: The only software that's worth making is software that does something new.

# 4 years ago
grizzli	is it all what I need ? I saw it but I was not sure that I can install only Tango Cpp lib…

# 4 years ago
Reynald	This provides only the Tango C++ library 9.3.3 version indeed. This is what you need to run C++ device servers with 9.3.3 version. Of course, you need a Tango Database server and a mariadb or mysql server too and this is not provided in this installer. But for this, you should be able to use the ones provided with older Tango 9 versions I think. Rosenberg's Law: Software is easy to make, except when you want it to do something new. Corollary: The only software that's worth making is software that does something new.

# 4 years ago

Reynald

This provides only the Tango C++ library 9.3.3 version indeed.
This is what you need to run C++ device servers with 9.3.3 version.

Of course, you need a Tango Database server and a mariadb or mysql server too and this is not provided in this installer. But for this, you should be able to use the ones provided with older Tango 9 versions I think.

Rosenberg's Law: Software is easy to make, except when you want it to do something new.
Corollary: The only software that's worth making is software that does something new.