Checking if the servers are running at least twice.

# 3 years ago
piotrtrojanowski	Hi, I got a message in astor control that the servers are running at least twice. I wonder how to fix it, because when checking information about tango starter, each server is run only once. I checked the attribute servers in tango/admin/ and everyone appeared only once. The same for tango database. Apart for that the servers are working properly, I can restart them. I tried to remove them and add them to the database again but it didn't help. Any suggestions how to fix it, what to check? Astor version: 7.3.5 Tango version: 9.3.3

# 3 years ago

Hi,
I got a message in astor control that the servers are running at least twice.
I wonder how to fix it, because when checking information about tango starter, each server is run only once.
I checked the attribute servers in tango/admin/ and everyone appeared only once. The same for tango database.
Apart for that the servers are working properly, I can restart them.
I tried to remove them and add them to the database again but it didn't help.

Any suggestions how to fix it, what to check?

Astor version: 7.3.5
Tango version: 9.3.3

# 3 years ago
Reynald	I think the Starter is somehow looking at the current list of processes on the host and detected at least 2 processes started with a command similar to the one used to start your device server. Let's take the TangoTest/test instance as example. If you start it with astor and you start another instance of TangoTest/test but using a different value for the TANGO_HOST environment variable, in this use case, for instance, Astor will show the warning you are seeing (this is a current limitation of this feature). You probably have another process started with the same binary name as your device server (might be a different executable) and using your device server instance name as parameter. If I were you, I would try to list all the running processes and see whether there is another one running with the same binary name and using the same instance name as parameter. Then I would look at the environment variables used by these processes to see whether they are using the same TANGO_HOST. On Linux: `strings /proc/<replace_with_pid>/environ` This feature is present in Astor because there is a small risk in Tango to start twice the same device server instance. There is a potential race condition which could occur at device server startup that could lead to a situation where the same device server is started twice. Astor is showing a warning to help you detect occurrences of such problem but is sometimes reporting some false alerts. Hoping this helps a bit. Reynald Rosenberg's Law: Software is easy to make, except when you want it to do something new. Corollary: The only software that's worth making is software that does something new.

# 3 years ago

Reynald

I think the Starter is somehow looking at the current list of processes on the host and detected at least 2 processes started with a command similar to the one used to start your device server.

Let's take the TangoTest/test instance as example.
If you start it with astor and you start another instance of TangoTest/test but using a different value for the TANGO_HOST environment variable, in this use case, for instance, Astor will show the warning you are seeing (this is a current limitation of this feature).

You probably have another process started with the same binary name as your device server (might be a different executable) and using your device server instance name as parameter.

If I were you, I would try to list all the running processes and see whether there is another one running with the same binary name and using the same instance name as parameter.
Then I would look at the environment variables used by these processes to see whether they are using the same TANGO_HOST.
On Linux:

strings /proc/<replace_with_pid>/environ

This feature is present in Astor because there is a small risk in Tango to start twice the same device server instance.
There is a potential race condition which could occur at device server startup that could lead to a situation where the same device server is started twice.
Astor is showing a warning to help you detect occurrences of such problem but is sometimes reporting some false alerts.

Hoping this helps a bit.
Reynald

Rosenberg's Law: Software is easy to make, except when you want it to do something new.
Corollary: The only software that's worth making is software that does something new.

# 3 years ago
piotrtrojanowski	Thanks for the answer! Unfortunately I do not see the another process running with the same binary name as my device server. I searched with 'htop' or 'ps'. I tried to turn off my server in astor, and checked the process list, but I also found nothing.

# 6 months ago
Hiro	Reynald This feature is present in Astor because there is a small risk in Tango to start twice the same device server instance. There is a potential race condition which could occur at device server startup that could lead to a situation where the same device server is started twice. Hi Reynald, I was digging in the forum when I found your post. We are currently experiencing an issue with the starter device which starts twice one of our device. It occurs far more often (but still randomly) with one type of our device, on one specific machine (windows 10, Tango 9.2.2, Starter version in old Tango 8) For example, we may restart the device from astor three times in a row and have the starter starting every time 2 instances of the device (we manually kill one of them between restart. Else, restart would kill only one of the instance without restarting the second that would be considered as a device already running etc.) then, we spent hours restarting the device with like an issue every 5/10 minutes. We tried changing several parameters of the starter related to timeout etc. but when things seemed to improve we reverted our changes to confirm that and now everything run without any issue for the last hours. That random behaviour is driving me nuts and I'm now quite interested by what you call "race condition" and what can be done to mitigate that issue. Guillaume DI FRANCO Thales - Software Engineer Manager

# 6 months ago

Hiro

Reynald
This feature is present in Astor because there is a small risk in Tango to start twice the same device server instance.
There is a potential race condition which could occur at device server startup that could lead to a situation where the same device server is started twice.

Hi Reynald,

I was digging in the forum when I found your post. We are currently experiencing an issue with the starter device which starts twice one of our device. It occurs far more often (but still randomly) with one type of our device, on one specific machine (windows 10, Tango 9.2.2, Starter version in old Tango 8)

For example, we may restart the device from astor three times in a row and have the starter starting every time 2 instances of the device (we manually kill one of them between restart. Else, restart would kill only one of the instance without restarting the second that would be considered as a device already running etc.) then, we spent hours restarting the device with like an issue every 5/10 minutes.

We tried changing several parameters of the starter related to timeout etc. but when things seemed to improve we reverted our changes to confirm that and now everything run without any issue for the last hours.

That random behaviour is driving me nuts and I'm now quite interested by what you call "race condition" and what can be done to mitigate that issue.

Guillaume DI FRANCO
Thales - Software Engineer Manager