Tango Feature Request 4: Defining a standard Tango REST API
vhardion avatar
hardion
Hi tangoers,

During the 29th Tango Meeting we announced the possibility to make a standard Tango REST API.
This feature request has been written in a more formal way below as a tentative for helping the discussion.
Please give any feedback about the definition of this TFR and I will update this page as soon as I can.

I will open a new topic in the forum to discuss especially about the API smile.

Cheers,
Vincent

Code name

Scope
The scope of this feature request is focus on the definition of a REST API for Tango.
This will include the definition of the HTTP request (URL, content, …)

Context
Several web application have been develop for Tango and a REST api has been often used.
Here is a list of different existing implementations:
  • mtango - https://bitbucket.org/hzgwpn/mtango/wiki/mTango%20REST%20API
    • mTango REST API allows you to do the following:
      list all devices available through the API get info about device, its attributes or commands read/write attributes execute commands
    You can find a very complete description on the wiki of the project

  • Solaris Tango REST
    • Generic Tango web server

    Please find more about Jive on Android project from Solaris:
    Android app:
    https://github.com/lukaszmitka/jive_for_android
    https://github.com/lukaszmitka/restful_jive_for_android
    REST interface:
    https://github.com/lukaszmitka/restful_tango_interface_2

  • MAXIV BPM application
    • Specific Tango web server
    (TODO)

  • Gotan - https://github.com/hardion/GoTan
    • Tango module for Gotan.
    The purpose is to access the Tango world through the Gotan REST server
    A document describing the api: https://www.dropbox.com/s/9nqeabizxlmj6zp/Gotan%20-%20Tango%20REST.pdf?dl=0

    Problem to solve
    There are many way to define an API using the architecture REST. Also there are different constraints among them the Stateless is the most important. RESTful is defining a set of rule to guarantee a compatibility between all of the Restful software. How to handle command with RESTful compatibility can be one of the topics among others.
    Also Tango already defines a standard API on how to access devices, attributes, properties… That can be matched with a Tango Rest Api.

    All the web client application won't be compatible if all the different Tango REST server have a different interface.

    Goal and milestones:
    The goal is to collect the different experiences to define a stable first version of the standard.
    First of all a proposition with different option will be released then to collect the feedback from the overall community. This will define a alpha version of the TANGO REST API
    Finally a document will release before the Tango meeting the 30rd for approval from the board. This will define a beta version of the TANGO REST API.

    Output:
    The first version of this API is expected to be release with:
      A complete description of the API written in a document, online manual,… A set of compliance test with a possible automatic test set (Optional)Some possible reference implementation for the only purpose of education and given as an example. Different language can be proposed. Release some implementation guidelines, i.e. caching and optimizations advises. For instance, implementation must be clever enough to combine several requests, i.e. several clients read the same attribute. This is implemented in mTangoREST server.

    Benefit:
    A standard API provide an interoperability between the different implementation of the web server and the client. The leverage of the standard will allow to create an active ecosystem.
    The expectation is to grow the web development of Tango in a coherent way for the users.

    Possible later improvement:
    Performance test: Can be part of the compliance test

    Edit:
    - added links for the Solaris project (Lukasz)
    - added implementation guidelines (Igor)
    Vincent Hardion
    Control System
    MAX IV Laboratory
    Edited 9 years ago
    lzytniak avatar
    lzytniak
    		Hi Tangoers,

    Please find more about Jive on Android project from Solaris:
    Android app:
    REST interface:

    All feedback is most welcome smile
    Kind Regards,
    Lukasz Zytniak

    Control Software Engineer
    —————————————————————
    Attachments: 1.00Mb 1.00Mb 1.00Mb 1.00Mb 186Kb 400Kb
    ikhokhriakov avatar
    Ingvord
    		Hi All,

    Here I shortly highlight the ideas implemented in mTangoREST API. For more details please refer to the page.

    1. Attributes

    GET {API_ROOT}/device/sys/tg_test/1/double_scalar – access to the 'double_scalar' attr of the TangoTest device. This is simple GET request. The response looks like the following:

    
{
  "argout":3.14,
  "quality":"VALID",
  "timestamp":1404817953999
}


    PUT request writes an attribute:

    PUT {API_ROOT}/device/sys/tg_test/1/double_scalar_w=2%2E78

    NOTE: that a client is responsible for encoding special characters, i.e. %2E => '.' in this particular case.

    Neither DELETE nor POST are not allowed.

    1.1 Image attributes:

    As our applications are browser oriented images are encoded into JPEG and BASE64 and injected into response:

    
{
  "argout":"data:/jpeg;base64,/9j/4AAQSkZJRgABAgAAAQABAAD…",
  "quality":"VALID",
  "timestamp":1404817953999
}

    On the server side byte array received from a remote Tango device is wrapped with BufferedImage without copying the array and is immediately written into the response.

    This approach does not support TIFF images. So the server side must ensure to convert all images into JPEG first.

    2. Commands

    Command execution is always a GET request, even with an argument:

    GET {API_ROOT}/device/sys/tg_test/1/DevString=Hello%20World%20!!!

    And this was a major question when I first started to design the API. GET was chosen just for simplicity – one can execute commands directly from the browser's address bar.

    Only GET is supported.

    3. Events

    GET {API_ROOT}/device/sys/tg_test/1/long_scalar_w.change

    Events implementation is based on the Comet specification, i.e. server blocks request till it gets a notification from the remote Tango device. This approach may utilize all available connections in heavily events oriented applications. Therefore client and server must implement multiplexing/demultiplexing of events' requests. Plus it might be useful to have a timeout so that if server does not get a notification within the specified period of time - it releases the request and responses with an old value. In this case client may resend the request.

    Other supported event types are: periodic, user, archive.

    Only GET is supported.

    4. Family/domain/device browsing

    As Tango hierarchy perfectly matches REST's idea of a resource tree this case is straight forward:

    GET {API_ROOT}/devices

    lists all the devices defined in the Tango DB attached to this mTango host:

    
{
  "argout":["dserver/DataBaseds/2",
                  "dserver/Starter/local-starter",
                  "dserver/TangoAccessControl/1",
                  "dserver/TangoTest/test",
                  "dserver/WebCam/local",
                  "local/webcam/0",
                  "sys/access_control/1",
                  "sys/database/2",
                  "sys/tg_test/1",
                  "tango/admin/local-starter"],
  "timestamp":1432556335002}

    GET {API_ROOT}/device/

    lists all available domains in the Tango DB:

    
{
  "argout":["dserver","local","sys","tango"],
  "timestamp":1432556231252
}

    GET {API_ROOT}/device/{domain}

    lists all families in the Tango DB

    GET {API_ROOT}/device/{domain}/{family}

    lists all members of the family in the Tango DB.

    GET {API_ROOT}/device/{domain}/{family}/{member}

    displays status of the device:

    
{
  "state":"RUNNING",
  "status":"The device is in RUNNING state."
}

    Only GET is supported.

    4. Attributes', commands' infos

    GET {API_ROOT}/device/{domain}/{family}/{member}/attributes

    lists all attributes of the device.

    GET {API_ROOT}/device/{domain}/{family}/{member}/{attribute}/info

    displays info of the attribute.

    Same for the commands:

    GET {API_ROOT}/device/{domain}/{family}/{member}/commands

    GET {API_ROOT}/device/{domain}/{family}/{member}/{command}/info

    4. Caching

    Caching in terms of HTTP response header is not supported. But mTangoREST server performs a binary caching of the response for 200 ms.

    This was a quick overview of the REST API implemented in mTangoREST API. For more details please refer to the page.

    Apart from API specification I think it is important to release some implementation guidelines, i.e. caching and optimizations advises. For instance, implementation must be clever enough to combine several requests, i.e. several clients read the same attribute. This is implemented in mTangoREST server.

    And do we really need so much time?

    In other words currently we will only get a beta version in an year. I believe this task can be accomplished much faster: in three month release alpha version, then perform a beta before next Tango meeting so board can agree on Release Candidate 1.

    Cheers,

    Igor.
    Edited 9 years ago
    vhardion avatar
    hardion
    Ingvord
    Apart from API specification I think it is important to release some implementation guidelines, i.e. caching and optimizations advises. For instance, implementation must be clever enough to combine several requests, i.e. several clients read the same attribute. This is implemented in mTangoREST server.

    And do we really need so much time?

    In other words currently we will only get a beta version in an year. I believe this task can be accomplished much faster: in three month release alpha version, then perform a beta before next Tango meeting so board can agree on Release Candidate 1.

    Thanks for your feedback.
    Great idea for the implementation guidelines. I will add that to the first post.
    Effectively we can have a stable version before the next Tango meeting. An approval from the community can be done online and just have a official presentation to the board.smile

    /Vincent
    Vincent Hardion
    Control System
    MAX IV Laboratory
    ikhokhriakov avatar
    Ingvord
    		Hi,

    Today I have looked through the Solaris REST project. Here is a small overview of it:

    Request:
    
GET http://localhost:8080/restfultango/rest/localhost:10000/Device/false
    This gives a full device list stored in the db:
    
{
  "numberOfDevices":8,
  "device0":"DEVELOPMENT\/WEBCAM\/0", 
  "device1":"DSERVER\/DATABASEDS\/2",
  "device2":"DSERVER\/TANGOACCESSCONTROL\/1",
  "device3":"DSERVER\/TANGOTEST\/TEST",
  "device4":"DSERVER\/WEBCAM\/DEVELOPMENT",
  "device5":"SYS\/ACCESS_CONTROL\/1",
  "device6":"SYS\/DATABASE\/2",
  "device7":"SYS\/TG_TEST\/1",
  "connectionStatus":"OK",
  "Operation time":0
}

    Attribute write/read:

    Attribute write request:
    
PUT restfultango/rest/localhost:10000/Device/sys/tg_test/1/write_attribute/double_scalar_w/3.14

    Response:
    
{"connectionStatus":"OK"}

    Read attribute request:
    
restfultango/rest/localhost:10000/Device/sys/tg_test/1/read_attribute/double_scalar_w

    Response:
    
{
  "devName":"sys\/tg_test\/1",
  "attName":"double_scalar_w",
  "attValue":"measure date: 14\/06\/2015 15:00:30 + 137ms\nquality: VALID\nRead:\t3.14\n",
  "connectionStatus":"OK"
}

    API also provides other interesting features:

    <device-prefix>/ping_device. NOTE from now on <device-prefix> = tango/rest/{host}:{port}/Device/{domain}/{family}/{name}

    <device-prefix>/get_device_info

    Attributes:
    <device-prefix>/get_attribute_list;<device-prefix>/plot_attribute/{attr_name} [Reads attribute and returns its values as array or set of arrays depending if attribute is of type SPECTRUM or IMAGE.]; PUT <device-prefix>/read_attributes

    Tango proxy related:
    <device-prefix>/set_source;<device-prefix>/get_source;<device-prefix>/set_timeout_milis
    /{value}

    Commands:
    <device-prefix>/command_list_query;PUT <device-prefix>/command_inout/{command_name}/{argin};<device-prefix>/black_box/{number_of_commands};PUT <device-prefix>/extract_plot_data/{command_name}

    Dealing with properties:
    <device-prefix>/get_property_list;<device-prefix>/get_property/{prop_name};PUT <device-prefix>/put_property/{prop_name}/{prop_val}

    Interesting ideas of this API:

    Provided access to properties (not implemented in mTangoREST)

    Provided access to proxy control like source, timeout (not implemented in mTangoREST)

    Arguments passed as a part of URL, i.e. <device-prefix>/write_attribute/{attr_name}/{argin} (in mTangoREST arguments are passed via '=', i.e. PUT <device-prefix>/{attr_or_command_name}={argin})

    My thoughts on this API:

    First of all this is not a REST API meaning that it does not follow REST specification. For instance, a tree structure of the resources ain't satisfied, i.e. instead of <device-prefix>/attributes we have <device-prefix>/get_attribute_list. API overflowed with redundant case specific commands, like 'plot_attribute', 'put_property', 'command_inout' and so on.

    Return value: "measure date: 14\/06\/2015 15:00:30 + 137ms\nquality: VALID\nRead:\t3.14\n” seems very strange to me saying the least. This is a string client must parse! Other issue is the date – it should be simple timestamp, because it is much more convenient for the client then to do something with it.

    Errors handling:
    API does not return anything more than “Attribute double_xxx not read. Unable to connect with device sys\/tg_test\/1” [NOTE: there is no such attribute 'double_xxx'], i.e. in case of any error raised by a tango proxy on the serer side a client will get “Unable to connect”

    Tango related features:

    API exports any Tango data base to the outside world. This is very questionable approach, hence very interesting if not 'Device' anchor in between, i.e. GET http://localhost:8080/tango/rest/localhost:10000/sys/tg_test/1 seems much more elegant in this case. Another issue with this anchor is that it is capitalized, i.e. server won't match anything if one asks for …/localhost:10000/device/sys/…

    Tango hierarchy is not supported: GET restfultango/rest/localhost:10000/device/sys/tg_test returns 404

    Commands execution always requires argin, even if it is a command(void). Moreover in case of void it requires special argin=DevVoidArgument

    No events.

    No TangoAccessControl integration.



    Conclusions

    API definitely provides some interesting ideas could be brought into Tango REST API specification. Like access to properties and control of the Tango proxy.


    P.S. next week I will look through GOTAN REST API. Meanwhile I will appreciate if other involved people look through mTangoREST, GOTAN and Solaris's RESTful Tango so we can start to discuss common things and how to combine them into a single Tango REST API
    Edited 9 years ago
    ikhokhriakov avatar
    Ingvord
    		I would like to rise here several questions for further discussion:

    1) should API export Tango database? Or API must be limited with only one DB?

    2) Mapping Tango hierarchy to REST API. Strictly following REST specification we will end up with something like this:

    
GET http://localhost:8080/tango/rest/domains/sys/families/tg_test/members/1/attributes/double_scalar

    Which is quite weired. To avoid this weariness I suggest to replace 'resources' anchor with actual resource:

    
GET http://localhost:8080/tango/rest/sys/tg_test/1/double_scalar

    So that each resource implements has many relationship, i.e.

    
GET http://localhost:8080/tango/rest/sys


    Returns all families of the sys-domain.

    3) Should command be a GET or PUT request?

    4) Passing arguments: there are three possibilities I see so far: in request's body; as a part of URL; or as a parameter of URL (double_scalar?argin=3.14). So which one should we choose?
    Edited 9 years ago
    ikhokhriakov avatar
    Ingvord
    		Unfortunately I was not able to full-scale test the GoTan project. As the only request it servers is 
    GET http://localhost:8080/gotan/objects

    This one returns an array of devices defined in the Tango DB:

    
["dserver/DataBaseds/2","dserver/Starter/local-starter","dserver/TangoAccessControl/1","dserver/TangoTest/test","dserver/WebCam/local","local/webcam/0","sys/access_control/1","sys/database/2","sys/tg_test/1","tango/admin/local-starter"]

    Any other GET request to the server returns 405 (Method is not allowed). And POST an attribute, e.g. POST http://localhost:8080/gotan/sys/tg_test/1/double_scalar crashes the server:

    
11:52:38.958 [ClientMessageReceptor2] INFO  jacorb.orb.iiop - Client-side TCP transport to 192.168.2.100:2179 closed.
11:52:38.958 [ClientMessageReceptor2] DEBUG jacorb.giop.conn - ClientGIOPConnection to 192.168.2.100:2179 (4d0f2471): will wait until connected
2015-06-20	12:33:04	127.0.0.1	-	-	8080	PUT	/gotan/sys/tg_test/1/double_scalar		405	487	4	4	http://localhost:8080	Apache-HttpClient/4.3.2 (java 1.5)	-
2015-06-20	12:33:33	127.0.0.1	-	-	8080	GET	/gotan/sys/tg_test/1/double_scalar		405	487	0	2	http://localhost:8080	Apache-HttpClient/4.3.2 (java 1.5)	-
Stopping the internal [HTTP/1.1] server

    But OK, since it is only a proof of the concept and API is very well described in this document (https://www.dropbox.com/s/9nqeabizxlmj6zp/Gotan%20-%20Tango%20REST.pdf?dl=0) Its implementation can be omitted.

    The only thing from my point of view worth to stress is that API is a little bit messy in terms of using http methods. For instance, attribute write must be PUT not POST, as it does not create anything new, just writes a new value, i.e. updates an attribute. Read attribute must be GET and so on.

    Anyway next week I will prepare a draft document that defines REST API for Tango.
    ikhokhriakov avatar
    Ingvord
    		REST API specification proposal ver. 0.1

    Any input is welcome!
    avatar par défaut
    mikegu
    		Hi,

    It has been useful for me to look at this since I am involved in a project where we are going to use a rest api as well. It looks a well though api, I just have a few comments/questions:

  • Device Commands: In the forum it is specified that commands are still to be decided if they are GET or PUT requests (with a first choice for GET), however in the proposal api it is being used POST. I like the idea of being able to test easily directly through the browser, however I feel more inclined to think a command as a PUT request since you are acting on a device, i.e., triggering the execution of whatever actions the command has.

  • When talking about partial answers, the GET request ~hangs from the device only? The following requests would still be valid?
    • GET /mtango/rest/device/sys/tg_test/1/ampli?fields=argout
GET /mtango/rest/device/sys/tg_test/1/ampli/info?fields=unit, format
GET /mtango/rest/device/sys/tg_test/1/info?fields=classname
    It migth be easier to filter in this way.

  • Managing tango database data, such as get db info, export devices, etc. Therefore
    • GET /mtango/rest/database/info  (answer: number of devices exported, etc.)
PUT/GET /mtango/rest/database/commands/export?argin=device_name
  • Properties: when writing new values for several properties, any property not specified will be deleted. I feel it is a bit risky and prone to property losses. I would suggest to leave the rest of the properties unchanged, a safer approach.

    • I hope it these comments help you somehow, cheers.

    Mikel
    Edited 9 years ago
    ikhokhriakov avatar
    Ingvord
    		Hi Mikel,

    Thanks for your input!

    mikegu
  • Device Commands: In the forum it is specified that commands are still to be decided if they are GET or PUT requests (with a first choice for GET), however in the proposal api it is being used POST. I like the idea of being able to test easily directly through the browser, however I feel more inclined to think a command as a PUT request since you are acting on a device, i.e., triggering the execution of whatever actions the command has.

    • I agree - PUT seems a little bit more natural, as it updates the "state" of the device, i.e. executes a command.


    mikegu
  • When talking about partial answers, the GET request ~hangs from the device only? The following requests would still be valid?

    • There will be a generic filter implementation for any response, i.e. one can always request only that portion of the response which is actually needed. In terms of implementation this will be a servlet filter on the server side (at least I think to implement it in this way).

    mikegu
  • Managing tango database data, such as get db info, export devices, etc. Therefore

    • Tango Database is just another Tango device. You can access it using the same API.

    mikegu
  • Properties: when writing new values for several properties, any property not specified will be deleted. I feel it is a bit risky and prone to property losses. I would suggest to leave the rest of the properties unchanged, a safer approach.

    • I agree that this approach is quite misleading and counterintuitive. I can not image any scenario where it can be useful.

    In fact in the next edition of the proposal I will remove PATCH request at all. Apart from introducing unnecessary complexity there is also a problem - it might not be implemented everywhere. Like in raw Java HTTP library.
     
    Register or login to create to post a reply.