This space has been made publicly available. No login is required to access this information. Only place information classified as UNRESTRICTED in this space.

Overview

Team Wombat has provided feedback to Tango community during their release candidate phase for the 9.5.0 release cycle by testing the release candidates against SKAO module tests. Overall the testing has been a success, we found two issues that have been fixed in the 9.5.0 release.

This is the first time that SKAO has provided feedback during the release candidate phase of a Tango release cycle. The feedback has directly improved the 9.5.0 release, especially for use at SKAO. SKAO should continue to provide this feedback for future release candidate phases, however, we should aim to automate the testing process.

Testing details

The pytango python module that is used at SKAO is split into two separate projects:

  1. cppTango -- the core implementation of the Tango kernel, written in C++
  2. PyTango -- a python wrapper around cppTango

Recently, these two projects have had synchronised release cycles where cppTango makes a release and PyTango will release with the same version number shortly after (aiming for ~1 month). This means for SKAO there are two release candidate phases to provide feedback for, one for cppTango and one for PyTango.

Testing cppTango 9.5.0 Release Candidates (SKAO Jira issue WOM-134)

We started this project at short notice for the cppTango 9.5.0 release candidate phases so we did less testing than for the PyTango release candidates (below). The cppTango project made 4 release candidates for the 9.5.0 starting on 2023-09-03 before finally releasing on 2023-10-09. Due to the time constraints we only did tests with RC3 and RC4. All our testing passed for both release candidates.

To test a cppTango release candidate with the SKAO modules we created an ska-tango-image branch using pytango wheels from the artefacts of the pytango CI for the develop branch (which was tracking the cppTango release candidates). The ska-tango-image CI then produced container images and helm charts which we could use to test other modules.

Modules used for testing

With the ska-tango-image images we ran the make python-test tests of the following modules. Where provided the CI pipeline is a run with RC4, if no CI pipeline is provided the modules was only tested locally:

  • ska-tango-images -- pipeline
  • ska-tango-base -- pipeline
  • ska-tango-examples -- pipeline
  • ska-tango-testing
  • ska-tmc-common
  • ska-csp-lmc-common

We also ran the make k8s-test tests for ska-tango-example.

Issues found

DevInt removal

In cppTango 9.5.0 the DEV_INT enumerator was removed from the CmdArgType enumeration as it was unused in the kernel (https://gitlab.com/tango-controls/cppTango/-/merge_requests/1128). The corresponding python value tango.CmdArgType.DevInt was also removed in the develop branch of PyTango.

As the enumerator was unused in the kernel, it was impossible to actually use it with Tango. However, in ska-tango-base there were a couple of references to tango.CmdArgType.DevInt that needed removing. Other modules depending on ska-tango-base needed an updated version of the module with the references removed.

For most modules we tested we used a patched version of the ska-tango-base main branch (created by e.g. the pipeline above), however, for ska-tango-examples we required updating the 0.12.1 release of ska-tango-base (the pipeline generating the patched 0.12.1 release is https://gitlab.com/ska-telescope/ska-tango-base/-/pipelines/1024280480).

The removal of DevInt is an intentional breaking change that is still present in the 9.5.0 release. It is possible that other SKAO modules will require updating when migrating to 9.5.0, although this is unlikely as DevInt could not really be used with Tango.

PyTango 9.5.0 Release Candidates (SKAO Jira issue WOM-138)

The PyTango project made 3 release candidates starting on 2023-10-27 before releasing 9.5.0 on 2023-11-23. We tested an provided feedback for all three release candidates. The RC3 release candidate passed all out tests.

For the testing we also created an ska-tango-images branch to build container images and helm charts with the CI. However, rather than installing pytango wheels from the PyTango CI, we instead used the prerelease wheels publish to PyPI.

Modules used for testing

The make python-test tests of the following modules was used for testing. Linked pipelines are examples of a passing run with RC3, if no pipeline is linked the tests were only run locally.  These instructions were followed for the PyTango release candidate testing.

We also ran the make k8s-test tests for the following modules

At the time of testing ska-low-mccs's make k8s-test tests were temporarily broken, so we decided to not run these tests this time around. However, going forward we would like to include this test.

In order to run the ska-low-mccs-pasd/ska-low-mccs-spshw make k8s-test tests we needed to generate an updated version of the ska-low-mccs-k8s-test-runner container image/chart (provided by ska-low-mccs-common).

Issues found

DevInt removal

As with the cppTangp 9.5.0 release candidate testing, we needed to use updated versions of ska-tango-base with references to DevInt removed. For ska-tango-examples we used a patched version of ska-tango-base 0.12.1 from the following pipeline: https://gitlab.com/ska-telescope/ska-tango-base/-/pipelines/1078584849.

New PyTango type hinting

With the 9.5.0 release PyTango has introduced a new feature to use python type hints to determine the types of Tango commands and attributes (https://gitlab.com/tango-controls/pytango/-/merge_requests/568).

Unfortunately, the original implementation of this feature (in 9.5.0rc1) was incompatibly with the use of self type hints (which we use extensively at SKAO). With this new feature the ska-tango-base make python-test tests fail. After reporting the issue to the PyTango team it was fixed for 9.5.0rc2 (https://gitlab.com/tango-controls/pytango/-/merge_requests/627).

Although, the self type hints now work for the 9.5.0 release this new type hinting feature requires (perhaps only in some cases) that type hints are "correct", that is they correspond to valid Tango types. For example, we found issues with ska-tmc-common using Any as a type hint.

Regression setting dynamics attributes to None

When testing ska-low-mccs-pasd with 9.5.0rc2 we got segfaults when setting the value of dynamic Tango attributes to None. This was introduced with https://gitlab.com/tango-controls/pytango/-/merge_requests/582.

After reporting it was fixed for 9.5.0rc3 with https://gitlab.com/tango-controls/pytango/-/merge_requests/634.

Plans going forward: Automation

The testing process described above was quite laborious, having to be repeated for each release candidate. We should try to automate this process for the next Tango release cycle.

Tentative requirements for automation

With the experience gained from running the release candidate tests manually we have come up with a first draft at what is required for a automated release candidate testing.

We believe the testing automation should:

  • Be able to specify the version of pytango to use either a prerelease wheel on pypi or a wheel from a pytango CI pipeline. These two sources are required for PyTango and cppTango testing respectively.
  • Use fixed versions of SKA modules. At the start of a release candidate phase, we should fix the versions we are testing so we can confidently verify that fixes to cppTango/PyTango have worked.
  • Be able to run the different steps of the tests locally for debugging.
  • Be able to apply patches for each module. As we found with ska-tango-base when testing 9.5.0, sometimes we need to update modules due to intentional breaking changes in PyTango.
  • Be able to try out new cppTangp/PyTango features. We did not do this for this release, however, we should be able to test new Tango features in an SKAO setting. Perhaps by updating ska-tango-examples.
  • Ideally, run in a single pipeline. It can be quite hard to keep track of several modules with different branches, it would be nice if there was only one place to keep track of.

Conclusion

SKAO testing of the cppTango/PyTango 9.5.0 release candidates was a success and improved the final PyTango 9.5.0 release. We should look to automate this process to make it easier to do for the next release.

Write a comment...