Skip to content

feat: Use workflows for config and log management operations #2844

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 22 commits into from
May 31, 2024
Merged

feat: Use workflows for config and log management operations #2844

merged 22 commits into from
May 31, 2024

Conversation

didier-wenzek
Copy link
Contributor

@didier-wenzek didier-wenzek commented Apr 26, 2024 •
edited
Loading

Proposed changes

Update the log and config operations so those can be used by the workflow system,
enabling the ability to trigger a log or config update as a sub operation.

  • Make log and config message types similar to those used for the workflows
  • Dynamically register Restart operation
  • Dynamically register Software operations
  • Make the LogManagerActor acting on LogUploadCmd instead of MqttMessage
  • Make the ConfigManagerActor acting on ConfigSnapshotCmd & ConfigUpdateCmd instead of MqttMessage
  • Enable log operation workflows
  • Enable config operation workflows
  • Add system tests checking that LogUpload can be triggered as sub operations
  • Add system tests checking that LogUpload can be overridden
  • Add system tests checking that ConfigSnapshot can be triggered as sub operations
  • Add system tests checking that ConfigSnapshot can be overridden
  • Add system tests checking that ConfigUpdate can be triggered as sub operations
  • Add system tests checking that ConfigUpdate can be overridden

Types of changes

  • Bugfix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Improvement (general improvements like code refactoring that doesn't explicitly fix a bug or add any new functionality)
  • Documentation Update (if none of the other choices apply)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)

Paste Link to the issue

#2836

Checklist

  • I have read the CONTRIBUTING doc
  • I have signed the CLA (in all commits with git commit -s)
  • I ran cargo fmt as mentioned in CODING_GUIDELINES
  • I used cargo clippy as mentioned in CODING_GUIDELINES
  • I have added tests that prove my fix is effective or that my feature works
  • I have added necessary documentation (if appropriate)

Further comments

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Apr 26, 2024 •
edited
Loading

Robot Results

✅ Passed ❌ Failed ⏭️ Skipped Total Pass % ⏱️ Duration
450 0 3 450 100 1h13m12.911536s

This was referenced Apr 29, 2024
Merged
Merged
@didier-wenzek didier-wenzek force-pushed the feat/config-and-log-management-workflows branch from 1b59fad to 21e55cb Compare April 29, 2024 13:16
@reubenmiller reubenmiller changed the title Feat: Config and log management workflows feat: Config and log management workflows Apr 30, 2024
@didier-wenzek didier-wenzek mentioned this pull request May 2, 2024
Merged
11 tasks
@didier-wenzek didier-wenzek force-pushed the feat/config-and-log-management-workflows branch from 340cf1a to 581d6d7 Compare May 3, 2024 11:44
@didier-wenzek didier-wenzek temporarily deployed to Test Pull Request May 3, 2024 11:44 — with GitHub Actions Inactive
@codecov Codecov
Copy link

codecov bot commented May 3, 2024 •
edited
Loading

Codecov Report

Attention: Patch coverage is 65.20270% with 206 lines in your changes are missing coverage. Please review.

Project coverage is 78.0%. Comparing base (0b974d5) to head (15b7e26).
Report is 3 commits behind head on main.

Additional details and impacted files
Files Coverage Δ
crates/common/mqtt_channel/src/topics.rs 86.2% <ø> (ø)
crates/core/c8y_api/src/json_c8y.rs 92.4% <ø> (ø)
crates/core/tedge_actors/src/builders.rs 96.6% <100.0%> (+19.6%) ⬆️
crates/core/tedge_actors/src/channels.rs 84.3% <ø> (+13.7%) ⬆️
crates/core/tedge_agent/src/agent.rs 0.0% <ø> (ø)
...tes/core/tedge_agent/src/software_manager/actor.rs 62.7% <100.0%> (+3.4%) ⬆️
...tedge_agent/src/tedge_operation_converter/tests.rs 94.5% <100.0%> (+0.6%) ⬆️
crates/core/tedge_api/src/lib.rs 100.0% <100.0%> (ø)
crates/core/tedge_api/src/workflow/mod.rs 60.5% <100.0%> (ø)
...s/c8y_mapper_ext/src/operations/config_snapshot.rs 90.5% <100.0%> (ø)
... and 20 more

... and 4 files with indirect coverage changes

@didier-wenzek didier-wenzek force-pushed the feat/config-and-log-management-workflows branch from 581d6d7 to 672fd42 Compare May 7, 2024 12:43
@didier-wenzek didier-wenzek temporarily deployed to Test Pull Request May 14, 2024 13:09 — with GitHub Actions Inactive
@didier-wenzek didier-wenzek temporarily deployed to Test Pull Request May 14, 2024 13:21 — with GitHub Actions Inactive
@didier-wenzek didier-wenzek force-pushed the feat/config-and-log-management-workflows branch from fa4db4f to a988410 Compare May 14, 2024 14:19
@didier-wenzek didier-wenzek temporarily deployed to Test Pull Request May 14, 2024 14:19 — with GitHub Actions Inactive
@didier-wenzek didier-wenzek temporarily deployed to Test Pull Request May 15, 2024 10:16 — with GitHub Actions Inactive
@didier-wenzek didier-wenzek temporarily deployed to Test Pull Request May 15, 2024 11:09 — with GitHub Actions Inactive
@didier-wenzek didier-wenzek force-pushed the feat/config-and-log-management-workflows branch from c1cb694 to 0bce80d Compare May 15, 2024 15:53
@didier-wenzek didier-wenzek temporarily deployed to Test Pull Request May 15, 2024 15:53 — with GitHub Actions Inactive
@didier-wenzek
Dynamically register the restart operation
615c815 
Now the workflow actor (aka TedgeOperationConverterActor) has no more
specific code related to the restart operation.

Signed-off-by: Didier Wenzek <didier.wenzek@free.fr>
@didier-wenzek
Rename Command::try_from into try_from_bytes
1b2b6cf 
To avoid name conflicts with TryFrom<Command>

Signed-off-by: Didier Wenzek <didier.wenzek@free.fr>
@didier-wenzek
Dynamically register software operations
aa97ce0 
Now the workflow actor (aka TedgeOperationConverterActor) has no more
specific code related to any specific operation.

Signed-off-by: Didier Wenzek <didier.wenzek@free.fr>
@didier-wenzek
Dynamically register log upload operation
17156b9 
Signed-off-by: Didier Wenzek <didier.wenzek@free.fr>
@didier-wenzek
Improve ConfigOperation to be used as main ConfigManagerActor input
b2b0285 
This is a preparation step to make the ConfigManagerActor MQTT
independent and ready to be supported by agent workflows.

Signed-off-by: Didier Wenzek <didier.wenzek@free.fr>
@didier-wenzek
Introduce ConfigOperationData to be used as main ConfigManagerActor o…
f2f7eaf 
…utput

This is a preparation step to make the ConfigManagerActor MQTT
independent and ready to be supported by agent workflows.

Signed-off-by: Didier Wenzek <didier.wenzek@free.fr>
@didier-wenzek
Remove unsafe unwrap
ae0465b 
A topic filter cannot be safely cast into a topic.

Signed-off-by: Didier Wenzek <didier.wenzek@free.fr>
@didier-wenzek
Remove ConfigManagerActor dependency on MQTT
b8d4e81 
The builder still expect an MQTT connection.
This will be removed when connecting ConfigManagerActor to the workflow
system.

Signed-off-by: Didier Wenzek <didier.wenzek@free.fr>
@didier-wenzek
Dynamically register config snapshot and upload operations
dc71c52 
These operations can now be triggered as a sub-workflow.

Signed-off-by: Didier Wenzek <didier.wenzek@free.fr>
@didier-wenzek
Fix capability messages of builtin operations
846e5f0 
Capability messages were erroneously sent twice for log upload, config
snapshot and config update.

Signed-off-by: Didier Wenzek <didier.wenzek@free.fr>
@didier-wenzek
Fix log messages
7ae8ff6 
Signed-off-by: Didier Wenzek <didier.wenzek@free.fr>
@didier-wenzek
Bypass the operation file cache for config update
df1eb41 
The operation file cache has to be redesigned as it's currently not
compatible with the workflow mechanism.

- The operation file cache runs on the main device while the workflow
  system runs on a child device: hence the communication must be over
  MQTT or HTTP.
- However, the operation-specific actors run behind the workflow actor
  and only the latter receives MQTT messages. A new mechanism is
  required for some messages to be pushed to the operation-specific
  actors.
- Furthermore, the config update workflow currently use the same state
  (executing) for two different stages (before and after cache).
  This prevents the config update operation to be properly handled by
  workflow.

I propose to disable - at least temporary, this cache for config
updates.

Signed-off-by: Didier Wenzek <didier.wenzek@free.fr>
@didier-wenzek
Test log upload operation workflow
32aba13 
Signed-off-by: Didier Wenzek <didier.wenzek@free.fr>
@didier-wenzek
Test config_snapshot operation workflow
2119aac 
Signed-off-by: Didier Wenzek <didier.wenzek@free.fr>
@didier-wenzek
Test config_update operation workflow
15b7e26 
Signed-off-by: Didier Wenzek <didier.wenzek@free.fr>
@didier-wenzek didier-wenzek force-pushed the feat/config-and-log-management-workflows branch from ca0f4cd to 15b7e26 Compare May 31, 2024 07:00
@didier-wenzek didier-wenzek temporarily deployed to Test Pull Request May 31, 2024 07:00 — with GitHub Actions Inactive
reubenmiller
reubenmiller approved these changes May 31, 2024
View reviewed changes
Copy link
Contributor

@reubenmiller reubenmiller left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved.

Great job (especially for the sustained effort!)

@didier-wenzek didier-wenzek added this pull request to the merge queue May 31, 2024
Merged via the queue into thin-edge:main with commit 08577c9 May 31, 2024
@didier-wenzek didier-wenzek deleted the feat/config-and-log-management-workflows branch May 31, 2024 07:47
@reubenmiller reubenmiller added theme:configuration Theme: Configuration management theme:troubleshooting Theme: Troubleshooting and remote control labels May 31, 2024
@didier-wenzek didier-wenzek mentioned this pull request May 31, 2024
Closed
@reubenmiller reubenmiller changed the title feat: Config and log management workflows feat: Use workflows for config and log management operations Aug 1, 2024
@reubenmiller reubenmiller added theme:workflows Theme: Workflow engine topics and removed theme:workflows Theme: Workflow engine topics labels Aug 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@albinsuresh albinsuresh albinsuresh approved these changes

@reubenmiller reubenmiller reubenmiller approved these changes

@jarhodes314 jarhodes314 Awaiting requested review from jarhodes314 jarhodes314 is a code owner

@rina23q rina23q Awaiting requested review from rina23q rina23q is a code owner

@gligorisaev gligorisaev Awaiting requested review from gligorisaev

Assignees
No one assigned
Labels
theme:configuration Theme: Configuration management theme:troubleshooting Theme: Troubleshooting and remote control
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@didier-wenzek @albinsuresh @reubenmiller