diff --git a/.yamllint.yaml b/.yamllint.yaml index 4e14abc..8d5e583 100644 --- a/.yamllint.yaml +++ b/.yamllint.yaml @@ -2,7 +2,9 @@ extends: default rules: line-length: - max: 150 + max: 120 + ignore: | + mkdocs.yml truthy: check-keys: false comments: diff --git a/README.md b/README.md index f6c3a81..93bf92f 100644 --- a/README.md +++ b/README.md @@ -10,7 +10,7 @@ You need uv to run this, take a look at [this](https://docs.astral.sh/uv/getting The project uses pre-commit to enforce YAML and Markdown codestyle, install that with: -```bash +```shell uv run pre-commit install ``` @@ -19,7 +19,7 @@ The expected repos are listed in the mkdocs.yml config file. Comment out these parts if you dont need them. Otherwise, you can clone and install everything you need with: -```bash +```shell ./setup_subprojects.sh ``` @@ -27,7 +27,7 @@ Now you can edit the documentation. To render the documentation locally, run: -```bash +```shell uv run --no-sync mkdocs serve ``` diff --git a/docs/about.md b/docs/about.md index a42cf5f..48455cf 100644 --- a/docs/about.md +++ b/docs/about.md @@ -8,14 +8,14 @@ orchestrating order, administration and provisioning processes. ## Core Principles of WFO - - Open Source and Community-Driven – developed by and for the NREN community, - but also adopted by organisations beyond it. - - User-Friendly and Modular – lightweight, opinionated, and easy to integrate - into existing systems. - - Sustainability and Continuity – working towards a self-sustaining - organisation to maintain and enhance the core software. - - Support and Collaboration – encouraging adoption and supporting the - community through documentation, workshops, and best practices. +- Open Source and Community-Driven – developed by and for the NREN community, + but also adopted by organizations beyond it. +- User-Friendly and Modular – lightweight, opinionated, and easy to integrate + into existing systems. +- Sustainability and Continuity – working towards a self-sustaining + organization to maintain and enhance the core software. +- Support and Collaboration – encouraging adoption and supporting the + community through documentation, workshops, and best practices. ## Governance and project documentation diff --git a/docs/architecture/framework.md b/docs/architecture/framework.md new file mode 100644 index 0000000..88173b1 --- /dev/null +++ b/docs/architecture/framework.md @@ -0,0 +1,33 @@ +The Workflow Orchestrator programme contains multiple components for both the frontend and backend, as shown below: + +![Screenshot](../img/base-orchestrator-setup.png) + +## Backend + +### Orchestrator Core + +The `orchestrator-core` component is an open-source backend component, which defines the ruleset for product modeling and workflows. The `orchestrator-core` is a mandatory component to have a functional workflow orchestrator application. This component is written in Python and makes use of the Fastapi framework. The `orchestrator-core` repo can be found [here](https://github.com/workfloworchestrator/orchestrator-core). The `orchestrator-core` cannot be run standalone, as it contains no definition of any products and workflows. + +### Workflow Orchestrator + +The `Workflow Orchestrator` is the custom implementation of the orchestrator backend. It is the application which defines all your products, workflows and tasks to create/modify/terminate/validate your product instances, the so-called `subscriptions`. All product modeling, tasks/workflows and subscription details are stored in the `orchestrator-coredb` which is part of the orchestrator-core package. This custom implementation of the workflow orchestrator uses the `orchestrator-core` as sub-module. + +With the two backend components set up correctly, you'll have a running Workflow Orchestrator instance accessible via the API. Additionally, with minimal effort, you can have a fully functional frontend application running on top of it. + +### Example Orchestrator + +An example of the `Workflow Orchestrator` using the `orchestrator-core` with some example products and workflow are available [here](https://github.com/workfloworchestrator/example-orchestrator). + +## Frontend + +### Workflow Orchestrator UI library + +The Workflow Orchestrator UI can also be split into 2 major components. A frontend library called the `components orchestrator-ui` library as available on [npm](https://www.npmjs.com/package/@orchestrator-ui/orchestrator-ui-components). And the out-of-the-box `Workflow Orchestrator UI`, which uses the npm library in it's pages. The example of this frontend is the `example-orchestrator-ui` and can be found [here](https://github.com/workfloworchestrator/example-orchestrator-ui). In most cases the example orchestrator is the best deployment model to start with, as is contains a fully functional userinterface, while you can focus your effort on developing products, workflows and tasks. + +### More advanced UI deployment models + +By tweaking the `example-orchestrator-ui` it is possible to easily add extra pages, cards on dashboard page, or change the rending of certain resource type. Examples of the possible changes is shown [here](orchestrator-ui.md) . This will leverage the default architecture, like shown below: + +![Screenshot](../img/custom-orchestrator-setup.png) + +Another approach could be to use individual components from the npm library and build your own application or integrate the components in an existing application. diff --git a/docs/architecture/input-forms.md b/docs/architecture/input-forms.md new file mode 100644 index 0000000..7ed6968 --- /dev/null +++ b/docs/architecture/input-forms.md @@ -0,0 +1,235 @@ +# Forms - from a frontend perspective + +Orchestrator Core contains a module called Pydantic Forms. Pydantic Forms allows for configuration of input forms to collect user input needed for the execution of a workflow. The module contains a frontend part that displays the forms automatically and handles submission and showing validation errors. This documentation describes what happens on the frontend side of this process. + +## Initiating a workflow from frontend + +A workflow can be initiated by doing a POST call to `/processes/` + +The steps that happen to initiate a workflow on the frontend are: + +- A `POST` request to `/processes/` with an empty payload +- The backend determines what input values are missing and sends a response with http status code `510` and a payload containing a [JSON6Schema definition][2] describing the form to display. See [Example JSON6Schema response](#example-json6schema-response) +- The frontend uses the [pydantic-forms library][1] to parse the JSON response into a form to display +- The [AutofieldLoader function][3] is called for each of the form.properties in the JSON response. This function uses the properties `type` and `format` to determine what kind of field will be displayed. + +!!! info + The following documentation is out of date after the migration from `uniforms` to `pydantic-forms`. + Development ticket [orchestrator-ui-library#2381](https://github.com/workfloworchestrator/orchestrator-ui-library/issues/2381) will update this. + +In the example JSON response below one of the properties is + +```json +"customer_id": { + "default": "c9b5e717-0b11-e511-80d0-005056956c1a", + "format": "customerId", + "title": "Customer Id", + "type": "string", + "uniforms": { + "disabled": true, + "value": "c9b5e717-0b11-e511-80d0-005056956c1a" + } +} +``` + +In the autoFieldFunction this maps to a CustomerField. + +```tsx +export function autoFieldFunction( + props, + uniforms, +) { + const { allowedValues, checkboxes, fieldType, field } = props; + const { format } = field; + + switch (fieldType) { + ... + case String: + switch (format) { + ... + case 'customerId': + return CustomerField; + ... + } + ... +``` + +The CustomerField is a React component that is provided by the Orchestrator Component Library. +It's passed the complete property object so it can use them to adjust it's behaviour. + +- A `POST` with the form values is made to the same `/processes/` endpoint +- The response status code can be: + - `400: Form invalid` Invalid values have been detected because the validators the backend runs have failed. An error message is shown. + - `510: FormNotComplete` There is another step. This response contains another json response containing a form. + - `201: Created` The workflow was initiated successfully. The response contains a workflow id and the user is redirected to the workflow detail page + +**Note**. For forms that have multiple steps the user input for each step is accumulated in local frontend state and posted to `/processes/` on each step. The endpoint will receive all available user inputs on each step and determine what other user input it still needs or if it's ready to start the workflow. + +**Note 2** The Orchestrator Component library contains fields that are marked as deprecated and live in a folder named `deprecated`. These contain field types that are very specific to workflows that are in use by SURF. There are plans to remove these from the general purpose components library. + +**Note 3** There are plans to make it easier to extend this functionality to add custom field types and extend the switch statement in the autoFieldFunction to include these custom `types` or `formats` + +## Backend: Creating a workflow that generates a form that asks for user input + +Creating workflows is described in other parts of this documentation in more detail. The practical steps and those that are relevant to the frontend +are these + +- A mapping between a function and a `processes/` endpoint is added to the `workflows/init.py` file + +```python + LazyWorkflowInstance("surf.workflows.core_link.create_core_link", "create_core_link") +``` + +This makes `POST` requests to `processes/create_core_link` call `surf.core_lint.create_core_link` with the `POST` payload + +- The `create_core_link` function is decorated with the `create_workflow` decorator. It provides workflow orchestrator functionality. + +```python +@create_workflow("Create Core Link", initial_input_form=initial_input_form_generator) +def create_core_link() -> StepList: + return ( + begin + >> step 1 + ... + >> step last + ) +``` + +- If the POST request contains no `user_input` the value provided to `initial_input_form` is called, in this case the function `initial_input_form_generator` + +```python +def initial_input_form_generator(product_name: str) -> FormGenerator: + user_input = yield step_1_form(product_name) + + ... + + user_input_ports = yield step_2_form(product_name, ..) + + return ( + ... result from step ... + ) +``` + +- The functionality provided by the workflow orchestrator decorator makes every yield statement pass if the required user input data is passed in or return a response to the client with a `510` status code and a payload containing the definition for the form to display in [JSON Schema 6 format](https://json-schema.org/draft-06/json-schema-release-notes) + +- An example of what the `step_1_form` function could look like. + +```python +def step_1_form(product_name: str) -> type[FormPage]: + class SpeedChoice(Choice): + _10000 = ("10000", "10 Gbps") + _40000 = ("40000", "40 Gbps") + _100000 = ("100000", "100 Gbps") + _400000 = ("400000", "400 Gbps") + + class CreateCoreLinkSpeedForm(FormPage): + model_config = ConfigDict(title=product_name) + + organization: SurfnetOrganisation + + label_core_link_settings: Label + divider_1: Divider + + core_link_service_speed: SpeedChoice = SpeedChoice._400000 + isis_metric: IsisMetric = 20 + + return CreateCoreLinkSpeedForm +``` + +The type specified for each property (eg divider_1: Divider) determines what `type` property it gets in the resulting JSON 6 Schema. There are a set number of property types that can be provided and that are automatically handled by the frontend by default. This is extendable. + +- The response for a POST call without user input to `/processes/create_core_link` is + +## Example JSON6Schema response: + +```json +{ + "type": "FormNotCompleteError", + "detail": [not relevant] + "traceback": [not relevant] + "form": { + "$defs": { + "SpeedChoice": { + "enum": [ + "10000", + "40000", + "100000", + "400000" + ], + "options": { + "10000": "10 Gbps", + "100000": "100 Gbps", + "40000": "40 Gbps", + "400000": "400 Gbps" + }, + "title": "SpeedChoice", + "type": "string" + } + }, + "additionalProperties": false, + "properties": { + "customer_id": { + "default": "c9b5e717-0b11-e511-80d0-005056956c1a", + "format": "customerId", + "title": "Customer Id", + "type": "string", + "uniforms": { + "disabled": true, + "value": "c9b5e717-0b11-e511-80d0-005056956c1a" + } + }, + "label_corelink_settings": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "format": "label", + "title": "Label Corelink Settings", + "type": "string" + }, + "divider": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "format": "divider", + "title": "Divider", + "type": "string" + }, + "corelink_service_speed": { + "allOf": [ + { + "$ref": "#/$defs/SpeedChoice" + } + ], + "default": "400000" + }, + "isis_metric": { + "default": 20, + "maximum": 16777215, + "minimum": 1, + "title": "Isis Metric", + "type": "integer" + } + }, + "title": "SN8 Corelink", + "type": "object" + }, + "title": [not relevant], + "status": 510 +} +``` + +[1]: https://www.npmjs.com/package/pydantic-forms +[2]: https://json-schema.org/draft-06/json-schema-release-notes +[3]: https://github.com/workfloworchestrator/orchestrator-ui-library/blob/main/packages/orchestrator-ui-components/src/components/WfoForms/AutoFieldLoader.tsx diff --git a/docs/architecture/orchestrator-ui.md b/docs/architecture/orchestrator-ui.md new file mode 100644 index 0000000..6062117 --- /dev/null +++ b/docs/architecture/orchestrator-ui.md @@ -0,0 +1,280 @@ +# Workflow Orchestrator UI + +The default Workflow Orchestrator UI app should offer sufficient functionality to start working with, and experiencing, +the workflow orchestrator. For this an +[example-orchestrator-ui](https://github.com/workfloworchestrator/example-orchestrator-ui) is available to start +working with a ready to deploy example workflow orchestrator backend. + +At the same time the UI is developed with the concept in mind that any user of the workflow orchestrator can customize +the UI to meet their own requirements. There are two possible ways to accomplish this: + +- Overriding components +- Using components from the npm UI library + +## Overriding components + +The first solution is based on using the orchestrator-ui library in its full extend and just add/tweak components. +Examples of this approach would be: + +- render certain resource type differently then the npm normally does +- add menu items to the navigation +- add summary cards to the dashboard page + +An example of a custom orchestrator-ui is shown below, which shows a custom summary card and additional menu items +compared to the standard orchestrator-ui. + +## Using components from the npm UI library + +The second solution will probably require more work, but could be interesting to extend an existing application with +orchestrator components. + +Both customization solutions rely on the npm package of the components library published in +[npm](https://www.npmjs.com/package/@orchestrator-ui/orchestrator-ui-components). This package contains the pages and +components that are meant to be used in an app that serves the frontend to a workflow orchestrator backend. + +To have a development setup where both the source code of the app and the source code of this package are available +have a look at the +[Orchestrator UI library repository](https://github.com/workfloworchestrator/orchestrator-ui-library) at the location +`packages/orchestrator-ui`. + +## Example screenshots of orchestrator-ui + +### Standard orchestrator-ui + +![Screenshot](../img/Standard-orchestrator-ui.png) + +### Custom orchestrator-ui + +- showing additional summary card component (in-maintenance core link) +- additional menu items ![Screenshot](../img/Custom-orchestrator-ui-using-override.png) + +### Env variables + +The env variables from .env.example are meant to work with an example workflow orchestrator backend. They can be +changed as needed. At the moment of writing, variable available are: + +- `ENVIRONMENT_NAME`: "Development", "Production", or anything in between. +- `ORCHESTRATOR_API_HOST` (string): The base url of the workflow orchestrator engine. +- `ORCHESTRATOR_API_PATH` (string): The path to the api from the base url. +- `ORCHESTRATOR_GRAPHQL_HOST` (string): The base url to the graphql server. +- `ORCHESTRATOR_GRAPHQL_PATH` (string): The path to the graphql endpoint. +- `ORCHESTRATOR_WEBSOCKET_URL` (string): The url to the websocket server that emits cache invalidation events. +- `AUTH_ACTIVE` (Boolean): If authorization is active or not. +- `USE_WEBSOCKET` (Boolean): Establishes a websocket connection that allows the backend to send cache key invalidation + messages to let the frontend now something has changed on the backend. +- `USE_THEME_TOGGLE` (Boolean): Show a toggle that allows a user to switch from light to dark theme and back. +- `SHOW_WORKFLOW_INFORMATION_LINK` (Boolean): Show a information icon on the workflow detail pages that allows for + linking to an external documentation system. +- `WORKFLOW_INFORMATION_LINK_URL` (string): The url used to build the url that the information icon links to. The + format of the link is: ``. The contents at this url should be maintained + in the external documentation system. + +An up-to-date definition can be found [in the code repository](https://github.com/workfloworchestrator/orchestrator-ui-library/blob/main/packages/orchestrator-ui-components/src/types/types.ts) +in the object defined as `OrchestratorConfig`. + +### Starting a workflow from the orchestrator UI - User input forms + +Creating a Workflow orchestrator workflow is explained here: [Creating a workflow][5]. + +Once a workflow is created it will - automatically - show up in the dropdown list that opens from `+ New Subscription`. +When selected a request will be made to the /processes/ endpoint which will return a form definition +describing the form that needs to be displayed to collect user input. The form definition is turned into a form +automatically. + +The example UI contains an example form that shows an example of a form definition and the form it renders at +`http://localhost:3000/example-form`. + +### Deployment to other environments + +For deployments to your own environment it's recommended to copy or fork the code of the [Example orchestrator UI][1] +repository into your own code repository, add your customization and configuration and deploy from there. + +### Extensibility: + +The Orchestrator UI allows for a number of customization options: + +#### Adding extra pages + +The Orchestrator UI is based on NextJs and its [pages router][6]. Files that are added to the `pages` folder are +automatically rendered at the URL `/`. For example a file a the location `pages/cats.tsx` will cause the +contents of the file to be rendered at the location `http://localhost/cats`. +Please note the pages routes is deprecated by NextJS and might be replaced by the newer [app router][7]. + +#### Adding menu items + +The file `pages/_app.tsx` contains the rendering of `WfoPageTemplate` component. This component takes a callback called +`overrideMenuItems` which receives the current menu items and can alter them. + +```tsx + { + return [ + ...currentMenuItems, + { custom_menu_items } + ] + } + } +> +``` + +#### Adding a custom logo + +The file `pages/_app.tsx` contains the rendering of `WfoPageTemplate` component. This component takes a callback called +`getAppLogo`. The React component that is returned by this function gets displayed as the app logo. + +```tsx + { + return
LOGO
+ }} +> +``` + +#### Authorization and RBAC + +The Orchestrator UI library provides a standard isAllowed handler that wraps certain components and can be used to wrap +pages. The default behaviour is to always return true allowing everything inside it to show. By providing a custom +implementation you can implement your own rules for allowing or disallowing things. + +```tsx + {... custom rules ... }}> + + .... + + +``` + +#### Overriding fields, sections and start page content + +Field values are displayed as plain strings by default and the sections on a page have a set order. For some fields and +sections you might want more control, this is made possible by supplying a `orchestratorComponentOverride` object to the +`StoreProvider` in `_app.tsx`. Currently this is possible for the subscription detail and the start page only. + +The `orchestratorComponentOverride` has these options + +```tsx + ReactElement[]; + }; + subscriptionDetail?: { + valueOverrides?: Record ReactNode>; + generalSectionConfigurationOverride?: ( + defaultSections: WfoSubscriptionDetailGeneralConfiguration[], + subscriptionDetail: SubscriptionDetail, + ) => WfoSubscriptionDetailGeneralConfiguration[]; + }; + } +> +``` + +##### Available functions + +**startpage: summaryCardConfigurationOverride**: A function that gets the default items as input and should return a +new list of items + +**subscriptionsDetail: valueOverrides**: A function that should supply a field name to function mapping where the +function will be called with the field value when the field is rendered. + +**subscriptionDetail: generalSectionConfigurationOverride**: A function that receives the default sections and the +subscription detail object and returns a new list of sections + +### Component library + +The repository that publishes the npm package that is used to supply the layout components is +[Orchestrator UI Component library][8]. Next to pages it exports page components that can be used to build custom pages. + +### Theming + +#### Customizing the theme + +The Workflow Orchestrator frontend ships with a default theme leveraging the theming mechanism of Elastic UI. This +theme can be partially adjusted or completely overridden. + +As part of the boilerplate code, the `_app.tsx` file applies a `defaultOrchestratorTheme` object to the `EuiProvider`. + +```tsx + + + ... + + +``` + +The default defaultOrchestratorTheme object contains adjustments of the standard theme provided by Elastic UI and can +be imported from the +[@orchestrator-ui/orchestrator-ui-components](https://www.npmjs.com/package/@orchestrator-ui/orchestrator-ui-components) +package. + +To make small adjustments, simply use +[`defaultOrchestratorTheme`](https://github.com/workfloworchestrator/orchestrator-ui-library/blob/main/packages/orchestrator-ui-components/src/theme/defaultOrchestratorTheme.ts) +as a base and override the desired properties: + +```tsx +import { EuiThemeModifications } from '@elastic/eui'; +import { defaultOrchestratorTheme } from '@orchestrator-ui/orchestrator-ui-components'; + +function CustomApp(...) { + const myTheme: EuiThemeModifications = { + ...defaultOrchestratorTheme, + colors: { + DARK: { + primary: '#FF69B4', + }, + LIGHT: { + primary: '#32CD32', + }, + }, + }; + + return ( + + + ... + + + ) +} +``` + +The usage of defaultOrchestratorTheme is not required, a new `EuiThemeModifications` can also be made from scratch or +using the [helper tool](https://eui.elastic.co/#/theming/customizing-themes) on the EUI website. + +#### Color Mode + +The `color` property of the theme object contains a `DARK` and `LIGHT` object representing the color mode. The +`_app.tsx` file contains a mechanism to switch and store the color mode. In any given component the +`useOrchestratorTheme` hook can be used to get the current color mode. For more convenience, there is also the +`isDarkThemeActive` boolean: + +```tsx +import { useOrchestratorTheme } from '@orchestrator-ui/orchestrator-ui-components'; + +const WfoAnyComponent: FC = (...) => { + const { + colorMode, // type: EuiThemeColorModeStandard + isDarkThemeActive // type: boolean + } = useOrchestratorTheme(); + + return(...); +} +``` + +[1]: https://github.com/workfloworchestrator/example-orchestrator-ui +[2]: https://nextjs.org +[3]: https://react.dev +[4]: https://www.npmjs.com/package/@orchestrator-ui/orchestrator-ui-components +[5]: https://workfloworchestrator.org/orchestrator-core/architecture/application/workflow +[6]: https://nextjs.org/docs/pages +[7]: https://nextjs.org/docs/app/building-your-application/routing +[8]: https://github.com/workfloworchestrator/orchestrator-ui-library diff --git a/docs/contact.md b/docs/contact.md index a2739ea..4742f20 100644 --- a/docs/contact.md +++ b/docs/contact.md @@ -1,7 +1,6 @@ +We'd love to hear your thoughts, inspiration and ideas! Feel free to contact us via one of the +channels below: -We'd love to hear your thoughts, inspiration and ideas! Feel free to contact us via one of the channels below: -

- -Email: [workfloworchestrator.board@commonsconservancy.org](mailto:workfloworchestrator.board@commonsconservancy.org')
-Github: [https://github.com/workfloworchestrator/](https://github.com/workfloworchestrator/)
-Discord: ![Discord](https://img.shields.io/discord/1295834294270558280?style=flat&logo=discord&label=discord&link=https%3A%2F%2Fdiscord.gg%fQkQn5ajFR) +- Email: [workfloworchestrator.board@commonsconservancy.org](mailto:workfloworchestrator.board@commonsconservancy.org') +- Github: [https://github.com/workfloworchestrator/](https://github.com/workfloworchestrator/) +- Discord: [![Discord](https://img.shields.io/discord/1295834294270558280?style=flat&logo=discord&label=discord&link=https%3A%2F%2Fdiscord.gg%fQkQn5ajFR)](https://discord.gg/fQkQn5ajFR) diff --git a/docs/img/Custom-orchestrator-ui-using-override.png b/docs/img/Custom-orchestrator-ui-using-override.png new file mode 100644 index 0000000..4d55fde Binary files /dev/null and b/docs/img/Custom-orchestrator-ui-using-override.png differ diff --git a/docs/img/Standard-orchestrator-ui.png b/docs/img/Standard-orchestrator-ui.png new file mode 100644 index 0000000..e4bde5f Binary files /dev/null and b/docs/img/Standard-orchestrator-ui.png differ diff --git a/docs/img/base-orchestrator-setup.png b/docs/img/base-orchestrator-setup.png new file mode 100644 index 0000000..6953ef6 Binary files /dev/null and b/docs/img/base-orchestrator-setup.png differ diff --git a/docs/img/custom-orchestrator-setup.png b/docs/img/custom-orchestrator-setup.png new file mode 100644 index 0000000..c12de4d Binary files /dev/null and b/docs/img/custom-orchestrator-setup.png differ diff --git a/docs/index.md b/docs/index.md index eed4495..7aa8025 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,34 +1,55 @@ # Workflow Orchestrator programme -Welcome to the Workflow Orchestrator programme homepage. This opensource collaboration develops software, tools and best practices for automating and orchestrating networks. Our vision is to create an ecosystem of Software that enables users to Automate and Orchestrate their network. - -## Documentation -Find workflow & product modeling, reference documentation and workshop information of the Workflow Orchestrator [here](https://workfloworchestrator.org/orchestrator-core). - - -## Core tooling -The Core tooling developed by the programme: - -- [Orchestrator-Core](https://github.com/workfloworchestrator/orchestrator-core): This Python program leverages the power of FastAPI to create an orchestration engine. Downloads: -[![pypi-downloads](https://static.pepy.tech/badge/orchestrator-core)](https://pepy.tech/project/orchestrator-core). -- [Orchestrator-UI](https://github.com/workfloworchestrator/orchestrator-ui-library): Component library for our NextJS -app on top of the Orchestrator-core. Downloads: -[![npm-downloads](https://img.shields.io/npm/dt/%40orchestrator-ui%2Forchestrator-ui-components)](https://github.com/workfloworchestrator/orchestrator-ui-library). -- [Orchestrator Example UI](https://github.com/workfloworchestrator/example-orchestrator-ui/): Example UI with a NextJS -implementation of our component library. - -## Other Software -Other relevant software maintained by the Workflow Orchestrator programme: - -- [LSO](https://workfloworchestrator.org/lso): This application provides an API layer on top of Ansible playbooks. -- [Example Orchestrator](https://github.com/workfloworchestrator/example-orchestrator): This repository houses a -Docker-compose running a full stack of the Orchestrator, UI and Netbox. It includes examples our best (coding) practices -and an example integration with Netbox. -- [PyNSO-Restconf](https://workfloworchestrator.org/pynso-restconf): A thin client for interfacing with Cisco NSO using -RESTCONF. -- [Pydantic-Forms](https://github.com/workfloworchestrator/pydantic-forms): A library that includes standardised Python -Form classes that can be used when generating form components from JSON-schema. + +Welcome to the Workflow Orchestrator programme homepage. This open-source collaboration develops software, tools and +best practices for automating and orchestrating networks. Our vision is to create an ecosystem of Software that enables +users to Automate and Orchestrate their network. + +

Production ready Workflow Orchestration to manage product lifecycle and workflows. Easy to use, +built on open source software.

+ +The Workflow Orchestrator software ecosystem is maintained by its members and all individual code contributors. More +information about the contributors can be found [here](members.md). + +## Orchestration + +When do you orchestrate and when do you automate? The answer is you probably need both. Automation helps you execute +repetitive tasks reliably and easily. Orchestration adds a layer and allows you to add more intelligence to the tasks +you need to automate and to have a complete audit log of changes. + +## Goal + +Workflow Orchestrator provides a framework with which you can manage service orchestration for your end-users. The +framework helps and guides you through the steps from automation to orchestration. The Workflow Orchestrator allows you +to define products to which users can subscribe. This helps you to intelligently manage their lifecycle, with the use +of creation, modification, validation, and termination workflows. + +## Tooling + +The WFO programme maintains an entire ecosystem of tooling, a non-comprehensive list in no +particular order: + +- [Orchestrator-Core](https://github.com/workfloworchestrator/orchestrator-core): This Python + program leverages the power of FastAPI to create an orchestration engine. Downloads: + [![pypi-downloads](https://static.pepy.tech/badge/orchestrator-core)](https://pepy.tech/project/orchestrator-core). +- [Orchestrator-UI](https://github.com/workfloworchestrator/orchestrator-ui-library): Component + library for our NextJS app on top of the Orchestrator-core. Downloads: + [![npm-downloads](https://img.shields.io/npm/dt/%40orchestrator-ui%2Forchestrator-ui-components)](https://github.com/workfloworchestrator/orchestrator-ui-library). +- [Orchestrator Example UI](https://github.com/workfloworchestrator/example-orchestrator-ui/): + Example UI with a NextJS implementation of our component library. +- [LSO](https://workfloworchestrator.org/lso): This application provides an API layer on top of + Ansible playbooks. +- [Example Orchestrator](https://github.com/workfloworchestrator/example-orchestrator): This + repository houses a Docker-compose running a full stack of the Orchestrator, UI and Netbox. It + includes examples our best (coding) practices and an example integration with Netbox. +- [PyNSO-Restconf](https://workfloworchestrator.org/pynso-restconf): A thin client for interfacing + with Cisco NSO using RESTCONF. +- [Pydantic-Forms](https://github.com/workfloworchestrator/pydantic-forms): A library that includes + standardized Python Form classes that can be used when generating form components from + JSON-schema. - [SuPA](https://workfloworchestrator.org/SuPA): An NSI Ultimate provider agent with a gRPC API. -- [PolyNSI](https://github.com/workfloworchestrator/polynsi): A bidirectional SOAP to gRPC translating proxy server for the NSI protocol. +- [PolyNSI](https://github.com/workfloworchestrator/polynsi): A bidirectional SOAP to gRPC + translating proxy server for the NSI protocol. ### Join the community + You can find this community on Discord. Feel free to join [us](https://discord.gg/fQkQn5ajFR) ![Discord](https://img.shields.io/discord/1295834294270558280?style=flat&logo=discord&label=discord&link=https%3A%2F%2Fdiscord.gg%2fQkQn5ajFR) diff --git a/docs/members.md b/docs/members.md index 86eceba..5f84435 100644 --- a/docs/members.md +++ b/docs/members.md @@ -1,5 +1,5 @@ # Programme Members -Organisations that are involved with the Workflow Orchestrator programme are: +Organizations that are involved with the Workflow Orchestrator programme are: ## Partner @@ -20,4 +20,4 @@ Organisations that are involved with the Workflow Orchestrator programme are: [REANNZ](https://reannz.co.nz) [Internet2](https://internet2.edu) -If you or your organisation would like to contribute please get in [touch](contact.md)! +If you or your organization would like to contribute please get in [touch](contact.md)! diff --git a/docs/workshops/example-orchestrator/bootstrap.md b/docs/workshops/example-orchestrator/bootstrap.md index cda5650..1235a35 100644 --- a/docs/workshops/example-orchestrator/bootstrap.md +++ b/docs/workshops/example-orchestrator/bootstrap.md @@ -5,12 +5,12 @@ setup in our docker-compose.yml file so that you don't have to think about how t The following Docker images are used in this workshop: -* [orchestrator-core](https://github.com/workfloworchestrator/orchestrator-core/pkgs/container/orchestrator-core): The workflow orchestrator step engine. -* [example-orchestrator-ui](https://github.com/workfloworchestrator/example-orchestrator-ui/pkgs/container/example-orchestrator-ui): An example GUI implementation for the orchestrator-core. -* [netbox](https://docs.netbox.dev/en/stable/): An open source IPAM and NSoT system. -* [postgres](https://hub.docker.com/_/postgres): The PostgreSQL object-relational database system. -* [redis](https://redis.io/): An open source in-memory data store. -* [containerlab](https://containerlab.dev/): A free network topology simulator that uses containerized +- [orchestrator-core](https://github.com/workfloworchestrator/orchestrator-core/pkgs/container/orchestrator-core): The workflow orchestrator step engine. +- [example-orchestrator-ui](https://github.com/workfloworchestrator/example-orchestrator-ui/pkgs/container/example-orchestrator-ui): An example GUI implementation for the orchestrator-core. +- [netbox](https://docs.netbox.dev/en/stable/): An open source IPAM and NSoT system. +- [postgres](https://hub.docker.com/_/postgres): The PostgreSQL object-relational database system. +- [redis](https://redis.io/): An open source in-memory data store. +- [containerlab](https://containerlab.dev/): A free network topology simulator that uses containerized Network Operating Systems. ## Prerequisites @@ -20,13 +20,11 @@ The following software is required on your machine to follow this workshop: - [docker-compose](https://docs.docker.com/compose/install/) - [containerlab](https://containerlab.dev/install/) - ### macOS on Apple Silicon !!! note - This only applies to **ARM / Apple Silicon (M-series) Macs**. On Linux and - Intel machines a standard Docker install works — skip to - [Step 1](#step-1-cloning-the-repo). + This only applies to **ARM / Apple Silicon (M-series) Macs**. On Linux and Intel machines a standard Docker install + works — skip to [Step 1](#step-1-cloning-the-repo). **Docker runtime (colima + virtiofs).** On Apple Silicon the containerlab SR Linux nodes need a Docker VM that mounts host @@ -63,53 +61,53 @@ colima ssh -- free -h # Mem total should be ~8 GB setup. Quit Docker Desktop (and disable "Start at login") before using colima. !!! warning "`mountType` / `vmType` are fixed when the VM is created" - The `--mount-type` / `--vm-type` flags are silently ignored on an existing VM. - If your colima VM was created with `sshfs`, recreate it — this destroys the - VM's containers, images and volumes: + The `--mount-type` / `--vm-type` flags are silently ignored on an existing VM. If your colima VM was created with + `sshfs`, recreate it — this destroys the VM's containers, images and volumes: ```shell colima delete && colima start --vm-type vz --mount-type virtiofs --memory 8 --cpu 4 ``` - Memory and CPU, by contrast, *can* be changed on a plain restart: - `colima stop && colima start --memory 8 --cpu 4`. + Memory and CPU, by contrast, *can* be changed on a plain restart: `colima stop && colima start --memory 8 --cpu 4`. -Once the runtime is up, run the `containerlab` commands in this guide from the -maintainers' container — there is no native `containerlab` binary on Apple Silicon: +Once the runtime is up, run the `containerlab` commands in this guide from the maintainers' container — there is no +native `containerlab` binary on Apple Silicon: !!! info - For MacOS ARM there is no `containerlab` executable. - You can run the `containerlab` commands in this guide from within this container provided by the maintainers. - ```shell - docker run --rm -it --privileged \ - --network host \ - -v /var/run/docker.sock:/var/run/docker.sock \ - -v /var/run/netns:/var/run/netns \ - -v /var/lib/docker/containers:/var/lib/docker/containers \ - --pid="host" \ - -v $(pwd):$(pwd) \ - -w $(pwd) \ - ghcr.io/srl-labs/clab bash - ``` + For MacOS ARM there is no `containerlab` executable. You can run the `containerlab` commands in this guide from + within this container provided by the maintainers. + + ```shell + docker run --rm -it --privileged \ + --network host \ + -v /var/run/docker.sock:/var/run/docker.sock \ + -v /var/run/netns:/var/run/netns \ + -v /var/lib/docker/containers:/var/lib/docker/containers \ + --pid="host" \ + -v $(pwd):$(pwd) \ + -w $(pwd) \ + ghcr.io/srl-labs/clab bash + ``` ## Step 1 - Cloning the repo The first step is to clone the Example orchestrator repository using: -``` +```shell git clone https://github.com/workfloworchestrator/example-orchestrator.git ``` + At this point, you have a functional environment to start play with. This includes: -* The orchestrator (core and GUI) -* NetBox (the entire stack including database, workers, etc...) -* LSO (to run ansible playbooks) -* An example containerlab topology based on Nokia SRlinux. -* Some examples of Ansible playbooks +- The orchestrator (core and GUI) +- NetBox (the entire stack including database, workers, etc...) +- LSO (to run ansible playbooks) +- An example ContainerLab topology based on Nokia SR Linux. +- Some examples of Ansible playbooks The directory contains at least these files and directories: -``` +```shell $ ls -1 alembic.ini ansible @@ -139,7 +137,7 @@ Use the command matching your OS to check if any are in use. On Linux you can use `netstat` or `ss`: -``` +```shell # net-tools netstat -tulnp | grep -E ':80|:3000|:4000|:5432|:5678|:8000|:8001|:8080' @@ -149,7 +147,7 @@ ss -tulnp | grep -E ':80|:3000|:4000|:5432|:5678|:8000|:8001|:8080' On macOS: -``` +```shell lsof -nP -iTCP -sTCP:LISTEN | grep -E ':80 |:3000 |:4000 |:5432 |:5678 |:8000 |:8001 |:8080 ' ``` @@ -189,7 +187,7 @@ You should be able to view the applications here: 3. NetBox (admin|admin): [NetBox: http://localhost:8000](http://localhost:8000) !!! note - Take your time to familiarise with the applications and make sure they are working correctly. +Take your time to familiarize with the applications and make sure they are working correctly. If anything is wrong, inspect the results of these commands: @@ -202,18 +200,18 @@ You should be able to view the applications here: docker compose logs -tf -n 5 ``` -## Step 4 - Containerlab +## Step 4 - ContainerLab -Now that we have our orchestrator stack running, we can spin up the containerlab topology: +Now that we have our orchestrator stack running, we can spin up the ContainerLab topology: -``` +```shell cd clab containerlab deploy ``` At the end of this process we can use `containerlab inspect` to check the status of our topology: -``` +```text ╭───────────────────────┬──────────────────────────────┬─────────┬────────────────╮ │ Name │ Kind/Image │ State │ IPv4/6 Address │ ├───────────────────────┼──────────────────────────────┼─────────┼────────────────┤ @@ -233,14 +231,14 @@ At the end of this process we can use `containerlab inspect` to check the status And with the command: -``` +```shell containerlab graph ``` This will serve a nice rendering of the topology on port [50080](http://localhost:50080). !!! info - When using containerlab's docker image, this does not work. + When using the ContainerLab docker image, this does not work. The topology we are going to use is something like this one: @@ -248,10 +246,10 @@ The topology we are going to use is something like this one: The Example orchestrator used in this workshop already has a number of products pre-configured and ready to be used: -* Nodes (including Ansible to deploy example config) -* Core-links (including Ansible to deploy/delete example config) -* Ports -* L2VPN +- Nodes (including Ansible to deploy example config) +- Core-links (including Ansible to deploy/delete example config) +- Ports +- L2VPN We can start feeding initial data into the environment and run some workflows! @@ -261,7 +259,7 @@ We can start feeding initial data into the environment and run some workflows! To completely reset your environment and start from scratch, follow these steps. -If you had containerlab deployed, destroy the deployed nodes: +If you had ContainerLab deployed, destroy the deployed nodes: ```shell containerlab destroy diff --git a/docs/workshops/example-orchestrator/create-your-own.md b/docs/workshops/example-orchestrator/create-your-own.md index 72e8367..63bbb40 100644 --- a/docs/workshops/example-orchestrator/create-your-own.md +++ b/docs/workshops/example-orchestrator/create-your-own.md @@ -4,7 +4,5 @@ To cap off this workshop we will create a new product and workflows by using the Orchestrator provides the user. In this scenario you will create a product that is very similar to the provided L2VPN product, but constrained to two interfaces. In other words a L2 Point-to-Point circuit. -## L2 Point-to-Point model - -{{ external_markdown('https://raw.githubusercontent.com/workfloworchestrator/orchestrator-core/main/docs/architecture/product_modelling/l2_point_to_point.md', -'') }} +A description of what this L2 product could look like can be found +[in this section of the documentation](../../../orchestrator-core/architecture/product-modeling/l2-point-to-point/). diff --git a/docs/workshops/example-orchestrator/domain-models.md b/docs/workshops/example-orchestrator/domain-models.md index c19d5da..5e84128 100644 --- a/docs/workshops/example-orchestrator/domain-models.md +++ b/docs/workshops/example-orchestrator/domain-models.md @@ -2,9 +2,10 @@ ## Introduction -First read the [Architecture; TL;DR](../../orchestrator-core/architecture/tldr.md) section of the orchestrator core documentation to get an overview of the -concepts that will be covered. +First read the [Architecture; TL;DR](../../orchestrator-core/architecture/tldr.md) section of the orchestrator core +documentation to get an overview of the concepts that will be covered. ## Products + {{ external_markdown('https://raw.githubusercontent.com/workfloworchestrator/example-orchestrator/main/README.md', '## Products') }} diff --git a/docs/workshops/example-orchestrator/execute-workflows.md b/docs/workshops/example-orchestrator/execute-workflows.md index 67492dc..8e08613 100644 --- a/docs/workshops/example-orchestrator/execute-workflows.md +++ b/docs/workshops/example-orchestrator/execute-workflows.md @@ -1,35 +1,38 @@ The topology in the previous section will be used in the workshop as an example of what a network could look like. -Obviously it is possible to create any "physical" topology you like and build the matching "logical" topology -using the Workflow Orchestrator. +Obviously it is possible to create any "physical" topology you like and build the matching "logical" topology using the +Workflow Orchestrator. ### Putting initial data in place -The first thing we will do is populate NetBox with some initial data such as Manufacturers and Device types, -as well as networks allocated for Loopback and Core-links addressing. +The first thing we will do is populate NetBox with some initial data such as Manufacturers and Device types, as well as +networks allocated for Loopback and Core-links addressing. -To do this, navigate to the **Tasks** page, click **New task** and then select **NetBox Bootstrap**. -An empty form will show where you can click **Start task**. +To do this, navigate to the **Tasks** page, click **New task** and then select **NetBox Bootstrap**. An empty form will +show where you can click **Start task**. -Once the workflow has successfully run, we can log in to NetBox (admin/admin) and check the situation: within the **Devices** menu you can see that the **Device Types** and **Manufacturers** now contain some definitions. +Once the workflow has successfully run, we can log in to NetBox (admin/admin) and check the situation: within the +**Devices** menu you can see that the **Device Types** and **Manufacturers** now contain some definitions. -In the **IPAM >> Prefixes** menu we are going to reserve the first address of the loopback networks since certain network devices don't like "network addresses" to be used as loopback addresses. +In the **IPAM >> Prefixes** menu we are going to reserve the first address of the loopback networks since certain +network devices don't like "network addresses" to be used as loopback addresses. -* From the IPv4 prefix `10.0.127.0/24` we allocate the address `10.0.127.0/32` with status `Reserved` and description `RESERVED` -* From the IPv6 prefix `fc00:0:0:127::/64` we allocate the address `fc00:0:0:127::/128` with status `Reserved` and description `RESERVED` +- From the IPv4 prefix `10.0.127.0/24` we allocate the address `10.0.127.0/32` with status `Reserved` and description + `RESERVED` +- From the IPv6 prefix `fc00:0:0:127::/64` we allocate the address `fc00:0:0:127::/128` with status `Reserved` and + description `RESERVED` ### Deploying the nodes -Now we should be able to deploy our routers using the `create node` workflow in the Orchestrator. -To do this, click the **New subscription** button and select **node Nokia**. -This is going to create a new subscription of the "node Nokia" product. -We will have to fill in an initial form. +Now we should be able to deploy our routers using the `create node` workflow in the Orchestrator. To do this, click the +**New subscription** button and select **node Nokia**. This is going to create a new subscription of the "node Nokia" +product. We will have to fill in an initial form. -* **Customer**: SURF -* **Node type**: Select the first option -* **Node role**: Provider -* **Site**: Amsterdam -* **Node status**: active -* **Node name**: `clab-orch-demo-ams-pe` +- **Customer**: SURF +- **Node type**: Select the first option +- **Node role**: Provider +- **Site**: Amsterdam +- **Node status**: active +- **Node name**: `clab-orch-demo-ams-pe` Click **Next**, review the summary page, and click **Next** again to start the workflow. @@ -44,15 +47,15 @@ ssh clab-orch-demo-ams-pe -l admin ``` !!! note - If you deployed containerlab through the docker image, run the ssh commands from the same container. + If you deployed ContainerLab through the docker image, run the ssh commands from the same container. For example, observe the IP addresses assigned to the loopback interface, e.g. `show interface lo1.0`. This corresponds to what you see in NetBox in the **Devices >> Devices** menu. -Repeat this process for the other 2 containerlab nodes: +Repeat this process for the other 2 ContainerLab nodes: -* `clab-orch-demo-lon-pe` -* `clab-orch-demo-par-p` +- `clab-orch-demo-lon-pe` +- `clab-orch-demo-par-p` ### Deploying core links @@ -68,7 +71,7 @@ Assuming that the result was successful, click **Resume workflow**. You can log in to the router and check the status of ISIS using: -``` +```text show network-instance default protocols isis adjacency show network-instance default protocols isis interface ``` diff --git a/docs/workshops/example-orchestrator/generator.md b/docs/workshops/example-orchestrator/generator.md index 602a61e..b6f04ac 100644 --- a/docs/workshops/example-orchestrator/generator.md +++ b/docs/workshops/example-orchestrator/generator.md @@ -9,7 +9,7 @@ the workflow and start developing on implementing the business logic. Open the Example Orchestrator directory and list the templates directory. It should look similar to this: -```bash +```shell ❯ ls -l templates total 32 -rw-r--r-- 1 boers001 staff 2687 Mar 7 11:28 core_link.yaml @@ -26,7 +26,7 @@ L2-Point-to-Point model and workflows by configuring it with this yaml file. Create a new file in the template directory called `l2-p2p.yaml` -```bash +```shell touch templates/l2-p2p.yaml ``` @@ -49,7 +49,6 @@ reuse as many of the product blocks already existing in the orchestrator as poss generator due to the different limits on the amount of SAPS that can be connected to the Virtual Circuit of the L2 P2P product. - #### Yaml file ```yaml config: summary_forms: true @@ -102,7 +101,7 @@ reuse as many of the product blocks already existing in the orchestrator as poss To help generate the correct file exec into the running container and activate the Python venv: -```bash +```shell docker compose exec -it orchestrator bash source .venv/bin/activate ``` @@ -111,7 +110,7 @@ source .venv/bin/activate Run the command to generate the domain models product blocks: -```bash +```shell python main.py generate product-blocks -cf templates/l2-p2p.yaml --no-dryrun ``` @@ -120,28 +119,29 @@ The `--no-dryrun` option will immediately write the files to the `products/produ product and defines the strict hierarchy of virtual circuit and saps. #### Product + Now create the product. -```bash +```shell python main.py generate product -cf templates/l2-p2p.yaml --no-dryrun ``` This will create the file `products/product_types/l2_p2p.py`. When looking at this file you can see it created the domain model, fixed inputs and imported the correct product blocks to be used in this subscription. - #### Workflows Now generate the workflows. This command will always create 4 sets of workflows `create`, `modify`, `terminate` and `validate`. These can be implemented as the user sees fit. !!! warning - There is a bug in `generate workflows` which makes it overwrite the contents of workflows/shared.py. - Make sure to backup the contents, or restore it afterwards. - Bug is tracked in https://github.com/workfloworchestrator/orchestrator-core/issues/1757. + There is a bug in `generate workflows` which makes it overwrite the contents of workflows/shared.py. Make sure to + backup the contents, or restore it afterwards. Bug is tracked in + https://github.com/workfloworchestrator/orchestrator-core/issues/1757. Run the command: -```bash + +```shell python main.py generate workflows -cf templates/l2-p2p.yaml --no-dryrun --force ``` @@ -150,25 +150,28 @@ files. Furthermore it will populate the files in `workflows.l2_p2p`. Feel free t already has done. #### Database migrations + As a final step the user must generate and run the migrations to wire up the database. This is done as follows. -```bash +```shell python main.py generate migration -cf templates/l2-p2p.yaml python main.py db upgrade heads ``` ### Step 4 - Profit + If this has been executed without errors, you should be able to create a new subscription for the l2-p2p product by running the create workflow through the UI. All it does is create the domain model and fill it in with some rudimentary values from the input form, but it's a starting point. Users can now go into the workflow source code and start implementing steps to provision the resource that is being created by the create workflow. Take some time in the orchestrator UI to see what has been configured. -* Metadata pages -* Action menu -* Available workflows +- Metadata pages +- Action menu +- Available workflows ### Step 5 - Bonus + Implement a new step in the create workflow that manipulates the subscription in a certain way. An example could be to change the subscription description, or any other value you can think of that exists in the subscription. diff --git a/docs/workshops/example-orchestrator/overview.md b/docs/workshops/example-orchestrator/overview.md index 44d4e2b..1b6a328 100644 --- a/docs/workshops/example-orchestrator/overview.md +++ b/docs/workshops/example-orchestrator/overview.md @@ -9,22 +9,22 @@ as well as teaching you how to relate products to other products, using the depe Workflow Orchestrator. !!! tip - Knowledge of the Python programming language, Docker, and the Unix command line interface are prerequisites for this workshop. - + Knowledge of the Python programming language, Docker, and the Unix command line interface are prerequisites for + this workshop. ## Topics -* **Installation** +- **Installation** Detailed instructions are given on how to prepare your environment and install the orchestrator and GUI using docker compose. -* **Start applications** +- **Start applications** Outline how to start the Workflow Orchestrator backend and GUI using docker compose. -* **Bootstrapping the applications and familiarisation** +- **Bootstrapping the applications and familiarization** Through a simple network node and network circuit scenario, a set of products is created showing how domain models are defined. - * **Domain models** + - **Domain models** Explains the benefits of the use of domain models and shows how the hierarchy of products, product blocks, fixed inputs and resource types are used to create product subscriptions for customers. -* **Workflows** +- **Workflows** Introduces the workflow concept and how the create, modify, terminate, and validate workflows relate to the product model, using the example orchestrator's node workflows as a reference. -* **L2 Point-to-Point product modelling and workflow** +- **L2 Point-to-Point product modeling and workflow** For the L2 Point-to-Point product, we will make the CREATE workflow by using the product generator. The use of input forms is explained as part of defining the create workflow. By using this method you should be able to quickly get up to speed and start coding quickly. @@ -33,9 +33,9 @@ Workflow Orchestrator. This workshop uses the following folder layout: -```text +```shell ├── migrations -│ └── versions +│ ├── versions │ └── schema ├── products │ ├── product_blocks @@ -54,32 +54,32 @@ This workshop uses the following folder layout: ## Workshop software architecture -The workshop combines a number of opensource software components that can provision a simulated network -running in containerlab. The following diagram shows the logical components of the application and how the data +The workshop combines a number of open source software components that can provision a simulated network +running in ContainerLab. The following diagram shows the logical components of the application and how the data flows. In reality there are a number of extra services like Postgres and Redis that store the application data of the Orchestrator, NetBox and LSO. ### Software used in the workshop -* **The orchestrator**: This includes the UI and python backend that will run all workflows. All data is persisted in a Postgres database. Redis is used for event broadcasting and synchronisation purposes. -* **[NetBox](https://docs.netbox.dev/en/stable/)**: NetBox is the source of truth for this network topology. It contains all resources that are known +- **The orchestrator**: This includes the UI and python backend that will run all workflows. All data is persisted in a Postgres database. Redis is used for event broadcasting and synchronization purposes. +- **[NetBox](https://docs.netbox.dev/en/stable/)**: NetBox is the source of truth for this network topology. It contains all resources that are known in the topology: Interfaces, Nodes, IP addresses etc. The Orchestrator will configure NetBox but also retrieve resources from it. -* **[LSO](https://github.com/workfloworchestrator/lso)**: The Network Resource Manager (NRM) of this topology. This software is an API abstraction on top of +- **[LSO](https://github.com/workfloworchestrator/lso)**: The Network Resource Manager (NRM) of this topology. This software is an API abstraction on top of ansible that integrates well with the orchestrator. It is responsible for running ansible jobs to provision the topology. -* **[containerlab](https://containerlab.dev/)**: This software will manage the (virtual) network topology running the Network Operating System +- **[containerlab](https://containerlab.dev/)**: This software will manage the (virtual) network topology running the Network Operating System of the workshop. Below the network topology is explained. ![Software topology](../images/Software-topology.drawio.png) ## Workshop topology -Once you have installed the example orchestrator with containerlab integration enabled (explained in the next section), +Once you have installed the example orchestrator with ContainerLab integration enabled (explained in the next section), we will build the workshop topology that can be used to actually see packets flow. The workflows that you will run in the following steps will do the following: -* Seed NetBox -* Provision two PE nodes -* Create an IS-IS cloud to signal MPLS LSPs with backbone links -* Provision customer Ports that can be used in network services +- Seed NetBox +- Provision two PE nodes +- Create an IS-IS cloud to signal MPLS LSPs with backbone links +- Provision customer Ports that can be used in network services The topology will be as follows: diff --git a/mkdocs.yml b/mkdocs.yml index 44095d9..1eeec58 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -6,6 +6,10 @@ nav: - About: about.md - Contact: contact.md - Leadership: leadership.md + - Architecture: + - The Framework: architecture/framework.md + - Orchestrator UI: architecture/orchestrator-ui.md + - Input Forms: architecture/input-forms.md - Community: - Members: members.md - Meetings: meetings.md @@ -91,7 +95,6 @@ plugins: - external-markdown - open-in-new-tab - render_swagger - - macros - include-markdown - mkdocstrings: default_handler: python @@ -138,6 +141,11 @@ plugins: "orchestrator-core/workshops/advanced/node-validate.md": "workshops/example-orchestrator/node-validate.md" "orchestrator-core/workshops/advanced/create-your-own.md": "workshops/example-orchestrator/create-your-own.md" "orchestrator-core/workshops/advanced/generator.md": "workshops/example-orchestrator/generator.md" + "orchestrator-core/architecture/framework.md": "architecture/framework.md" + "orchestrator-core/architecture/orchestration/orchestrator-ui.md": "architecture/orchestrator-ui.md" + "orchestrator-core/architecture/application/forms-frontend.md": "architecture/input-forms.md" # orchestrator-ui-library plugins - termynal - awesome-pages + # workfloworchestrator.org general plugin that has to be loaded after awesome-pages + - macros