Pipelines
Pipeline is function to process data. It can be used to load, transform, enrich, filter, and route data. Pipelines are the core of the data processing in the system.
Create Pipeline
1) Open Integration Detail Page
2) Open "Pipelines" tab and click on the "Create Pipeline" button
3) Select Application Instance
Choose an application instance from the available registered applications. If an instance does not exist, the user can create a new one directly from this interface.
4) Select Pipeline:
Choose pipeline type from available options for the selected application instance. For most applications,
there are two types of pipelines: "Import" and "Export", but the available options may vary depending on the application.
The system will provide pipeline description and guidance on how to configure the specific pipeline.
5) Set up pipeline: On the pipeline setup page, users can configure the pipeline by providing the necessary information such as pipeline name, source and target entities, and other required settings. On the right side of the page, users can see the "Setup guide" that provides detailed instructions on how to configure the pipeline.
6) Set up schedule: You can define how often the pipeline should run (e.g., every 15 minutes, once a day). The system will automatically run the pipeline according to the defined schedule.
If you want create pipeline without schedule, you can click "Create without schedule" button.
Pipeline List
Pipelines are displayed on the "Pipelines" tab of the Integration Detail Page. Users can quickly see the name, related entities, last run, status, and actions for each pipeline.
Navigation
- Filter pipelines by name using the search bar.
- Pagination to navigate through the list pages.
- Click on a pipeline status to open logs for pipeline last run. If the pipeline failed, the logs will contain the error message.
Actions
- Edit: Modify the pipeline details.
- Delete: Remove the pipeline from the system.