CI Tool for Data Marketplace

CI Tool for NNDM

Capturing or onboarding the Data Product(s) & Contract(s) via the CI tool created by the NNDM Team has the following benefits: - Built-in artifact ID generation that is in the form of a governed uniform resource name (for source systems, teams, products, and contracts). - Single pipeline to upload multiple products and associated contracts. - Pluggable to existing repository that contains data contracts within a folder. - Validates Data Product and Data Contract inputs. - Docker image can be used to run it locally.

Artefacts

Prerequisites: - Source System is registered in ServiceNow (ex. Snowflake) and was onboarded to NNDM by the NNDM Tech Team. - Data Product Team is created & their unique API key is created & communicated by NNDM Team. - Repository is created and owned by LoB.

ARTIFACTS DESCRIPTION LINK
NNDM CI Tool Repository Git repository to see a working example of the CI tool, as well to import the documents into one's own repository Data Marketplace CI Tool Package
Data Contract YAML (Template) Template for a standard Data Contract which can be used as a starting point dataContract.yaml
_Config.toml file Configuration file template that defines the related source system, the Data Product Team, as well as the Data Product(s) Specification(s). config.toml
Pipeline.yaml (CI tool) • Example of a pipeline that includes a step using the CI Tool to do the following actions in the backend:
• Create Data Product(s) & Data Contract(s) for the defined team
• Onboard & associate the specified Source System
• Check for duplicate contracts		 azure-pipelines.yml
API Key API Key for a specific Data Product Team, which is created in NNDM by an ADMIN, and then shared to the Data Product Team in a safe manner Must be requested from NNMD admin/tech support
Docker Image Used to run your pipeline locally Must be requested from NNMD admin/tech support

Steps to capture Data Product(s) & Data Contract(s) in NNDM via the NNDM CI Tool

Step 1:
In your repository, establish a Service Connection to link the container registry with the CI tool’s Docker image so that you can access the required Docker image.
- Docker Registry: https://rescontainerregistry.azurecr.io
- Docker ID: will be provided by NNDM team.
- Docker Password: Will be shared separately.
- Service Connection Name: nndm-container-rescontainerregistry

Step 2:
Locate your Data Contracts or fill out the Data Contract YAML Template (one per Data Contract) and store them in a folder in your repository.

Step 3:
Import/Copy the config.toml file into the root of your repository and fill it out with the Source System ServiceNow ID, NNDM Data Product Team ID, Data Product Information, and the relative path to the previously created Data Contract folder.
- For this step, we have onboarded a Data Product Team and Source System in advance and filled the information in the config.toml:
- Source System ServiceNow ID: 15565 (Example)
- NNDM Data Product Team ID: urn:team:platform:NNDM(domain:test) (Example)

Step 4:
Copy the pipeline into your repository or add it as a step in your existing pipeline.

Step 5:
Configure the required variables for the pipeline:
- NNDM_INSTANCE: Target instance of NNDM to which the product and contracts should be uploaded.
- DEV, TEST, VAL, or PROD
- NNDM_API_KEY: The generated API key from the onboarding for data product team.
- NNDM_CONFIG_FILE_PATH: Name of the Config.toml file

Step 6:
Create a service connection at project level to access the docker image of CI tool present in the Azure Container Registry, NNDM Admin will provide you with the admin and password.

CI Pipeline Code

YAML Pipeline Configuration for NNDM CI Tool

trigger:
- main
pool:
  vmImage: ubuntu-latest
steps:
- checkout: self
  clean: true
  fetchDepth: full
- task: Docker@2
  displayName: ACR Login
  inputs:
    containerRegistry: 'nndm-container-rescontainerregistry'
    command: 'login'
- script: |
    # Run the Docker container
    output=$(docker run -t --name NNDM_INSTANCE="${NNDM_INSTANCE}" -e NNDM_API_KEY="${NNDM_API_KEY}" -e NNDM_MOUNT_PATH=/app/mount -e NNDM_CONFIG_FILE_PATH="${NNDM_CONFIG_FILE_PATH}" -v "${Build.Repository.LocalPath}:/app/mount rescontainerregistry.azurecr.io/devops/nndm-ci:0.2.38x86)
    echo "$output"

    # A JSON output is expected from the tool, so checking if there is any output.
    if [ -z "$(echo "$output" | jq -n 'inputs')" ]; then
        printf "No JSON output found.\n"
        exit 1
    fi

    # Gets the last output from the console and checks if there is any error.
    last_json=$(echo "$output" | jq -n 'last(inputs)')
    error_flag=$(echo "$last_json" | jq -r '.error_flag')
    if [ "$error_flag" == "true" ]; then
        printf "Encountered an error:\n%s\n" "$last_json"
        exit 1
    else
        printf "Products and contracts have been uploaded to NNDM successfully.\n"
    fi
    displayName: 'Running Docker'
  - task: Docker@2
    displayName: ACR Logout
    inputs:
      containerRegistry: 'nndm-container-rescontainerregistry'
      command: 'logout'

Steps for Setting Up the CI Pipeline

Step 1:
Copy the Pipeline Code: Copy the CI pipeline code provided by NNDM team.

Step 2:
Access Your Repository:
- Navigate to the Azure DevOps portal.
- Select your project and then go to the Repos section.

Step 3:
Create a New Pipeline:
- In your repository, go to the azure-pipelines.yml file (or create a new one).
- Paste the copied YAML code into the file.

Step 4:
Configure Environment Variables:
- Ensure that the following environment variables are set in your Azure DevOps pipeline settings:
- NNDM_INSTANCE: Your NNDM instance.
- NNDM_API_KEY: The API key for accessing the NNDM service.
- NNDM_CONFIG_FILE_PATH: The path to your config.toml file.

Step 5:
Submit Changes: Save and commit your changes to trigger the pipeline.

How the Pipeline Works

  • Trigger: The pipeline is automatically triggered by changes to the main branch of your repository.

  • Checkout Phase:

  • The pipeline checks out the source repository code.

  • ACR Login:

  • Logs into the Azure Container Registry to access the necessary Docker images.

  • Run Docker:

  • The pipeline runs a Docker container with environment variables such as your NNDM instance and API key.
  • The output from the Docker container is processed. If a JSON output is expected but not returned, the pipeline fails.

  • The pipeline checks for any errors in the output. If any errors are detected, it will also fail and provide logs for troubleshooting.

  • ACR Logout:

  • Finally, it logs out of the Azure Container Registry.

Creating a Service Connection to Access the CI Tool Docker Image

To enable the CI pipeline to access Docker images, you need to create a service connection. Follow the steps below:

Step 1:
Navigate to Project Settings:
- Go to your Azure DevOps project.
- Click on Project Settings located at the bottom left corner.

Step 2:
Access Service Connections:
- Under the Pipelines section, click on Service connections.

Step 3:
Add a New Service Connection:
- Click on the New service connection button.
- Select Docker Registry from the list of available service connection types.

Step 4:
Enter Required Information:
- Fill in the details as follows:
- Registry type: Select either Docker Hub or Azure Container Registry.
- Docker Registry: Enter the URL for your registry, e.g., https://rescontainerregistry.azurecr.io.
- Docker ID: Enter your registry username, typically the name of your registry. - Docker Password: Provide the password associated with your Docker ID.
- Email (optional): Enter your email address.
- Service Connection Name: Give your service connection a descriptive name, such as nndm-container-reslcontainerregistry.
- Description (optional): Add a description for the service connection if needed.
- Grant access permission to all pipelines: Check this box if you want to allow access to all pipelines.

Step 5: Save the Connection:
- Click the Save button to create the service connection.