Skip to content

Solution Design#

Overview#

Solution Design is the critical phase where your finalized requirements are transformed into a comprehensive technical blueprint. This phase bridges the gap between business needs and implementation, resulting in detailed architecture documentation and formal approval through the Technical Design Authority (TDA) process.

What you will learn#

After completing this chapter, you will understand how to:

  • Transform requirements into detailed technical architecture designs
  • Create comprehensive data models that support scalable data products
  • Design robust data contracts and lineage mechanisms for NNDM integration
  • Navigate the Technical Design Authority approval process effectively
  • Document architectural decisions using Architecture Decision Records (ADR)
  • Plan integration patterns and data governance frameworks
  • Prepare detailed implementation blueprints for development teams

Key Personas & Stakeholders - RACI Matrix#

Activity Solution Architect Data Engineer Data Product Manager Business Analyst GDAI Architecture Team
Low-Level Design Creation R R C I C
Data Modeling A R C C C
Data Contract Design C R A C C
ADR Documentation A R C I C
TDA Submission A C C I C
TDA Review & Approval A R I I A
Design Updates Post-TDA A R C I C

R = Responsible, A = Accountable, C = Consulted, I = Informed

Prerequisites#

Key Foundation Documents

Ensure you have access to the high-level architecture from feasibility assessment and the detailed requirements specification before beginning solution design work.

Step-by-Step Process#

Step 1: Architecture Foundation Setup#

Objective: Establish the architectural foundation by refining high-level designs into detailed technical specifications.

  1. Review and Refine High-Level Architecture:

    • Take the high-level architecture from feasibility assessment
    • Identify areas requiring detailed specification
    • Map requirements to architectural components

  2. Define Solution Components:

    • Data Sources: Detailed specifications of all source systems
    • Integration Layer: APIs, connectors, and data ingestion mechanisms
    • Processing Layer: Transformation engines, compute resources, and orchestration
    • Storage Layer: Data lakes, warehouses, and persistence strategies
    • Consumption Layer: APIs, dashboards, and output interfaces

  3. Apply Architectural Patterns:

    • Implement medallion architecture (bronze, silver, gold layers)
    • Define microservices boundaries where applicable
    • Plan for scalability and fault tolerance

Architecture Best Practice

Expected Outcome: Detailed architecture diagram with component specifications and integration flows.

Step 2: Data Modeling and Schema Design#

Objective: Create comprehensive data models that support your data product requirements while enabling future scalability.

Logical Data Modeling:

  • Import the approved conceptual model into Erwin
  • Expand entities with attributes, data types, and business rules
  • Define primary and foreign key relationships and cardinality
  • Document assumptions and design decisions for traceability
  • Determine level of normalization needed in each layer (bronze, silver or gold). Typically bronze is a straight dump from source, silver follows more ER style modelling while Gold has more dimensional modelling to cater to different needs.
  • If the data product can have multiple Gold Products, then consider silver to have the dimensional modelling.

Hold "Alt" / "Option" to enable Pan & Zoom
Logical Data Model

Learn more about logical data modeling here.

Physical Data Modeling:

  • Define partitioning strategies for performance
  • Plan indexing and optimization approaches
  • Design for incremental processing patterns for example control tables.

Hold "Alt" / "Option" to enable Pan & Zoom
Physical Data Model

Learn more about about physical modelling here.

Data Model Documentation:

  • Entity relationship diagrams
  • Attribute definitions and business rules
  • Data lineage mapping
  • Scalability considerations
  • EDM Learning & Support Page – Refer to this page for foundational knowledge on data modelling concepts and step-by-step guidance on navigating the Erwin tool

Model Validation:

  • Review with business stakeholders for accuracy
  • Validate against source system capabilities
  • Ensure compliance with data governance standards

Critical Success Factor

Reference the Data Modeling Best Practices document to ensure your models follow organizational standards and can accommodate future requirements.

Expected Outcome: Complete logical and physical data models with comprehensive documentation

Step 3: Data Contract and Lineage Design#

Objective: Design robust data contracts and establish mechanisms for tracking data lineage throughout the data product lifecycle.

  1. Input Data Contracts

  2. Processing Data Contracts:

    • Define transformation rules and business logic
    • Specify quality validation requirements
    • Document error handling procedures

  3. Output Data Contracts:

    • Define consumer interface specifications
    • Establish SLA commitments
    • Document usage patterns and limitations

  4. Lineage Mechanism Design:

    • Plan automated lineage capture points
    • Design metadata storage strategy
    • Define lineage visualization approaches

Expected Outcome: Comprehensive data contracts covering all pipeline stages with automated lineage tracking design

Step 4: Integration and Security Architecture#

Objective: Design secure, efficient integration patterns and implement comprehensive security measures.

  1. Integration Architecture:

    • Source System Integration:

      • API specifications and authentication methods
      • Batch vs. streaming ingestion patterns
      • Error handling and retry mechanisms

    • Internal Integration:

      • Service-to-service communication patterns
      • Event-driven architecture considerations
      • Data synchronization strategies

  2. Security Architecture

  3. Compliance Design:

    • GDPR/privacy regulation compliance
    • Industry-specific compliance requirements
    • Audit trail and logging strategies

Expected Outcome: Detailed integration patterns with comprehensive security and compliance measures

Step 5: Requesting assistance from GDAI Architecture Team or suggesting improvements based on design (optional)#

In case the Data Product Technical Team encounters a platform related problem or is having difficulty selecting a tool for a particular use-case or has figured out a smarter solution to existing platforms' restrictions, an ADR can be raised to GDAI Architecture as outlined here

Step 6: Architecture Decision Record (ADR) Creation#

Objective: Document all significant architectural decisions with rationale and alternatives considered.

  1. ADR Structure Setup: Follow the GDAI Architecture ADR process for consistency.

  2. Key ADRs to Create

  3. Document Design Decisions:

    • Technology stack selections
    • Architecture pattern choices
    • Security approach decisions
    • Performance optimization strategies

Expected Outcome: Complete set of ADRs documenting all major architectural decisions

Step 7: Technical Design Authority (TDA) Preparation and Submission#

Objective: Prepare comprehensive documentation for TDA review and successfully obtain architectural approval.

  1. Pre-TDA Review:

    • Validate design against TDA template requirements
    • Conduct internal review with solution architects
    • Address any obvious gaps or inconsistencies

  2. TDA Submission Package:

    • Executive Summary: High-level overview and business justification
    • Detailed Architecture: Complete technical specifications
    • ADR Collection: All architectural decision records
    • Risk Assessment: Identified risks and mitigation strategies
    • Implementation Plan: High-level delivery approach

  3. TDA Presentation Preparation:

    • Create clear, concise presentation materials
    • Prepare for common TDA questions
    • Have technical deep-dive materials ready

TDA Success Tips

Review previous successful TDA submissions for similar projects. Focus on clearly articulating the business value and technical rationale for your architectural choices.

Expected Outcome: TDA submission package ready for formal review

Step 8: TDA Review and Design Refinement#

Objective: Successfully navigate the TDA review process and incorporate feedback into final design.

  1. TDA Review Process:

    • Present design to TDA board
    • Address questions and concerns
    • Provide additional clarification as needed

  2. Incorporate TDA Feedback:

    • Update architecture based on TDA observations
    • Revise ADRs to reflect approved changes
    • Update requirements specification if needed

  3. Final Design Approval:

    • Obtain formal TDA approval
    • Document any conditions or recommendations
    • Update project documentation with approved design

  4. Post-TDA Activities:

    • Communicate approval to delivery team
    • Update project plans based on final design
    • Begin detailed implementation planning

Expected Outcome: TDA-approved solution design ready for implementation

Success Metrics & Checkpoints#

  • Architecture Completeness: All system components and integrations fully specified
  • Data Model Validation: Logical and physical models reviewed and approved by stakeholders
  • Data Contract Definition: Complete contracts covering all pipeline stages
  • Security Compliance: All security and compliance requirements addressed in design
  • ADR Documentation: All major decisions documented with clear rationale
  • TDA Approval: Formal architecture approval obtained from TDA board
  • Design Traceability: Clear mapping from requirements to architectural components

Common Challenges & Solutions#

  • Challenge: Requirements changes during design phase affecting architectural decisions
    Solution: Implement change control process with formal approval for scope modifications
    Prevention: Ensure requirements are thoroughly validated before beginning detailed design

  • Challenge: TDA feedback requiring significant architectural changes
    Solution: Engage with GDAI Architecture team early for informal review and guidance
    Prevention: Use established patterns and reference architectures as starting points

  • Challenge: Complex integration requirements with legacy systems
    Solution: Design abstraction layers and implement staged migration approaches
    Prevention: Conduct thorough analysis of existing systems during feasibility assessment

  • Challenge: Balancing performance requirements with governance and security needs
    Solution: Work closely with governance and security teams to find acceptable compromises
    Prevention: Include all stakeholders in early design discussions

Next Steps#

After obtaining TDA approval and finalizing your solution design:

  1. Update Project Documentation: Ensure all project artifacts reflect the approved design
  2. Begin Detailed Implementation Planning: Break down the design into specific development tasks
  3. Set Up Development Environment: Configure infrastructure and tools based on approved architecture
  4. Initiate Data Marketplace Registration: Begin the process to register your data product for discoverability

The next phase will focus on Building the Data Product based on your approved solution design.

Additional Resources#