Building a Cross-Repository Test Automation Pipeline: From Manual QA Nightmares to Automated Excellence

TL;DR

Transformed our SDK release process from manual, error-prone testing to a fully automated cross-repository testing pipeline using GitHub Actions and Playwright. Result: 80% reduction in testing time, zero release-blocking bugs in the last 6 months, and engineers who actually sleep well during release weeks.

The Problem: When SDK Testing Becomes a Bottleneck

Your team maintains a JavaScript SDK used by hundreds of applications. Every release means manually testing core functionality, web popups, inbox features, and API integrations across different versions and environments.

Our pre-automation reality:

• 2-3 days of manual testing per release
• Multiple environments to validate (development, staging, various SDK versions)
• Human error leading to missed edge cases
• Release anxiety because something always broke in production
• Delayed releases due to testing bottlenecks

The Solution: A Cross-Repository Automation Architecture

We built a system where our main SDK repository can trigger comprehensive tests in a separate automation repository. This architecture separates concerns effectively:

SDK Repository (Source Code)
     ↓ [Commit with trigger]
GitHub Actions Bridge
     ↓ [Cross-repo workflow dispatch]  
Automation Repository (Test Suite)
     ↓ [Execute comprehensive tests]
Results & Artifacts

Key Components

1. Smart Trigger Detection

# SDK Repository Workflow
on:
  push:
    branches: [main, develop]
  pull_request:
    types: [opened, synchronize]

jobs:
  check-trigger:
    runs-on: ubuntu-latest
    steps:
      - name: Check for test trigger
        run: |
          COMMIT_MESSAGE="$"
          if [[ "$COMMIT_MESSAGE" == *"[run-test]"* ]]; then
            echo "trigger=true" >> $GITHUB_OUTPUT
          fi

2. Cross-Repository Communication

# Trigger automation tests in separate repo
- name: Trigger automation tests
  if: steps.check.outputs.trigger == 'true'
  run: |
    # Clone automation repo
    git clone https://$@github.com/org/automation-repo.git
    cd automation-repo
    
    # Create trigger commit with test parameters
    git commit --allow-empty -m "Test SDK branch:$ [run-test]"
    git push origin main

3. Dynamic Test Environment Configuration

// Automation Repository - Dynamic URL construction
const buildTestURL = (baseURL, params) => {
  const { branch, version, accountId, region, token } = params;
  const sdkParam = version ? `sdkVersion=${version}` : `sdkBranch=${branch}`;
  
  return `${baseURL}?region=${region}&accountId=${accountId}&token=${token}&${sdkParam}`;
};

// Usage in tests
const testURL = buildTestURL('https://test-app.com/automation', {
  branch: process.env.BRANCH || 'main',
  version: process.env.VERSION,
  accountId: 'test-account',
  region: 'us1',
  token: 'test-token'
});

Implementation Deep Dive

1. Test Suite Structure

We organized our tests into logical modules that mirror our SDK's architecture:

tests/
├── core/
│   ├── initialization.spec.js    # SDK bootstrap & config
│   ├── publicAPI.spec.js         # Core API methods
│   └── eventTracking.spec.js     # Analytics & events
├── features/
│   ├── webPopups.spec.js         # Modal & overlay features
│   ├── webInbox.spec.js          # In-app messaging
│   └── nativeDisplay.spec.js     # Native UI components
└── utils/
    ├── testHelpers.js            # Shared test utilities
    └── constants.js              # Environment configs

2. Playwright Configuration for SDK Testing

// playwright.config.js
module.exports = {
  testDir: './tests',
  timeout: 60000,
  retries: 2,
  
  use: {
    baseURL: process.env.TEST_URL,
    screenshot: 'only-on-failure',
    trace: 'retain-on-failure',
    video: 'retain-on-failure'
  },
  
  projects: [
    {
      name: 'chromium',
      use: { ...devices['Desktop Chrome'] }
    },
    {
      name: 'webkit',
      use: { ...devices['Desktop Safari'] }
    }
  ],
  
  reporter: [
    ['html', { outputFolder: 'playwright-report' }],
    ['junit', { outputFile: 'test-results.xml' }]
  ]
};

3. Smart Parameter Extraction

The automation system can parse different types of test triggers:

# Test specific branch
git commit -m "Feature update [run-test] branch:feature-xyz"

# Test specific version  
git commit -m "Release prep [run-test] version:2.1.0"

# Test with multiple parameters
git commit -m "Hotfix [run-test] branch:hotfix-123 region:eu1"

# Parameter extraction logic
- name: Extract test parameters
  run: |
    COMMIT_MESSAGE="$"
    
    # Extract branch
    if [[ "$COMMIT_MESSAGE" =~ branch:([^ ]*) ]]; then
      echo "BRANCH=${BASH_REMATCH[1]}" >> $GITHUB_OUTPUT
    fi
    
    # Extract version
    if [[ "$COMMIT_MESSAGE" =~ version:([^ ]*) ]]; then
      echo "VERSION=${BASH_REMATCH[1]}" >> $GITHUB_OUTPUT
    fi

4. Comprehensive Test Coverage

Our test suite covers critical SDK functionality:

// Core SDK initialization test
test('SDK initializes with correct configuration', async ({ page }) => {
  await page.goto(testURL);
  await page.waitForFunction(() => window.clevertap !== undefined);
  
  const isInitialized = await page.evaluate(() => {
    return window.clevertap.isInitialized();
  });
  
  expect(isInitialized).toBe(true);
});

// Feature-specific test
test('Web popup displays correctly', async ({ page }) => {
  await page.goto(testURL);
  await setupSDK(page);
  
  // Trigger popup
  await page.evaluate(() => {
    window.clevertap.displayPopup('campaign-id');
  });
  
  // Verify popup appearance
  await expect(page.locator('[data-testid="web-popup"]')).toBeVisible();
  await expect(page.locator('.popup-content')).toContainText('Expected Content');
});

Results & Impact

Metrics That Matter

• Testing Time: Reduced from 2-3 days to 30 minutes
• Test Coverage: Increased from ~40% to 95% of critical paths
• Release Frequency: From monthly to weekly releases
• Bug Escape Rate: Reduced by 90% in production

Process Transformation

Before (Manual Process)

Developer creates PR → Manual testing (2-3 days) → 
Bugs found → Fix bugs → Re-test manually → 
Release (with fingers crossed)

After (Automated Process)

Developer creates PR → Automated tests (30 mins) →
Results in GitHub → Fix if needed → Re-test automatically →
Confident release

Team Benefits

• Developer Confidence: No more release anxiety
• Faster Iteration: Quick feedback on feature branches
• Better Documentation: Tests serve as living specifications
• Team Morale: Engineers can focus on features, not manual testing

Key Technical Patterns

1. Separation of Concerns

• SDK Repository: Focuses on code, lightweight workflow triggers
• Automation Repository: Dedicated to comprehensive testing, reporting

2. Dynamic Environment Management

• URL parameter-based configuration
• Environment-specific test data
• Branch/version specific testing

3. Robust Error Handling

// Retry logic for flaky network operations
const waitForSDKReady = async (page, retries = 3) => {
  for (let i = 0; i < retries; i++) {
    try {
      await page.waitForFunction(() => window.sdk?.isReady, { timeout: 10000 });
      return;
    } catch (error) {
      if (i === retries - 1) throw error;
      await page.waitForTimeout(2000);
    }
  }
};

4. Comprehensive Reporting

• HTML reports for developers
• JUnit XML for CI integration
• Screenshots and traces for debugging
• Artifact retention for historical analysis

Lessons Learned

What Worked Well

1. Clear trigger syntax makes it easy for developers to request tests
2. Cross-repository architecture keeps concerns separated
3. Dynamic configuration allows testing any branch/version combination
4. Rich reporting makes debugging failures straightforward

Challenges We Overcame

1. GitHub token permissions - Required careful scope management
2. Test flakiness - Added proper waits and retry logic
3. Environment synchronization - Used URL parameters for dynamic config
4. Debugging failures - Comprehensive artifact collection was key

Implementation Guide

Phase 1: Basic Setup

• [ ] Create separate automation repository
• [ ] Set up GitHub Actions workflows
• [ ] Configure cross-repository access tokens
• [ ] Implement basic trigger detection

Phase 2: Test Development

• [ ] Choose testing framework (Playwright, Cypress, etc.)
• [ ] Create test environment configuration
• [ ] Write core functionality tests
• [ ] Add reporting and artifact collection

Phase 3: Enhancement

• [ ] Add parameter parsing for branches/versions
• [ ] Implement comprehensive error handling
• [ ] Set up monitoring and notifications
• [ ] Create documentation and runbooks

GitHub Actions Workflow Template

name: Cross-Repo Test Trigger

on:
  push:
    branches: [main]

jobs:
  trigger-tests:
    if: contains(github.event.head_commit.message, '[run-test]')
    runs-on: ubuntu-latest
    steps:
      - name: Trigger automation
        run: |
          curl -X POST \
            -H "Authorization: token $" \
            -H "Accept: application/vnd.github.v3+json" \
            https://api.github.com/repos/your-org/automation-repo/actions/workflows/test.yml/dispatches \
            -d '{"ref":"main","inputs":{"branch":"$"}}'

Test Structure Template

import { test, expect } from '@playwright/test';

test.describe('SDK Core Functionality', () => {
  test.beforeEach(async ({ page }) => {
    const testURL = buildTestURL(process.env.BASE_URL, {
      branch: process.env.BRANCH || 'main',
      version: process.env.VERSION
    });
    
    await page.goto(testURL);
    await waitForSDKReady(page);
  });

  test('core initialization', async ({ page }) => {
    // Your test logic here
  });
});

Conclusion

Building this cross-repository automation pipeline transformed our release process from a source of stress into a competitive advantage. The key insight? Automation isn't just about saving time—it's about enabling confidence.

When you can test comprehensively and quickly, you ship faster, sleep better, and build better products. Start small, automate incrementally, and focus on the tests that matter most to your users.

Resources:

• Playwright Documentation
• GitHub Actions Workflows