Data Flow Architecture

This document explains how data flows through the Next.js applications in the monorepo, specifically examining the architecture used in www.schwab.com which integrates multiple packages for data fetching, transformation, and rendering.

Overview

The data flow architecture follows a clean separation of concerns across multiple packages:

Fetching: @schwab/fetch - Retrieves data from external APIs (primarily Drupal CMS)
Schema Validation: @schwab/schema - Provides Zod schemas for type safety and validation
Transformation: @schwab/transformer - Converts external data formats to UI-compatible schemas
UI Rendering: @schwab/ui - React components that render transformed data

Data Flow Diagram

Data Flow Pipeline

1. External Data Sources

The applications primarily fetch data from:

Drupal CMS: Content management system accessed via JSON:API
Feature Flags: LaunchDarkly for dynamic configuration
Analytics: Various tracking and measurement services

2. Data Fetching Layer (`@schwab/fetch`)

The fetch package provides typed functions for retrieving data from external sources:

// Example: Fetching a story from Drupal
import { getStory } from '@schwab/fetch/getStory';
import { EDrupalSource } from '@schwab/schema/native-enums/EDrupalSource';

const story = await getStory(EDrupalSource.Education, '/story/market-insights');

Key fetch functions discovered:

getStory() - Retrieves individual story/article content
getRelatedContent() - Fetches related articles and content
getProgrammaticCta() - Gets call-to-action content
Multiple Drupal API integrations through drupalFetch()

3. Schema Validation (`@schwab/schema`)

All external data is validated using Zod schemas to ensure type safety:

// Fetched data schema (from external API)
export const FetchedStorySchema = z.object({
  uuid: z.string(),
  title: z.string(),
  subtitle: z.string().optional(),
  summary: z.string(),
  components: z.array(z.unknown()),
  // ... more properties
});

// UI-ready schema (after transformation)
export const StorySchema = z.object({
  role: z.nativeEnum(ERole),
  identifier: z.string(),
  heading: z.string(),
  subheading: z.string().optional(),
  components: z.array(ComponentSchema),
  // ... transformed properties
});

4. Data Transformation (`@schwab/transformer`)

The transformer package converts external data formats into UI-compatible schemas:

import { transformStory } from '@schwab/transformer/transform/cmap-api/transformStory';

// Transform fetched data to UI schema
export function transformStory(data: TFetchedStory): TStory {
  const storyHeroImage = transformStoryHeroImage(data);
  const authors = transformStoryAuthor(data.authors);
  const components = transformStoryComponents(data.components, data.urlAlias);
  
  return mapComponent<TStory>({
    role: ERole.Story,
    heading: data?.title,
    subheading: data.subtitle,
    body: data.summary,
    image: storyHeroImage,
    authors,
    components,
    // ... more transformed fields
  });
}

Transformation capabilities:

Component Transformation: Individual story components (text, images, videos, etc.)
Media Processing: Image optimization, aspect ratio handling
Content Structuring: Converting CMS structure to React component props
URL Processing: Link prefixing and path resolution

5. UI Components (`@schwab/ui`)

The UI package provides React components that consume transformed data:

import StoryLoader from '@schwab/ui/StoryLoader';
import Story from '@schwab/ui/Story';

// Server component that orchestrates data loading
export default async function StoryLoader({ urlAlias }: TStoryLoader) {
  const story = await getStory(EDrupalSource.Education, urlAlias);
  
  return (
    <Story
      heading={story.data?.heading}
      subheading={story.data?.subheading}
      components={story.data?.components}
      // ... other props
    />
  );
}

Next.js Integration Pattern

App Router Implementation

The applications use Next.js App Router with server components for optimal performance:

// apps/www.schwab.com/src/app/[code]/learn/story/[...alias]/page.tsx
export default async function Page(props: { params: Promise<TStoryStaticParams> }) {
  const params = await props.params;
  const urlDetails = parseUrl(params.alias);

  return (
    <StoryLoader
      urlAlias={`/story/${urlDetails.urlAlias}`}
      pageNumber={urlDetails.pageNumber}
      pageParams={params}
      allFlags={allFlags}
    />
  );
}

Metadata Generation

Pages generate SEO metadata using the same data flow:

export async function generateMetadata(props): Promise<Metadata> {
  const params = await props.params;
  const urlDetails = parseUrl(params.alias);
  
  const story = await getStory(EDrupalSource.Education, `/story/${urlDetails.urlAlias}`);
  
  return {
    title: story.data?.pageTitle,
    description: story.data?.pageDescription,
    openGraph: {
      title: story.data?.heading,
      description: story.data?.body,
      images: [story.data?.image?.src],
    },
  };
}

Package Integration Points

1. Path Mapping Configuration

The monorepo uses TypeScript path mapping for clean imports:

// tsconfig.json paths
{
  "#components/*": ["./src/components/*"],
  "#functions/*": ["./src/functions/*"],
  "@schwab/fetch/*": ["../../../packages/fetch/src/*"],
  "@schwab/transformer/*": ["../../../packages/transformer/src/*"],
  "@schwab/ui/*": ["../../../packages/ui/src/*"]
}

2. Server-Only Operations

Data fetching and transformation occur server-side only:

import 'server-only'; // Ensures code doesn't run in browser

// Server component with data fetching
export default async function ServerComponent() {
  const data = await getStory(); // Server-only operation
  return <ClientComponent data={data} />;
}

Data Flow Examples

Story Page Flow

Request: User visits /learn/story/investment-basics
Parsing: parseUrl() extracts story alias from URL
Fetching: getStory() retrieves data from Drupal API
Validation: FetchedStorySchema validates raw API response
Transformation: transformStory() converts to UI schema
Validation: StorySchema validates transformed data
Rendering: Story component renders final UI

Component-Level Flow

// 1. Fetch raw data
const rawStory = await drupalFetch('/api/story/123');

// 2. Validate fetched data
const fetchedStory = FetchedStorySchema.parse(rawStory);

// 3. Transform to UI schema
const uiStory = transformStory(fetchedStory);

// 4. Validate transformed data
const validatedStory = StorySchema.parse(uiStory);

// 5. Render component
return <Story {...validatedStory} />;

Error Handling

The data flow includes comprehensive error handling:

// Fetch with error handling
const story = await getStory(source, urlAlias);

if (!story?.isOk) {
  notFound(); // Next.js 404 handling
}

// Zod validation with error handling
try {
  const validStory = StorySchema.parse(transformedData);
} catch (error) {
  return <ZodErrorOutput error={error} />;
}

Performance Optimizations

Static Generation

Pages use static generation where possible:

export const dynamic = 'force-static';

export async function generateStaticParams() {
  // Return static paths for build-time generation
}

Parallel Data Loading

Multiple data sources are loaded in parallel:

const [story, relatedContent, programmaticCta] = await Promise.all([
  getStory(source, urlAlias),
  getRelatedContent(source, urlAlias),
  getProgrammaticCta(source, urlAlias),
]);

Package Dependencies

The data flow relies on these key packages:

@schwab/fetch: Data retrieval functions
@schwab/schema: Zod schemas for validation
@schwab/transformer: Data transformation functions
@schwab/ui: React components
@schwab/utilities: Helper functions (parseUrl, stripHTML, etc.)

Summary

The data flow architecture provides:

Type Safety: End-to-end TypeScript with Zod validation
Performance: Server-side data fetching with static generation
Maintainability: Clear separation of concerns across packages
Scalability: Modular architecture supporting multiple applications
Developer Experience: Path mapping and consistent patterns across the monorepo

This architecture enables the team to build performant, type-safe web applications while maintaining clean code organization and developer productivity.

Overview​

Data Flow Diagram​

Data Flow Pipeline​

1. External Data Sources​

2. Data Fetching Layer (@schwab/fetch)​

3. Schema Validation (@schwab/schema)​

4. Data Transformation (@schwab/transformer)​

5. UI Components (@schwab/ui)​

Next.js Integration Pattern​

App Router Implementation​

Metadata Generation​

Package Integration Points​

1. Path Mapping Configuration​

2. Server-Only Operations​

Data Flow Examples​

Story Page Flow​

Component-Level Flow​

Error Handling​

Performance Optimizations​

Static Generation​

Parallel Data Loading​

Package Dependencies​

Summary​