[{"_1":2,"_34":-5,"_35":-5},"loaderData",{"_3":4},"routes/_main.tags.$tag",{"_5":6,"_33":19},"posts",[7],{"_8":9,"_10":11,"_12":13,"_14":15,"_16":17,"_27":28,"_29":30,"_31":32},"slug","flexible-schema","title","Flexible schema using Drizzle and Zod","description","How to combine SQL structure with document database flexibility using SQLite JSON capabilities, Drizzle ORM, and Zod validation for evolving data schemas.","date","2024-10-02","tags",[18,19,20,21,22,23,24,25,26],"database","drizzle","javascript","orm","react","schema-design","sqlite","typescript","zod","content","

In many applications, data structures can evolve rapidly or vary significantly between entries. Relational database schemas can be limiting in these scenarios, often requiring migrations for even minor changes.

With some experimenting, I found an approach that combines the structure of SQL with the flexibility of document databases, using SQLite's JSON capabilities and Drizzle ORM.

The Concept

The core idea is to create a table with two main columns:

A type column to categorize the data
A data column to store the actual content as JSON

This structure allows for:

Storing varied data structures in a single table
Adding new data types without altering the table schema
Maintaining queryability on common fields

Implementation

Let's implement this concept using SQLite and Drizzle ORM. We'll create a generic items table that could store various types of data.

import { z } from 'zod'\nimport { sqliteTable, json, text, index } from 'drizzle-orm/sqlite-core'\n\n// Define possible item types\nexport const ItemTypeEnum = z.enum(['user', 'product', 'order'])\nexport type ItemType = z.infer<typeof ItemTypeEnum>\n\n// Define schemas for each item type\nexport const UserSchema = z.object({\n    type: z.literal(ItemTypeEnum.enum.user),\n    name: z.string(),\n    email: z.string().email(),\n    age: z.number().optional(),\n})\n\nexport const ProductSchema = z.object({\n    type: z.literal(ItemTypeEnum.enum.product),\n    name: z.string(),\n    price: z.number(),\n    categories: z.array(z.string()),\n})\n\nexport const OrderSchema = z.object({\n    type: z.literal(ItemTypeEnum.enum.order),\n    user_id: z.number(),\n    products: z.array(z.number()),\n    total: z.number(),\n    status: z.enum(['pending', 'shipped', 'delivered']),\n})\n\n// Combine all item types into a discriminated union\nexport const ItemSchema = z.discriminatedUnion('type', [\n    UserSchema,\n    ProductSchema,\n    OrderSchema,\n])\nexport type ItemSchemaType = z.infer<typeof ItemSchema>\n\n// Define the SQLite table\nexport const Items = sqliteTable(\n    'items',\n    {\n        type: text('type').generatedAs(`data ->> '$.type'`).$type<ItemType>(),\n        data: json('data').$type<ItemSchemaType>().notNull(),\n    },\n    (table) => ({\n        type_index: index('idx_type').on(table.type),\n    })\n)\n

Code Explanation

We define an enum ItemTypeEnum for possible item types. This can be easily extended in the future.
For each item type, we create a Zod schema (e.g., UserSchema, ProductSchema). These schemas define the structure and validation rules for each type of item.
The ItemSchema combines all item types into a discriminated union, allowing for type-safe handling of different item structures.
The SQLite table Items is defined with two main columns:\n
- type: A generated column that extracts the type from the JSON data
- data: The JSON column that stores the actual item data
\n
We use Drizzle's $type method to ensure type safety between TypeScript and the database.
An index is created on the type column for efficient querying.

Practical Examples

Inserting Data

Here's how you might insert different types of items into the database:

import { db } from './db' // Assume this is your Drizzle database instance\nimport { Items, ItemSchema } from './schema' // From the previous example\n\nasync function insertItem(item: z.infer<typeof ItemSchema>) {\n    try {\n        const validatedItem = ItemSchema.parse(item)\n        await db.insert(Items).values({\n            data: validatedItem,\n        })\n        console.log('Item inserted successfully')\n    } catch (error) {\n        console.error('Error inserting item:', error)\n    }\n}\n\n// Example usage:\ninsertItem({\n    type: 'user',\n    name: 'John Doe',\n    email: 'john@example.com',\n    age: 30,\n})\n\ninsertItem({\n    type: 'product',\n    name: 'Widget',\n    price: 19.99,\n    categories: ['electronics', 'gadgets'],\n})\n

Reading and Rendering Data in React

Now let's create a React component that fetches and renders a list of items, using the type field to determine how to display each item:

import React, { useState, useEffect } from 'react';\nimport { db } from './db';\nimport { Items, ItemSchemaType } from './schema';\n\nconst ItemList: React.FC = () => {\n  const [items, setItems] = useState<ItemSchemaType[]>([]);\n\n  useEffect(() => {\n    const fetchItems = async () => {\n      const fetchedItems = await db.select().from(Items);\n      setItems(fetchedItems.map(item => item.data as ItemSchemaType));\n    };\n    fetchItems();\n  }, []);\n\n  const renderItem = (item: ItemSchemaType) => {\n    switch (item.type) {\n      case 'user':\n        return (\n          <div className="user-item">\n            <h3>{item.name}</h3>\n            <p>Email: {item.email}</p>\n            {item.age && <p>Age: {item.age}</p>}\n          </div>\n        );\n      case 'product':\n        return (\n          <div className="product-item">\n            <h3>{item.name}</h3>\n            <p>Price: ${item.price.toFixed(2)}</p>\n            <p>Categories: {item.categories.join(', ')}</p>\n          </div>\n        );\n      case 'order':\n        return (\n          <div className="order-item">\n            <h3>Order #{item.user_id}</h3>\n            <p>Total: ${item.total.toFixed(2)}</p>\n            <p>Status: {item.status}</p>\n          </div>\n        );\n      default:\n        return <div>Unknown item type</div>;\n    }\n  };\n\n  return (\n    <div className="item-list">\n      {items.map((item, index) => (\n        <div key={index} className="item">\n          {renderItem(item)}\n        </div>\n      ))}\n    </div>\n  );\n};\n\nexport default ItemList;\n\n

In this example:

We use a useEffect hook to fetch items from the database when the component mounts.
The renderItem function uses a switch statement on the item.type to determine how to render each item. This demonstrates how we can have different UI representations for each item type, all stored in the same table.
TypeScript's discriminated union feature ensures that we have access to the correct properties for each item type within the respective case blocks.
We're treating all items as ItemSchemaType, which is the union type of all possible item schemas. This allows for type-safe access to properties based on the type field.

Conclusion

This flexible schema approach allows us to store diverse data structures in a single table while maintaining type safety and the ability to query efficiently. By using Zod for runtime validation and TypeScript for static typing, we can ensure data integrity while keeping our database schema flexible.

The React example demonstrates how this approach can be particularly powerful in front-end applications, allowing for dynamic rendering based on the data type without needing separate API endpoints or tables for each data type.

Remember to consider performance implications when querying large datasets, and always validate data on insertion to maintain data integrity.

Benefits and Use Cases

This approach is particularly useful for:

Applications with evolving data models
Systems that need to store varied, user-defined structures
Prototyping and rapid development where the final data model is not yet established

It allows you to add new item types or fields without altering the table structure, while still maintaining some level of structure and queryability.

","excerpt","In many applications, data structures can evolve rapidly or vary significantly between entries. Relational database schemas can be limiting in these scenarios, often requiring migrations for even...","readingTime",5,"tag","actionData","errors"]

Posts tagged "drizzle"

Flexible schema using Drizzle and Zod