Video Library

Video Library is a quick prototype of the automated video analysis pipeline designed to turn raw video footage into structured, searchable metadata. By leveraging free AI models (Gemma 3) and vector databases (pgvector), it enables semantic search and deep content understanding of your video library.

🚀 Features

• Automated Ingestion: Watches specific directories for new video content.
• Frame Extraction: Intelligent frame sampling using ffmpeg to capture key visual moments.
• AI-Powered Analysis: Uses Gemma 3 (via Ollama) to analyze visual content against a rich, production-grade schema (cinematography, mood, lighting, narrative beat, etc.).
• Semantic Search: Generates vector embeddings for video descriptions, allowing for conceptual search (e.g., "find a tense scene with low-key lighting").
• Structured Storage: Stores comprehensive metadata in a PostgreSQL database equipped with the pgvector extension.
• Modular Architecture: Clean separation of concerns between ingestion, analysis, and storage.

🛠️ Prerequisites

Before you begin, ensure you have the following installed:

• Node.js: v20 or higher.
• PostgreSQL: With the pgvector extension installed and enabled.
• Ollama: Running locally or on a network-accessible server.
  • Models required:
    • gemma3-128k:latest (for visual analysis)
    • nomic-embed-text (for vector embeddings)
• FFmpeg: (Managed automatically via @ffmpeg-installer/ffmpeg, but system-level installation is helpful for debugging).

📦 Installation

1. Clone the repository:

   git clone https://github.com/yourusername/video-library.git
   cd video-library

2. Install dependencies:

   npm install
   # or
   pnpm install

3. Configure Environment: Create a .env file in the root directory (based on .env.example if available, otherwise use the template below):

   DB_USER=your_db_user
   DB_HOST=localhost
   DB_PASSWORD=your_db_password
   DB_PORT=5432
   DB_NAME_VIDEOLIBRARY=videolibrary_db
   DB_NAME_PGVECTOR=pgvector_test
   OLLAMA_BASE_URL=http://127.0.0.1:11434
   OLLAMA_MODEL=gemma3-32k:latest
   OLLAMA_EMBED_MODEL=nomic-embed-text

🗄️ Database Setup

You need to create the clips table in your PostgreSQL database (pgvector_test or whatever you named it in .env).

Execute the following SQL commands:

-- Enable the extension if not already enabled
CREATE EXTENSION IF NOT EXISTS vector;

-- Create the table
CREATE TABLE clips (
    id SERIAL PRIMARY KEY,
    filepath TEXT NOT NULL,
    description TEXT,
    mood TEXT[], -- Storing array of strings
    shot_type TEXT,
    keywords TEXT[],
    embedding VECTOR(768), -- Dimension depends on your model (nomic-embed-text is usually 768)
    file_hash TEXT UNIQUE, -- SHA-256 hash of the video file for deduplication
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

Note: Adjust the VECTOR dimension if you use a different embedding model.

🏃 Usage

1. Place Videos: Drop your video files (.mp4, .mov, etc.) into the data/incoming/ directory.

2. Run Ingestion: Start the ingestion worker. It will process all files in data/incoming/ one by one.

   node run-ingestion.js

3. Process Flow:

   • Files move to data/processing/.
   • Frames are extracted.
   • AI analyzes frames and generates metadata.
   • Embeddings are created.
   • Data is saved to the DB.
   • Original files move to data/completed/.
   • Failed files move to data/failed/ for inspection.

4. Search: You can search your video library using the CLI tool. It uses vector similarity to find the most relevant clips based on meaning, not just keywords.

   node search-cli.js "a tense scene with dark lighting"

   Output example:

   Found 2 matches:
   
   1. [Score: 0.1234] data/completed/scene_01.mp4
      📝 A dark, moody alleyway scene...
      🎭 Mood: Tense, Suspenseful
   

🧪 Testing

The project includes a test suite for the frame extractor and AI integration (using mocks).

npm test

📂 Project Structure

• ingestion-worker.js: Main orchestration logic.
• frame-extractor.js: FFmpeg wrapper for frame capture.
• gemma-analyzer.js: Logic for interacting with Ollama/Gemma.
• data/metadata-prompt.js: Defines the detailed JSON schema for AI analysis.
• db/pg-pools.js: Database connection management.
• api/ollamaGemma3Request.js: Low-level API client for Ollama.
• data/: Directory structure for video processing stages.

🤝 Contributing

Contributions are welcome! Please fork the repository and submit a pull request.

1. Fork the repo.
2. Create your feature branch (git checkout -b feature/amazing-feature).
3. Commit your changes (git commit -m 'Add some amazing feature').
4. Push to the branch (git push origin feature/amazing-feature).
5. Open a Pull Request.

📄 License

ISC License.