CSV Row Chunking

CSV row chunking is a method of splitting CSV files based on the number of rows, rather than character count. This approach treats each row (or group of rows) as a semantic unit, preserving the integrity of individual records while enabling efficient processing of tabular data.

Code

import asyncio
from agno.agent import Agent
from agno.knowledge.chunking.row import RowChunking
from agno.knowledge.knowledge import Knowledge
from agno.knowledge.reader.csv_reader import CSVReader
from agno.vectordb.pgvector import PgVector

db_url = "postgresql+psycopg://ai:ai@localhost:5532/ai"

knowledge_base = Knowledge(
    vector_db=PgVector(table_name="imdb_movies_row_chunking", db_url=db_url),
)

asyncio.run(knowledge_base.add_content_async(
    url="https://agno-public.s3.amazonaws.com/demo_data/IMDB-Movie-Data.csv",
    reader=CSVReader(
        chunking_strategy=RowChunking(),
    ),
))  

# Initialize the Agent with the knowledge_base
agent = Agent(
    knowledge=knowledge_base,
    search_knowledge=True,
)

# Use the agent 
agent.print_response("Tell me about the movie Guardians of the Galaxy", markdown=True)

Usage

1. Create a virtual environment

   Open the Terminal and create a python virtual environment.

   Mac

   python3 -m venv .venv
   source .venv/bin/activate
   

2. Install libraries

   pip install -U sqlalchemy psycopg pgvector agno
   

3. Run PgVector

   docker run -d \
     -e POSTGRES_DB=ai \
     -e POSTGRES_USER=ai \
     -e POSTGRES_PASSWORD=ai \
     -e PGDATA=/var/lib/postgresql/data/pgdata \
     -v pgvolume:/var/lib/postgresql/data \
     -p 5532:5432 \
     --name pgvector \
     agno/pgvector:16
   

4. Run Agent

   Mac

   python cookbook/knowledge/chunking/csv_row_chunking.py
   

CSV Row Chunking Params

Parameter	Type	Default	Description
rows_per_chunk	int	100	The number of rows to include in each chunk.
skip_header	bool	False	Whether to skip the header row when chunking.
clean_rows	bool	True	Whether to clean and normalize row data.
include_header_in_chunks	bool	False	Whether to include the header row in each chunk.
max_chunk_size	int	5000	Maximum character size for each chunk (fallback limit).