CodeNote.net

Self-hosting the OSS Version of Firecrawl

#scraping #oss #llm
Tadashi Shigeoka ·  Sat, September 7, 2024

I tried self-hosting the OSS version of Firecrawl, so I’ll introduce how to do it.

Prerequisites

Preparation

gh repo clone mendableai/firecrawl && cd firecrawl/
git checkout refs/tags/v1.0.0

Setting Up Firecrawl OSS Version for Local Environment

Setup .env

cp apps/api/.env.example .env
10c10
< USE_DB_AUTHENTICATION=true
---
> USE_DB_AUTHENTICATION=false

Start Firecrawl in Local Environment

docker compose up -d

I confirmed that the following URLs can be opened:

Crawling Example

curl -X POST http://localhost:3002/v1/crawl \
    -H 'Content-Type: application/json' \
    -d '{
      "url": "https://docs.firecrawl.dev"
    }'

Scraping Example

curl -X POST http://localhost:3002/v1/scrape \
    -H 'Content-Type: application/json' \
    -d '{
      "url": "https://docs.firecrawl.dev"
    }'

That’s all from the Gemba about trying to self-host the OSS version of Firecrawl.