Build a LinkedIn
Lead Scraper

# Create the project
npm create vite@latest social-engager -- --template react-ts
cd social-engager

# Install dependencies
npm install @supabase/supabase-js tailwindcss postcss autoprefixer
npm install lucide-react react-router-dom

# Init Tailwind
npx tailwindcss init -p

-- Core tables

CREATE TABLE profiles (
  id UUID REFERENCES auth.users PRIMARY KEY,
  email TEXT,
  full_name TEXT,
  created_at TIMESTAMPTZ DEFAULT NOW()
);

CREATE TABLE projects (
  id UUID DEFAULT gen_random_uuid() PRIMARY KEY,
  name TEXT NOT NULL,
  created_by UUID REFERENCES profiles(id),
  created_at TIMESTAMPTZ DEFAULT NOW()
);

CREATE TABLE monitors (
  id UUID DEFAULT gen_random_uuid() PRIMARY KEY,
  project_id UUID REFERENCES projects(id),
  name TEXT NOT NULL,
  mode TEXT NOT NULL,          -- 'keyword' | 'profile' | 'posts' | 'direct'
  input TEXT NOT NULL,         -- keyword string, profile URL, or post URLs
  webhook_url TEXT,             -- where to send leads
  schedule TEXT,                -- 'daily' | 'weekly' | 'monthly' | null
  is_active BOOLEAN DEFAULT true,
  created_at TIMESTAMPTZ DEFAULT NOW()
);

CREATE TABLE queue_jobs (
  id UUID DEFAULT gen_random_uuid() PRIMARY KEY,
  monitor_id UUID REFERENCES monitors(id),
  project_id UUID REFERENCES projects(id),
  status TEXT DEFAULT 'queued',  -- queued | running | completed | failed | awaiting_approval
  payload JSONB,
  result JSONB,
  error TEXT,
  started_at TIMESTAMPTZ,
  completed_at TIMESTAMPTZ,
  created_at TIMESTAMPTZ DEFAULT NOW()
);

CREATE TABLE runs (
  id UUID DEFAULT gen_random_uuid() PRIMARY KEY,
  monitor_id UUID REFERENCES monitors(id),
  project_id UUID REFERENCES projects(id),
  job_id UUID REFERENCES queue_jobs(id),
  leads_found INTEGER DEFAULT 0,
  new_leads INTEGER DEFAULT 0,
  cost DECIMAL(10,4) DEFAULT 0,
  posts_scraped INTEGER DEFAULT 0,
  details JSONB,
  created_at TIMESTAMPTZ DEFAULT NOW()
);

CREATE TABLE seen_engagers (
  id UUID DEFAULT gen_random_uuid() PRIMARY KEY,
  monitor_id UUID REFERENCES monitors(id),
  post_url TEXT NOT NULL,
  engager_profile_url TEXT NOT NULL,
  first_seen_at TIMESTAMPTZ DEFAULT NOW(),
  UNIQUE(monitor_id, post_url, engager_profile_url)
);

• Supabase (public)
• VITE_SUPABASE_URL
• VITE_SUPABASE_ANON_KEY

• Supabase (service role)
• SUPABASE_URL
• SUPABASE_KEY
• Limadata
• LIMADATA_API_KEY
• Apify
• APIFY_API_KEY

social-engager/
│
├── src/                             ← React frontend
│   ├── App.tsx                      ← Routes + auth gating
│   ├── pages/
│   │   ├── Dashboard.tsx            ← Monitor management
│   │   ├── Queue.tsx                ← Job queue + approval flow
│   │   ├── History.tsx              ← Run execution history
│   │   ├── Trigger.tsx              ← Manual job triggering
│   │   └── Login.tsx                ← Supabase auth
│   ├── components/
│   │   ├── MonitorCard.tsx          ← Display monitor config + stats
│   │   ├── MonitorForm.tsx          ← Create/edit monitors
│   │   ├── UsageBanner.tsx          ← Cost + usage tracking
│   │   └── Layout.tsx               ← Navigation shell
│   ├── hooks/
│   │   ├── useMonitors.ts           ← CRUD for monitors
│   │   ├── useQueue.ts              ← Job enqueueing + approval
│   │   ├── useRuns.ts               ← Execution history queries
│   │   ├── useUsageStats.ts         ← Cost tracking per project
│   │   └── useJobNotifications.ts   ← Realtime status via Supabase
│   └── lib/
│       ├── supabase.ts              ← Supabase client init
│       ├── types.ts                 ← TypeScript interfaces
│       └── costs.ts                 ← Cost calculation helpers
│
├── backend/                         ← Python worker
│   ├── worker.py                    ← Main job processor (runs 24/7)
│   ├── Dockerfile                   ← Python 3.11-slim for Railway
│   └── requirements.txt             ← supabase, requests, python-dotenv
│
└── supabase/                        ← Database schema + migrations
    ├── setup.sql                    ← Full production schema
    └── migrations/                  ← Incremental changes

1. 1
   Monitor creation form

   Select mode (keyword, profile, posts, direct). Enter the input (keyword string, LinkedIn profile URL, or post URLs). Optionally set a webhook URL and schedule (daily, weekly, monthly).

2. 2
   Monitor cards

   Display each monitor with its mode, input, schedule, and stats (total leads found, last run date, cost to date). Toggle active/inactive.

3. 3
   Manual trigger button

   Run any monitor on demand. Creates a queue_job with status "queued" and the monitor's config as payload.

1. 1
   Job queue page

   List all jobs with status badges: queued (waiting), running (in progress), completed (done), failed (error), awaiting_approval (needs confirmation).

2. 2
   Approval flow

   When estimated leads exceed 1,000, the worker sets status to "awaiting_approval". The dashboard shows the estimated count and cost. User clicks "Approve" to re-queue the job.

3. 3
   Realtime status updates

   Subscribe to Supabase realtime on the queue_jobs table. Status changes appear live without refreshing.

// Supabase realtime subscription for job status
const channel = supabase
  .channel('job-updates')
  .on('postgres_changes', {
    event: 'UPDATE',
    schema: 'public',
    table: 'queue_jobs',
    filter: `project_id=eq.${projectId}`,
  }, (payload) => {
    updateJobInState(payload.new);
  })
  .subscribe();

# worker.py - main loop

import time
from supabase import create_client

supabase = create_client(SUPABASE_URL, SUPABASE_KEY)

while True:
    # Pick up the oldest queued job
    job = supabase.table("queue_jobs") \
        .select("*") \
        .eq("status", "queued") \
        .order("created_at") \
        .limit(1) \
        .execute()

    if job.data:
        process_job(job.data[0])
    else:
        time.sleep(5)

1. 1
   Keyword mode

   Search LinkedIn for posts matching the keyword. Run two parallel searches (by relevance and by recency) to get a broader set. Take the top posts by engagement count. Scrape commenters + likers from each post.

2. 2
   Profile mode

   Fetch the profile's recent posts via Limadata. For each post, scrape commenters + likers with Apify. Useful for tracking competitors or your own team's content.

3. 3
   Posts mode (recurring)

   Take the post URLs from the monitor config. Check if engagement count has changed since last run (smart skip). If new engagement detected, scrape and deduplicate against seen_engagers. Only return NEW people.

4. 4
   Direct mode (one-time)

   Take the post URL, scrape all commenters + likers in one shot. No dedup tracking, no repeat logic. One-time full pull.

def process_job(job):
    # 1. Mark job as running
    update_status(job["id"], "running")

    # 2. Get post URLs based on mode
    if job["mode"] == "keyword":
        posts = search_posts_by_keyword(job["input"])
    elif job["mode"] == "profile":
        posts = get_profile_posts(job["input"])
    else:
        posts = job["input"]  # direct post URLs

    # 3. Check approval threshold
    estimated = estimate_engagers(posts)
    if estimated > 1000 and not job.get("large_scrape_approved"):
        update_status(job["id"], "awaiting_approval")
        return

    # 4. Scrape engagers from each post (parallel, max 3 concurrent)
    all_engagers = []
    for post in posts:
        commenters = scrape_commenters(post["url"])
        likers = scrape_likers(post["url"])
        all_engagers.extend(commenters + likers)

    # 5. Deduplicate against seen_engagers
    new_leads = deduplicate(job["monitor_id"], all_engagers)

    # 6. Deliver via webhook
    if job["webhook_url"] and new_leads:
        send_to_webhook(job["webhook_url"], new_leads)

    # 7. Record results
    create_run(job, leads_found=len(all_engagers), new_leads=len(new_leads))
    update_status(job["id"], "completed")

def deduplicate(monitor_id, engagers):
    new_leads = []
    for engager in engagers:
        # Check if we've seen this person on this post before
        existing = supabase.table("seen_engagers") \
            .select("id") \
            .eq("monitor_id", monitor_id) \
            .eq("engager_profile_url", engager["profile_url"]) \
            .execute()

        if not existing.data:
            # New lead - record it and add to output
            supabase.table("seen_engagers").insert({
                "monitor_id": monitor_id,
                "post_url": engager["post_url"],
                "engager_profile_url": engager["profile_url"],
            }).execute()
            new_leads.append(engager)

    return new_leads

1. 1
   Calculate cost per job

   Track API calls and leads scraped. Cost = (engagers x cost per engager) + (API calls x cost per call). Store on each run record.

2. 2
   Monthly lead limit

   Set a monthly cap (e.g., 5,000 leads). Query the runs table for the current billing cycle. Block new jobs when approaching the limit. Show a warning banner at 80%.

3. 3
   Per-job lead cap

   Hard limit per individual job (e.g., 5,000 leads). Anything above 1,000 requires the approval flow before running.

1. 1
   Build the project

   Run npm run build. This creates a dist/ folder with the static site.

2. 2
   Deploy to Netlify

   Drag the dist/ folder to Netlify, or connect your GitHub repo for automatic deploys on every push.

3. 3
   Add environment variables

   Set VITE_SUPABASE_URL and VITE_SUPABASE_ANON_KEY in Netlify's environment variable settings.

1. 1
   Create a Dockerfile

   Python 3.11-slim base image. Copy requirements.txt and worker.py. Install dependencies. Set the entrypoint to run worker.py.

2. 2
   Push to Railway

   Connect your GitHub repo to Railway. It detects the Dockerfile and deploys automatically. Set environment variables in Railway's dashboard.

3. 3
   Configure restart policy

   Set the service to auto-restart on failure with a retry limit. The worker should run continuously.

# Dockerfile
FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY worker.py .
CMD ["python", "worker.py"]