- collect-channel-data: gangnamUnni scraping no longer requires
verified=true. Fallback: Firecrawl search for gangnamunni.com URL
when discover-channels failed to verify. Solves empty ratings/reviews.
- generate-report: Perplexity prompt now explicitly requests leadDoctor
(name, specialty, rating, reviewCount) and staffCount in clinicInfo.
- transformReport: clinicInfo type extended with leadDoctor + staffCount;
transformation prefers clinic.leadDoctor over doctors[0] fallback.
Root cause: clinic_registry table not yet in DB → discover-channels
always falls back to API search → gangnamUnni URL not found →
collect-channel-data skips gangnamUnni → all clinic metrics empty.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Each analysis run now creates a dedicated folder in Supabase Storage:
clinics/{domain}/{reportId}/
├── scrape_data.json (discover-channels: website scrape + Perplexity)
├── channel_data.json (collect-channel-data: all channel API results)
└── report.json (generate-report: final AI-generated report)
Screenshots also moved from {reportId}/{id}.png to:
clinics/{domain}/{reportId}/screenshots/{id}.png
Migration: 20260407_clinic_data_storage.sql creates 'clinic-data' bucket
(private, 10MB/file, JSON only). All writes are non-fatal — pipeline
continues even if Storage upload fails.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
## 문제
Firecrawl이 반환하는 스크린샷 URL은 GCS Signed URL로 7일 후 만료.
리포트에 저장된 이미지 URL이 일주일 후 전부 깨짐 (403 Access Denied).
## 해결
collect-channel-data의 Vision 단계에 아카이빙 스텝 추가.
캡처 직후 base64(이미 메모리에 있음)를 Supabase Storage에 영구 업로드.
### 처리 흐름 (변경 후)
1. captureAllScreenshots() → GCS URL + base64 반환 (기존)
2. [신규] archiveTasks: base64 → Supabase Storage 업로드 (병렬)
- 경로: screenshots/{reportId}/{screenshotId}.png
- 성공 시 ss.url을 영구 Supabase URL로 in-place 교체
- 실패 시 non-fatal — GCS URL fallback으로 Vision 분석 계속 진행
3. runVisionAnalysis() — base64 여전히 메모리에 있어 정상 실행 (기존)
4. channelData.screenshots 저장 시 영구 URL 사용 (자동)
- archived: true/false 플래그 추가 (모니터링용)
### 비용/성능
- 추가 API 호출 없음 (base64 이미 캡처 시 다운로드됨)
- 업로드: ~1-3초/장 (병렬), 5MB limit, PNG/JPEG/WebP 허용
- 버킷: public (URL만 있으면 열람) + 서비스 역할만 업로드 가능
## 마이그레이션
supabase/migrations/20260407_screenshots_storage.sql
- screenshots 버킷 생성 (public, 5MB limit)
- RLS: public read / service_role write
- delete_old_screenshots() 함수: 90일 이상 된 파일 정리 (pg_cron 연동 가능)
## 타입
ScreenshotResult.archived?: boolean 필드 추가 (영구 vs GCS fallback 구분)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
## P0 버그 수정 (즉시 영향)
### fix(collect-channel-data): 강남언니 rating 오변환 제거
- 기존: `rating ≤ 5 → ×2` 로직으로 4.8/10을 9.6/10으로 잘못 변환
- Firecrawl 프롬프트가 이미 0-10 반환 지시 → rawValue 직접 신뢰
### fix(generate-report): Perplexity 단일 fetch → fetchWithRetry
- maxRetries:2, backoffMs:[5000,15000], timeoutMs:90s 설정
- 기존: 일시적 429/타임아웃 시 리포트 생성 전체 실패
## P1 기능 추가 (데이터 품질)
### feat(collect-channel-data): channel_snapshots health_score 계산
- `computeHealthScore(channel, data)` 함수 추가 (채널별 0-100 스코어)
- Instagram: followers 기반 선형 보간 + posts bonus
- YouTube: subscribers 기반 + video count bonus
- 강남언니: rating×7 + reviews bonus (max 30pt)
- Google Maps: rating×12 + reviews bonus (max 40pt)
- Naver Blog: presence (50pt) + 언급 수 bonus (max 30pt)
- 모든 channel_snapshots INSERT에 health_score 포함
### feat(collect-channel-data): 네이버 블로그 공식 컨텐츠 스크랩 추가
- 기존: Naver Search API로 3rd-party 언급만 수집
- 추가: Registry에서 확인된 공식 블로그 URL을 Firecrawl로 직접 스크랩
- 총 게시글 수, 최근 게시물 (제목/날짜/요약), 카테고리 추출
- 실패 시 non-critical — 기존 Naver Search 결과는 항상 유지
## docs: PIPELINE_IMPROVEMENT_PLAN 감사 결과 반영
- Sprint 0 (Vision), Sprint 1, Sprint 2 완료 표시
- WP-10, WP-11 완료 표시
- 2026-04-07 전수 감사 섹션 추가 (구현 완료/수정/남은 Gap 표)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
YouTube now verifies all candidates and picks best match by channel title.
Facebook tries all candidates with domain-name fallback when Firecrawl returns empty.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Hero button: gray when empty, accent gradient when URL entered
- generate-report: force-inject gangnamUnniStats from Vision Analysis
into channelAnalysis (score, rating, reviews, doctors)
- Add gangnamunni Vision data to prompt context
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Previous chunked btoa approach encoded each chunk independently,
producing corrupted base64 that Gemini couldn't parse (returned {})
- Now builds complete binary string first, then encodes once with btoa
- Added screenshot debug info to channel errors for diagnostics
- Confirmed: foundingYear 2004, doctors, gangnamunni data all extracted
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- All fetch calls to Supabase Edge Functions now include
Authorization: Bearer <anon_key> (was missing → 401 errors)
- Fix Firecrawl screenshot API: remove invalid screenshotOptions,
use "screenshot@fullPage" format (v2 API compatibility)
- Fix screenshot response handling: v2 returns URL not base64,
now downloads and converts to base64 for Gemini Vision
- Add about page to Vision Analysis capture targets
- Add retry utility, channel error tracking, pipeline resume,
enrichment retry, EmptyState improvements (Sprint 2-3)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Vision analysis addresses the critical gap that text-only scraping
misses ~40% of clinic website information (founding year in banners,
doctor photos, certification marks, social icons in images).
Sprint 0 adds: Firecrawl screenshot → Gemini Vision → structured
data extraction for founding year, doctors, certifications, services,
social icons, floating buttons, brand colors, slogans.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Added A4 parallel Firecrawl call with actions: [wait 3s, scrape]
to execute JavaScript and extract social button href URLs from
header/footer. This is the most reliable source — most Korean
clinics have Facebook/Instagram/YouTube/Blog icons in their nav.
Results merged as Source 3 (buttonHandles) alongside HTML links,
JSON extraction, and API searches.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
WP-1: YouTube channel ID regex {20,} → {22} (exactly 24 chars)
WP-2: Naver Place category filtering in enrich-channels (성형/피부)
WP-3: Google Maps stores mapsUrl separately from clinicWebsite
WP-4: Naver Blog separates officialBlogUrl from search results
WP-5: 강남언니 rawRating + normalized rating (≤5 → ×2), Firecrawl
prompt explicitly states "out of 10, NOT out of 5"
WP-6: Perplexity model centralized in _shared/config.ts (env override)
WP-7: Apify Instagram timeout 30s → 45s
Frontend: transformReport uses mapsUrl and officialBlogUrl when available
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Comprehensive audit of discover→collect→generate pipeline found:
- 16 silent failures, 8 data quality issues, 0 error recovery, 6 API issues
- Organized into 4 sprints (15 WPs, ~11h total)
- Each WP has file locations, changes, and verification criteria
- Checkbox format for progress tracking
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Google Maps: was using gm.website (clinic's own site) → now always
generates maps.google.com/search URL
Naver Blog: was linking to first search result post (random personal
blog) → now links to Naver blog search results page
Naver Place: np.link was the clinic's own website, not Naver Place →
now generates map.naver.com search URL. Also fixed collect-channel-data
to search with "성형외과" suffix and match by category (성형/피부) to
avoid same-name dental clinics.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Facebook pageName now links to facebook.com/{url} with ExternalLink icon.
OtherChannels: moved ExternalLink from right-end to inline with channel
name, matching the Instagram/YouTube pattern.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
staffCount is the number of registered doctors from 강남언니, not total
staff. Label changed from 의료진(medical staff) to 전문의(specialists)
to accurately reflect the data source. Diagnosis message also updated.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
YouTubeAudit: handle links to youtube.com/@{handle} with ExternalLink icon
InstagramAudit: handle links to instagram.com/{handle} with ExternalLink icon
ClinicSnapshot: domain is now clickable link, phone is tel: link
OtherChannels: Google Maps generates search URL, Naver Blog links to
first blog post or search results (previously empty string)
transformReport: fills missing URL fields for Google Maps and Naver Blog
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Includes every prompt in production across 3 pipeline phases:
- Phase 1 (discover): 7 data sources with exact prompts
- Phase 2 (collect): 5 API calls + 4 market analysis queries
- Phase 3 (generate): full report generation prompt template
Added prompt engineering learnings:
- Short prompts outperform long ones on Perplexity sonar
- sonar > sonar-pro for channel search
- English clinic name in parentheses improves international results
- Verify strategy: keep unverified handles as candidates
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Instagram HEAD requests often fail (rate limiting, blocking) causing
valid handles to be dropped. Now all discovered handles are kept
(verified or not) and Apify attempts collection on all of them.
Apify's own scraper validates existence more reliably than HEAD requests.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Split queries performed worse. The proven working pattern is:
- Single query with Korean+English clinic name
- "검색해서 찾아줘. 검색 결과에서 발견된 계정을 모두 알려줘" phrasing
- All channels in one request
- English name in parentheses helps Perplexity find international accounts
Tested: "그랜드성형외과 (Grand Plastic Surgery)" → finds Instagram,
YouTube, Facebook, TikTok, Naver Blog all in one call.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Single mega-query returns empty results. Split into:
B4a. Instagram + YouTube (most important, focused search)
B4b. Facebook + TikTok + Naver Blog + Kakao
B4c. 강남언니 + review platforms
Each query is short and focused — matches the proven pattern of
2-5 keyword searches that Perplexity handles well.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Long system prompt caused sonar-pro to return empty results.
Reverted to sonar model with short, proven prompt pattern that
matches the user's successful manual test.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replaced simple "find handles" prompt with comprehensive research agent:
- Model: sonar → sonar-pro (advanced multi-step web search)
- System prompt: full research methodology with 2-3 keyword searches,
URL fetching, quantitative data extraction
- Output: structured JSON with channels (handles + follower counts +
subscriber counts) + platforms (강남언니 rating, reviews)
- Research results saved to scrape_data.onlinePresenceResearch for
downstream use in collect-channel-data and generate-report
Added _shared/researchPrompt.ts with prompt template + builder.
Updated agent documentation in doc/.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
B4 Perplexity: rewrote from narrow "find social accounts" to broad
"Online Presence 종합 분석" — finds Instagram, YouTube, Facebook,
TikTok, Naver, Kakao, 강남언니, 바비톡 in one query.
B5 Apify Instagram: generates handle candidates from clinic name
(english name, domain, _official, _ps, _clinic variants) and directly
checks each via Apify instagram-profile-scraper. Finds accounts that
web search misses.
Removed redundant B4b (platform presence) — now merged into B4.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
API results may contain null, numbers, or objects instead of strings.
Now coerces all values to strings before processing.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replaced Perplexity-only approach with 5 parallel direct API searches:
B1. YouTube Data API: search?type=channel&q={clinicName} → find channel
B2a. Naver Blog API: search blog.json → find official Naver blog
B2b. Naver Web API: search webkr.json → find Instagram/YouTube/Facebook URLs
B3. Firecrawl Search: web search → extract social URLs from results
B4. Perplexity: supplement — catch what direct APIs missed
All 5 sources run in parallel after Stage A (Firecrawl scrape for clinicName).
Results merged + deduplicated + verified. Perplexity is now a fallback,
not the primary source.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Perplexity prompts changed from "find verified accounts" (returns all
null) to "search and report what you find" (returns actual handles).
Added clinicName resolution: Firecrawl Korean → English → Perplexity
URL-to-name lookup → domain fallback.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Previously Firecrawl and Perplexity ran in parallel, so Perplexity
received raw URL instead of clinic name → poor search results.
Now:
Stage A: Firecrawl scrape+map (parallel) → extract clinicName from HTML
Stage B: Perplexity searches using extracted clinicName → finds Instagram,
YouTube, Facebook handles that Firecrawl HTML parsing missed
Stage C: Merge 3 sources + verify all handles
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
discover-channels: extractHandle('youtube') now detects UC* channel IDs
and returns them without @ prefix (previously @UC... caused verify fail)
verifyHandles: verifyYouTube uses cleanHandle for UC* check, requests
part=id,snippet for richer data
collect-channel-data: if channelId missing but handle present, resolves
via forHandle/forUsername lookup or direct UC* detection before skipping
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Halved blob movement (30px→15px, 8vw→4vw) and reduced scale
(1.1→1.05). Using translate3d forces GPU compositing layer,
preventing main-thread layout recalculation that causes jitter.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Added will-change:transform, contain:layout style, and
backface-visibility:hidden to all blob animations. This promotes
blobs to their own GPU compositing layer, preventing them from
triggering main-thread reflow/repaint that causes page shaking.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Animated blobs with translate+scale overflow their containers,
causing horizontal scrollbar to flicker and page to jitter.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Uses scrollbar-width:thin + 6px webkit scrollbar to minimize
layout shift from the default ~15px scrollbar. Keeps original
design intact while reducing visual asymmetry.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Reverts to the original design with full-color button (no disabled
opacity), original background gradients, and original blob decorations.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
discover-channels: new extractHandle() validates each handle belongs to
its platform (rejects hospital-internal URLs like /idtube/view being
treated as YouTube). Extracts handles from full URLs correctly.
collect-channel-data: explicit Record<string,unknown> typing for DB JSON
fields — fixes TypeScript property access on VerifiedChannels from DB.
verifyHandles: fix TikTok double-URL concatenation.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
All layout changes (blob modifications, overflow-x/clip, scrollbar-gutter)
reverted to the original version that was working correctly.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Removes scrollbar-gutter:stable which pushed content left.
overlay scrollbar floats over content with zero layout impact.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Haewon Kam