Instagram HEAD requests often fail (rate limiting, blocking) causing
valid handles to be dropped. Now all discovered handles are kept
(verified or not) and Apify attempts collection on all of them.
Apify's own scraper validates existence more reliably than HEAD requests.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Split queries performed worse. The proven working pattern is:
- Single query with Korean+English clinic name
- "검색해서 찾아줘. 검색 결과에서 발견된 계정을 모두 알려줘" phrasing
- All channels in one request
- English name in parentheses helps Perplexity find international accounts
Tested: "그랜드성형외과 (Grand Plastic Surgery)" → finds Instagram,
YouTube, Facebook, TikTok, Naver Blog all in one call.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Single mega-query returns empty results. Split into:
B4a. Instagram + YouTube (most important, focused search)
B4b. Facebook + TikTok + Naver Blog + Kakao
B4c. 강남언니 + review platforms
Each query is short and focused — matches the proven pattern of
2-5 keyword searches that Perplexity handles well.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Long system prompt caused sonar-pro to return empty results.
Reverted to sonar model with short, proven prompt pattern that
matches the user's successful manual test.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replaced simple "find handles" prompt with comprehensive research agent:
- Model: sonar → sonar-pro (advanced multi-step web search)
- System prompt: full research methodology with 2-3 keyword searches,
URL fetching, quantitative data extraction
- Output: structured JSON with channels (handles + follower counts +
subscriber counts) + platforms (강남언니 rating, reviews)
- Research results saved to scrape_data.onlinePresenceResearch for
downstream use in collect-channel-data and generate-report
Added _shared/researchPrompt.ts with prompt template + builder.
Updated agent documentation in doc/.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
B4 Perplexity: rewrote from narrow "find social accounts" to broad
"Online Presence 종합 분석" — finds Instagram, YouTube, Facebook,
TikTok, Naver, Kakao, 강남언니, 바비톡 in one query.
B5 Apify Instagram: generates handle candidates from clinic name
(english name, domain, _official, _ps, _clinic variants) and directly
checks each via Apify instagram-profile-scraper. Finds accounts that
web search misses.
Removed redundant B4b (platform presence) — now merged into B4.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
API results may contain null, numbers, or objects instead of strings.
Now coerces all values to strings before processing.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replaced Perplexity-only approach with 5 parallel direct API searches:
B1. YouTube Data API: search?type=channel&q={clinicName} → find channel
B2a. Naver Blog API: search blog.json → find official Naver blog
B2b. Naver Web API: search webkr.json → find Instagram/YouTube/Facebook URLs
B3. Firecrawl Search: web search → extract social URLs from results
B4. Perplexity: supplement — catch what direct APIs missed
All 5 sources run in parallel after Stage A (Firecrawl scrape for clinicName).
Results merged + deduplicated + verified. Perplexity is now a fallback,
not the primary source.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Perplexity prompts changed from "find verified accounts" (returns all
null) to "search and report what you find" (returns actual handles).
Added clinicName resolution: Firecrawl Korean → English → Perplexity
URL-to-name lookup → domain fallback.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Previously Firecrawl and Perplexity ran in parallel, so Perplexity
received raw URL instead of clinic name → poor search results.
Now:
Stage A: Firecrawl scrape+map (parallel) → extract clinicName from HTML
Stage B: Perplexity searches using extracted clinicName → finds Instagram,
YouTube, Facebook handles that Firecrawl HTML parsing missed
Stage C: Merge 3 sources + verify all handles
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
discover-channels: extractHandle('youtube') now detects UC* channel IDs
and returns them without @ prefix (previously @UC... caused verify fail)
verifyHandles: verifyYouTube uses cleanHandle for UC* check, requests
part=id,snippet for richer data
collect-channel-data: if channelId missing but handle present, resolves
via forHandle/forUsername lookup or direct UC* detection before skipping
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
discover-channels: new extractHandle() validates each handle belongs to
its platform (rejects hospital-internal URLs like /idtube/view being
treated as YouTube). Extracts handles from full URLs correctly.
collect-channel-data: explicit Record<string,unknown> typing for DB JSON
fields — fixes TypeScript property access on VerifiedChannels from DB.
verifyHandles: fix TikTok double-URL concatenation.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Restructured the entire analysis pipeline from AI-guessing social
handles to deterministic 3-phase discovery + collection + generation.
Phase 1 (discover-channels): 3-source channel discovery
- Firecrawl scrape: extract social links from HTML
- Perplexity search: find handles via web search
- URL regex parsing: deterministic link extraction
- Handle verification: HEAD requests + YouTube API
- DB: creates row with verified_channels + scrape_data
Phase 2 (collect-channel-data): 9 parallel data collectors
- Instagram (Apify), YouTube (Data API v3), Facebook (Apify)
- 강남언니 (Firecrawl), Naver Blog + Place (Naver API)
- Google Maps (Apify), Market analysis (Perplexity 4x parallel)
- DB: stores ALL raw data in channel_data column
Phase 3 (generate-report): AI report from real data
- Reads channel_data + analysis_data from DB
- Builds channel summary with real metrics
- AI generates report using only verified data
- V1 backwards compatibility preserved (url-based flow)
Supporting changes:
- DB migration: status, verified_channels, channel_data columns
- _shared/extractSocialLinks.ts: regex-based social link parser
- _shared/verifyHandles.ts: multi-platform handle verifier
- AnalysisLoadingPage: real 3-phase progress + channel panel
- useReport: channel_data column support + V2 enrichment merge
- 강남언니 rating: auto-correct 5→10 scale + search fallback
- KPIDashboard: navigate() instead of <a href>
- Loading text: 20-30초 → 1-2분
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Naver Blog search: collect blog post results for clinic name (total count + top 10 posts)
- Naver Place search: collect place info (name, category, address, telephone)
- Multi-account Instagram: AI prompt requests all IG accounts (국내/해외)
- enrich-channels: process multiple IG handles with fallback per handle
- transformReport: merge multiple IG accounts into instagramAudit.accounts[]
- generate-report: socialHandles.instagram now array of handles
- Hero/CTA: transition-all → transition-shadow for instant click response
- Hero/CTA: disabled state when URL is empty (opacity-50 + cursor-not-allowed)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- enrich-channels: add 강남언니 scraping module (search + structured JSON extraction)
- Collects: rating/10, reviews, doctors with ratings, procedures, certifications
- transformReport: merge 강남언니 data into clinicSnapshot + otherChannels
- Updates lead doctor info, certifications, and review counts from real data
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- enrich-channels: Instagram fallback — auto-try _ps, .ps, _clinic suffixes when <100 followers
- enrich-channels: YouTube URL normalization via normalizeYouTubeChannel (handles /c/, /user/, @handle)
- enrich-channels: Google Maps multi-query search for better hit rate
- generate-report: AI-found social handles prioritized over Firecrawl scrape
- generate-report: Added socialMedia field to AI prompt for accurate handle discovery
- normalizeHandles: Added normalizeYouTubeChannel for /c/, /user/, /channel/, @handle URLs
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- P1-5: Add kpiTargets schema to AI prompt, use AI-generated goals instead of hardcoded multipliers
- P1-6: Extend website channelAnalysis with trackingPixels, snsLinksOnSite, additionalDomains, mainCTA
- P1-7: ClinicProfilePage fetches data from DB by report ID instead of hardcoded VIEW clinic data
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add normalizeInstagramHandle() utility (Edge + browser) to strip URLs, @ prefixes
- generate-report: normalize handles before saving, persist socialHandles in report JSONB
- enrich-channels: normalize Instagram handle before Apify call (defense in depth)
- useReport: recover socialHandles + channelEnrichment from DB on direct URL access
- ReportPage: skip redundant enrichment when data already exists in DB
Fixes: Instagram enrichment failing due to URL-format handles passed to Apify
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Replace mock useReport() with real Supabase API data pipeline
- Add transformReport.ts to map API responses to MarketingReport type
- Add useEnrichment() hook for background channel data enrichment
- Replace Apify YouTube scraper with YouTube Data API v3
- Add mergeEnrichment() for progressive data loading
- Add EmptyState component for graceful empty data handling
- Add socialHandles to generate-report metadata
- Graceful empty data in ClinicSnapshot, YouTube, Instagram, Facebook
- Add Supabase Edge Functions and DB migrations
- Add developer handoff documentation
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Haewon Kam