Common Crawl Foundation

non-profit

Verified

https://commoncrawl.org

AI & ML interests

Crawled data and metadata

Recent Activity

malteos updated a dataset 3 days ago

commoncrawl/commonlid-cache_commonlid_nano

malteos updated a bucket 15 days ago

commoncrawl/test-bucket

malteos new activity 17 days ago

commoncrawl/commonlid-results:Add py3langid results on commonlid_nano

View all activity

commoncrawl 's Spaces 6

README

Explore Common Crawl's metadata and experimental datasets

Web Graph Embeddings

What is actually inside llms.txt

Analyze llms.txt files for content insights

Web Graph kNN Spot-check

Nearest neighbours of a host in the CC host-graph embedding

CommonLID Leaderboard

View the CommonLID language ID leaderboard

cc-citations

Scientific articles using or citing Common Crawl data