Sold by: Common Crawl
Open data
|
Deployed on AWS
A corpus of web crawl data composed of over 300 billion web pages.
Overview
A corpus of web crawl data composed of over 300 billion web pages.
Features and programs
Open Data Sponsorship Program
This dataset is part of the Open Data Sponsorship Program, an AWS program that covers the cost of storage for publicly available high-value cloud-optimized datasets.
Pricing
This is a publicly available data set. No subscription is required.
Legal
Content disclaimer
Vendors are responsible for their product descriptions and other product content. AWS does not warrant that vendors' product descriptions or other product content are accurate, complete, reliable, current, or error-free.
Delivery details
AWS Data Exchange (ADX)
AWS Data Exchange is a service that helps AWS easily share and manage data entitlements from other organizations at scale.
Open data resources
Available with or without an AWS account.
- How to use
- To access these resources, reference the Amazon Resource Name (ARN) using the AWS Command Line Interface (CLI). Learn more
- Description
- Crawl data (WARC and ARC format)
- Resource type
- S3 bucket
- Amazon Resource Name (ARN)
- arn:aws:s3:::commoncrawl
- AWS region
- us-east-1
- AWS CLI access (No AWS account required)
- aws s3 ls --no-sign-request s3://commoncrawl/
Resources
Vendor resources
Support
Managed By
How to cite
Common Crawl was accessed on DATE from https://registry.opendata.aws/commoncrawl .
License
This data is available for anyone to use under the Common Crawl Terms of Use
Similar products
Use Common Crawl (AWS Data Exchange for AWS Lake Formation Test Product) to understand how to interact with data made available via AWS Lake Formation.
Diffbot's mission is to structure the world wide web to create the first complete map of public human knowledge.
A large Japanese language corpus created through preprocessing Common Crawl data
This product has charges associated with it for seller support. Scrapy is a powerful, open-source Python framework designed for web crawling, web scraping, and data extraction at scale. It is widely used in data engineering, SEO analysis, price monitoring, market research, and building large datasets from websites.