$ curl -d '{
"url": "https://techcrunch.com",
"query": "Get top 5 latest news",
"inferFields": ["category","headline","imageUrl","publishedAt","author","shortDescription"],
"output": "json"
}' \
https://api.webscraperapi.ai/v1/scrape
[
{
"category": "Enterprise",
"headline": "Google announces Axion, its first custom Arm-based data center processor",
"imageUrl": "https://techcrunch.com/wp-content/uploads/2019/03/GettyImages-1128091493.jpeg?w=430&h=230&crop=1",
"publishedAt": "2024-04-09T12:00:47",
"author": "Frederic Lardinois",
"shortDescription": "Google says its Axion instances offer 30% better performance than other Arm-based instances from competitors."
},
{
"category": "Startups",
"headline": "Kiki World, a beauty brand that uses web3 for customer co-creation and ownership, raises $7M from a16z",
"imageUrl": "https://techcrunch.com/wp-content/uploads/2024/04/KIKI-Chrome-03.jpg?w=430&h=230&crop=1",
"publishedAt": "2024-04-09T12:00:47",
"author": "Marina Temkin",
"shortDescription": "If you think that choosing a nail polish color or which ingredients go into your face cream cannot have anything to do with blockchain, think again. Kiki World, a beauty startup launched last year..."
},
{
"category": "AI",
"headline": "Google open-sources tools to support AI model development",
"imageUrl": "https://techcrunch.com/wp-content/uploads/2019/04/IMG_20190409_075753-1.jpg?w=430&h=230&crop=1",
"publishedAt": "2024-04-09T12:00:44",
"author": "Kyle Wiggers",
"shortDescription": "Google is launching Jetstream, a new engine to run generative AI models, and MaxDiffusion, a collection of reference implementations of various diffusion models."
},
{
"category": "Enterprise",
"headline": "New Google Vids product helps create a customized video with an AI assist",
"imageUrl": "https://techcrunch.com/wp-content/uploads/2024/04/Screenshot-2024-04-08-at-3.44.11 PM.png?w=430&h=230&crop=1",
"publishedAt": "2024-04-09T12:00:39",
"author": "Ron Miller",
"shortDescription": "All of the major vendors have been looking at ways to use AI to help customers develop creative content. On Tuesday at the Google Cloud Next customer conference in Las Vegas, Google introduced a ne..."
},
{
"category": "AI",
"headline": "Google’s Gemini Pro 1.5 enters public preview on Vertex AI",
"imageUrl": "https://techcrunch.com/wp-content/uploads/2024/03/google-gemini.jpg?w=430&h=230&crop=1",
"publishedAt": "2024-04-09T12:00:31",
"author": "Kyle Wiggers",
"shortDescription": "Its headlining feature is the amount of context that it can process: 1 million tokens, which is equivalent to around 700,000 words or around 30,000 lines of code."
}
]
Integrate in minutes
We have idiomatic SDKs for your favorite languages. Install and start scraping.
import { WebScraperApiClient } from 'webscraperapi/client';
const client = new WebScraperApiClient({ apiKey: 'YOUR_API_KEY' });
(async function() {
const data = await client.scrape({
url: 'https://remoteok.com/remote-python-jobs?order_by=salary',
query: 'Get jobs details',
inferFields: ["salaryRangeInUSD", "title", "datePosted", "url", "description"],
output: 'json'
});
console.log(JSON.stringify(data[0], null, 2));
// {
// "salaryRangeInUSD": [70000, 110000],
// "title": "Manager Software Engineering Navigator",
// "datePosted": "2024-03-26T11:00:08+00:00",
// "url": "https://remoteok.com/addepar",
// "description": "The Role\n\nThe Navigator team at Addepar is growing!\n\nNavigator is a quantitative liquidity risk projection tool that is one of the key components of Addepar's multi-product ecosystem. Brought into the Addepar family as part of an acquisition in 2021, Addepar has continued to double down on investing in this critical add-on, as we've scaled well beyond the multi-million ARR goal first set for this product line.\n\nAs we continue to scale, a need has opened up for a hands-on Engineering Manager who would bring to bear their unique expertise with the existing Engineering team, with exciting opportunities to extend the scope for this role longer term, as the product continues to push onward into future global expansion mode.\n\nAddepar takes a market-based approach to pay. A successful candidate’s starting pay will be determined based on the role, job-related skills, experience, qualifications, work location, and market conditions. The range displayed on each job posting reflects the\n Apply now and work remotely at Addepar",
// }
})();No code? No problem. Just type=SCRAPEURL("URL", "QUERY")
in Google Sheets to get data from any website
=SCRAPEURL("URL", "QUERY")A1
fx=SCRAPEURL("https://www.cars.com/shopping/results/?makes[]=toyota&sort=list_price_desc", "Get all Toyotas")
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
A
B
C
D
E
F
=SCRAPEURL("https://www.cars.com/shopping/results/?makes[]=toyota&sort=list_price_desc", "Get all Toyotas")
Monitor websites and receive email alerts when the content changes
Web scraping is hard
CAPTCHAs
Rate-limiting
JavaScript generated page
IP blocked
Retrying connections
Cleaning data
Messy HTML
Changing HTML structure
Changing CSS selectors
Pagination
Infinite scrolling
We made it easy. For you.
Scrape the web using English.