主题
Oxylabs 抓取器
Oxylabs 抓取器可让你轻松访问相应来源中的信息。请参阅下方可用来源列表：
Amazon ProductAmazon SearchGoogle SearchUniversal
安装
在此处创建 Oxylabs 账户以获取凭证。
shell
pip install 'crewai[tools]' oxylabs查看 Oxylabs 文档 以了解更多关于 API 参数的信息。
OxylabsAmazonProductScraperTool
Example
python
from crewai_tools import OxylabsAmazonProductScraperTool
# make sure OXYLABS_USERNAME and OXYLABS_PASSWORD variables are set
tool = OxylabsAmazonProductScraperTool()
result = tool.run(query="AAAAABBBBCC")
print(result)参数
query- 10 位符号的 ASIN 代码。domain- Amazon 的域名本地化设置。geo_location- Deliver to 位置。user_agent_type- 设备类型和浏览器。render- 设置为html时启用 JavaScript 渲染。callback_url- 指向你的回调端点的 URL。context- 针对特殊需求的附加高级设置和控制项。parse- 设置为 true 时返回解析后的数据。parsing_instructions- 定义你自己的解析和数据转换逻辑，该逻辑将在 HTML 抓取结果上执行。
进阶示例
python
from crewai_tools import OxylabsAmazonProductScraperTool
# make sure OXYLABS_USERNAME and OXYLABS_PASSWORD variables are set
tool = OxylabsAmazonProductScraperTool(
config={
"domain": "com",
"parse": True,
"context": [
{
"key": "autoselect_variant",
"value": True
}
]
}
)
result = tool.run(query="AAAAABBBBCC")
print(result)OxylabsAmazonSearchScraperTool
Example
python
from crewai_tools import OxylabsAmazonSearchScraperTool
# make sure OXYLABS_USERNAME and OXYLABS_PASSWORD variables are set
tool = OxylabsAmazonSearchScraperTool()
result = tool.run(query="headsets")
print(result)参数
query- Amazon 搜索词。domain- Amazon 的域名本地化设置。start_page- 起始页码。pages- 要检索的页数。geo_location- Deliver to 位置。user_agent_type- 设备类型和浏览器。render- 设置为html时启用 JavaScript 渲染。callback_url- 指向你的回调端点的 URL。context- 针对特殊需求的附加高级设置和控制项。parse- 设置为 true 时返回解析后的数据。parsing_instructions- 定义你自己的解析和数据转换逻辑，该逻辑将在 HTML 抓取结果上执行。
进阶示例
python
from crewai_tools import OxylabsAmazonSearchScraperTool
# make sure OXYLABS_USERNAME and OXYLABS_PASSWORD variables are set
tool = OxylabsAmazonSearchScraperTool(
config={
"domain": 'nl',
"start_page": 2,
"pages": 2,
"parse": True,
"context": [
{'key': 'category_id', 'value': 16391693031}
],
}
)
result = tool.run(query='nirvana tshirt')
print(result)OxylabsGoogleSearchScraperTool
Example
python
from crewai_tools import OxylabsGoogleSearchScraperTool
# make sure OXYLABS_USERNAME and OXYLABS_PASSWORD variables are set
tool = OxylabsGoogleSearchScraperTool()
result = tool.run(query="iPhone 16")
print(result)参数
query- 搜索关键词。domain- Google 的域名本地化设置。start_page- 起始页码。pages- 要检索的页数。limit- 每页要检索的结果数量。locale-Accept-Language请求头的值，用于更改你的 Google 搜索页面 Web 界面语言。geo_location- 结果应适配的地理位置。正确使用此参数对于获取正确的数据极其重要。user_agent_type- 设备类型和浏览器。render- 设置为html时启用 JavaScript 渲染。callback_url- 指向你的回调端点的 URL。context- 针对特殊需求的附加高级设置和控制项。parse- 设置为 true 时返回解析后的数据。parsing_instructions- 定义你自己的解析和数据转换逻辑，该逻辑将在 HTML 抓取结果上执行。
进阶示例
python
from crewai_tools import OxylabsGoogleSearchScraperTool
# make sure OXYLABS_USERNAME and OXYLABS_PASSWORD variables are set
tool = OxylabsGoogleSearchScraperTool(
config={
"parse": True,
"geo_location": "Paris, France",
"user_agent_type": "tablet",
}
)
result = tool.run(query="iPhone 16")
print(result)OxylabsUniversalScraperTool
Example
python
from crewai_tools import OxylabsUniversalScraperTool
# make sure OXYLABS_USERNAME and OXYLABS_PASSWORD variables are set
tool = OxylabsUniversalScraperTool()
result = tool.run(url="https://ip.oxylabs.io")
print(result)参数
url- 要抓取的网站 url。user_agent_type- 设备类型和浏览器。geo_location- 设置代理的地理位置以检索数据。render- 设置为html时启用 JavaScript 渲染。callback_url- 指向你的回调端点的 URL。context- 针对特殊需求的附加高级设置和控制项。parse- 设置为true时返回解析后的数据，前提是存在针对所提交 URL 页面类型的专用解析器。parsing_instructions- 定义你自己的解析和数据转换逻辑，该逻辑将在 HTML 抓取结果上执行。
进阶示例
python
from crewai_tools import OxylabsUniversalScraperTool
# make sure OXYLABS_USERNAME and OXYLABS_PASSWORD variables are set
tool = OxylabsUniversalScraperTool(
config={
"render": "html",
"user_agent_type": "mobile",
"context": [
{"key": "force_headers", "value": True},
{"key": "force_cookies", "value": True},
{
"key": "headers",
"value": {
"Custom-Header-Name": "custom header content",
},
},
{
"key": "cookies",
"value": [
{"key": "NID", "value": "1234567890"},
{"key": "1P JAR", "value": "0987654321"},
],
},
{"key": "http_method", "value": "get"},
{"key": "follow_redirects", "value": True},
{"key": "successful_status_codes", "value": [808, 909]},
],
}
)
result = tool.run(url="https://ip.oxylabs.io")
print(result)