How to Scrape DuckDuckGo Search Results in Python: Step-by-Step Guide

DuckDuckGo is best known as a privacy-focused search engine. However, its search results can also be useful for developers building research tools, SEO workflows, and applications that need to discover relevant webpages.

Suppose you want to collect search result titles, URLs, and descriptions for a list of keywords. Manually copying each result quickly becomes inefficient.

Python offers a more practical approach.

In this guide, you’ll learn how to scrape DuckDuckGo search results in Python using the ddgs package, organize results into a CSV file, apply regional settings, and handle common errors.

We’ll also compare Python search libraries with browser scraping and managed SERP APIs. Finally, we’ll explain when proxies are useful and when they add unnecessary complexity.

The examples focus on small-scale, permitted research rather than bypassing search engine restrictions.

Can You Scrape DuckDuckGo Search Results With Python?

Yes. Python tools can retrieve DuckDuckGo search results and organize the returned information into structured records.

A typical search result contains:

• Page title.
• Destination URL.
• Short description or snippet.

For example, a Python application might convert search results into the following structure:

{
  "keyword": "python web scraping",
  "title": "Example Python Tutorial",
  "url": "https://example.com/tutorial",
  "snippet": "Learn the basics of web data extraction.",
  "region": "us-en"
}

This example is illustrative rather than a live DuckDuckGo result.

Structured records are easier to store, filter, compare, and analyze than manually copied search pages.

However, the collection method matters.

DuckDuckGo does not provide a general-purpose public web-search API equivalent to the interfaces offered by many commercial SERP data providers.

Instead, developers commonly use third-party libraries or services.

Before choosing an approach, review its maintenance status, access requirements, and suitability for your project.

Three Ways to Get DuckDuckGo Search Data

There are three common approaches.

For a practical Python tutorial, we’ll use ddgs.

It provides a convenient way to request search results without building an HTML parser from scratch.

Nevertheless, the library is an independent open-source project, not an official DuckDuckGo API.

Step 1: Install Python and DDGS

First, make sure Python is installed.

The current DDGS package requires Python 3.10 or later.

You can check your Python version using:

python --version

Next, install the library:

python -m pip install -U ddgs

Older tutorials may recommend:

pip install duckduckgo_search

However, the duckduckgo_search package has been renamed to ddgs. New projects should use the newer package and import path.

The library’s installation instructions and supported features are available in its official project repository.

Step 2: Run Your First DuckDuckGo Search

Create a Python file named:

duckduckgo_search.py

Add the following code:

from ddgs import DDGS

query = "python web scraping"

results = DDGS().text(
    query,
    region="us-en",
    max_results=5,
    backend="duckduckgo"
)

for result in results:
    print("Title:", result.get("title"))
    print("URL:", result.get("href"))
    print("Description:", result.get("body"))
    print("-" * 50)

Run the script:

python duckduckgo_search.py

The library requests search results and returns structured dictionaries containing information such as titles, URLs, and snippets.

Why Set backend=”duckduckgo”?

This parameter is particularly important in newer versions of DDGS.

The library has evolved into a metasearch tool that supports several search engines.

Its default backend can select from multiple providers. Therefore, results returned by the default configuration are not necessarily exclusive to DuckDuckGo.

By explicitly specifying:

backend="duckduckgo"

you tell the library which search backend to use.

This matters when your research specifically targets DuckDuckGo search results rather than general web search.

Step 3: Save DuckDuckGo Results to CSV

Printing results is useful for testing, but structured storage becomes more practical when you need to analyze multiple searches.

CSV files can be opened in Excel, imported into databases, or processed with pandas.

Here’s a complete example:

import csv
from datetime import datetime, timezone

from ddgs import DDGS

query = "residential proxies"
region = "us-en"

results = DDGS().text(
    query,
    region=region,
    max_results=10,
    backend="duckduckgo"
)

timestamp = datetime.now(timezone.utc).isoformat()

with open(
    "duckduckgo_results.csv",
    "w",
    newline="",
    encoding="utf-8-sig"
) as file:
    writer = csv.writer(file)

    writer.writerow([
        "keyword",
        "region",
        "collected_at",
        "title",
        "url",
        "snippet"
    ])

    for result in results:
        writer.writerow([
            query,
            region,
            timestamp,
            result.get("title", ""),
            result.get("href", ""),
            result.get("body", "")
        ])

print("Results saved successfully.")

This creates a CSV file containing each result’s keyword, region, collection timestamp, title, destination URL, and snippet.

The additional metadata matters because search results can change over time.

For example, a result collected in November may differ from one collected several weeks later.

Recording the collection date helps you interpret those differences.

Step 4: Search DuckDuckGo by Region

DuckDuckGo supports regional search preferences.

For example, users may want to compare results associated with the United States and the United Kingdom.

In DDGS, you can specify the region parameter.

from ddgs import DDGS

regions = ["us-en", "uk-en"]
query = "best project management tools"

for region in regions:
    print(f"\nRegion: {region}")

    results = DDGS().text(
        query,
        region=region,
        max_results=5,
        backend="duckduckgo"
    )

    for result in results:
        print(result.get("title"))

The region parameter influences the search preferences used for the request.

However, changing the search region does not change your public IP address.

This distinction is important for SEO research.

A region setting helps request location-oriented search results. Meanwhile, a geographically targeted proxy changes the network exit location.

Neither guarantees that your results exactly match what every local user sees.

For a detailed explanation of these differences, see our DuckDuckGo Proxy Guide.

Step 5: Collect Multiple Keywords

Once a single search works, you can organize a small batch of research queries.

The following example includes conservative pacing and stops when the service reports a rate limit.

import csv
import time

from ddgs import DDGS

keywords = [
    "python automation",
    "web scraping tools",
    "seo rank tracking"
]

rows = []

with DDGS(timeout=15) as client:
    for keyword in keywords:
        try:
            results = client.text(
                keyword,
                region="us-en",
                max_results=5,
                backend="duckduckgo"
            )

            for result in results:
                rows.append({
                    "keyword": keyword,
                    "title": result.get("title", ""),
                    "url": result.get("href", ""),
                    "snippet": result.get("body", "")
                })

        except Exception as error:
            print(f"Stopped after error: {error}")
            break

        time.sleep(5)

with open(
    "keyword_results.csv",
    "w",
    newline="",
    encoding="utf-8-sig"
) as file:
    writer = csv.DictWriter(
        file,
        fieldnames=[
            "keyword",
            "title",
            "url",
            "snippet"
        ]
    )

    writer.writeheader()
    writer.writerows(rows)

The delay reduces unnecessary request bursts, but it does not guarantee that requests will be accepted or that a particular rate is permitted.

For a production application, use documented rate limits and access requirements. Also, avoid assuming that catching an exception means the collection job should automatically retry.

When a provider restricts automated access, stop and review the appropriate supported options.

Step 6: Remove Duplicate Search Results

Different keywords can return the same webpage.

For example, a tutorial may appear for both:

• python automation
• web scraping tools

If your goal is to build a list of unique URLs, deduplication helps prevent unnecessary repeated processing.

A simple approach uses the URL as the unique identifier:

unique_results = {}

for result in rows:
    url = result["url"]

    if url and url not in unique_results:
        unique_results[url] = result

clean_results = list(unique_results.values())

print("Unique URLs:", len(clean_results))

This approach is straightforward, but URL-level deduplication has limitations.

For instance, the same page may appear with tracking parameters or alternate URL formats.

For larger datasets, you may also need URL normalization, canonical URL checks, or content-based deduplication.

Additionally, remember that removing duplicate URLs can discard information about which keywords produced each result.

If keyword overlap matters, preserve a separate keyword-to-URL relationship rather than keeping only one record per URL.

Python DDGS vs. Manual HTML Scraping

Another way to collect search information is to request a search-result webpage and parse its HTML.

Python libraries such as Requests and Beautiful Soup can help with permitted webpage extraction.

However, manual HTML parsing introduces several challenges.

Page Structure Changes

Search pages can change their HTML structure without notice.

As a result, CSS selectors that work today may stop finding the expected elements later.

Dynamic Content

Some interfaces depend on JavaScript or asynchronous requests.

A simple HTTP client may not reproduce the same browser behavior.

Data Validation

A successful request does not mean the response contains the search results you expected.

For example, the response could contain an error page, an unexpected language, or incomplete data.

Maintenance

Custom parsers require ongoing testing.

Therefore, a maintained library or authorized SERP API may be more practical when you only need structured results.

Manual extraction makes more sense when the task requires examining permitted raw HTML or elements that a structured interface does not expose.

Common DuckDuckGo Scraping Errors

Even a simple Python script can encounter unexpected errors.

The correct response depends on the underlying cause.

1. ModuleNotFoundError: No Module Named ‘duckduckgo_search’

You may be following an older tutorial.

The package has moved to ddgs.

Install the current package:

python -m pip install -U ddgs

Then use:

from ddgs import DDGS

Also make sure that you installed the package in the same Python environment used to run your script.

2. Empty Search Results

An empty result list does not always mean your script is broken.

Possible explanations include:

• The query returns few matching pages.
• The selected region changes the results.
• The search backend is temporarily unavailable.
• The library encounters a source-side response it cannot parse.

Start by testing one simple, common query.

Then check the selected backend and library version.

3. Rate Limit Errors

Automated requests may trigger restrictions or receive rate-limit responses.

When this happens, stop the affected collection process and review the source’s permitted access methods.

Avoid escalating request volume or switching IP addresses simply to defeat restrictions.

For recurring commercial workloads, an authorized search-data provider may be more suitable.

4. Timeout Errors

Timeouts can occur because of network instability, slow responses, or service availability problems.

You can configure a reasonable timeout in DDGS:

from ddgs import DDGS

with DDGS(timeout=15) as client:
    results = client.text(
        "python programming",
        max_results=5,
        backend="duckduckgo"
    )

If the service remains unavailable, investigate the connection and stop rather than creating an uncontrolled retry loop.

5. Results Do Not Match the Expected Region

Check the region argument first.

Next, review language preferences and whether your research requires a particular network location.

Keep in mind that regional search settings and proxy IP location are separate factors.

Do You Need Proxies for DuckDuckGo Scraping?

Not always.

For occasional permitted research queries, an ordinary connection may be sufficient.

Proxies become relevant when a legitimate collection or testing workflow requires specific network characteristics, such as:

• A particular geographic exit location.
• A stable network identity.
• Application-specific routing.
• Controlled network testing across regions.

For example, a market research team may need to inspect how destination websites behave when accessed from different countries.

A location-specific proxy can support that type of testing.

However, a proxy does not make automated scraping automatically permitted, and it does not guarantee successful requests.

Residential Proxies

Residential proxies can be useful for authorized geographic research when the network location matters.

QuarkIP’s Residential Proxies offer geographic targeting options that may support suitable regional research workflows.

Static ISP Proxies

Static ISP proxies provide a consistent outbound IP address.

They can be useful when your research environment requires repeatable network conditions or longer-running sessions.

Datacenter Proxies

Datacenter proxies often provide predictable infrastructure for permitted technical testing and accessible public sources.

They may be sufficient when residential network characteristics are unnecessary.

For a broader comparison, see our guide to SEO Proxies.

DuckDuckGo Python Library vs. SERP API

If your application depends on search results in production, infrastructure reliability becomes more important.

Consider the differences:

For smaller experiments, DDGS offers a convenient starting point.

However, a managed SERP API may be easier to maintain when your application needs predictable output, support, and documented service terms.

Before selecting a provider, review its DuckDuckGo coverage, data freshness, permissible uses, and pricing model.

Best Practices for DuckDuckGo Data Collection

A reliable search-data workflow should focus on quality rather than simply collecting as many results as possible.

Record Your Search Parameters

Save the keyword, search region, backend, and collection timestamp.

This makes later comparisons more meaningful.

Validate the Output

Check that results contain valid URLs, titles, and expected fields.

A successful function call does not guarantee useful data.

Avoid Duplicate Processing

Deduplicate URLs when appropriate, while preserving keyword relationships if your analysis needs them.

Handle Errors Explicitly

Distinguish timeouts, empty results, package errors, and source-side restrictions.

Avoid unlimited automatic retries.

Choose a Suitable Data Source

Use a library for small experiments when permitted.

For applications requiring documented access and reliable delivery, evaluate a suitable authorized provider.

Protect Credentials

If you configure a proxy, keep authentication details in environment variables or another appropriate secrets-management system rather than committing them to public repositories.

FAQ

Can I scrape DuckDuckGo with Python?

Python libraries such as DDGS can retrieve search results in a structured format. However, developers should review the applicable access rules and the third-party library’s limitations before using it.

Is duckduckgo_search deprecated?

The package has been renamed to ddgs. For new projects, use pip install ddgs and from ddgs import DDGS.

Is DDGS an official DuckDuckGo API?

No. DDGS is an independent open-source metasearch library, not an official DuckDuckGo web-search API.

How do I get DuckDuckGo search results as JSON?

DDGS returns structured Python dictionaries. You can use Python’s built-in json module to serialize those records into JSON files or API responses.

Final Thoughts

Scraping DuckDuckGo search results with Python can be useful for prototypes, research scripts, and applications that need structured search information.

For a simple workflow, the ddgs library lets you retrieve titles, URLs, and snippets without building a custom HTML parser.

As your requirements grow, however, data quality becomes just as important as retrieval.

Record search parameters, validate outputs, manage duplicates, and handle errors responsibly.

When geographic testing is required, a proxy may help control the network location. For recurring production use, an authorized SERP API may provide a more maintainable solution.

Ultimately, the best approach is the one that delivers reliable, appropriately sourced data with the least unnecessary infrastructure.