Elasticsearch Integration in Python — Core Concepts

Master the elasticsearch-py client: indexing documents, building queries, handling pagination, and understanding how Python apps leverage Elasticsearch effectively.

Elasticsearch is a distributed search and analytics engine built on Apache Lucene. Python integrates with it through the official elasticsearch-py client, enabling full-text search, structured queries, and aggregations from Python applications.

Connection basics

from elasticsearch import Elasticsearch

es = Elasticsearch(
    "https://localhost:9200",
    basic_auth=("elastic", "password"),
    ca_certs="/path/to/ca.crt"
)

print(es.info())  # Cluster name, version, status

For cloud-hosted Elasticsearch:

es = Elasticsearch(
    cloud_id="my-deployment:base64encodedstring",
    api_key="your-api-key"
)

Indexing documents

Documents are JSON objects stored in indices (similar to database tables).

doc = {
    "title": "Python Async Deep Dive",
    "content": "Asyncio enables concurrent I/O operations...",
    "tags": ["python", "async"],
    "published": "2026-03-15"
}

es.index(index="articles", id="1", document=doc)

For bulk indexing (essential for performance with many documents):

from elasticsearch.helpers import bulk

actions = [
    {"_index": "articles", "_id": str(i), "_source": doc}
    for i, doc in enumerate(documents)
]
bulk(es, actions, chunk_size=500)

Searching

Full-text search

results = es.search(
    index="articles",
    query={
        "match": {
            "content": "async programming"
        }
    }
)

for hit in results['hits']['hits']:
    print(f"{hit['_score']:.2f} — {hit['_source']['title']}")

Multi-field search

results = es.search(
    index="articles",
    query={
        "multi_match": {
            "query": "python testing",
            "fields": ["title^3", "content", "tags^2"],
            "type": "best_fields"
        }
    }
)

The ^3 boosts title matches to 3x importance, making title matches rank higher.

Filtered queries

Combine full-text search with exact filters:

results = es.search(
    index="articles",
    query={
        "bool": {
            "must": {"match": {"content": "machine learning"}},
            "filter": [
                {"term": {"tags": "python"}},
                {"range": {"published": {"gte": "2026-01-01"}}}
            ]
        }
    }
)

must clauses affect relevance scoring. filter clauses are yes/no checks that don’t affect scores — they’re also cached for speed.

Pagination

Offset-based (simple, limited)

results = es.search(index="articles", query=query, from_=20, size=10)

Works for the first 10,000 results (configurable limit). Beyond that, use search_after.

Search-after (deep pagination)

results = es.search(
    index="articles",
    query=query,
    sort=[{"published": "desc"}, {"_id": "asc"}],
    size=10
)

# Next page
last_hit = results['hits']['hits'][-1]
results = es.search(
    index="articles",
    query=query,
    sort=[{"published": "desc"}, {"_id": "asc"}],
    search_after=last_hit['sort'],
    size=10
)

Aggregations

Elasticsearch can compute analytics alongside search results:

results = es.search(
    index="articles",
    query={"match_all": {}},
    aggs={
        "tags_breakdown": {
            "terms": {"field": "tags.keyword", "size": 20}
        },
        "monthly_count": {
            "date_histogram": {"field": "published", "calendar_interval": "month"}
        }
    },
    size=0  # We only want aggregations, not documents
)

Common misconception

People treat Elasticsearch as a primary database. It’s not designed for that. It lacks true ACID transactions, and data can be briefly inconsistent after writes (near-real-time, usually 1 second). Use it as a search layer alongside your primary database, with a sync pipeline keeping them aligned.

One thing to remember: Elasticsearch gives Python apps powerful search capabilities — but it works best as a specialized search index, not a replacement for your main database.

pythonelasticsearchelasticsearch-py