Home › Resources › NCBI E-utilities Tutorial
Bioinformatics · Interactive · Works offline

Query NCBI Programmatically:
The E-utilities Playground

Learn how esearch, efetch and esummary fit together, then experiment with a fully simulated API, generate Biopython / EDirect / cURL code, and walk through a query-to-FASTA case study. By Fiaraz Iqbal, Former Headteacher & Examiner.

01Architecture of a query

Hover, focus or tap each stage to see what happens. Press “Animate flow” to watch a request travel.

02Foundations

The five core utilities, the rules of polite access, and three ways to call them.

🪪 Identify yourself

Always send tool and email. NCBI uses them to contact you about a misbehaving script rather than blocking your IP. In Biopython: Entrez.email = "you@uni.ac.uk".

🔑 API key = 3× faster

Without a key: 3 requests/second. With a free key from your NCBI account: 10 requests/second. Pass api_key= or set NCBI_API_KEY for EDirect.

🗄️ History server

Use usehistory=y in esearch to get WebEnv + query_key, then efetch in batches with retstart/retmax instead of posting thousands of IDs.

🐍 Biopython Bio.Entrez

Idiomatic Python wrapper; parses XML into dicts and feeds SeqIO.

💻 EDirect (CLI)

Unix pipes: esearch | efetch | xtract. Handles history and batching for you.

🌐 Raw REST

Plain HTTP GET with cURL or fetch() against eutils.ncbi.nlm.nih.gov.

03E-utilities API simulator

Change any control: the URL and payload update instantly. All responses are self-contained mock data (accessions are illustrative), so nothing leaves your browser.

HTTP GET




Response











04Multi-language code generator


Mirrors the simulator settings above. Change them and the code regenerates.




query.py







05Case study: from query to FASTA


Retrieve human BRCA1 RefSeq mRNAs. Run each step in order; the WebEnv token from step 2 is reused in step 3.






Request




Output
Press “Run step”.











06Common pitfalls & troubleshooting


Tick items off as you apply them in your own scripts.


HTTP 429 “Too Many Requests”: use exponential backoff


Wait base × 2^attempt seconds (plus random jitter), capped. Adjust the sliders, or run a live demo of 15 simultaneous requests against a 3 req/s limit.










backoff.py


 Retry only on 429 / 5xx, never on 400
 Add jitter so parallel workers don't retry together
 Register an API key for 10 req/s




Nested XML vs. retmode="text" / rettype="fasta"


For sequences, ask for plain FASTA and parse with SeqIO: no XML tree walking. Use XML only when you need structured fields such as features or citations.




 Prefer rettype="fasta", retmode="text" for sequences
 Use Entrez.read() only for XML responses; handle.read() for text
 Remember efetch has no JSON for most databases; use esummary retmode=json




Memory: stream big downloads, batch with retstart/retmax






batch.py


 Write each batch to disk immediately; never hold millions of records in memory
 For complete genome assemblies use the NCBI Datasets / FTP downloads instead of efetch
 Keep WebEnv and query_key; they expire after roughly 8 hours of inactivity









Copied!