bdpe_fetch_chunks {BigDataPE}R Documentation

Fetch data from the BigDataPE API in chunks

Description

This function retrieves data from the BigDataPE API iteratively in chunks. It uses bdpe_fetch_data as the base function and supports limits for the total number of records to fetch and the size of each chunk.

Usage

bdpe_fetch_chunks(
  base_name,
  total_limit = Inf,
  chunk_size = 500000L,
  query = list(),
  verbosity = 0L,
  endpoint = "https://www.bigdata.pe.gov.br/api/buscar"
)

Arguments

base_name

A string specifying the name of the dataset associated with the token.

total_limit

An integer specifying the maximum number of records to fetch. Default is Inf (all available data).

chunk_size

An integer specifying the number of records to fetch per chunk. Default is 50000

query

A named list of additional query parameters to filter the API results. Default is an empty list.

verbosity

An integer specifying the verbosity level for the API requests. Values are: - 0: No verbosity (default). - 1: Minimal verbosity, showing request status. - 2: Detailed verbosity, including request and response details.

endpoint

A string specifying the API endpoint URL. Default is "https://www.bigdata.pe.gov.br/api/buscar".

Value

A tibble containing all the data retrieved from the API.

Examples

## Not run: 
# Store a token for the dataset
bdpe_store_token("dengue_dataset", "token")

# Fetch up to 500 records in chunks of 100
data <- bdpe_fetch_chunks("dengue_dataset", total_limit = 500, chunk_size = 100)

# Fetch all available data in chunks of 200
data <- bdpe_fetch_chunks("dengue_dataset", chunk_size = 200)

## End(Not run)

[Package BigDataPE version 0.0.96 Index]