6  Import Data Live using qualtrRics

How to automate data export and import

Abstract

This chapter demonstrates how to programmatically download survey data from Qualtrics using the Qualtrics API. Researchers learn to authenticate with Qualtrics, initiate data exports, and download responses directly into R for real-time analysis and reporting. This approach enables automated data updates and live dashboards that reflect the most current survey responses without manual downloads. Using this approach, a research team should be able to run analyses during the last week before the poster is submitted to maximize the data collection window.

Keywords

import, qualtrics, live

Tip📖 qualtrRics resources

• Documentation
• Qualtrics website

ImportantAdvanced Module

This is an advanced chapter and not required for your report or manuscript!

6.1 Benefits of qualtRics Continuous Importing of Survey Data

By importing data continously using the qualtRics R package, you do not need to export from qualtrics and upload/import a data file. You can generate with up-to-date data instaneously for a:
- Quarto report - Plots in a shiny app - Quarto presentation, such as a live demonstration with survey data collected moments ago or to show what your code can do

6.2 Getting necessary qualtrics information

Qualtrics Info

When you log into Qualtrics, you must retrieve specific data from your account as well as live data from the survey. Here, we’ve clicked our user icon in the top right and gone to “Account Settings.” You’re going to need these strings later.

Next, you’re going to need your survey ID. When you click on the survey that contains the live data, the URL will look something like this: https://binghamton.yul1.qualtrics.com/survey-builder/SV_6mOC1SaJbecAuP0/edit The survey ID is the string between survey-builder/ and /edit, in this case, being “SV_6mOC1SaJbecAuP0”

Getting into the actual code, we have the following steps: ## Import necessary packages:

library(httr)
library(jsonlite)
library(readr)

6.3 Save important info to variables:

api_token <- "1V0X29BHlwPfSYPsUMQ5d8tQfeydhsVBcGQ2v9Bw"
data_center <- "yul1"
survey_id <- "SV_6mOC1SaJbecAuP0"

base_url <- paste0("https://", data_center, ".qualtrics.com/API/v3")
headers <- add_headers(
  "X-API-TOKEN" = api_token,
  "Content-Type" = "application/json"
)

In this chunk, after saving our API key and survey ID info, we construct the URL that we are going to scrape using the paste0 function (which just combines the arguments into a single string with no spaces between them). We add these specific headers because they are necessary arguments for the API call.

6.4 Start the export job

start_response_export <- function(survey_id) {
  url <- paste0(base_url, "/surveys/", survey_id, "/export-responses")
  body <- list(format = "csv")  # Use CSV format here
  res <- POST(url, headers, body = toJSON(body, auto_unbox = TRUE))
  stop_for_status(res)
  content(res, as = "parsed", simplifyVector = TRUE)
}

This chunk creates the API endpoint and allows us to begin exporting. It also gives us a progress report on how close the data is to being exported!

6.5 Poll for export completion

check_export_progress <- function(progress_id) {
  url <- paste0(base_url, "/surveys/", survey_id, "/export-responses/", progress_id)
  repeat {
    res <- GET(url, headers)
    stop_for_status(res)
    progress <- content(res, as = "parsed", simplifyVector = TRUE)
    if (progress$result$status == "complete") {
      return(progress$result$fileId)
    } else if (progress$result$status == "failed") {
      stop("Export failed")
    }
    Sys.sleep(2)
  }
}

Once the export is ready, this chunk returns the ID for the file that we can now download from the API.

6.6 Download and read CSV file

daily_download_responses_csv <- function(file_id) {
  url <- paste0(base_url, "/surveys/", survey_id, "/export-responses/", file_id, "/file")
  res <- GET(url, headers)
  stop_for_status(res)
  tmp <- tempfile(fileext = ".zip")
  writeBin(content(res, "raw"), tmp)
  unzip_dir <- tempdir()
  unzip(tmp, exdir = unzip_dir)
  csv_file <- list.files(unzip_dir, pattern = "*.csv", full.names = TRUE)
  read_csv(csv_file)
}

This chunk unpacks the output into a .csv file which makes it ready for analysis!

6.7 Main execution

export_response <- start_response_export(survey_id)
progress_id <- export_response$result$progressId
file_id <- check_export_progress(progress_id)
responses_df <- daily_download_responses_csv(file_id)

Finally, we set up our dataframes. These dataframes are now in your environment and can update charts or tables every time you reload the export!