Use Ethnio for design research

• Status: Active

• Last Modified: Nov 14, 2023

• Related Issue: #117

• Deciders: Andy Cochran, [Designer/researcher]

• Tags: design, research, recruitment, incentives, scheduling

Context and Problem Statement

Simpler Grants would benefit from more robust tools for conducting user research (e.g. interviews and usability tests).

Decision Drivers

• Participant recruitment and management — The tool(s) should allow for robust management of research participants. We need the ability to track, sort, and filter participants and our interaction with them in a central database.

• Incentives disbursement — It's important ethically that we compensate research participants for their time. We need a tool that allows us to easily send electronic rewards (e.g. gift cards) to participants after research sessions. This tool should include tracking and reporting.

• Scheduling — Larger research efforts require a lot of calendar coordination to provide participants with options that also work for facilitators. The tool should be able to schedule 1:1s, group sessions, and multi-part sessions; account for time zones; allow syncing of facilitators' calendars; integrate with Zoom; and send emails to participants (bonus: SMS) to both schedule and remind them of their scheduled time.

• Screeners / Intercepts — Calls-to-action on the site should allow users to sign up as potential research participants or provide real-time feedback as they're using the site. Screeners allow us to analyze the pool of participants so that they can be funnelled to the right engagement opportunity (usability study, interview, focus group, etc) based on their answers to a set of survey questions that help identify the their archetype. Intercepts allow us to create controlled triggers that gather the right feedback in specific parts of a user's journey.

• Research data repository — The tool(s) might also include a place to store, manage, analyze, and share customer insights. However, of these decsion drivers, this may most easily be done through existing tools (see Use Mural for design diagrams and whiteboarding), and insights/findings should be stored in a long-term location kepr separate from PII.

Options Considered

tool/feature	Ethnio	Tremendous	Qualtrics	GreatQuestion	Touchpoints	Dovetail	Giftbit	Typeform	SurveyMonkey
User Mgmt	✅	❌	✅	✅	❌	❌	❌	❌	❌
Incentives	✅	✅	🛠️	🛠️	❌	❌	✅	❌	❌
Schedule/Cal	✅	❌	✅	✅	❌	❌	❌	❌	❌
Screeners	✅	❌	✅	✅	✅	❌	❌	✅	✅
Intercepts	✅	❌	✅	✅	✅	❌	❌	❌	❌
Data repo	🛠️	❌	✅	✅	✅	✅	❌	❌	❌
Nava exp.	✅	✅	❌	✅	✅	✅	❌	❌	❌

• ✅ Built-in

• 🛠️ Integration available

• ❌ Not included

Decision Outcome

Chosen option: Ethnio, because as a single tool it satisfies most decision drivers and has been validated by Nava experience.

Positive Consequences

Ethnio is a user research CRM for tracking participant activity. Nava has used it with much success on other projects. It addresses many needs in one tool with the following features:

• Participant pool — Manage participants with tags, notes, segments and filters. Keep a record of key interactions.

• Incentives — Participants receive compensation in their preferred method. Weekly or monthly reports on spending. Prepaid balance never expires.

• Screeners — Post or email screeners that recruit participants with qualifying questions and filtering logic and optional feedback surveys.

• Intercepts — Embed targeted intercepts in desktop and mobile viewports to recruit or survey site users in specific locations or devices.

• Scheduling — Invite, schedule, and remind participants. Combined team calendar with Outlook and Google integration.

There is potential (depending on procurement) for Ethnio to be used both by this Simpler.Grants.gov project and HHS's other grants improvement efforts, which would let them share a single participant pool within a centralized database where participants can be tagged/labeled and their activity can be tracked (incentives might also be funded in a single location).

Ethnio has the ability to create surveys of various types (open-ended feedback, targeted, screening potential participants), which could also satisfy the "Method & technology for collecting targeted feedback" ADR issue.

Other features of note:

• Internationalized and localized — Ethnio is fully i18n-friendly. Screeners, intercepts, and inentives/scheduling emails can all be given manual translations through advanced options in the editor (with the exception of right-to-left support).

• Section 508 and WCAG 2.0 (AA)

• GDPR- and CCPA-compliant data retention, expiration, and portability

Cost: $82+/seat/month (note: The Enterprise account is required for Participant Pool feature. It has an 8 seat minimum, which should more than suffice)

Negative Consequences

Ethnio does not include a research data repository. However, this is the one single feature need unmet, and shared text docs, internal wiki, and Mural will suffice for storing and analyzing insights.

Pros and Cons of other options

Tremendous

Tremendous is only for incentives disbursement. Nava has experience using this. Free to use, Tremendous lets the user select their preferred method: cash, prepaid card, gift card, or donation.

Cost: Free (Only pay for the incentives loaded into it.)

• Pros

  • Simple

  • Free

  • Simple reporting

  • WCAG 2.0

  • W-9 form collection

• Cons

  • Only addresses a single need. Requires we either procure other tools (e.g. for participant mgmt) or handle those needs through existing tools and manual processes (see caveat under Priority). However, it should be noted that Tremendous has integrations with other tools (e.g. Qualitrics, Great Question, etc).

Qualtrics

Qualtrics is used for feedback forms on HHS.gov. Includes a user research CRM (among other features). Integrates with Tremendous, Slack, Figma, calendars, Zoom.

Unknowns: Might it possible for both Simpler Grants teams (Nava and HHS staff) to access this tool? If seats are available, we could use it for participant recruiting & management, as a research data repository, and integrate with Tremendous for incentives.

Cost: Pricing not clear (enterprise only?)

• Pros

  • Already in use for www

  • FedRAMP authorization / NIST 800-53

• Cons

  • Not specifically for user research

  • Unintuitive UI complicated by unnecessary features (people mgmt, HR, product mgmt, etc)

  • Nava has no previous experience with it

GreatQuestion

GreatQuestion is the all-in-one solution. It has all the features we need. It even includes unmoderated usability testing (note: Nava would recommend maze.co if we were to pick a single-purpose tool for usability tests, and choosing other tools does not preclude us from conducting unmoderated tests in the future). Integrates with Tremendous and other tools. GDPR/HIPAA compliant.

Cost: $35/seat/month "Team" plan may suffice.

• Pros

  • Has everything

  • Intuitive interface

• Cons

  • Nava has little experience with it (a free trial for a single project test)

Other tools explored

Surveys tools (more appropriate for #306):

• Touchpoints — Just surveys, really. PRA-friendly, including forms that already have clearance. Free to use for federal projects. However, required .gov or .mil email address to admin.

• Typeform — Basic surveys. Custom branding. High price point.Integrates w/ Tremendous, Dovetail. Only feature it provides is surveys.

• SurveyMonkey — Integrates with Tremendous, Dovetail. Only feature it provides is surveys.

Data repo tools:

• Dovetail — Industry-standard research data repository. Lots of tool integrations. Hefty price tag. Likely not necessary for such a small team. Only feature it provides is a data repo.

Incentives:

• Giftbit — Similar functionality to Tremendous (which is preferred by Nava). Is recommended by 18F/TSS.

Unmoderated testing (not yet a requirement, can be considered later):

• maze — Industry-standard Figma testing. Nava's recommended platform for unmoderated usability testing (likely not needed in the near-term).