Simpler.Grants.gov Public Wiki
Grants.govSimpler.Grants.govGitHubDiscourse
  • 👋Welcome
  • GET INVOLVED
    • Why open source?
    • How to contribute code
    • How to file issues
      • Report a bug
      • Request a feature
      • Report a security vulnerability
    • Community guidelines
      • Code of Conduct
      • Reporting and removing content
      • Incident response protocol
    • Community events
      • Fall 2024 Coding Challenge
        • Event Submissions & Winners
      • Spring 2025 Collaborative Coding Challenge
        • Event Submissions & Winners
    • Communication channels
  • Product
    • Roadmap
    • Deliverables
      • 🏁Static site soft launch
      • 🏁Static site public launch
      • 🏁GET Opportunities
      • 🏁Open source onboarding
      • 🏁Co-Design Group planning
    • Decisions
      • ADR Template
      • ADRs
        • Dedicated Forum for Simpler.Grants.gov Community
        • Recording Architecture Decisions
        • Task Runner for the CI / CD Pipeline
        • API Language
        • Use Figma for design prototyping
        • ADR: Chat
        • DB Choices
        • API Framework and Libraries
        • Back-end Code Quality Tools
        • Front-end Language
        • Communications Tooling: Wiki Platform
        • Use Mural for design diagrams and whiteboarding
        • Ticket Tracking
        • Front-end Framework
        • Front-end Code Quality Tools
        • Front-end Testing & Coverage
        • Backend API Type
        • Front-end Testing & Coverage
        • Deployment Strategy
        • Use U.S. Web Design System for components and utility classes
        • FE server rendering
        • Use NPM over Yarn Architectural Decision Records
        • U.S. Web Design System in React
        • Communications Tooling: Video Conferencing
        • Back-end Production Server
        • Communications Tooling: Analytics Platform
        • Commit and Branch Conventions and Release Workflow
        • Cloud Platform to Host the Project
        • Infrastructure as Code Tool
        • Data Replication Strategy & Tool
        • HHS Communications Site
        • Communications Tooling: Email Marketing
        • Communications Tooling: Listserv
        • Use Ethnio for design research
        • Uptime Monitoring
        • Database Migrations
        • 30k ft deliverable reporting strategy
        • Public measurement dashboard architecture
        • Method and technology for "Contact Us" CTA
        • E2E / Integration Testing Framework
        • Logging and Monitoring Platform
        • Dashboard Data Storage
        • Dashboard Data Tool
        • Search Engine
        • Document Storage
        • Document Sharing
        • Internal Wiki ADR
        • Shared Team Calendar Platform
        • Cross-Program Team Health Survey Tool
        • Adding Slack Users to SimplerGrants Slack Workspace
        • Repo organization
        • Internal knowledge management
        • Migrate Existing API Consumers
      • Infra
        • Use markdown architectural decision records
        • CI/CD interface
        • Use custom implementation of GitHub OIDC
        • Manage ECR in prod account module
        • Separate terraform backend configs into separate config files
        • Database module design
        • Provision database users with serverless function
        • Database migration architecture
        • Consolidate infra config from tfvars files into config module
        • Environment use cases
        • Production networking long term state
    • Analytics
      • Open source community metrics
      • API metrics
  • DESIGN & RESEARCH
    • Brand guidelines
      • Logo
      • Colors
      • Grid and composition
      • Typography
      • Iconography
      • Photos and illustrations
    • Content guidelines
      • Voice and tone
    • User research
      • Grants.gov archetypes
  • REFERENCES
    • Glossary
  • How to edit the wiki
Powered by GitBook
On this page
  • Context and Problem Statement
  • Decision Drivers
  • Options Considered
  • Decision Outcome
  • Positive Consequences
  • Negative Consequences
  • Pros and Cons of the Options
  • PostgreSQL
  • OpenSearch

Was this helpful?

Edit on GitHub
  1. Product
  2. Decisions
  3. ADRs

Search Engine

PreviousDashboard Data ToolNextDocument Storage

Last updated 1 month ago

Was this helpful?

  • Status: Active

  • Last Modified: 2024-10-03

  • Related Issue:

  • Deciders: Michael, Kai, Aaron

  • Tags: search, api, backend, infra, opensearch, elastic, database

Context and Problem Statement

Simpler needs to surface Grant Opportunities to Grant Seekers to ensure that they're able to discover grants that could support their communities. This begins with a powerful and performant search page with strong filtering and sorting capabilities, but also includes the ability to do recommendation and other second-order matching/similarity functionality that goes beyond search. We want to select a search backend that will support rapid development, and long term features and success.

Decision Drivers

  • Better Search relevance for grant seakers

  • Ease of data ingestion/querying of data via the API

  • Operational ease/performance

  • Project mandate to lean in to Open Source

Options Considered

  • Roll our own search engine in

Decision Outcome

Chosen option: "OpenSearch" because:

  • Rolling our own is more complex than we would like, even relative to deploying Opensearch, so that eliminates PostgreSQL

  • Elastic had licensing issues/controversy that they've since walked back but has lost the trust of the Open Source community

  • OpenSearch largely tracks Elastic, but has a dedicated community and is direcly committed to avoiding situations like Elastic experienced.

Positive Consequences

  • Using a dedicated search product provides a ton of functionality and features out of the box

  • An isolated search engine allows us to scale/tune independently of the operational database.

  • The "search queries" have documented syntax, an existing testing/troubleshooting environment, and can share expertise across Nava teams.

Negative Consequences

  • Additional infra/spend compared to utilizing an existing product we're already hosting

Pros and Cons of the Options

PostgreSQL

Utilize some PostgreSQL supported SQL syntax to do searches as SQL SELECT queries

  • Pros

    • No additional infra/cost

    • Infra already exists

    • Works with existing data we're already tracking, no data sync needed

  • Cons

    • Not a core/frequently utilized portion of the product functionality

    • Still wasn't really "utilizing" the existing data/skillset as special indexes/syntax are needed

    • Creates issues scaling since it would require scaling the entire DB instance to account for search traffic

    • Harder to support new search features, onboard new developers, etc.

    • Slow queries

Elastic Search

Stand up Elastic Search, pump Opportunity data in, use search DSL to power search

  • Pros

    • Managed AWS service

    • Scales up/down independently of DB load

    • Purpose built for exact use case

    • Fast queries

    • Developed/Supported by a commercial entity

  • Cons

    • Additional infra/cost

    • Recent licensing/open source feud/issues hurt the product's standing in the developer/customer realms

    • Data from DB must be kept up-to-date in the search engine

OpenSearch

Stand up OpenSearch, pump Opportunity data in, use search DSL to power search

  • Pros

    • Managed AWS service

    • Scales up/down independently of DB load

    • Purpose built for exact use case

    • Fast queries

    • Actively developed/supported by the Open Source Community

    • Open Source/Licensing in good standing

  • Cons

    • Additional infra/cost

    • Data from DB must be kept up-to-date in the search engine

    • Features/improvements may lag Elastic Search a bit since it's not commercially funded

#2200
PostgreSQL
Elastic Search
OpenSearch