Skip to main content

Warnings, Limitations, and Regulatory Considerations

It is crucial to understand the limitations and potential risks associated with using Phlox. Phlox is a documentation tool, not a certified medical device: all AI-generated content (notes, letters, extracted data) is a draft and must be reviewed by the treating clinician before use.

Usage Warnings and Disclaimers​

  1. Actively developed open source: Phlox is open-source software under active development and may contain bugs, inconsistencies, and security vulnerabilities. It has not been evaluated by any regulatory authority as a medical device, and has not undergone the formal testing and quality assurance expected of medical devices.

  2. AI Hallucinations and Inaccuracies: Phlox relies on Large Language Models (LLMs) for various functions. LLMs are known to hallucinate, meaning they can generate outputs that are plausible but factually incorrect or nonsensical. This is especially true for smaller, locally-run models.

    • Critical Verification: ALL AI-generated outputs from Phlox (clinical notes, summaries, correspondence, reference searches, chart insights, etc.) MUST be independently verified by qualified medical professionals using trusted, primary sources. Do not rely solely on AI-generated information for clinical documentation.
    • Risk of Misinformation: AI-generated content may contain inaccurate medical information or misinterpret clinical context. Using unverified AI output in clinical practice could lead to patient harm.
    • Speaker labels are best-effort: the live agent's speaker tags (S1–S4) are heuristic hints, not authoritative attribution — they can mislabel or miss speakers, especially with overlapping speech or similar voices.
    • Grounding is a review aid, not a guarantee: grounded notes check note lines against the transcript, but the check itself is model-based and can miss contradictions or lend false confidence — a supported verdict does not certify clinical accuracy. Independent verification remains mandatory.
  3. Not a Certified Medical Device: Phlox is not a certified medical device and has not been evaluated or approved by any regulatory bodies (e.g. FDA, TGA, MHRA). It is a documentation tool: it drafts content for clinician review and is not designed to make clinical decisions.

  4. Regulatory Compliance Is Deployment-Specific (HIPAA, GDPR, TGA, etc.): Regulatory compliance is a property of your deployment, not of this software. Phlox keeps all data on hardware you control — encrypted at rest, with no telemetry or third-party services by default — but you are responsible for ensuring your setup meets the obligations that apply to your practice.

    • Security & Privacy: Phlox is a local-first application with basic multi-user support — it is not a hardened clinical system. While it does include security controls — per-user accounts with admin/clinician roles, encryption at rest (SQLCipher), an audit log of API activity, rate limiting, security headers, and optional proxy authentication (see Security) — these are not a substitute for the comprehensive administrative, physical, and technical safeguards required for regulatory compliance.
    • Audit logging is basic: the audit log captures request metadata (method, path, status, actor, timing) but not a full compliance-grade audit trail with immutable retention or tamper-evidence.
    • Database Encryption: database encryption protects data at rest but is not, by itself, a comprehensive security measure.
    • Transcription Data: consider the privacy implications of your chosen Whisper transcription service, especially if using cloud-based APIs.
  5. Authentication and Access Control: Phlox has built-in username/password accounts with admin and clinician roles, and login is required for Docker deployments (see Users & Authentication). This is basic access control, not compliance-grade identity management:

    • Docker/web: first-run setup creates the admin account; further users are added by admins. There is no SSO/OIDC, MFA, or account-recovery flow.
    • Desktop (Tauri): a single implicit admin protected by a random local bearer token, plus a database passphrase.

    Exposing Phlox to the public internet without a reverse proxy or VPN is highly discouraged and poses a significant security risk. On desktop, the database passphrase must be re-entered on every launch — Phlox deliberately does not cache it in the OS keychain, so that physical access to the machine does not grant access to the database.

  6. MCP Server and External Tool Security: When connecting external MCP servers or enabling built-in tools that call external APIs (PubMed, Wikipedia):

    • Patient data may be transmitted to third-party services depending on configuration.
    • The per-server "Allow sensitive data" toggle (off by default) strips patient-identifying information from MCP arguments, but is best-effort and not guaranteed to catch all PHI.
    • External services may log, cache, or retain transmitted data.
    • You are responsible for ensuring that any external service complies with applicable privacy regulations.
  7. Intended Use: Phlox is intended for:

    • Clinical documentation support: drafting notes, letters, and summaries for review and sign-off by the treating clinician.
    • Educational purposes: to explore the potential of AI in administrative workflows and learn about LLMs and related technologies.
    • Development and Research: as a platform for further development, research, and contribution to open-source medical AI tools.

By using Phlox, you acknowledge and accept these warnings and limitations. You assume full responsibility for any use of Phlox and agree to use it ethically, responsibly, and in compliance with all applicable laws and regulations.

If you are unsure about any aspect of these warnings or the regulatory implications of using Phlox, obtain professional advice and implement all necessary safeguards before using it in a clinical setting.