Skip to main content

Import Data into Accoil

Accoil’s data import system enables high-throughput bulk ingestion of historical user, group, and event data to power your analytics.

Simon Herd avatar
Written by Simon Herd
Updated over 2 weeks ago

Overview

Whether you’re onboarding a new customer, backfilling records, or importing from a legacy system, this feature provides a reliable and monitored pipeline with automatic retries and detailed status tracking.

Import Types & Required Fields

Accoil supports three distinct import types. Each file must contain only one type of data and must include the required fields for that type. Mixing types (e.g. Events and Groups) in a single file — whether CSV or ZST — is not supported.

1. Events Import

Used to import user interaction and activity data, such as page views, feature usage, or custom tracking events.

Required Fields:

  • type (string) — Event type (track, page_view, etc.)

  • name (string) — Human-readable event name

  • timestamp (string) — ISO 8601 format (e.g., 2025-07-29T10:30:00Z)

  • user_id (string) — Unique user identifier

Example CSV:

type,name,timestamp,user_id 
page_view,Reports Viewed,2025-07-29T10:30:00Z,user_123 
track,Report Downloaded,2025-07-29T11:45:00Z,user_456 
track,Feature Used,2025-07-29T14:20:00Z,user_789

2. Groups Import

Used to import account or organization-level data and traits.

Required Fields:

  • group_id (string) — Unique group/account identifier

  • user_id (string) — Associated user identifier

  • timestamp (string) — ISO 8601 formatted timestamp

  • traits (JSON string) — Group attributes in valid JSON format

Example CSV:

group_id,user_id,timestamp,traits 
company_123,user_123,2025-07-29T00:00:00Z,"{""name"": ""Acme Corp"", ""industry"": ""technology""}" 
company_456,user_456,2025-07-29T00:00:00Z,"{""name"": ""Beta Inc"", ""industry"": ""finance""}"

3. User Identifies Import

Used to update user workspace attributes like plan, role, or contact info.

Required Fields:

  • user_id (string) — Unique user identifier

  • timestamp (string) — ISO 8601 format

  • traits (JSON string) — Key-value attributes of the user

Example CSV:

user_id,timestamp,traits
user_123,2025-07-29T10:00:00Z,"{""role"": ""admin"", ""plan"": ""premium"", ""email"": ""john@acme.com"", ""name"": ""John Doe""}"
user_456,2025-07-29T11:00:00Z,"{""role"": ""user"", ""plan"": ""basic"", ""email"": ""jane@beta.com"", ""name"": ""Jane Smith""}"
user_789,2025-07-29T12:00:00Z,"{""role"": ""manager"", ""plan"": ""pro"", ""email"": ""bob@gamma.com"", ""name"": ""Bob Johnson""}"

Supported File Formats

Accepted File Types

Accoil supports two file types for data import:

  • CSV (.csv)
    A plain text file with comma-separated values, encoded in UTF-8. Each row represents one record, and each file must correspond to a single import type (e.g., Events, Groups, or Identifies).
    ​

  • Zstandard-compressed CSV (.zst)
    A .zst file is a compressed version of a CSV file using the Zstandard algorithm. This format is ideal for uploading large datasets quickly and efficiently.

⚠️ Important Notes:

  • A .zst file must contain only one underlying CSV file, and that file must adhere to the correct schema for a single import type.

  • Do not combine multiple import types (e.g., Events and Groups) into a single file. Each import type must be uploaded as its own separate file, even if compressed.

  • The CSV inside the .zst must still meet all header, formatting, and field requirements described above.

  • Extra columns in the CSV beyond these will be ignored.

✅ Why use ZST?
Zstandard is a modern compression algorithm that dramatically reduces file size without compromising speed. It’s ideal for uploading millions of rows efficiently, especially under the 100MB file limit.

Usage on mac:

brew install zstd       # To install the necessary libraries
zstd yourfile.csv       # To compress a file to zstd
zstd -d yourfile.zst    # To uncompress a zstd file

Usage on windows

Download from https://github.com/facebook/zstd/releases
zstd yourfile.csv       # To compress a file to zstd
zstd -d yourfile.zst    # To uncompress a zstd file

File Size & Encoding

  • Maximum File Size: 100MB (applies to both .csv and .zst)

  • Required Encoding: UTF-8

  • Processing: Files are streamed during processing, so large datasets don’t overwhelm memory.

Certainly! Here's the Import Types & Required Fields section, cleanly formatted and ready for your documentation:

Processing Flow

  1. Upload: Your .csv or .zst file is stored and queued for processing.

  2. Validation: Headers and data rows are checked for format and completeness.

  3. Queueing: Each row is pushed to a background queue for delivery to the analytics engine.

  4. Delivery: Rows are sent to Accoil’s ingestion APIs asynchronously.

  5. Completion or Failure: Import completes when all rows succeed—or is marked as failed based on retry rules.

Retry Logic & Failure Conditions

Accoil ensures reliable delivery with smart retries and robust failure detection.

1. Immediate Failures (No Retry)

Some errors fail fast:

  • Invalid Headers: CSV headers must exactly match the required format (case-sensitive). Any mismatch results in immediate failure.

  • File Parsing Errors: Corrupt files or unreadable content (e.g., invalid encoding) also fail without retry.

2. Row-Level Retries (With Limits)

Each row is sent individually to Accoil’s analytics engine. If a row fails due to transient issues, Accoil will retry it up to 5 times.

Common retryable scenarios include:

  • API failures (timeouts, 5xx responses)

  • Network issues

  • Temporary rate limits

Retry Rules:

  • MAX_RETRIES: 5 retries per failed row

  • MAX_PROCESSING_MINUTES: Rows stuck in “processing” for over 5 minutes are retried

  • Exponential Backoff: Delays between retries increase over time

3. Timeouts & Stuck Imports

Accoil monitors progress at the import level to catch systemic issues.

  • No Progress for 30 Minutes: If no rows are updated in 30 minutes, the import is marked failed

📌 Failure reason: “Import stuck and needs investigation”

4. Final Failure Conditions

An import is marked as failed if:

  • There are failed rows,

  • All failed rows have exhausted their 5 retry attempts,

  • No rows remain in pending or processing states.

📌 Failure reason: “Has failures but used maximum retries on all rows”

5. Skipped Rows

Rows are skipped (not retried or failed) if:

  • Their timestamp is older than 90 days

  • The row is completely empty

Skipped rows are counted and reported in import statistics but don’t impact the import’s success/failure status.

Success Conditions

An import is considered complete when:

  • All rows are in a completed state

  • No pending, processing, or failed rows remain

Monitoring & Visibility

In the Accoil UI, you’ll see real-time status updates:

  • Import Statuses:

    • pending: Awaiting processing

    • processing: Actively being parsed/sent

    • completed: All rows succeeded

    • failed: Import stopped due to retries or timeouts

  • Row Metrics:

    • Total rows processed

    • Rows successfully ingested

    • Failed rows (after retries)

    • Skipped rows (e.g., old timestamps)

💡 Tip: Failure reasons are shown directly in the UI for rapid diagnosis.

Best Practices for Data Imports

Before You Upload

✅ Validate that your headers match exactly (case-sensitive)
✅ Make sure timestamps are in ISO 8601 format (e.g., 2025-07-29T10:30:00Z)
✅ Confirm UTF-8 encoding
✅ Use .zst compression for large files to save time
✅ Validate JSON inside traits fields using a linter or test parser

During & After Upload

  • Monitor status in the import dashboard

  • Check for skipped rows due to timestamp cutoffs

  • Review failure reasons if an import fails

  • Retry with corrected data if needed

Did this answer your question?