[6.x] Feature/import

[i-just]

It imports data from JSON, CSV or XML files into Craft CMS elements or Eloquent models.

Core concepts:

• Data Types => the data sources this feature can work with
• Transformers => normalise and manipulate incoming data, and sometimes map it to existing “slots”
• Importers => do the heavy lifting. Each importer knows how to write data into one specific target (e.g. Entries, Assets, Users, System Messages)
• Import Plans => a named, ordered list of steps. Each step pairs an importer with its own file, transformer, batch size, settings and mapping

Extensibility:

• you can register extra data types and importer types
• you can manipulate your data any way you want via transformers
• you can add a CLI command for your own importer by extending CraftCms\Cms\Import\Commands\Import

Data Types:

• there are 3 built-in data types: JSON, XML and CSV
• extra types can be registered via the RegisterDataTypes event

Transformers:

• there are 3 built-in transformers, all related to element importing:
  - ElementTransformer => used by default by all elements
  - EntryTransformer => default when importing Entries
  - AssetTransformer => default when importing Assets
• ElementTransformer auto-matches incoming keys to attributes and field handles even when the names aren’t exact (e.g. Plain Text / plain_text → plainText)
• you can create your own transformers and use them to manipulate the data however you wish. For example, you can uppercase every value imported into myPlainTextField, or run complex checks to decide what
  value myOtherField gets

Importers:

• each importable target has its own concrete importer class. Being importable means having an importer registered:
  - ElementImporter (abstract) => EntryImporter, AssetImporter, UserImporter. Each implements targetClass(). Common settings: site (required) and fieldLayout
  - EntryImporter adds section and entryType
  - AssetImporter adds volume
  - ModelImporter (abstract) => SystemMessageImporter is the only built-in.
• Address has no importer, so it’s never a standalone target. It can only be imported as nested content (Addresses field / User addresses)
• each importer defines its own settings form, validation rules and batch size (default 5, 0 disables batching)
• extra importer types (e.g. for plugin element types or models) can be registered via the RegisterImporterTypes event

Import plans:

• there are 2 kinds of import plans:
  - editable => created via the Control Panel and stored in the database (import_plans)
  - non-editable => file-based, stored in config/craft/import.php and committed to your repository. Invalid file-based plans are skipped and logged
• for each step you:
  - choose the importer and fill in its settings. For example, for an Entry importer, you pick the site, section and entry type
  - specify the file that holds the data you want to import
  - optionally choose a transformer and a batch size
  - define which importable attribute and field maps to which incoming value (map)
  - choose which values should be used to match against existing elements (matchCriteria)
  - choose which values should be cleared out if they’re empty or missing from the incoming data (clearableItems)
  - choose whether you’d like to keep nested content that’s missing from the incoming data (keepMissingNestedElements)
• steps run sequentially, and each step runs as its own batch, so one bad step doesn’t kill the whole plan. Per-item failures are logged and skipped, not fatal
• editable plans can be duplicated

Mapping UI

• a mapping table with Destination / Incoming data / Match / Clear columns. Container fields (Matrix, Addresses, Content Block) open a nested slideout, which can nest to any depth
• the incoming-data select is a combobox that shows the value from the file’s first row next to each option
• the most likely incoming column is pre-selected for each destination
• attributes marked canBeSet: false (e.g. id, uid) can only be used for matching, not set

Match criteria — three tiers, last wins

1. Step-level — from the UI or file config. Resolved once per step.
2. In the data — a matchCriteria key on the row, [systemHandle => incomingKey].
   Also looks inside $data['fields'].
3. Transformer — additionalMatchCriteria(), already resolved to actual values.

Clearable items

An array shaped like the match criteria. If a field or attribute is

• marked clearable and its value is missing or empty in the incoming data, it’s forced to null and explicitly applied
• not marked clearable and its value is empty in the incoming data, its key is unset() entirely, so the existing value is left untouched.

Importing:

• you can trigger an import via the UI: CP > Import > Run, for editable and file-based plans alike
• or via the CLI, with one command per importer:
  - artisan craft:import:entry {file} [--site=] [--section=] [--entryType=] [--transformer=] [--matchCriteria=]
  - artisan craft:import:asset {file} [--site=] [--volume=] …
  - artisan craft:import:user {file} …
  - artisan craft:import:system-message {file} …
  - any options you leave out are prompted for

Making things importable:

• Elements & models: register a concrete ElementImporter / ModelImporter subclass via RegisterImporterTypes
• CraftCms\Cms\Support\Attributes\Importable marks an element’s properties as importable. Its flags are excludeFromUiMapping, isContainer, canBeMatchCriteria, canBeCleared and canBeSet
• CraftCms\Cms\FieldLayout\Contracts\ImportableFieldLayoutElementInterface makes field layout elements importable
• CraftCms\Cms\Field\Contracts\ImportableElementContainerFieldInterface lets container fields be imported correctly

Events:

RegisterDataTypes, RegisterImporterTypes, DataImporting / DataImported, ImportPlanSaving / ImportPlanSaved, ImportDispatching / ImportDispatched. The *ing events are cancellable.

Permissions:

There’s a permissions group import with the following permissions: viewImportPlans, saveImportPlans, deleteImportPlans, triggerImportPlans.

Related issues

CMS-2390