Data Import

Screenshots on this page are taken from the Chinese UI. Menu, field and button names in the text use the English UI labels.

Target Audience

The target audience of this document may include:

• Developers and implementers of this system
• User data administrators who are preparing to import data

Feature Description

This system follows the principle that everything is data: metadata such as domain models, forms, menus, permissions and dynamic logic can be imported from CSV files just like business data, and an import can create, update and delete records.

TIP

The CSV parsing format of this system is based on the RFC 4180 (opens new window) standard, and additionally uses the backslash (\) as the escape character; see Escaping Special Characters.

Seed data generally refers to a set of initial data inserted into the database when an application is initialized or deployed. The design principle of the Muyan low-code platform is that everything, including customization code, is data, and an entire business system can be built from seed data.

The platform currently has no data export feature; neither the UI nor the API can export data as CSV.

If you prefer learning by example, go directly to A Complete Example. For the CSV columns of each type of platform object, see CSV Import Templates.

Import Methods

Three import methods are currently supported:

Method	Use case
Automatic Import at System Startup	Development environments and plugin development, with seed data maintained together with code
Plugin Data Import	Publishing and installing a set of data as a plugin package
Import Data from a List Page	Importing data into a model in batch from the UI

Automatic Import at System Startup

Each time the backend starts, it imports in the following order:

1. The platform's own seed data (/app/platform_data in the image). It is imported again only when the platform version (AppVersion) changes; it is imported on every startup in the development environment (GRAILS_ENV=development), when the environment variable FORCE_RELOAD_SEED_DATA=true is set, or when the system configuration debugDynamicLogic.forceRefreshDomainDefinition is true.
2. The application's seed data directory (see Seed Data Directory): first the plugin packages in the plugins subdirectory, then the CSV files in the csv subdirectory.

Before each CSV file is imported, file change detection is performed and files whose content has not changed are skipped, so restarting the backend service is enough to apply changes to CSV files.

Plugin Data Import

A plugin package is a zip file whose root directory contains:

Path	Description
PLUGIN_INFO	Plugin information in JSON, including name, version, description and dependsOnPlugins (names and minimum versions of the plugins it depends on)
csv/	The plugin's seed data CSV files
libs/	The plugin's jar files
sql/	Optional; SQL executed before and after import, see Run SQL Before and After Import

A plugin can be imported in two ways:

• Put it in the plugins subdirectory of the seed data directory; it is imported automatically at system startup.
• Upload it with the Import dynamic plugin action on the Development > Plugins list (requires the DEVELOPER role).

The parameter window of the Import dynamic plugin action has three items:

Parameter	Description
Plugin file	The plugin zip package to import
Ignore MD5 Check	When turned on, file change detection is skipped and every CSV file in the plugin is imported again. It only takes effect when the plugin package is actually imported: if a plugin with the same name already exists in the system, the new plugin version is not higher, the file name does not contain SNAPSHOT, and Overwrite Conflict is not turned on, the whole plugin package is ignored and turning this on does not import any CSV either (see the import rules below)
Overwrite Conflict	When turned on, this plugin package overwrites the plugin with the same name in the system regardless of version, and the plugin's CSV files skip conflict detection during import and overwrite the values in the database directly (see Conflict Detection and Overwrite)

Import rules:

• Multiple plugins are sorted by the dependencies declared in PLUGIN_INFO, and plugins that others depend on are imported first. The import fails if a dependency does not exist, its version does not satisfy the requirement, or there is a circular dependency.
• If a plugin with the same name already exists in the system, it is overwritten only when the new plugin's version is higher, the plugin file name contains SNAPSHOT, or Overwrite Conflict is turned on; otherwise the plugin package is ignored.
• The CSV files in a plugin are also imported in the import order, except that HierarchyRole, DynamicLogicEngine, DynamicLogicType, DynamicLogic, RoleRequirement, DomainClass, DomainClassField and ExtendedDomainClass are moved to the front.
• A plugin whose file name contains SNAPSHOT behaves as if Overwrite Conflict were turned on: it forcibly overwrites the plugin with the same name, and conflicts are forcibly overwritten when its CSV files are imported.

For how to develop and package plugins, see Plugin Development.

Import Data from a List Page

The platform has a built-in object action Import domain data from csv that imports CSV files into the model of a list form. Its label is 导入数据, which has no English translation, so the English UI also shows it in Chinese. By default this action is not bound to any form, so no import button is visible in the default UI; you need to bind it to the target list form first:

1. Open Development > Actions > DynamicActionDynamicForm and click Create.
2. For action, select 导入数据; for form, select the target list form; fill in displaySequence and save.

The following example uses the platform's built-in user groups and binds 导入数据 to the user group list form List of groups:

WARNING

In 1.0.0-beta18, the dateCreated and lastUpdated fields of this create window are marked as required. You need to pick any time (for example click Now) to save; the system overwrites them with the actual time when saving. You can also bind it with CSV; see the DynamicActionDynamicForm section in CSV Import Templates.

After binding, a 导入数据 button appears above the Business Config > Group list. Prepare a CSV file, for example Group.csv:

name(*),roles.name[:]
市场部,[ROLE_USER]
客服部,[ROLE_USER]

Click 导入数据, select one or more CSV files in the upload box, choose whether to turn on Overwrite Conflict, and click Submit. The upload box is titled Plugin files, which is a wording issue in the current version; what you select here are CSV files:

After the import finishes, refresh the list and you can see the two new user groups:

• The target model is determined by the Object type of the list form, not by the file name.
• The action runs asynchronously. When it finishes, the system sends a message notification, and the import result can be viewed in Exec History > Import.
• This method does not perform file change detection; every submission is imported.
• When Overwrite Conflict is turned on, conflicting lines are also overwritten by the CSV.

Known issues in 1.0.0-beta18

• When importing data into a dynamic model (a model of type DYNAMIC in Development > Classes, such as the customers and sales opportunities in the step-by-step tutorial), every line currently fails with the reason ScopedValue not bound. Both automatic import at system startup and the 导入数据 action are affected; the platform's built-in models (such as users, user groups, forms and menus) are not, which is why the example above uses user groups.
• Even if all lines fail, the 导入数据 action still reports "Execute succeed"; rely on the status in Exec History > Import. In the result window after submission, the HTML "Execution record: <a href='/DynamicActionExecRecord/…'…" is displayed as raw text instead of being rendered as a link; the execution record can be opened from the "Execute log" link in the system message received afterwards.
• In the UI, only the DEVELOPER role and above can use 导入数据: the action's parameter form Import domain data from csv RequestForm requires the DEVELOPER role, so ordinary users cannot open the parameter form after clicking the button (the API returns 403), and Exec History > Import, where the import results are shown, also requires DEVELOPER. The action definition itself only requires the USER role, which matters only when the action API is called directly. Either way, the import does not check the create, update or delete permissions of the target model.

Seed Data Directory

The root directory of the application's seed data is configured with seedData.folder in application.yml. By default it reads the environment variable SEED_DATA_FOLDER, and falls back to /app/data when that is not set:

seedData:
  folder: ${SEED_DATA_FOLDER:/app/data}

In the docker compose development project provided with the platform, for example, SEED_DATA_FOLDER is set to /app/plugin/data, and the subdirectories under codes/data in the project are mounted under that directory.

The directory structure is as follows:

<seedData.folder>
├── Tenant.csv            --> Optional, list of tenants created when the system starts for the first time
├── csv                   --> Seed data CSV files, see "File Name" for naming rules
│   ├── DomainClass.csv
│   ├── DomainClassField.csv
│   └── ...
├── plugins               --> Optional, plugin packages imported automatically at startup
├── sql                   --> Optional
│   ├── before_import.sql --> Executed before importing CSV files
│   └── after_import.sql  --> Executed after importing CSV files
├── groovy                --> Usually holds dynamic logic source code, referenced by the code(F) column in CSV
├── css                   --> Usually holds CSS for display themes, referenced by the css(F) column of DynamicTheme
└── attachments           --> Usually holds attachments to import, such as the logo used by a display theme

groovy, css and attachments are only conventional directory names. CSV files reference them by file path, and paths are relative to the seed data root directory.

TIP

The directory is no longer divided into subdirectories by runtime environment or tenant. Tenant.csv has only one column, name(*). It is used only when no tenant is specified with the system property gorm.tenantId (set by the environment variable TENANT_ID in the docker image), and creates the tenants that do not yet exist in the database.

Import Order

The CSV files in a directory are imported in the following order:

1. Files with a prefix, such as 000-DynamicLogicEngine.csv and 006-DomainClass.csv, are sorted by file name and imported first. The rule is: the part of the file name before the first _ contains exactly one -; the part before - is the prefix and the part after it is the model name. The prefix is usually a number to make sorting easy.
2. Models with a predefined order: DisplayComponentModule, DynamicLogicEngine, DynamicLogicType, DynamicConfig, DynamicLogic, HierarchyRole, RoleRequirement, DynamicObjectHook, DomainClass, DomainClassField, DynamicFormType, DynamicForm, DynamicMenu, DynamicActionGroup, DynamicAction, imported in this order.
3. Other files: sorted automatically by the relationships between models, with models that others depend on imported first. For example, when Group.csv, User.csv and UserGroup.csv all exist, UserGroup.csv is imported after the other two.

If the automatically detected dependencies are not enough, you can specify them with loadAfter in the model definition:

• Dynamic models: set loadAfter in the domain model's extended information (extInfo); see Dynamic Domain Model for details.
• The platform's built-in models (GORM models): define a static property loadAfter in the class, for example DynamicAction:

  // Means DynamicLogic and DynamicActionGroup must be imported before DynamicAction
  static loadAfter = [DynamicLogic, DynamicActionGroup]

If a type of object does not depend on any other object and should be imported as early as possible, set loadAfter to an empty array, for example I18nType:

  static loadAfter = []

File Change Detection

Before importing a CSV file, the system calculates the md5 of the file content and looks for the most recent finished import record of this model with the same md5. If that record's status is "Success" or "No data imported", the file is skipped.

TIP

• The check is based on file content, not on file name or modification time; even adding a comment line makes the file be imported again.
• The system only looks at the most recently finished record among those with the same md5: if it is "Success" or "No data imported", the file is skipped; if it failed or partially failed, the file is imported again. So when you change a file back to an earlier version that was imported successfully, it is not imported again; in that case, add a comment line to change the file content.
• A file whose last import failed or partially failed is imported again in full on the next startup.

Run SQL Before and After Import

If sql/before_import.sql exists under the seed data root directory (or the plugin package root directory for a plugin), the system executes it before importing the CSV files; if sql/after_import.sql exists, it is executed after importing the CSV files.

• ${TENANT} in the SQL is replaced with the current tenant identifier.
• It is executed once for each tenant.
• An SQL execution failure is only logged as an error and does not interrupt the import.

CSV File Format

The following sections describe how to prepare data in CSV files.

File Name

The file name determines the target model: take the part of the file name before the first _, then remove - and the numeric prefix before it, to get the model's short name (without the package name). For example, the following files are all imported into DynamicForm:

• DynamicForm.csv
• DynamicForm_crm.csv
• 011-DynamicForm_action.csv

The .csv extension must be lowercase. Dynamic models use the domain model's short name, for example Customers_2026.csv.

Header Row

In every CSV file to be imported, the first line must be the header row, which describes the structure of the data in the CSV file. The following is an example of a header row:

username(*),password,name,accountLocked,DELETE_FLAG

• The name of an ordinary column is a field name of the model.
• Columns with the suffix (*), such as username(*), are lookup fields; see Lookup Fields.
• Columns containing ., such as group.name, are associated object fields; see Associated Object Queries.
• Columns with the suffix (F), such as code(F), mean the value of the column is a file path, and the file content is read as the field value during import. Paths are relative to the seed data root directory (or the plugin package root directory for a plugin).
• A column named DELETE_FLAG is used to delete data; see Deleting Objects.
• A column named OVERWRITE_FLAG is used to force an overwrite; see Conflict Detection and Overwrite.
• If the type of a field is attachment (StorageFieldValue), the value of the column is the file path of the attachment to import.

WARNING

If the header row contains a column name that does not exist in the model, the whole file fails to import; for automatic import at system startup, the files after it in the same directory and after_import.sql are not executed either. The dynamic field columns (ending with (#)) from before 1.0 have been removed since 1.0, and they also cause the whole file to fail.

Comment Lines

Lines starting with a semicolon (;) are comment lines and are ignored during import:

name(*),implies.name[:]
;; Sales role, includes the regular user role
ROLE_SALES,[ROLE_USER]

Lookup Fields

To support creating or updating existing records through CSV files, all columns with the suffix (*) in the header row are recognized as lookup fields. The system uses the values of all lookup fields to query existing records with strict equality:

• No record found: a new record is created, and the line is recorded as inserted in the import record.
• Exactly one record found: the record is updated with the values from the CSV, and the line is recorded as updated; if the values have not changed, it is recorded as not updated.
• More than one record found: the line is skipped and recorded as skipped in the import record.

TIP

If the CSV file being imported has no lookup field at all, update and delete are not available, and all records are treated as new.

Associated Object Queries

group.name, for example, means that when importing the object's group field (a Group object), the value in this CSV column is matched against the name field of Group, and the object found is associated:

• No associated object found: the line fails to import.
• Exactly one found: the import of the line continues.
• More than one found: the line fails to import.

An association field can also be a lookup field, such as user.username(*).

For one-to-many and many-to-many collection fields, write the value as [value1,value2] and the column name as roles.name[] or roles.name ([] can be omitted; values are separated by commas by default, and if a value contains a comma the whole cell must be wrapped in double quotes). You can also specify the separator in the brackets; for example, roles.name[:] means values are separated by : and written as [ROLE_A:ROLE_B], so no quotes are needed.

If the column name contains only the field name and no lookup field:

• Single associated object (for example customer): the system uses queryField in the extended information of the associated model as the lookup field; if the associated model has no queryField configured either, the value of the column cannot be resolved and the line fails to import.
• Collection field (for example roles): associated objects are looked up by their name field by default.

WARNING

When importing an object, the associated objects it depends on must already exist in the system; otherwise the line fails to import.

Null Values

An empty cell, or the literal NULL, is imported as a null value.

Escaping Special Characters

• If the content of a column contains a comma (,) or a line break, the column must be wrapped in double quotes (").
• Inside content wrapped in double quotes, a double quote written twice ("") represents one double quote; it can also be escaped with a backslash (\"). For example, the JSON value {"labelField": "name"} is written in CSV as "{""labelField"": ""name""}".
• The backslash (\) is the escape character: \\ represents one backslash, \n and \t are converted to a line break and a tab, and a backslash followed by any other character is kept as is (for example C:\data is unchanged). When a value needs to keep literal content like \n, write the backslash as \\.

Whitespace Handling

When reading data from a CSV file, the system automatically strips leading and trailing whitespace from each column.

Deleting Objects

CSV import supports deleting existing data, as follows:

• Add a DELETE_FLAG column to the header row of the CSV file;
• For the lines to delete, set the value of the DELETE_FLAG column to Y (the value rules are the same as for Boolean values).

WARNING

The header row of the CSV file must contain lookup fields that uniquely identify a record for deletion to work correctly.

For lines marked for deletion:

• 0 existing records found: a warning is logged and nothing is done.
• Multiple existing records found: the line is skipped.
• No lookup field in the file: none of the lines marked for deletion deletes any data.

Conflict Detection and Overwrite

After each line is imported successfully (created or updated), the system records the original content of the line in Exec History > Import Record. When the same record is imported again later, the system compares three versions: the current value in the database, the CSV line from the last import, and the CSV line of this import.

When importing in overwrite-conflict mode (Overwrite Conflict is turned on in the 导入数据 or Import dynamic plugin action, or the plugin file name contains SNAPSHOT), successfully imported lines are not written to Exec History > Import Record. On the next import in normal mode, the system still compares against the records left by the earlier import.

Changed in UI	Changed in CSV	Result
No	Yes	Updated normally
Yes	No	Skipped, the change made in the UI is kept (counted as updated in the DB but not in the CSV in the import record)
Yes	Yes	Treated as a conflict, not updated, and the line is recorded as a conflict line

In the following cases, conflict detection is skipped and the CSV overwrites the value in the database directly:

• The OVERWRITE_FLAG column of the line is Y;
• Overwrite Conflict is turned on in the 导入数据 action;
• The file name of the plugin package contains SNAPSHOT.

TIP

Data that has never been imported through CSV (for example a form created in the UI) has no previous import record the first time it is updated with CSV, so it is not treated as a conflict.

Data Type Mapping

The following lists how field types of the platform's built-in models map to values in CSV:

Field type	Value in CSV
String	Imported as is
Boolean	See Boolean
Integer, Long, Double, BigDecimal	Number
java.time.LocalDate, LocalDateTime, ZonedDateTime, OffsetDateTime, java.util.Date	See Date and Time
Enum	See Enum Types
org.springframework.http.HttpMethod	GET, POST, etc.
tech.muyan.storage.StorageFieldValue	File path of the attachment
Associated object	See Associated Object Queries
Set, List collections	[value1,value2], see Associated Object Queries

Fields of dynamic models are converted according to the data type (dataType) of the domain model field, following the same rules.

The platform provides no extension point for registering custom type conversions.

Boolean

For Boolean fields, values map as follows; values outside this range cause the line to fail:

Value in the CSV file	Imported value
Y, y, Yes, YES, true, TRUE, T, t, 是, 1	true
N, n, No, NO, false, FALSE, F, f, 否, 0	false

Date and Time

The following formats are supported. For the meaning of each letter in the formats, see SimpleDateFormat (opens new window):

 "yyyyMMdd"
 "dd-MM-yyyy"
 "yyyy-MM-dd"
 "MM/dd/yyyy"
 "yyyy/MM/dd"
 "dd MMM yyyy"
 "dd MMMM yyyy"
 "yyyyMMddHHmm"
 "yyyyMMdd HHmm"
 "dd-MM-yyyy HH:mm"
 "yyyy-MM-dd HH:mm"
 "MM/dd/yyyy HH:mm"
 "yyyy/MM/dd HH:mm"
 "dd MMM yyyy HH:mm"
 "dd MMMM yyyy HH:mm"
 "yyyyMMddHHmmss"
 "yyyyMMdd HHmmss"
 "dd-MM-yyyy HH:mm:ss"
 "yyyy-MM-dd HH:mm:ss"
 "MM/dd/yyyy HH:mm:ss"
 "yyyy/MM/dd HH:mm:ss"
 "dd MMM yyyy HH:mm:ss"
 "dd MMMM yyyy HH:mm:ss"
 "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'"

Enum Types

Enum fields only accept enum constant names, and the case must match exactly. For example, the type column of DynamicMenu takes FORM, not Form as shown in the UI. Other values cause the line to fail.

Attachment Types

For attachment columns, fill in the relative file path from the seed data root directory. During import, the system saves the file as an attachment (StorageFieldValue) and associates it with the imported object.

Viewing Import Records

The import of each CSV file is recorded in Exec History > Import; viewing it requires the DEVELOPER role. The first row in the screenshot below is the record of the user group import above:

An import record stores the following information:

• The type of the imported object, the md5 of the imported CSV file, and the header row
• The import status, one of:
  • Success (SUCCESS)
  • Partially fail (PARTIALLY_FAIL)
  • Failed (FAILED)
  • Running (RUNNING)
  • No data imported (NO_DATA_IMPORTED)
  • Not start (NOT_START)
  • Partially success (PARTIALLY_SUCCESS, deprecated)
• Import start and finish time
• The ids and counts of inserted, updated, deleted, not updated, conflicted, failed, and updated-in-the-DB-but-not-in-the-CSV objects
• The original CSV content of failed, skipped, conflicted, and updated-in-the-DB-but-not-in-the-CSV lines
• The log of the import process

WARNING

In 1.0.0-beta18, the Updated ids column of the data import list shows [object Object], and clicking Details on a row causes a page error. Failure reasons can be viewed in the Failed lines and Logs columns of the list.

In addition, Start date and Finish date in the data import list are displayed in UTC as is, without conversion to the browser's time zone, while Imported at in the Import Record list below is converted to local time. When viewed in UTC+8, the same import shows times 8 hours apart in the two lists (16:16 in the first row of the screenshot above and 00:16 in the first two rows of the screenshot below are the same import).

Each successfully imported line is stored in Exec History > Import Record, and conflict detection relies on these records:

A Complete Example

The following example creates a sales role, a sales department user group and two salespeople, and adds the salespeople to the user group. Create the following files in the csv directory (the _crm suffix of the file names can be anything):

HierarchyRole_crm.csv:

name(*),implies.name[:]
;; Sales role, includes the regular user role
ROLE_SALES,[ROLE_USER]

RoleRequirement_crm.csv:

name(*),hasPermissionRoles.name[:],customLogic.name
SALES,[ROLE_SALES],

Group_crm.csv:

name(*),roles.name[:]
销售部,[ROLE_SALES]

User_crm.csv:

username(*),password,name,accountLocked,DELETE_FLAG
[email protected],Crm@2026,王芳,N,N
[email protected],Crm@2026,刘洋,否,N

UserGroup_crm.csv:

user.username(*),group.name(*)
[email protected],销售部
[email protected],销售部

After the backend service restarts, the import order is: HierarchyRole_crm.csv, RoleRequirement_crm.csv (predefined order), then Group_crm.csv, User_crm.csv, UserGroup_crm.csv (sorted by dependencies; UserGroup depends on User and Group, so it comes last).

File Notes

• Mapping file names to models: in User_crm.csv, the part before the first _ is User, so it is imported into the User model; the _crm suffix only distinguishes files.
• Looking up existing records: in User_crm.csv, only username(*) is a lookup field, so existing users are looked up by username, updated if found and created otherwise; in UserGroup_crm.csv, the two association fields user.username(*) and group.name(*) are both lookup fields.
• Associated object queries: group.name(*) means the user group is matched by the name field of Group.

Field Notes

• implies.name[:], roles.name[:], hasPermissionRoles.name[:]
  • Collection fields; values are written in square brackets, with multiple values separated by :. For example, [ROLE_SALES] associates the role named ROLE_SALES.
  • The associated objects must already exist: ROLE_SALES is created in HierarchyRole_crm.csv, which is imported before Group_crm.csv according to the predefined order.
• accountLocked
  • A Boolean field; both N and 否 are converted to false, see Boolean.
• password
  • The user's initial password. During import the system encrypts it and saves it to the user table (note: the original CSV line, including the plaintext initial password, is saved in Exec History > Import Record, where roles DEVELOPER and above can see it; use only temporary initial passwords and require users to change them after the first login).
• DELETE_FLAG
  • The value N means create or update. To delete a user, change the value of that line to Y and import again.
• customLogic.name
  • Left empty, meaning the role requirement uses no custom logic and is judged by roles only.