Entities
"Entity" represents a structural unit designed to organize and store persistent data. Analogous to an index in Elasticsearch, or a collection in MongoDB. You can think of an entity as a table in a relational database, or a document in a NoSQL database.
Key Characteristics
- JSON data: Entities are JSON objects.
- Queryable: Entities can be queried using a mongo-like query language.
- Data Manipulation: Entities allow for standard CRUD (Create, Read, Update, Delete) operations on data records.
- Indexing: For improved query performance, data can be indexed on one or multiple fields.
- Views: Users can create data views that filter and display records according to specified criteria.
- Logging: Entities can be used to store logs from integration pipelines or other processes.
Use Cases
Entities can be used for various purposes such as:
- Data Storage: Store and manage data records from external systems
- Logging Data: Store logs from integration pipelines or other processes
- Materialized Views: Persistently store the results of complex Views
Entity List
The Entity List page displays all entities that have been created. Users can quickly see the entity name and type.
From Entity List page user can perform the following actions on each entity:
- Open List of Entity Records by clicking on the entity name
- Create new Entity
- See the number of Views related to the Entity in the "Views" button
- Open List of Entity Views by clicking on the "Views" button
- Open Dropdown list with all available Views by clicking on the arrow in the "Views" button
- Edit Entity by clicking on the "Edit" button
- Delete Entity by clicking on the "Delete" button
For navigation purposes, users can:
- Filter entities by name using the search bar
- Use pagination to navigate through the list pages
Create Entity
To create an entity we must specify the following:
- Name: Name of the entity. String from 3 to 64 characters
- Type: We must select Regular or Logs entity type.
- Regular entities are used to store data records
- Logs entities are used to store logs from integration pipelines or other processes. It has additional indexes and fields for logging purposes.
Date Settings
Date Settings is optional settings for date fields. If the entity data contains date fields, you must specify the source field, target field, source type and source format. BuiltAPI will automatically convert the date fields to BuiltAPI date format.
- source path: Path to the original date field.
- target path: Path to the converted field in the entity record
- type: Type of date field:
- ISO: ISO8601 format. Example: 2023-01-01T00:00:00Z
- Seconds: UNIX timestamp in seconds. Example: 1609459200
- Milliseconds: UNIX timestamp in milliseconds. Example: 1609459200000
- Custom: If the date format is not one of the above, you can specify a custom format. The format must be a valid JS time format string.
Example:
#original record
{
"name": "John Doe",
"age": 30,
"email": "[email protected]",
"dob": "2021-01-01T00:00:00Z"
}
In the above example, the "dob" field is in ISO8601 format. If we want to use dob field in query we must specify the date settings as follows:
In this example we create new entity "user" with date settings for "dob" field. We specify that the source field is "dob", the target field is "dob_iso" and the type is "ISO". Each time a record is created or updated, BuiltAPI will automatically convert the "dob" field from ISO8601 format to "dob_iso" field in BuiltAPI date format.
You can set "source path" and "target path" to the same value if you want to overwrite the original field.
Entity records
An entity record is a single JSON object that represents a data record in an Entity. See details in the Records section.
Entity types
Regular
Regular Entities are used to store data records. They can be used to store data from external systems, or to store data generated by integration pipelines. Entities can be used to materialize the results of Views.
Logs
Logs Entities are used to store logs from integration pipelines or other processes. Logs Entities have additional indexes and fields for logging purposes. Logs Entities can be used to store logs from integration pipelines, or to store logs from other processes.
Logs records creates automatically when you create an Integration, or can be created manually and set as logs entity in "Create Integration" form.
Logs records example
{
"level": "error",
"timestamp": "2024-07-04T20:56:52.502Z",
"workspaceId": "workspace-1234",
"integrationId": "integration-5678",
"pipelineId": "pipeline-91011",
"integrationName": "Integration Name",
"pipelineName": "Pipeline Name",
"pipelineRunId": "run-1213",
"message": "Request failed with status code 400",
"details": {
"id": "asset-1415",
"error": {
"name": "ConnectorError",
"type": "RESPONSE",
"details": {
"data": {
"errorMessages": "'type_of_use'"
},
"status": 400
},
"params": {
"method": "POST",
"baseUrl": "https://api.example.com",
"path": "/services/endpoint/",
"body": {
"data": [
{
"asset": {
"id": "asset-1415",
"type_of_use": null
},
"settings": {
"normilise_consumption": "Yes"
}
}
]
},
"allowStatuses": [
202
]
},
"message": "Request failed with status code 400"
}
}
}
Logs records always have the following fields:
- level: Log level. Can be "info", "warning", "error", "debug"
- timestamp: Timestamp when the log was created
- workspaceId: Workspace ID
- integrationId: Integration ID
- pipelineId: Pipeline ID
- integrationName: Integration Name
- pipelineName: Pipeline Name
- pipelineRunId: Pipeline Run ID
- message: Log message
- details: Additional details about the log
Entity Validation
Entity Validation ensures data integrity and consistency by allowing you to define and enforce validation rules for records within an Entity. This feature helps maintain data quality by checking records against a predefined schema (e.g., JSON Schema) during creation or updates.
Accessing Validation Settings
To configure validation for an Entity:
- Navigate to the Data section in the sidebar.
- Select the desired Entity from the list.
- Click on the "Set validation" button
Validator Editor
This is where you define the validation rules using a JSON-based schema definition (e.g., JSON Schema).
The editor interface includes:
Left Panel (Example/Schema Tabs):
- Example Tab: Shows a sample document from the entity. You can drag-and-drop fields from the example to the Validator editor, which can help in constructing rules by providing context and field paths.
- Schema Tab: Displays the inferred schema of the entity's documents. This can also be used to drag-and-drop fields into the editor.
Center Panel (Settings/Validator Tabs):
- Settings Tab: Contains the "Validation level" and "Validation action" selectors described above.
- Validator Tab: A text editor where you input or modify the JSON-based validation rules. The editor typically includes controls for formatting, searching, and managing the JSON content.
Right Panel (Valid/Invalid Tabs): This panel is for ad-hoc checking of existing data against the rules currently present in the editor, which may not yet be saved.
- Valid Tab: Lists documents that conform to the validation rules.
- Invalid Tab: Lists documents that do not conform to the validation rules.
- Refresh Button: Clicking this button re-evaluates all existing documents in the entity against the currently defined validation rules in the editor. This is useful for checking compliance after modifying rules or for assessing the state of existing data.
Buttons at the bottom of the Validation page:
- Remove validation: Clears the current validation rules and disables validation for the entity.
- Save: Saves the current validation configuration (level, action, and rules). These settings will then be applied to subsequent data operations.
- Close: Closes the validation settings page without saving any pending changes.
Configuration Options
The Validation provides several options to control how data validation is performed.
Validation Level
This setting in the "Settings" area determines the strictness of the validation process.
- Off: Validation is disabled. No checks are performed on documents.
- Moderate: validation only applies your rules to new documents and existing valid documents. Existing invalid documents are not affected.
- Strict: validation applies your rules to all document inserts and updates.
Validation Action
This setting, also in the "Settings" area, determines the action taken when a new document is inserted or an existing document is updated, and it fails to meet the validation criteria.
- Warning: If a document does not meet the criteria, the insert/update operation will proceed, but a warning will be logged (viewable in the Logs section).
- Error: If a document does not meet the criteria, the insert/update operation will be rejected, and an error will be returned, preventing the change.
Validator Examples (MongoDB Syntax)
BuiltAPI utilizes MongoDB's schema validation. You define rules using JSON, most commonly with the $jsonSchema operator. All field paths in the validator should refer to fields within the data object of your records (e.g., data.fieldName).
Here are some common validation examples:
Basic Type Checking and Required Fields This example ensures that name is a string, age is an integer, and email is a string and is required. isSubscribed is an optional boolean.
```json
{
"$jsonSchema": {
"bsonType": "object",
"required": [ "data.email" ],
"properties": {
"data": {
"bsonType": "object",
"properties": {
"name": {
"bsonType": "string",
"description": "must be a string"
},
"age": {
"bsonType": "int",
"minimum": 0,
"description": "must be an integer and not negative"
},
"email": {
"bsonType": "string",
"description": "must be a string and is required"
},
"isSubscribed": {
"bsonType": "bool",
"description": "must be a boolean"
}
}
}
}
}
}
```String Pattern (Regex) and Length Validate that zipCode is a string matching a 5-digit pattern and status is one of predefined values.
```json
{
"$jsonSchema": {
"bsonType": "object",
"properties": {
"data": {
"bsonType": "object",
"properties": {
"zipCode": {
"bsonType": "string",
"pattern": "^[0-9]{5}$",
"description": "must be a string of 5 digits"
},
"status": {
"enum": [ "active", "inactive", "pending" ],
"description": "can only be one of the enum values"
},
"description": {
"bsonType": "string",
"minLength": 10,
"maxLength": 200
}
}
}
}
}
}
```Number Ranges Ensure rating is a number between 1 and 5 (inclusive).
```json
{
"$jsonSchema": {
"bsonType": "object",
"properties": {
"data": {
"bsonType": "object",
"properties": {
"rating": {
"bsonType": "double",
"minimum": 1,
"maximum": 5,
"description": "must be a double between 1 and 5"
}
}
}
}
}
}
```Nested Object Validation Validate fields within a nested address object.
```json
{
"$jsonSchema": {
"bsonType": "object",
"properties": {
"data": {
"bsonType": "object",
"required": ["data.address"],
"properties": {
"address": {
"bsonType": "object",
"required": [ "street", "city" ],
"properties": {
"street": {
"bsonType": "string",
"description": "Street is required and must be a string"
},
"city": {
"bsonType": "string",
"description": "City is required and must be a string"
},
"zip": {
"bsonType": "string",
"pattern": "^[0-9]{5}$",
"description": "Zip code must be a 5-digit string"
}
}
}
}
}
}
}
}
```
Note: When referencing nested fields within the data object for required arrays or properties, ensure you are targeting the correct level. For address.street, the required array is within the address property definition.
A more direct way to define nested data properties might look like this for the validator:
```json
{
"$jsonSchema": {
"bsonType": "object",
"properties": {
"data": {
"bsonType": "object",
"required": ["address"],
"properties": {
"address": {
"bsonType": "object",
"required": [ "street", "city" ],
"properties": {
"street": {
"bsonType": "string",
"description": "Street is required and must be a string"
},
"city": {
"bsonType": "string",
"description": "City is required and must be a string"
}
}
}
}
}
}
}
}
```
If your actual data structure is `{"data": {"address": {"street": "123 Main St"}}}`, then in your $jsonSchema, when defining properties for data.address, the paths to street and city are relative to address.Array Validation Ensure tags is an array of strings, with at least one tag.
```json
{
"$jsonSchema": {
"bsonType": "object",
"properties": {
"data": {
"bsonType": "object",
"properties": {
"tags": {
"bsonType": "array",
"minItems": 1,
"uniqueItems": true,
"items": {
"bsonType": "string",
"description": "Each tag must be a string"
},
"description": "must be an array of unique strings and is required"
}
}
}
}
}
}
```Using MongoDB Query Operators (outside $jsonSchema) While $jsonSchema is powerful, MongoDB's native validation also allows other query operators. For example, to ensure a field startDate is before endDate (assuming both are Date types and exist):
```json
{
"data.endDate": { "$gte": "$data.startDate" }
}
```
How Validation Works
- When you Save the validation settings, the rules, level, and action are stored for the entity.
- All subsequent create or update operations on records within this entity will be subject to these validation rules.
- If an operation (like creating or updating a record) violates a validation rule and the "Validation Action" is set to "Error", the operation will fail and the data will not be saved/modified. If set to "Warning", the operation will proceed, and a log entry will be generated.