TaCustomTableWriter Plug-in

I. Quick Introduction

TACustomTableWriter can create custom tables to the Ta system.

In some cases, the data you need to use may not be represented in the form of user or event, such as some mapping tables, or some external data. If you need to use this part of data, you need to use the TACustomTableWriter plug-in Import custom data into the TA system.

II. Functions and Limitations

Only the custom table function of the TA system is supported.

III. Function Description

3.1 Sample configuration

{
  "job": {
    "setting": {
      "speed": {
        "channel": 1
      }
    },
    "content": [
      {
        "reader": {
          "name": "streamreader",
          "parameter": {
            "column": [
              {
                "value": 123456,
                "type": "long"
              },
              {
                "value": "123123",
                "type": "string"
              },
              {
                "value": "pay",
                "type": "string"
              },
              {
                "value": "2020-01-01 08:08:08",
                "type": "date"
              },
              {
                "value": "2020-01-01 08:08:08",
                "type": "date"
              },
              {
                "value": "2222",
                "type": "string"
              },
              {
                "value": "2020-01-01 08:08:08",
                "type": "date"
              },
              {
                "value": "test",
                "type": "string"
              },
              {
                "value": true,
                "type": "bool"
              }
            ],
            "sliceRecordCount": 10
          }
        },
        "writer": {
          "name": "ta-custom-table-writer",
          "parameter": {
            "appid": "6f9e64da5bc74792b9e9c1db4e3e3822",
            "table": "ta_custom_table",
            "tableDesc": "test table",
            "partitionValue": "2020-01-01",
            "writeMode": "overwrite",
            "column": [
              {
                "type": "double",
                "name": "id"
              },
              {
                "type": "string",
                "name": "distinct_id"
              },
              {
                "type": "string",
                "name": "event_name"
              },
              {
                "type": "string",
                "name": "time"
              },
              {
                "name": "event_time",
                "type": "string"
              },
              {
                "type": "string",
                "name": "account_id"
              },
              {
                "type": "string",
                "name": "testDate"
              },
              {
                "type": "string",
                "name": "timetest2"
              },
              {
                "type": "string",
                "name": "os_1"
              }
            ]
          }
        }
      }
    ]
  }
}

3.2 Parameter description

• appid
  • Description: project appid corresponding to the import table can be found in the background of the TA system. It must exist in the system and no project is offline.
  • Type: string
  • Required: Yes
  • Default value: none
• table
  • Description: import the table name into the TA system. Note: The table name cannot be repeated globally. It is recommended to add distinguishable prefixes or suffixes based on different projects. The table name can only be a combination of numbers, letters, and underlines.
  • Type: string
  • Required: Yes
  • Default value: none
• tableDesc
  • Description: import the comment of the table. It is recommended to configure this parameter when importing to facilitate the subsequent query of the table and clarify the meaning of the table
  • Type: string
  • Required: No
  • Default value: none
• partitionValue
  • Description: imported partition value, the custom table imported by the TA system will bring the partition field $pt by default, so the imported partition value must be specified when importing. Generally, it can be set as the imported data date, for example: 20180701, and also supports time macro replacement, for example: @[{yyyMMdd} - {1day}]. Section XXXX will introduce the specific use method
  • Type: string
  • Required: Yes
  • Default value: none
• writeMode
  • Description: TACustomTableWriter data cleaning processing mode before writing:
    • append, like a partitionValue partition, writes data accumulatively.
    • overwrite, delete the existing file under the partitionValue partition, and then write the data imported this time.
    • Required: Yes
    • Default value: none
• column
  • Description: field to write data
  • Required: Yes
  • Default value: none
    • where: name specifies the field name, type specifies the field type.

The user can specify the Column field information, configured as follows:

"column":
         [
             {
                "name": "userName",
                "type": "string"
             },
             {
                "name": "age",
                "type": "long"
             }
         ]

3.3 Type conversion

Currently, TACustomTableWriter supports the HIVE type. Please pay attention to check your type.

The following is a list of TACustomTableWriter conversions for HIVE data types:

DataX internal type	HIVE data type
Long	TINYINT,SMALLINT,INT,BIGINT
Double	FLOAT,DOUBLE
String	STRING,VARCHAR,CHAR
Boolean	BOOLEAN
Date	DATE,TIMESTAMP