Skip to content

Bulk Batch File Upload

Note

This section is applicable only for region Europe 1.

Bulk batch upload of objects in Integrated Data Lake adds the possibility to import or export data in bulk. It can be done from and to the external servers or programs to analyze or filter massive data. Bulk batch file upload uses the STS token provided by AWS with write access. It is required to specify the folder in which the bulk files need to be uploaded. Thereafter, the STS token with write access to that path (folder) is generated. Using this STS token, you can upload files in bulk. The validity for the STS token is up to 12 hours.

Set environment variables

Set the environment variables for your system:

Environment Paths on Windows

C:> setx AWS_ACCESS_KEY_ID XXXXXXXXXXXXXXXXXXXX C:> setx AWS_SECRET_ACCESS_KEY xxxxxxxxxxxxx/xxxxxxx/xxxxxxxxxxxxxxxxxx C:> setx AWS_SESSION_TOKEN=<>

Environment Paths on Linux

$ export AWS_ACCESS_KEY_ID=XXXXXXXXXXXXXXXXXXXX $ export AWS_SECRET_ACCESS_KEY=xxxxxxxxxxxxx/xxxxxxx/xxxxxxxxxxxxxxxxxx $ export AWS_SESSION_TOKEN=<>

Provide write permission

The tenantAdmin can access this API and provide write permission on folder or path, on the subtenants folder and on the root folder, except for the Time Series Import (TSI) folder.

1
POST /accessTokenPermissions

1
Content-Type: Content-Type: application/json

Request example:

1
2
3
4
5
{
  "subtenantId": "204a896c-a23a-11e9-a2a3-2a2ae2dbcce4",
  "path": "/myfolder1/myfolder2",
  "permission": "WRITE"
}

Response example:

1
2
3
4
5
6
7
{
  "subtenantId": "204a896c-a23a-11e9-a2a3-2a2ae2dbcce4",
  "id": "string",
  "path": "/myfolder1/myfolder2",
  "permission": "WRITE",
  "created": "2019-11-12T13:36:00.000Z"
}

Get a list of write permission folders

The tenantAdmin can get a list of all the folders that have the write permission. It is also possible to get a list of write permissions of the subtenants folders.

1
GET /accessTokenPermissions

1
Content-Type: application/json

Response example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
{
   "accessTokenPermissions": [
  {
    "subtenantId": "204a896c-a23a-11e9-a2a3-2a2ae2dbcce4",
    "id": "string",
    "path": "/myfolder1/myfolder2",
    "permission": "WRITE",
    "created": "2019-11-12T13:36:00.000Z"
  }
  ]
}

Get the list for write permission folders for the given id

You can get the list of folders that have the write permissions, for the given id. The tenantAdmin can also get the details of the request for subtenants. The subtenants can access this API to get details of the request belongs to their write folder.

1
GET /accessTokenPermissions/{id}

1
Content-Type: application/json

Response example:

1
2
3
4
5
6
7
{
  "subtenantId": "204a896c-a23a-11e9-a2a3-2a2ae2dbcce4",
  "id": "string",
  "path": "/myfolder1/myfolder2",
  "permission": "WRITE",
  "created": "2019-11-12T13:36:00.000Z""id": "0234sd34a23a-11e9-a2a3-2a2sdfw34ce4",
}

Revoke write permission

The TenantAdmin can access this API and provide delete permission on folders for the given id.

1
DELETE /accessTokenPermissions/{id}

1
Content-Type: application/json

Response example:

1
Status: 204 - deleted

Get the STS token with Write access

Request URL:

1
https://gateway.eu1.mindsphere.io/api/datalake/v3/generateAccessToken

Request example:

1
2
3
4
{
  "durationSeconds": 900,
  "permission": "WRITE"
}

Response example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
{
  "storageAccount": "dlbucketname",
  "storagePath": "data/ten=tenantname OR data/sub=subtenantId",
  "credentials": {
    "secretAccessKey": "AbCdEfGhIjKlMnOpQrStUvWxYz",
    "accessKeyId": "AbCdEfGhIjKlMnOpQrStUvWxYzAbCdEfGhIjKlMnOpQrStUvWxYz",
    "sessionToken": "AbCdEfGhIjKlMnOpQrStUvWxYzAbCdEfGhIjKlMnOpQrStUvWxYzAbCdEfGhIjKlMnOpQrStUvWxYzAbCdEfGhIjKlMnOpQrStUvWxYzAbCdEfGhIjKlMnOpQrStUvWxYzAbCdEfGhIjKlMnOpQrStUvWxYz"
  },
  "durationSeconds": 900,
  "permission": "WRITE"
}

Command and Output

The following command is used for Bulk upload:

1
2
3
4
5
aws s3 cp s3://ms-coldstorage-iot-integ-eu-central-1/000013bb5ed740f9950504854367e12a s3://estbuckettimeseries/ten=dide2/000013bb5ed740f9950504854367e12a --recursive

Completed 1.7 KiB/1.7 KiB (812 Bytes/s) with 1 file(s) remaining
copy: s3://ms-coldstorage-iot-integ-eu-central-1/000013bb5ed740f9950504854367e12a/propSetName=SPSNFRTestFields/date=2019-04-05/0900.parquet to s3://estbuckettimeseries/ten=dide2/000013bb5ed740f995050485436
7e12a/propSetName=SPSNFRTestFields/date=2019-04-05/0900.parquet

Note

The time required to upload files is dependent on the network bandwidth and location.

Any questions left?

Ask the community

Except where otherwise noted, content on this site is licensed under the MindSphere Development License Agreement.