Bulk Batch File Upload¶
Note
This section is applicable only for region Europe 1.
Bulk batch upload of objects in Integrated Data Lake adds the possibility to import or export data in bulk. It can be done from and to the external servers or programs to analyze or filter massive data. Bulk batch file upload uses the STS token provided by AWS with write access. It is required to specify the folder in which the bulk files need to be uploaded. Thereafter, the STS token with write access to that path (folder) is generated. Using this STS token, you can upload files in bulk. The validity for the STS token is up to 12 hours.
Set environment variables¶
Set the environment variables for your system:
Environment Paths on Windows
C:> setx AWS_ACCESS_KEY_ID XXXXXXXXXXXXXXXXXXXX
C:> setx AWS_SECRET_ACCESS_KEY xxxxxxxxxxxxx/xxxxxxx/xxxxxxxxxxxxxxxxxx
C:> setx AWS_SESSION_TOKEN=<
Environment Paths on Linux
$ export AWS_ACCESS_KEY_ID=XXXXXXXXXXXXXXXXXXXX
$ export AWS_SECRET_ACCESS_KEY=xxxxxxxxxxxxx/xxxxxxx/xxxxxxxxxxxxxxxxxx
$ export AWS_SESSION_TOKEN=<
Provide write permission¶
The tenantAdmin can access this API and provide write permission on folder or path, on the subtenants folder and on the root folder, except for the Time Series Import (TSI) folder.
1 | POST /accessTokenPermissions
|
1 | Content-Type: Content-Type: application/json
|
Request example:
1 2 3 4 5 | { "subtenantId": "204a896c-a23a-11e9-a2a3-2a2ae2dbcce4", "path": "/myfolder1/myfolder2", "permission": "WRITE" } |
Response example:
1 2 3 4 5 6 7 | { "subtenantId": "204a896c-a23a-11e9-a2a3-2a2ae2dbcce4", "id": "string", "path": "/myfolder1/myfolder2", "permission": "WRITE", "created": "2019-11-12T13:36:00.000Z" } |
Get a list of write permission folders¶
The tenantAdmin can get a list of all the folders that have the write permission. It is also possible to get a list of write permissions of the subtenants folders.
1 | GET /accessTokenPermissions
|
1 | Content-Type: application/json
|
Response example:
1 2 3 4 5 6 7 8 9 10 11 | { "accessTokenPermissions": [ { "subtenantId": "204a896c-a23a-11e9-a2a3-2a2ae2dbcce4", "id": "string", "path": "/myfolder1/myfolder2", "permission": "WRITE", "created": "2019-11-12T13:36:00.000Z" } ] } |
Get the list for write permission folders for the given id¶
You can get the list of folders that have the write permissions, for the given id. The tenantAdmin can also get the details of the request for subtenants. The subtenants can access this API to get details of the request belongs to their write folder.
1 | GET /accessTokenPermissions/{id}
|
1 | Content-Type: application/json
|
Response example:
1 2 3 4 5 6 7 | { "subtenantId": "204a896c-a23a-11e9-a2a3-2a2ae2dbcce4", "id": "string", "path": "/myfolder1/myfolder2", "permission": "WRITE", "created": "2019-11-12T13:36:00.000Z""id": "0234sd34a23a-11e9-a2a3-2a2sdfw34ce4", } |
Revoke write permission¶
The TenantAdmin can access this API and provide delete permission on folders for the given id.
1 | DELETE /accessTokenPermissions/{id}
|
1 | Content-Type: application/json
|
Response example:
1 | Status: 204 - deleted
|
Get the STS token with Write access¶
Request URL:
1 | https://gateway.eu1.mindsphere.io/api/datalake/v3/generateAccessToken
|
Request example:
1 2 3 4 | { "durationSeconds": 900, "permission": "WRITE" } |
Response example:
1 2 3 4 5 6 7 8 9 10 11 | { "storageAccount": "dlbucketname", "storagePath": "data/ten=tenantname OR data/sub=subtenantId", "credentials": { "secretAccessKey": "AbCdEfGhIjKlMnOpQrStUvWxYz", "accessKeyId": "AbCdEfGhIjKlMnOpQrStUvWxYzAbCdEfGhIjKlMnOpQrStUvWxYz", "sessionToken": "AbCdEfGhIjKlMnOpQrStUvWxYzAbCdEfGhIjKlMnOpQrStUvWxYzAbCdEfGhIjKlMnOpQrStUvWxYzAbCdEfGhIjKlMnOpQrStUvWxYzAbCdEfGhIjKlMnOpQrStUvWxYzAbCdEfGhIjKlMnOpQrStUvWxYz" }, "durationSeconds": 900, "permission": "WRITE" } |
Command and Output¶
The following command is used for Bulk upload:
1 2 3 4 5 | aws s3 cp s3://ms-coldstorage-iot-integ-eu-central-1/000013bb5ed740f9950504854367e12a s3://estbuckettimeseries/ten=dide2/000013bb5ed740f9950504854367e12a --recursive Completed 1.7 KiB/1.7 KiB (812 Bytes/s) with 1 file(s) remaining copy: s3://ms-coldstorage-iot-integ-eu-central-1/000013bb5ed740f9950504854367e12a/propSetName=SPSNFRTestFields/date=2019-04-05/0900.parquet to s3://estbuckettimeseries/ten=dide2/000013bb5ed740f995050485436 7e12a/propSetName=SPSNFRTestFields/date=2019-04-05/0900.parquet |
Note
The time required to upload files is dependent on the network bandwidth and location.
Any questions left?
Except where otherwise noted, content on this site is licensed under the MindSphere Development License Agreement.