Skip to content

Bulk Batch File Upload

Bulk batch upload of objects in Integrated Data Lake adds the possibility to import or export data in bulk. It can be done from and to the external servers or programs to analyze or filter massive data. Bulk batch file upload uses the STS token provided by AWS with write access. It is required to specify the folder in which the bulk files need to be uploaded. Thereafter, the STS token with write access to that path (folder) is generated. Using this STS token, you can upload files in bulk. The validity for the STS token is up to 12 hours.

Provide write permission

The tenantAdmin can access this API and provide write permission on folder or path, on the subtenants folder and on the root folder, except for the Time Series Import (TSI) folder.

1
POST /accessTokenPermissions

1
Content-Type: Content-Type: application/json

Request example:

1
2
3
4
5
{
  "subtenantId": "204a896c-a23a-11e9-a2a3-2a2ae2dbcce4",
  "path": "/myfolder1/myfolder2",
  "permission": "WRITE"
}

Response example:

1
2
3
4
5
6
7
{
  "subtenantId": "204a896c-a23a-11e9-a2a3-2a2ae2dbcce4",
  "id": "string",
  "path": "/myfolder1/myfolder2",
  "permission": "WRITE",
  "created": "2019-11-12T13:36:00.000Z"
}

Get a list of write permission folders

The tenantAdmin can get a list of all the folders that have the write permission. It is also possible to get a list of write permissions of the subtenants folders.

1
GET /accessTokenPermissions

1
Content-Type: application/json

Response example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
{
   "accessTokenPermissions": [
  {
    "subtenantId": "204a896c-a23a-11e9-a2a3-2a2ae2dbcce4",
    "id": "string",
    "path": "/myfolder1/myfolder2",
    "permission": "WRITE",
    "created": "2019-11-12T13:36:00.000Z"
  }
  ]
}

Get the list for write permission folders for the given id

You can get the list of folders that have the write permissions, for the given id. The tenantAdmin can also get the details of the request for subtenants. The subtenants can access this API to get details of the request belongs to their write folder.

1
GET /accessTokenPermissions/{id}

1
Content-Type: application/json

Response example:

1
2
3
4
5
6
7
{
  "subtenantId": "204a896c-a23a-11e9-a2a3-2a2ae2dbcce4",
  "id": "string",
  "path": "/myfolder1/myfolder2",
  "permission": "WRITE",
  "created": "2019-11-12T13:36:00.000Z""id": "0234sd34a23a-11e9-a2a3-2a2sdfw34ce4",
}

Revoke write permission

The TenantAdmin can access this API and provide delete permission on folders for the given id.

1
DELETE /accessTokenPermissions/{id}

1
Content-Type: application/json

Response example:

1
Status: 204 - deleted

Get the STS token with Write access

Request URL:

1
https://gateway.eu1.mindsphere.io/api/datalake/v3/generateAccessToken

Request example:

1
2
3
4
{
  "durationSeconds": 900,
  "permission": "WRITE"
}

Response example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
{
  "storageAccount": "dlbucketname",
  "storagePath": "data/ten=tenantname OR data/sub=subtenantId",
  "credentials": {
    "secretAccessKey": "AbCdEfGhIjKlMnOpQrStUvWxYz",
    "accessKeyId": "AbCdEfGhIjKlMnOpQrStUvWxYzAbCdEfGhIjKlMnOpQrStUvWxYz",
    "sessionToken": "AbCdEfGhIjKlMnOpQrStUvWxYzAbCdEfGhIjKlMnOpQrStUvWxYzAbCdEfGhIjKlMnOpQrStUvWxYzAbCdEfGhIjKlMnOpQrStUvWxYzAbCdEfGhIjKlMnOpQrStUvWxYzAbCdEfGhIjKlMnOpQrStUvWxYz"
  },
  "durationSeconds": 900,
  "permission": "WRITE"
}

Command and Output

The following command is used for Bulk upload:

1
2
3
4
5
aws s3 cp s3://ms-coldstorage-iot-integ-eu-central-1/000013bb5ed740f9950504854367e12a s3://estbuckettimeseries/ten=dide2/000013bb5ed740f9950504854367e12a --recursive

Completed 1.7 KiB/1.7 KiB (812 Bytes/s) with 1 file(s) remaining
copy: s3://ms-coldstorage-iot-integ-eu-central-1/000013bb5ed740f9950504854367e12a/propSetName=SPSNFRTestFields/date=2019-04-05/0900.parquet to s3://estbuckettimeseries/ten=dide2/000013bb5ed740f995050485436
7e12a/propSetName=SPSNFRTestFields/date=2019-04-05/0900.parquet

Note

The time required to upload files is dependent on the network bandwidth and location.

Any questions left?

Ask the community

Except where otherwise noted, content on this site is licensed under the MindSphere Development License Agreement.