How to Connect AWS S3 with Labellerr

​

Prerequisites

• AWS Account: You must have an active AWS account to access and utilize the IAM service. If you don’t have an account, you can sign up for one at the AWS website.
• IAM Permissions: Ensure that you have the necessary permissions to create IAM users and policies. You need to have IAM administrative access or consult with your AWS account administrator to grant you the required permissions.
• IAM User Creation: Decide on the user(s) you want to create in IAM. Consider the appropriate naming conventions and the level of access each user should have. Determine the number of users you need to create and the specific roles and permissions they require.
• User Credentials: Plan how you will manage the user credentials. Determine whether you will generate initial passwords for the users or require them to set their own passwords upon first login. Alternatively, you can leverage AWS Single Sign-On or other authentication mechanisms.
• IAM Policies: Identify the specific permissions and access levels required for each user. You can create custom policies to grant fine-grained permissions or leverage existing AWS managed policies. Define the policies that align with your security and access control requirements.
• IAM Best Practices: Familiarize yourself with IAM best practices, such as implementing the principle of least privilege, regularly reviewing and rotating credentials, and enabling multi-factor authentication (MFA) for added security.

​

Quick Reference

​

S3 URI Format

Format: s3://bucket-name/path/to/folder/
Example: s3://my-annotation-bucket/images/batch-001/

​

Required Permissions Summary

​

CREATE IAM USER

1

Access IAM Users

Go to IAM ⇒ Users and click to Add users button

2

Create User

Now write a User name and click to Next button

3

Set Permissions

In the Set Permissions section you will see the Permissions options. Choose Attach Policies Directly

After choosing the above option you can see the list of existing policies

4

Create Custom Policy

Create a new policy for the user to access the bucket. Now click on the Create policy button. After this you will be redirected to a window IAM ⇒ Policies ⇒ Create Policy as we can see following.

Choose the JSON option from the RIGHT-HAND-SIDE of the window. After choosing the JSON option you will see the window as follow

5

Configure Policy Permissions

Update the JSON as follow and replace the your-bucket-name with bucket-name

Ensure to include '' at the end of Resource URL in case of S3:GetObject & S3:PutObject like ‘arn:aws:s3:::bucket-name/’. The ’*’ ensures that the required permission is added to the sub-folders also

Now, update the JSON policy. We will create two permission sets: one for importing data from S3 and another for exporting data to S3.

​

A. Policy for Importing Data

To allow Labellerr to read data from your bucket, paste the following JSON code. Replace your-bucket-name with the actual name of your S3 bucket.

Copy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "S3ObjectAccess",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject"
      ],
      "Resource": "arn:aws:s3:::your-bucket-name/*"
    },
    {
      "Sid": "S3BucketReadAccess",
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket",
        "s3:GetBucketCors",
        "s3:GetBucketLocation"
      ],
      "Resource": "arn:aws:s3:::your-bucket-name"
    },
    {
      "Sid": "S3PutCorsAccess",
      "Effect": "Allow",
      "Action": [
        "s3:PutBucketCors"
      ],
      "Resource": "arn:aws:s3:::your-bucket-name"
    }
  ]
}

Note: If you prefer not to grant s3:PutBucketCors permission, you can set the CORS policy on your S3 bucket manually with the following configuration. For more details, refer to the official AWS documentation on CORS.

Copy

[
  {
      "AllowedHeaders": ["*.labellerr.com"],
      "AllowedMethods": ["GET", "HEAD"],
      "AllowedOrigins": ["*.labellerr.com"],
      "ExposeHeaders": [],
      "MaxAgeSeconds": 3600
   }
]

​

B. Policy for Exporting Data

To allow Labellerr to write (export) annotations back to your S3 bucket, add the following permissions to your policy. These are required in addition to the import permissions above.

Copy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "S3ObjectWriteDeleteAccess",
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:DeleteObject"
      ],
      "Resource": "arn:aws:s3:::your-bucket-name/*"
    },
    {
      "Sid": "S3BucketListAccess",
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket"
      ],
      "Resource": "arn:aws:s3:::your-bucket-name"
    }
  ]
}

6

Finalize Policy

Click on Next button

Fill the Policy name & Policy Description and click on Create policy button

After this the policy will be created and listed

7

Attach Policy to User

Now go to the previous user-creation window and after reloading and entering the User Name and selecting the option Attach policies directly we can see the policy that we created. Choose the same policy and click on Next button

Now click on Create user button and user will be created

​

Generate Access Key

1

Create Access Key

Now go to IAM ⇒ user ⇒ user_name and click on Create access key button.

2

Configure Access Key

Choose the option Application running outside AWS & click on Next . Enter a description and click on the Create access key button. Now you can copy the Access keys & Secret access key or download a CSV file for the same for future use while connecting with Labellerr

​

Creating Dataset & Annotation Project at LABELLERR

1

Select Amazon S3

While creating a project dataset choose Amazon S3 to connect data-source and click on Create new connection button.

2

Configure Connection

Fill in the details. Fill the Access key & Secret Access Key generated in previous steps along with s3 URI. To find the s3 URI go to your bucket and copy the path of the images folder. Also remove the s3:// prefix from the URL. Now click on connect button

If you fill in the details correctly you will receive this message.

If you messed up somewhere you will get an error.

After successfully testing the connection you will be required to fill the Connection Name and Connection Description(Optional).

​

Troubleshooting

from labellerr.core.connectors import LabellerrConnection
from labellerr.core.schemas import ConnectionType, DatasetDataType

connection = LabellerrConnection(client=client, connection_id="your_connection_id")
test_result = connection.test(
    path="s3://your-bucket/path/to/data/",
    connection_type=ConnectionType._IMPORT,
    data_type=DatasetDataType.image
)

[
  {
    "AllowedHeaders": ["*"],
    "AllowedMethods": ["GET", "HEAD"],
    "AllowedOrigins": ["*.labellerr.com"],
    "ExposeHeaders": [],
    "MaxAgeSeconds": 3600
  }
]

​

Related Guides