May 9, 2017

AWS ECR with Gradle

AWS Elastic Container Registry is a registry service for Docker images. It supports the Docker API for registries and therefore can be used with docker pull, docker push, etc. It also works with Gradle plugins such as gradle-docker-plugin.

The authentication mechanism proves to be a challenge though. ECR requires a temporary token to be obtained using IAM credentials in order to use the registry. This doesn’t work well for CI tools because the token has a 24-hour expiry. It is also cumbersome for developer use to request a new token every 24 hours.

Solution

One solution is to use a Gradle plugin that will manage the ECR token, requesting it using AWS credentials. This post describes the use of the gradle-aws-ecr-plugin that will manage ECR tokens for you.

The plugin uses the AWS SDK for Java. This means it will use the ~/.aws/credentials file if available. Environment variables or Java system properties may also be used.

IAM Policy

As of this writing, I could not find a security policy to attach to a group or user to allow read/write access to ECR. You can use the following inline policy, but note this allows all access. A discussion of security around ECR is out of the scope of this writing, but you may want to look into restricting this to only allowing read/write to your repositories.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1461121381000",
            "Effect": "Allow",
            "Action": [
                "ecr:*"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

Example

Using the plugin is straightforward. You will need both the gradle-docker-plugin and gradle-aws-ecr-plugin.

plugins {
  id "com.bmuschko.docker-remote-api" version "3.0.6"
  id "com.patdouble.awsecr" version "0.3.2"
}

Then set the Docker registry to your ECR URL. The URL is given by the AWS ECR console. Note that this clause is defined by the gradle-docker-plugin. Neither username nor password should be set, the gradle-aws-ecr-plugin will handle that.

docker {
    registryCredentials {
        url # 'https://123456789012.dkr.ecr.us-east-1.amazonaws.com'
    }
}

AWS Credential Store

The plugin uses the AWS SDK to get credentials by default, read http://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html for details. We’ll provide snippets here of how it works.

$ aws configure
AWS Access Key ID [None]: accesskey
AWS Secret Access Key [None]: secretkey
Default region name [None]: us-west-2
Default output format [None]:
$ ./gradlew  ...

Environment Variables

Most CI tools use environment variables for configuration. You can do this for your AWS credentials. Make sure your CI tool obfuscates the secret in logs and other displays.

$ export AWS*ACCESS_KEY*ID=accesskey
$ export AWS*SECRET_ACCESS*KEY=secretkey
$ ./gradlew  ...

System Properties

You can pass system properties to Gradle. Remember that command lines, including arguments, can generally be seen with system tools available to regular users. This is likely only useful for local testing.

$ ./gradlew -Daws.accessKeyId=accesskey -Daws.secretKey=secretkey ...

Gradle Configuration

In addition to the AWS SDK, the plugin will also take credentials as project properties.

$ ./gradlew -PawsAccessKeyId=accesskey -PawsSecretAccessKey=secretkey ...

Docker Tasks

All Docker tasks such as DockerPullImage, DockerPushImage, etc. that are configured with the ECR registry URL will get a temporary ECR token. No further configuration is necessary. It is possible to set the registry URL for individual tasks. For those tasks with a registry that is not ECR, the username and password will not be set with an ECR token.

Conclusion

AWS ECR is a good private registry. Following the AWS pricing model, billing is per use for storage and data. However, the temporary tokens are a challenge that is neatly solved with the gradle-aws-ecr-plugin.

If you’d like to contribute to the plugin, either by filing bugs or enhancements, or a PR, see https://bitbucket.org/double16/gradle-aws-ecr-plugin.

Links

About the Author

Patrick Double profile.

Patrick Double

Principal Technologist

I have been coding since 6th grade, circa 1986, professionally (i.e. college graduate) since 1998 when I graduated from the University of Nebraska-Lincoln. Most of my career has been in web applications using JEE. I work the entire stack from user interface to database.   I especially like solving application security and high availability problems.

One thought on “AWS ECR with Gradle

  1. Uri Shohet says:

    Hi Patrick,

    I’m running “./gradlew –no-daemon -Dorg.gradle.debug=true -PawsAccessKeyId=accessKey -PawsSecretAccessKey=secretKey populateECRCredentials” and I’m getting errors:

    :populateECRCredentials FAILED

    FAILURE: Build failed with an exception.

    * What went wrong:
    Some problems were found with the configuration of task ‘:populateECRCredentials’.
    > No value has been specified for property ‘registryUrl’.
    > No value has been specified for property ‘registryId’.

    — build.gradle —
    plugins {
    id “com.bmuschko.docker-remote-api” version “3.0.7”
    id “com.patdouble.awsecr” version “0.3.3”
    }
    docker {
    registryCredentials {
    url = ‘https://123456789012.dkr.ecr.us-east-1.amazonaws.com’ // my Anazon ID here of course
    }
    }
    import com.bmuschko.gradle.docker.tasks.image.DockerTagImage
    task tagWorkerImage(type: DockerTagImage) {
    force = true
    tag = ‘latest’
    imageId = ‘myImageName’
    repository = “123456789012.dkr.ecr.us-east-1.amazonaws.com/myImageName”
    }
    — build.gradle —

    I tried to debug it in IntelliJ – but it doesn’t stop on breakpoints for some reason.

    Could you please help me with this?

    Regards,
    Uri

    1. Uri Shohet says:

      Forgot to mention that I’m using Gradle 3.5, but when I tried 4.0 it didn’t help

    2. Please create an issue at https://bitbucket.org/double16/gradle-aws-ecr-plugin/issues and include the above info. Thanks.

    3. Uri Shohet says:

      I got it working.
      I was missing 2 things:
      * tagImage task is not RegistryCredentialsAware, so I had to add again pushWorkerImage(type: DockerPushImage)
      * in my previous attempts to run pushWorkerImage the imageName and tag were wrong – the imageName didn’t contain the AWS URL and tag did.

      I guess more informative error message would help 🙂

      Thanks for your efforts,
      Uri

      1. Akshay says:

        Hi Uri,

        I am stuck with the same problem.

        Can you share the format of the the imageName and tag you are currently using?

        I am getting below error after using DockerPushImage with imageName containing AWS url.

        Local repo:
        REPOSITORY TAG IMAGE ID CREATED SIZE
        cont-deploy-test latest e71a2a1e8004 21 seconds ago 326MB

        task dockerPushImage(type: DockerPushImage) {
        dependsOn dockerBuildImage
        imageName = “https://058029036333.dkr.ecr.eu-west-2.amazonaws.com/cont-deploy-test”
        tag = “cont-deploy-test:latest”
        }

        ERROR:
        Execution failed for task ‘:dockerPushImage’.
        > com.github.dockerjava.core.exception.InvalidRepositoryNameException (no error message)

        Thanks,
        Akshay

        1. Akshay, your ‘tag’ should not contain a colon (:). So the task should look like:
          task dockerPushImage(type: DockerPushImage) {
          dependsOn dockerBuildImage
          imageName = “https://058029036333.dkr.ecr.eu-west-2.amazonaws.com/cont-deploy-test”
          tag = “latest”
          }

          1. Akshay says:

            Patrick,

            I tried making above mentioned modifications but, still getting the same error message.
            Do I have to follow any specific naming convention while naming local image or AWS ECR repository?

            Thanks

Leave a Reply

Your email address will not be published.

Related Blog Posts
Natively Compiled Java on Google App Engine
Google App Engine is a platform-as-a-service product that is marketed as a way to get your applications into the cloud without necessarily knowing all of the infrastructure bits and pieces to do so. Google App […]
Building Better Data Visualization Experiences: Part 2 of 2
If you don't have a Ph.D. in data science, the raw data might be difficult to comprehend. This is where data visualization comes in.
Unleashing Feature Flags onto Kafka Consumers
Feature flags are a tool to strategically enable or disable functionality at runtime. They are often used to drive different user experiences but can also be useful in real-time data systems. In this post, we’ll […]
A security model for developers
Software security is more important than ever, but developing secure applications is more confusing than ever. TLS, mTLS, RBAC, SAML, OAUTH, OWASP, GDPR, SASL, RSA, JWT, cookie, attack vector, DDoS, firewall, VPN, security groups, exploit, […]