Device onboarding to AWS IoT using Virtual Private Cloud endpoints

Introduction

If you operate secure private networks—such as an assembly line’s operational technology (OT) network at a factory or government agency­—and intend to connect your devices to AWS, then you need to use X.509 client certificates for authenticating requests to AWS services—all while staying within the Virtual Private Cloud (VPC). In this post, we will demonstrate how to use the new capability, VPC endpoint (VPCe) for AWS IoT Core credential provider to manage component deployments into AWS IoT Greengrass-powered gateways running on private networks. Similarly, we will also demonstrate how to develop and connect devices that use AWS IoT Device SDK and require exchange of X.509 client certificates for security tokens to interact with other AWS services.

AWS IoT Core is a fully managed service that supports connectivity for billions of devices. Devices and software clients that connect to AWS IoT Core rely on X.509 certificates for authentication. However, other AWS services rely on security tokens to authenticate API calls. To avoid hardcoding credentials and/or tokens in the clients, AWS IoT Core provides AWS IoT Core credential provider endpoint. This endpoint allows clients, like AWS IoT Greengrass components, to exchange their X.509 certificates as security tokens to interact with other AWS services. Moreover, with AWS IoT Core’s credential provider endpoint capability, you can extend your operational networks to a VPC in AWS via Virtual Private Network (VPN), thus eliminating the need for your equipment to use public internet access to reach AWS IoT Core’s credential provider.

How customers are benefiting from the new capability

Pre-requisites

• Administrator access to an AWS account
• Basic AWS CLI skills
• Basic AWS IoT Greengrass and AWS IoT Core knowledge

Solution architecture

The following architecture represents a typical IoT infrastructure where devices using on-premises operational networks connect to AWS through a private network.

Walkthrough

In this walkthrough you will learn how to use AWS IoT Core credential provider to enable a VPCe connection either for AWS IoT Greengrass or IoT devices developed using the AWS IoT Device SDK. 

Note that you must implement the section “Create VPC endpoints” for both cases.

AWS IoT Greengrass

Create VPC endpoints

To establish an edge-to-cloud communication link completely over VPC, you must first setup AWS Direct Connect between your on-premises network infrastructure and your AWS VPC. For detailed implementation, please refer to AWS Direct Connect developer guide.

Once AWS Direct Connect is setup, there are 3 VPC Endpoints required for an IoT gateway to be provisioned, managed, and synchronized (component deployments) as an AWS IoT Greengrass gateway.

• AWS IoT Greengrass
• AWS IoT Core data
• AWS IoT Core credential provider

The AWS IoT Greengrass endpoint (com.amazonaws.[region].greengrass) is used to manage components, deployments, and devices from the AWS IoT Greengrass cloud service. Authentication and authorization with this endpoint are done using X.509 certificates as described in Device authentication and authorization for AWS IoT Greengrass.

The AWS IoT Core data endpoint (com.amazonaws.[region].iot.data) is used for interactions between AWS IoT Greengrass components and AWS IoT Core by publishing/subscribing to AWS IoT Core MQTT broker. Authentication and authorization with this endpoint is also done using X.509 certificates.

The AWS IoT Core credential provider endpoint (com.amazonaws.[region].iot.credentials) is used to communicate with other AWS services that do not support X.509 authentication and authorization, such as Amazon Simple Storage Service (Amazon S3) and Amazon Elastic Container Registry (Amazon ECR). In either case, the devices developed using the AWS IoT SDK or an AWS IoT Greengrass component, will call the AWS IoT Core credential provider endpoint using the X.509 certificate to authenticate and get authorized. The endpoint will then issue a temporary security token for the client to use in the call to the services not supporting X.509.

Calls to Amazon S3 and Amazon ECR are required during AWS IoT Greengrass component deployments as described in the following sequence diagram. The AWS IoT Greengrass component will also require a security token if it uses AWS SDKs to communicate with other cloud services not supporting X.509 certificates authentication and authorization.

Then, to accomplish an end-to-end VPCe communication, you must create these 3 VPC endpoints pointing to one or more private subnets selected to route traffic to and from AWS.

Each VPC endpoint will get an Internet Protocol (IP) address, one for each private subnet where the VPCe is created. We recommend selecting at least 2 subnets for high availability.

You can use the following AWS CLI commands to create the 3 VPC endpoints passing your private subnet as parameters or use the AWS Console to create them.

All following commands are for us-east-1 region.

1. Create VPC endpoints

aws ec2 create-vpc-endpoint --vpc-id  --vpc-endpoint-type Interface --service-name com.amazonaws.us-east-1.iot.credentials | jq -r ".VpcEndpoint.VpcEndpointId"
aws ec2 create-vpc-endpoint --vpc-id  --vpc-endpoint-type Interface --service-name com.amazonaws.us-east-1.greengrass | jq -r ".VpcEndpoint.VpcEndpointId"
aws ec2 create-vpc-endpoint --vpc-id  --vpc-endpoint-type Interface --service-name com.amazonaws.us-east-1.iot.data | jq -r ".VpcEndpoint.VpcEndpointId"

2. Get security groups associated with the VPC endpoints

Using the VPC endpoints IDs returned by the commands implemented in the step 1, apply the following command to get the security groups associated with them.

aws ec2 describe-vpc-endpoints --vpc-endpoint-ids <.iot.credentials VpcEndpointId> | jq -r ".VpcEndpoints[0].Groups[0].GroupId"
aws ec2 describe-vpc-endpoints --vpc-endpoint-ids <.greengrass VpcEndpointId> | jq -r ".VpcEndpoints[0].Groups[0].GroupId"
aws ec2 describe-vpc-endpoints --vpc-endpoint-ids <.iot.data VpcEndpointId> | jq -r ".VpcEndpoints[0].Groups[0].GroupId"

The security groupId will most likely be the same for all your VPC endpoints. If that is the case, you can run steps 3 and 4 just once.

3. Add ingress rule to the security groups

Using the Security Groups IDs returned by the commands in the step 2, run the following command to allow ingress communication from the IP range where your AWS IoT Greengrass device will be running.

In this blog post, for simplicity, we allow ingress from anywhere (0.0.0.0), but we recommend you limit the ingress to the CidrIp where your AWS IoT Greengrass device will be deployed.

aws ec2 authorize-security-group-ingress --group-id <.iot.credentials GroupID> --protocol tcp --cidr 0.0.0.0/0 --port 443
aws ec2 authorize-security-group-ingress --group-id <.greengrass GroupID> --protocol tcp --cidr 0.0.0.0/0 --port 443
aws ec2 authorize-security-group-ingress --group-id <.iot.data GroupID> --protocol tcp --cidr 0.0.0.0/0 --port 443

4. Add egress rule to the security groups

Using the Security Groups IDs returned by the commands executed in step 2, run the following command to allow egress communication to the IP range where your AWS IoT Greengrass device will be running.

In this blog post, for simplicity, we allow egress to anywhere (0.0.0.0), but we recommend you limit the egress to the CidrIp where your Greengrass device will be deployed.

aws ec2 authorize-security-group-egress --group-id <.iot.credentials GroupID> --protocol tcp --cidr 0.0.0.0/0 --port 443
aws ec2 authorize-security-group-egress --group-id <.greengrass GroupID> --protocol tcp --cidr 0.0.0.0/0 --port 443
aws ec2 authorize-security-group-egress --group-id <.iot.data GroupID> --protocol tcp --cidr 0.0.0.0/0 --port 443

5. Add subnets to the VPC endpoints

Once the VPC endpoints have being created and configured, you can follow the Amazon Virtual Public Cloud documentation to add or remove subnets to the VPC endpoint. We recommend configuring at least 2 subnets in each endpoint for HA.

6. Enable DNS support for the VPC

Run the following commands to enable DNS support in the VPC where the VPC endpoints were created.

aws ec2 modify-vpc-attribute --vpc-id  --enable-dns-support "{\"Value\":true}"
aws ec2 modify-vpc-attribute --vpc-id  --enable-dns-hostnames "{\"Value\":true}"

7. Create private hosted zone in Amazon Route 53

Next step is to configure a private hosted zone in Amazon Route 53.

First create a hosted zone for each VPC endpoint.

echo '{
"VPCRegion":"us-east-1",
"VPCId":""
}' > vpc.json

echo '{
"Comment": "PrivateZoneForVPCe",
"PrivateZone": true
}' > hostedzoneconfig.json

aws route53 create-hosted-zone --name credentials.iot.us-east-1.amazonaws.com --vpc file://vpc.json --caller-reference  --hosted-zone-config file://hostedzoneconfig.json |jq -r ".HostedZone.Id"|sed 's?^.*hostedzone/??g'
aws route53 create-hosted-zone --name iot.us-east-1.amazonaws.com --vpc file://vpc.json --caller-reference  --hosted-zone-config file://hostedzoneconfig.json |jq -r ".HostedZone.Id"|sed 's?^.*hostedzone/??g'

You can use the following online tool to generate the required UUIDs https://www.uuidgenerator.net/version1

Then you get:

a. VPC endpoints addresses

aws iot describe-endpoint --endpoint-type iot:CredentialProvider | jq -r '.endpointAddress'
aws iot describe-endpoint --endpoint-type iot:Data-ATS | jq -r '.endpointAddress'

b. VPC endpoints DNS names

aws ec2 describe-vpc-endpoints --vpc-endpoint-ids <.iot.credentials VpcEndpointId FROM STEP 1> | jq -r ".VpcEndpoints[0].DnsEntries[0].DnsName"
aws ec2 describe-vpc-endpoints --vpc-endpoint-ids <.iot.data VpcEndpointId FROM STEP 1> | jq -r ".VpcEndpoints[0].DnsEntries[0].DnsName"

c. Hosted Zone IDs

aws ec2 describe-vpc-endpoints --vpc-endpoint-ids <.iot.credentials VpcEndpointId FROM STEP 1> | jq -r ".VpcEndpoints[0].DnsEntries[0].HostedZoneId"
aws ec2 describe-vpc-endpoints --vpc-endpoint-ids <.iot.data VpcEndpointId FROM STEP 1> | jq -r ".VpcEndpoints[0].DnsEntries[0].HostedZoneId"

With the information from sections 7.a, 7.b, and 7.c, you can create the DNS record in the Private Hosted Zones.

Credential endpoint:

echo '{
"Comment": "Route traffic from Credential default endpoint to VPCe.",
"Changes": [
{
"Action": "UPSERT",
"ResourceRecordSet": {
"Name": "",
"DNSName": "",
"EvaluateTargetHealth": true
}
}
}
]
}' > credentialvpce.json

aws route53 change-resource-record-sets --hosted-zone-id  --change-batch file://credentialvpce.json

IMPORTANT

HostedZoneId in the credentialvpce.json is the ID you can find in the credentials VPCe console definition between parenthesis under the DNS Names section. It is also returned by the command “aws ec2 describe-vpc-endpoints —vpc-endpoint-ids <.iot.credentials VpcEndpointId FROM STEP 1> ……” from section 7.c

–hosted-zone-id parameter in the AWS CLI command is the Hosted Zone Id returned by the command “aws route53 create-hosted-zone —name credentials.us-east-1.iot.amazonaws.com …….” You can find this ID also in the Amazon Route 53 console, by selecting your hosted zone and navigating to the “Hosted zone details” section.

Data endpoint:

echo '{
"Comment": "Route traffic from IoT Core default data endpoint to VPCe.",
"Changes": [
{
"Action": "UPSERT",
"ResourceRecordSet": {
"Name": "",
"DNSName": "",
"EvaluateTargetHealth": true
}
}
}
]
}' > datavpce.json

aws route53 change-resource-record-sets --hosted-zone-id  --change-batch file://datavpce.json

Deploy and connect your AWS IoT Greengrass core device

You can now follow any of the supported methods to configure your AWS IoT Greengrass core device. In this scenario, you are most likely working in an environment without internet access to download packages into the machine where you are going to install AWS IoT Greengrass nucleus. We recommend following Install with manual provisioning.

Before running the install command:

sudo -E java -Droot="/greengrass/v2" -Dlog.store=FILE \
-jar ./GreengrassInstaller/lib/Greengrass.jar \
--init-config ./GreengrassInstaller/config.yaml \
--component-default-user ggc_user:ggc_group \
--setup-system-service true

You have to check your “./GreengrassInstaller/config.yaml” looks like the following:

system:
certificateFilePath: "/greengrass/v2/[your thing cert]"
privateKeyPath: "/greengrass/v2/[your thing private key]"
rootCaPath: "/greengrass/v2/AmazonRootCA1.pem"
rootpath: "/greengrass/v2"
thingName: "[your thing name]"
services:
aws.greengrass.Nucleus:
componentType: "NUCLEUS"
version: "[your NUCLEUS version]"
configuration:
awsRegion: "us-east-1"
iotRoleAlias: "[your GreengrassCoreTokenExchangeRoleAlias]"
iotDataEndpoint: "[the 'A' record created in Route53 for IoT Data Enpoint]"
iotCredEndpoint: "[the 'A' record created in Route53 for IoT Credential Endpoin]"
mqtt:
port: 443
greengrassDataPlaneEndpoint: "iotdata"
greengrassDataPlanePort: 443

Validate you are using VPCe

To validate you are using the VPCe and not traversing the public internet, run the following command from the AWS IoT Greengrass core machine:

nslookup [the 'A' record created in Route53 for IoT Credential Endpoin]
nslookup [the 'A' record created in Route53 for IoT Data Enpoint]

The IP addresses returned by the above commands should be in the range of the subnets associated with your VPC endpoints.

Then, you can run the same commands from your laptop. You should get public IPs for the endpoints.

AWS IoT devices

After you have implemented the steps in section “Create VPC endpoints,” you can also use AWS IoT Core credential provider through VPCe, directly from AWS IoT Core things running in your VPC. To test this option, follow the next steps.

1. Follow the step by step wizard in the console to create a single thing.

2. Follow the Authorizing direct calls to AWS services using AWS IoT Core credential provider documentation to create the required roles and policies, and attach them to the certificate created in the previous step.

3. Move the connect_device_package.zip file created during the wizard execution into the device where you are planning to use the AWS IoT Device SDK. For testing this, you can use an Amazon EC2 instance. In any case, confirm the device or EC2 instance is connected to a network or subnet within the VPC where the VPC endpoint was created. You can validate this running the following commands from the device or EC2 instance.

nslookup [the 'A' record created in Route53 for IoT Credential Endpoin]
nslookup [the 'A' record created in Route53 for IoT Data Enpoint]

The IP addresses returned by the above commands should be in the range of the subnets associated with your VPCe endpoints.

4. Run the following CURL command from the Authorizing direct calls to AWS services using AWS IoT Core credential provider documentation, pointing to the “A” record created in Amazon Route 53 in the credentials.iot.us-east-1.amazonaws.com private zone. It will look like your_aws_account_specific_prefix.credentials.iot.us-east-1.amazonaws.com

curl --cert your certificate --key your device certificate key pair -H "x-amzn-iot-thingname: your thing name" --cacert AmazonRootCA1.pem https://your endpoint /role-aliases/your role alias/credentials

Conclusion

With the new VPCe support for AWS IoT Core credential provider, you can now have end-to-end VPC communication between devices—either directly through AWS IoT Device SDKs or AWS IoT Greengrass—without needing to set up network proxies and complex firewall configurations. This simplified network infrastructure can help you reduce operational overhead costs and improve the security posture of your solution. To learn more, go to AWS IoT Greengrass and interface VPC endpoints (AWS PrivateLink)

About the authors

Vladi Salomon is a Principal IoT Data Architect with Amazon Web Services. He has 7+ years of experience in IoT architecture in different vertical like IIoT, Smart Home, Smart City and Mining as well as data warehousing and big data platform. In the latest years he got focus in how to bring AI to IoT through scalable MLOps platforms. As a member of AWS Professional Services, He works with customers of different scale and industries architecting and implementing a variety of end-to-end IoT solutions.

Victor Lesau is a Sr. Technical Product Manager at Amazon Web Services. He focuses on product strategy, roadmap planning, business analysis, customer engagement, and other product management areas of AWS IoT Core, AWS IoT Identity, and smart home initiatives.

Ben Omer is a Sr. Technical Product Manager at Amazon Web Services. He works with IoT device services and products including FreeRTOS, AWS IoT Greengrass, and KVS, with a focus on roadmap planning, feature exploration, and customer engagement.