Debug programming issues like you cross game levels

How I fixed aws SignatureDoesNotMatch error while setting up an application

Photo by Spikeball on Unsplash

It all started here.

I was trying to setup cortex metrics and was using the below config for alertmanager.

alertmanager:
external_url: http://random.com/alertmanager
enable_api: true
storage:
type: s3
s3:
endpoint: 172.17.0.1:9000
bucketnames: cortex/am
access_key_id: admin
secret_access_key: admin
signature_version: v4
insecure: true

The approach followed here in general is similar to how you debug any issue and not specific to cortex in itself.

Issue:

Cortex was failing to start with error SignatureDoesNotMatch.
During cortex start, cortex initialises alertmanager and which was fetching bucket info from minio to setup alerts and minio was throwing the error.

I was using minio as s3 store.So as we go through I will be using the terms s3 or minio synonymously.

Now lets debug this issue like how you cross levels in a game

Basic checks

The services are up and we are using right defaults

minio in itself was available on 172.17.0.1:9000 and I was able to create a bucket using minio cli client.The bucket policy is public for testing.

Error origin

2021-08-11T01:49:10:000 [403 Forbidden] s3.ListObjectsV2 172.17.0.1/cortex/am?delimiter=&list-type=2&prefix=alerts%2F  172.30.0.1        341µs       ↑ 93 B ↓ 643 B

The above is minio’s access log when cortex starts.
Tried using an incorrect access key in the alertmanager s3 config and got below error.It means request was reaching minio and its throwing the error.

The access key ID you provided does not exist in our records

Possible Cause

As the error was related to aws signature failure, I ran through the request fields which were used for aws signature generation.

  • Region was ruled out as us-east-1 was the default at minio and the same was explicitly specified in alertmanager config.
  • At minio the default signature is v4 and was able to verify that.

So most probably some fields such as request path or host or query params were causing the issue as these fields are used in signature generation and verification ?

Trace Request

I verified that through aws cli I was able to create and list buckets in minio.So even aws cli was working fine.

AWS_PROFILE=s3-store aws --debug --endpoint-url http://172.17.0.1:9000 s3 ls s3://cortex/am

Kool !

So lets compare the request from aws cli and from cortex.For this instead of actual minio endpoint I started nc listener on a random port(8000) and pointed aws cli endpoint and alertmanager s3 endpoint to this fake minio.

nc -l 8000
  • cortex request to fake minio
Logs from cortex to fake minio
GET /cortex%2Fam?delimiter=&list-type=2&prefix=alerts%2F HTTP/1.1
Host: my-system-ip:8000
User-Agent: aws-sdk-go/1.35.31 (go1.16.2; linux; amd64)
Authorization: AWS4-HMAC-SHA256 Credential=admin/20210810/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=3ebb08e65c83270ea50f4ef0ec7ae74d26b196cb1f706ec495c540496f787e79
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20210810T201626Z
Accept-Encoding: gzip
  • aws cli request to fake minio
Command
AWS_PROFILE=s3-store aws --debug --endpoint-url http://localhost:8000 s3 ls s3://cortex/am
aws cli to fake minio Logs
GET /cortex?list-type=2&prefix=am&delimiter=%2F&encoding-type=url HTTP/1.1
Host: 192.168.43.205:10920
Accept-Encoding: identity
User-Agent: aws-cli/2.0.45 Python/3.7.3 Linux/5.4.0-72-generic exe/x86_64.ubuntu.18 command/s3.ls
X-Amz-Date: 20210810T201104Z
X-Amz-Content-SHA256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Authorization: AWS4-HMAC-SHA256 Credential=admin/20210810/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=ef09dc9ac293f65a92ed45f87c3758251127d8f7c893982db4ec0906f2cb222d

So if you see except path both the logs are similar in most aspects (aws signature fields etc)

The major difference looks like the path.

cortex: /cortex%2Fam
aws cli: /cortex

Travel in the direction of the error :-)

Now that we have some info from the trace, I used cortex as the bucket name instead of cortex/am in alertmanager s3 config and it worked.Hoy ……

So when cortex s3 bucketname has / , the SignatureDoesNotMatch error was thrown.

Lets level up (part 2)

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store