Debug programming issues like you cross game levels
How I fixed aws SignatureDoesNotMatch error while setting up an application
It all started here.
level=error ts=2021–08–10T15:37:08.952533603Z caller=cortex.go:401 msg=”module failed” module=alertmanager err=”invalid service state: Failed, expected: Running, failure: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.\n\tstatus code: 403, request id: 1699FC02BBE5816F, host id: “
I was trying to setup cortex metrics and was using the below config for alertmanager.
The approach followed here in general is similar to how you debug any issue and not specific to cortex in itself.
Cortex was failing to start with error SignatureDoesNotMatch.
During cortex start, cortex initialises alertmanager and which was fetching bucket info from minio to setup alerts and minio was throwing the error.
I was using minio as s3 store.So as we go through I will be using the terms s3 or minio synonymously.
Now lets debug this issue like how you cross levels in a game
The services are up and we are using right defaults
minio in itself was available on 172.17.0.1:9000 and I was able to create a bucket using minio cli client.The bucket policy is public for testing.
2021-08-11T01:49:10:000 [403 Forbidden] s3.ListObjectsV2 172.17.0.1/cortex/am?delimiter=&list-type=2&prefix=alerts%2F 172.30.0.1 341µs ↑ 93 B ↓ 643 B
The above is minio’s access log when cortex starts.
Tried using an incorrect access key in the alertmanager s3 config and got below error.It means request was reaching minio and its throwing the error.
The access key ID you provided does not exist in our records
As the error was related to aws signature failure, I ran through the request fields which were used for aws signature generation.
- Region was ruled out as us-east-1 was the default at minio and the same was explicitly specified in alertmanager config.
- At minio the default signature is v4 and was able to verify that.
So most probably some fields such as request path or host or query params were causing the issue as these fields are used in signature generation and verification ?
I verified that through aws cli I was able to create and list buckets in minio.So even aws cli was working fine.
AWS_PROFILE=s3-store aws --debug --endpoint-url http://172.17.0.1:9000 s3 ls s3://cortex/am
So lets compare the request from aws cli and from cortex.For this instead of actual minio endpoint I started nc listener on a random port(8000) and pointed aws cli endpoint and alertmanager s3 endpoint to this fake minio.
nc -l 8000
- cortex request to fake minio
Logs from cortex to fake minio
GET /cortex%2Fam?delimiter=&list-type=2&prefix=alerts%2F HTTP/1.1
User-Agent: aws-sdk-go/1.35.31 (go1.16.2; linux; amd64)
Authorization: AWS4-HMAC-SHA256 Credential=admin/20210810/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=3ebb08e65c83270ea50f4ef0ec7ae74d26b196cb1f706ec495c540496f787e79
- aws cli request to fake minio
AWS_PROFILE=s3-store aws --debug --endpoint-url http://localhost:8000 s3 ls s3://cortex/amaws cli to fake minio Logs
GET /cortex?list-type=2&prefix=am&delimiter=%2F&encoding-type=url HTTP/1.1
User-Agent: aws-cli/2.0.45 Python/3.7.3 Linux/5.4.0-72-generic exe/x86_64.ubuntu.18 command/s3.ls
Authorization: AWS4-HMAC-SHA256 Credential=admin/20210810/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=ef09dc9ac293f65a92ed45f87c3758251127d8f7c893982db4ec0906f2cb222d
So if you see except path both the logs are similar in most aspects (aws signature fields etc)
The major difference looks like the path.
aws cli: /cortex
Travel in the direction of the error :-)
Now that we have some info from the trace, I used cortex as the bucket name instead of cortex/am in alertmanager s3 config and it worked.Hoy ……
So when cortex s3 bucketname has / , the SignatureDoesNotMatch error was thrown.
Lets level up (part 2)