Debug programming issues like you cross game levels — Part 2
Bucketnames should not have slashes.
I wanted to pass a subfolder to alertmanager s3 config and see how that works and that ended up in this issue !
Ok now that we have fixed the actual cortex server startup issue lets move onto the unintended error that came in our way !
Next battle -> SignatureDoesNotMatch, but why ?
Now lets summarise where we started.
Ok as we have seen earlier from minio access logs that alertmanager was invoking ListObjectsV2 api on startup.That endpoint was throwing SignatureDoesNotMatch error and which caused alertmanager failure.
So now we have to debug why that is occuring :-)
Lets try hitting the same api from aws cli, aws sdk for js and asws sdk for golang and see the behaviour.
- AWS Cli request
With aws cli, you can use the bucketpath for listing the objects and cli takes care of splitting the path into bucketname and prefix
$ AWS_PROFILE=s3-store aws --endpoint-url http://172.17.0.1:9000 s3 ls s3://cortex/am
PRE am/$ ./mc admin trace s3-store2021-08-14T17:31:10:000 [200 OK] s3.ListObjectsV2 172.17.0.1/cortex?list-type=2&prefix=am&delimiter=%2F&encoding-type=url 172.21.0.1 942µs ↑ 93 B ↓ 605 B
Output:InvalidBucket: Bucket names cannot contain forward slashes. Bucket: cortex/amat Request.validateBucketName (/../aws-s3-listobjects/node_modules/aws-sdk/lib/services/s3.js:234:30)
- AWS SDK For Golang
$ AWS_PROFILE=s3-store go run main.go
Got error retrieving list of objects:
operation error S3: ListObjectsV2, https response error StatusCode: 403, RequestID: 169B3166BFB42726, HostID: d173742c-5eb0-4c04-ae7a-a20a2f4cee37, api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.Traced Request by using fake minio (nc -l)GET /cortex%2Fam?list-type=2 HTTP/1.1
User-Agent: aws-sdk-go-v2/1.8.0 os/linux lang/go/1.16.7 md/GOOS/linux md/GOARCH/amd64 api/s3/1.12.0
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=123admin456/20210814//s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date, Signature=8a034b86175d2f7c5404ff29e9837e74260e996b516a127ee124ce2f39a810fb
So aws sdk for golang is accepting an invalid bucket name and also encoding the request path.So this is the module responsible for the incorrect behaviour.
So are we done yet ?
Next battle -> SDK
So 2 things here
- Why signature error ?
- Behaviour of aws sdk for golang
Kool lets check them in part 3.