S3 bucketnaming rules don’t allow slashes.
The key names can have slashes and that is how the logical folder structure is achieved.
Bucketnames should not have slashes.
I wanted to pass a subfolder to alertmanager s3 config and see how that works and that ended up in this issue !
Ok now that we have fixed the actual cortex server startup issue lets move onto the unintended error that came in our way !
Next battle -> SignatureDoesNotMatch, but why ?
Now lets summarise where we started.
Ok as we have seen earlier from minio access logs that alertmanager was invoking ListObjectsV2 api on startup.That endpoint was throwing SignatureDoesNotMatch error and which caused alertmanager failure.
So now we have to debug why that is occuring :-)
Lets try hitting the same api from aws cli, aws sdk for js and asws sdk for golang and see the behaviour.
- AWS Cli request
With aws cli, you can use the bucketpath for listing the objects and cli takes care of splitting the path into bucketname and prefix
$ AWS_PROFILE=s3-store aws --endpoint-url http://172.17.0.1:9000 s3 ls s3://cortex/am
PRE am/$ ./mc admin trace s3-store2021-08-14T17:31:10:000 [200 OK] s3.ListObjectsV2 172.17.0.1/cortex?list-type=2&prefix=am&delimiter=%2F&encoding-type=url 172.21.0.1 942µs ↑ 93 B ↓ 605 B
- AWS SDK for Javascript
Output:InvalidBucket: Bucket names cannot contain forward slashes. Bucket: cortex/amat Request.validateBucketName (/../aws-s3-listobjects/node_modules/aws-sdk/lib/services/s3.js:234:30)
So aws sdk for javascript was throwing the error straightaway and the code is here.Kool !
- AWS SDK For Golang
$ AWS_PROFILE=s3-store go run main.go
Got error retrieving list of objects:
operation error S3: ListObjectsV2, https response error StatusCode: 403, RequestID: 169B3166BFB42726, HostID: d173742c-5eb0-4c04-ae7a-a20a2f4cee37, api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.Traced Request by using fake minio (nc -l)GET /cortex%2Fam?list-type=2 HTTP/1.1
Host: localhost:8000
User-Agent: aws-sdk-go-v2/1.8.0 os/linux lang/go/1.16.7 md/GOOS/linux md/GOARCH/amd64 api/s3/1.12.0
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: eae36291-0fea-46ec-9d88-a61fe9259f4b
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=123admin456/20210814//s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date, Signature=8a034b86175d2f7c5404ff29e9837e74260e996b516a127ee124ce2f39a810fb
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20210814T124515Z
So aws sdk for golang is accepting an invalid bucket name and also encoding the request path.So this is the module responsible for the incorrect behaviour.
So are we done yet ?
Next battle -> SDK
So 2 things here
- Why signature error ?
- Behaviour of aws sdk for golang
Kool lets check them in part 3.
Ref
The ListObjectsV2 api expects a bucketname in path though in examples the virtual host style buckets are shown.The prefix is the one which contains the actual key that you are looking for.
