S3 as a Serverless Service

Chapter Comments
1

Link to chapter - https://serverless-stack.com/chapters/s3-as-a-serverless-service.html

2

Hi All,

I am trying to use this s3 serverless service by modifying the note taking app so that it compiles a users notes into a csv file and uploads it into the bucket created when this service was deployed. I have this feature defined as a new service in which the lambda function makes the query call defined in the list function from part I, iterates over the items in the query to push them into an array, joins the array with a ā€œ\nā€ separator, and then should take that string created by joining the elements of the array as the body of an s3.upload() call. The only problem is that it is not making that call.

Here is my function:

async function uploadNotes(csvBody) {
        const csvBuffer = Buffer.from(csvBody);
        let params = {Bucket: 'notes-app-mono-uploads-dev-mycsvbucket', Key: 'private/' + event.requestContext.identity.cognitoIdentityId + '/notes.csv', Body: csvBuffer};
        console.log("s3");
        console.log("csvBuffer: \n" + csvBuffer);
        s3.upload(params, function(err, data) {
            console.log("s3.upload function should be happening here");
            if (err) console.log(err, err.stack); // an error occurred
            else console.log(data);           // successful response
        })
    }

I set up a mock like the list mock from part I to test this. The output returns shows the ā€œs3ā€ log and the csvBuffer output with all the notes from my test user but the ā€œs3.upload function should be happening hereā€ console.log, the err console.log and the data console.log do not show up on the output.

My try block:

try {
         const result = await dynamoDbLib.call("query", dynamoParams);
         const csvString = await gatherNotes(result.Items);
        uploadNotes(csvString);
        return success({ status: true });
    } catch (e) {
        console.log(e);
        return failure({ status: false });
    }

After running the invoke local command to test this, the output also shows that it has return ā€œstatus: trueā€ object indicating that the lambda call was successful.

I am somewhat new to serverless but I feel like this should work. My auth service is set up to allow the authorized user to upload to the bucket just like it allows the same user to query the notes table. The query works but the upload doesnā€™t.

Anyone have any idea what I am missing? Any help would be much appreciated.

All the best,

Aaron

3

Iā€™m not entirely sure what is going on here but donā€™t you need an await on the s3.upload call?

4

Hey Jay!

Sorry about the word soup. Guess I should proofread before I post, amirite? Anyway, you were correct. That was the problem. I needed to add async in front of s3.upload. I also needed to add .promise() after s3.upload(params) call.

I ended up writing the bad code as follows:

async function uploadNotes(csvBody) {
    let csvBuffer = Buffer.from(csvBody);
    let params = {
        ACL: "public-read",
        Bucket: process.env.notesBucket,
        Key: 'private/' + event.requestContext.identity.cognitoIdentityId + '/notes.csv',
        Body: csvBuffer
    };
    try {
        const result = await s3.upload(params).promise();
        console.log(result);
    } catch (e) {
        console.log(e);
    }
}

I was able to find more help at forum.serverless.com.

Anyway, thanks again!

1 Like
5

Glad you figured it out!

6

Are you able to dynamically set environment variables in serverless? For example setting an S3_BUCKET variable where the value is Fn::ImportValue: ${self:custom.stage}-mybucket

7

Yeah AFAIK, you should be able to?

8

Hi @jayair

Iā€™ve tried doing what I described but the variables end up being undefined.

For example I wanted to access an imported SNS topic name in my code by setting it as an environment variable like this:

environment:
    UPLOAD_NOTIFICATION_TOPIC_ARN:
         Fn::ImportValue: ${self:custom.stage}-UploadsProcessingNotificationArn

Am I doing something wrong?

9

Iā€™m going to answer my own question. I was using the wrong ā€˜envrionmentā€™ section of the serverless yml file. To use imported values in your envrionment variables you need to be under provider/envrionment.

provider:
   name: aws
   region: eu-central-1
   environment:
       UPLOAD_NOTIFICATION_TOPIC_ARN:
            Fn::ImportValue: ${self:custom.stage}-UploadsProcessingNotificationArn

Hope this helps someone :slight_smile:

1 Like
10

Glad you figured it out!