Monitoring Deployments in Seed

[jayair]

Link to chapter - https://serverless-stack.com/chapters/monitoring-deployments-in-seed.html

[benwalker]

During Seed deployments, I am getting failures when promoting and when manually triggering a build.

Log Output

Build v4 started…
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed

0 0 0 0 0 0 0 0 --:–:-- --:–:-- --:–:-- 0
100 9507k 100 9507k 0 0 37.6M 0 --:–:-- --:–:-- --:–:-- 37.7M

====================
️ Init

2018/06/21 03:02:53 Loading production build

2018/06/21 03:02:55 Using Serverless Framework version 1.26.0

====================
Compile

2018/06/21 03:02:55 Installing Node Modules

$ yarn
yarn install v1.3.2
[1/4] Resolving packages…
[2/4] Fetching packages…
info fsevents@1.2.4: The platform “linux” is incompatible with this module.
info “fsevents@1.2.4” is an optional dependency and failed compatibility check. Excluding it from installation.
[3/4] Linking dependencies…
[4/4] Building fresh packages…
Done in 13.61s.

====================
Build

2018/06/21 03:03:09 Applying secret environment variables

====================
Deploy

2018/06/21 03:03:09 Deploying to: prod

$ SLS_DEBUG=* serverless deploy --stage prod --package promote-artifacts --force
Serverless: Load command run
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command create
Serverless: Load command install
Serverless: Load command package
Serverless: Load command deploy
Serverless: Load command deploy:function
Serverless: Load command deploy:list
Serverless: Load command deploy:list:functions
Serverless: Load command invoke
Serverless: Load command invoke:local
Serverless: Load command info
Serverless: Load command logs
Serverless: Load command login
Serverless: Load command logout
Serverless: Load command metrics
Serverless: Load command print
Serverless: Load command remove
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command slstats
Serverless: Load command plugin
Serverless: Load command plugin
Serverless: Load command plugin:install
Serverless: Load command plugin
Serverless: Load command plugin:uninstall
Serverless: Load command plugin
Serverless: Load command plugin:list
Serverless: Load command plugin
Serverless: Load command plugin:search
Serverless: Load command emit
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command webpack
Serverless: Load command offline
Serverless: Load command offline:start

Serverless Warning --------------------------------------

A valid file to satisfy the declaration ‘file(env.yml):prod,file(env.yml):default’ could not be found.

Serverless Warning --------------------------------------

A valid file to satisfy the declaration ‘file(env.yml):prod,file(env.yml):default’ could not be found.

Serverless Warning --------------------------------------

A valid service attribute to satisfy the declaration ‘self:custom.environment.stripeSecretKey’ could not be found.

Serverless: Invoke deploy
Serverless: Invoke aws:common:validate
Serverless: Invoke aws:common:moveArtifactsToTemp
Serverless: Invoke aws:deploy:deploy
Serverless: Uploading CloudFormation file to S3…
Serverless: Uploading artifacts…
Serverless: Validating template…
Serverless: Updating Stack…
Serverless: Checking Stack update progress…
…
Serverless: Operation failed!

Serverless Error ---------------------------------------

An error occurred: BillingLambdaVersionX2Ou8TdE7c5cYSAIeOaEktqNaA2CSa8DPTmkTNuzFI - A version for this Lambda function exists ( 1 ). Modify the function to create a new version…

Stack Trace --------------------------------------------

ServerlessError: An error occurred: BillingLambdaVersionX2Ou8TdE7c5cYSAIeOaEktqNaA2CSa8DPTmkTNuzFI - A version for this Lambda function exists ( 1 ). Modify the function to create a new version…
at provider.request.then (/sls-1.26.0/node_modules/serverless/lib/plugins/aws/lib/monitorStack.js:112:33)
From previous event:
at AwsDeploy.monitorStack (/sls-1.26.0/node_modules/serverless/lib/plugins/aws/lib/monitorStack.js:26:12)
at provider.request.then (/sls-1.26.0/node_modules/serverless/lib/plugins/aws/lib/updateStack.js:84:30)
From previous event:
at AwsDeploy.update (/sls-1.26.0/node_modules/serverless/lib/plugins/aws/lib/updateStack.js:84:8)
From previous event:
at AwsDeploy.BbPromise.bind.then (/sls-1.26.0/node_modules/serverless/lib/plugins/aws/lib/updateStack.js:101:12)
From previous event:
at AwsDeploy.updateStack (/sls-1.26.0/node_modules/serverless/lib/plugins/aws/lib/updateStack.js:95:8)
From previous event:
at AwsDeploy.BbPromise.bind.then (/sls-1.26.0/node_modules/serverless/lib/plugins/aws/deploy/index.js:135:39)
From previous event:
at Object.aws:deploy:deploy:updateStack [as hook] (/sls-1.26.0/node_modules/serverless/lib/plugins/aws/deploy/index.js:131:10)
at BbPromise.reduce (/sls-1.26.0/node_modules/serverless/lib/classes/PluginManager.js:372:55)
From previous event:
at PluginManager.invoke (/sls-1.26.0/node_modules/serverless/lib/classes/PluginManager.js:372:22)
at PluginManager.spawn (/sls-1.26.0/node_modules/serverless/lib/classes/PluginManager.js:390:17)
at AwsDeploy.BbPromise.bind.then (/sls-1.26.0/node_modules/serverless/lib/plugins/aws/deploy/index.js:101:48)
From previous event:
at Object.deploy:deploy [as hook] (/sls-1.26.0/node_modules/serverless/lib/plugins/aws/deploy/index.js:97:10)
at BbPromise.reduce (/sls-1.26.0/node_modules/serverless/lib/classes/PluginManager.js:372:55)
From previous event:
at PluginManager.invoke (/sls-1.26.0/node_modules/serverless/lib/classes/PluginManager.js:372:22)
at PluginManager.run (/sls-1.26.0/node_modules/serverless/lib/classes/PluginManager.js:403:17)
at variables.populateService.then (/sls-1.26.0/node_modules/serverless/lib/Serverless.js:102:33)
at runCallback (timers.js:794:20)
at tryOnImmediate (timers.js:752:5)
at processImmediate [as _immediateCallback] (timers.js:729:5)
From previous event:
at Serverless.run (/sls-1.26.0/node_modules/serverless/lib/Serverless.js:89:74)
at serverless.init.then (/sls-1.26.0/node_modules/serverless/bin/serverless:42:50)
at

…

Your Environment Information -----------------------------
OS: linux
Node Version: 8.10.0
Serverless Version: 1.26.0

2018/06/21 03:03:52 Error: exec: internal error

====================
Cleanup

2018/06/21 03:03:52 Post deploy cleanup

Do you have any idea what could cause this? I have found a few other github issues referencing this problem with suggestions, but nothing has worked and I’m almost out of deployments on my free tier Seed account.

[jayair]

It seems like a really weird issue that people have run into in the past.

Issue: Serverless deploy fails after update from 1.24.1 to 1.25.0

Did you see this error in your first deploy?

[bscaspar]

I am not seeing the error message details in my Lambda log, instead I see “module initialization error: ReferenceError” as the entire content of text in that log message where the error is in the example, see attached pic. Is this a change in the Seed logging? Or AWS logging? Or could it be where I put the gibberish.what; in my code? (I put it at the very top of the create.js file, first line)

[jayair]

Hmm not sure where the change is. But if you click on the ReferenceError line, what day you see?

[mikeyamato]

I’m seeing the same thing. Not sure why we can’t expand this part any longer.

[mikeyamato]

For those that don’t have a functions folder, place gibberish.what; below the import statements of create.js.

[hmpargi]

I am having same problem. Can’t expand Reference Error. Any Idea ?

[jayair]

Hmm I’m not sure why it’s not expanding. We’ll need to take a look at what’s going on.

[prgtrdr]

I’ve inserted the faulty code, committed and pushed it to dev. Seed then performs a build and gives me an error which is expected.

Serverless: Invoke package
Serverless: Invoke aws:common:validate
Serverless: Invoke aws:common:cleanupTempDir
Serverless: Invoke webpack:validate
Serverless: Invoke webpack:compile
Serverless: Bundling with Webpack...
ERROR in /tmp/seed/source/create.js
Module Error (from /tmp/seed/source/node_modules/eslint-loader/index.js):
/tmp/seed/source/create.js
  4:1  error  'gibberish' is not defined  no-undef
✖ 1 problem (1 error, 0 warnings)
  Error --------------------------------------------------
  Webpack compilation error, see above

But then, following the text, I try to manually promote to prod, and I receive this message:

You are trying to promote an unsuccessful build from the dev stage. Please fix the build and try again.

I can’t seem to find a way to override this, but it seems like the rest of the chapter depends on in being in prod. Has anyone else seen this?

[jayair]

Ah there’s been a change. The error is getting caught by the linter before you are able to push it.

I think we’ll need to try pushing some code that is faulty but doesn’t get caught by the linter. Try this line instead:

uuid.abc.gibberish;

[prgtrdr]

Thanks @jayair. Yes, this works.

I see you’ve already updated the repo with the corrected code. I’ve made some changes to clarify where to place the faulty code and updated the “Revert the Code” section with the corrected code.

Separately, I’m having the same issue reported by others here in that I don’t see the details of the error in the Seed Lambda Log. Here’s a screen shot:

I’m not able to expand the 2nd line, “module initialization error: TypeError” although the cursor does change to a hand pointer.

[jayair]

So this module initialization error usually means that the Lambda function failed to initialize. This is because the faulty code was inserted outside our function instead of inside it.

If you move that line inside the function, you’ll see the error with all the details.

[Elias-Serneels]

I’m wondering what exactly you mean by

Seed keeps track of your past builds and simply uses the previously built package to deploy it again.

Does it use the serverless rollback command internally or does it run serverless deploy again with the previous package?

[jayair]

This behavior has changed a bit but it used redeploy an older package. But that ended up causing some issues for a lot users. So instead we do a deploy with the commit you are rolling back to.

[Elias-Serneels]

Thanks for the reply jayair!

Can you expand on what the possible issues were with redeploying the previously built package?
To me that actually seems like the safer option instead of rebuilding and introducing variability.

[jayair]

Yeah so Serverless Framework does not generate proper build artifacts, they cannot be used across stages for example. And certain plugins might need to be run every time you are doing a deploy. This meant that we couldn’t use the older package for all of our users. Even though the desired behavior is to use the older package.

[Elias-Serneels]

Thank you for those insights jay

[piavgh]

@jayair: At the moment, I can’t even reproduce the 502 request because I can’t promote the faulty code to production stage.
seed.run catches this faulty code automatically. Is that correct?

[RichStone]

Hey,

I have a get.js function like this where I have forgotten to specify a partition key:

import * as dynamoDbLib from "../libs/dynamodb-lib";
import { success, failure } from "../libs/response-lib";

export async function main(event, context) {
  const params = {
    TableName: process.env.tableName,
    Key: {
      ProductId: event.pathParameters.id
    }
  };

  try {
    const result = await dynamoDbLib.call("get", params);
    console.log(result);
    if (result.Item) {
      return success(result.Item);
    } else {
      return failure({ status: false, error: "Item not found." });
    }
  } catch (e) {
    return failure({ status: false, error: e, errorStack: e.stackTrace });
  }
}

But like other posters in this thread, I only can see the 500 error in my dev tools and not in Seed or AWS logs. How can I achieve that?

This is not much information to work with: