Restricting access for a stage of a deployment to a single source ip

1

We are trying to limit access to our test stage deployment to only come from our VPN address. All other source IPs should be denied access to the StaticSite or Api requests that are stacks in our app. I tried to use default function permissions to allow when the request is from the specific source ip:

app.addDefaultFunctionPermissions([
    new iam.PolicyStatement({
      actions: [ "cloudformation:*", "s3:*", "logs:*", "iam:*", "apigateway:*", "lambda:*", "events:*", "dynamodb:*" ],
      effect: iam.Effect.ALLOW,
      resources: ["*"],
      condition: {
        IpAddress: {
          "aws:SourceIp": "123.123.123.123/32"
        }
      }
    })
  ]);

That didn’t work – I could request the site and apis from any public IP address. I also tried the inverse of above - to deny if not from the specific IP:

app.addDefaultFunctionPermissions([
    new iam.PolicyStatement({
      actions: [ "cloudformation:*", "s3:*", "logs:*", "iam:*", "apigateway:*", "lambda:*", "events:*", "dynamodb:*" ],
      effect: iam.Effect.DENY,
      resources: ["*"],
      condition: {
        NotIpAddress: {
          "aws:SourceIp": "123.123.123.123/32"
        }
      }
    })
  ]);

What am I missing or is there a different with serverless-stack to setup an IP restriction for a specific stage of a deployment?