흰 스타렉스에서 내가 내리지

CloudFormation 으로 서버리스 알리미 구축하기 본문

AWS

CloudFormation 으로 서버리스 알리미 구축하기

주씨. 2024. 9. 14. 02:30
728x90

만들어볼 서버리스 알리미 아키텍처입니다. 

해야할 게 엄청 많지만, 차근차근 진행해 봅시다. 

 

먼저, CloudFormation 의 템플릿을 만들어봅니다. 

 

vim notifier-template.yml

 

# S3 버킷 생성과 IAM Role 

먼저, S3 버킷을 만들고, 버킷과 SNS 에 접근할 수 있는 정책이 할당된 IAM Role 을 만드는 것부터 해봅니다. 

AWSTemplateFormatVersion: '2010-09-09'
Parameters:
  SubscriptionEmail:
    Type: String
    Description: "The email address to subscribe to the SNS topic"
    
Resources:
  MyS3Bucket:
    Type: 'AWS::S3::Bucket'
    Properties:
      BucketName: !Sub 'youth-house-notifier-${AWS::AccountId}-${AWS::Region}'

  MyIAMRole:
    Type: 'AWS::IAM::Role'
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: 'ec2.amazonaws.com'
            Action: 'sts:AssumeRole'
      Policies:
        - PolicyName: 'S3AccessPolicy'
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - 's3:GetObject'
                  - 's3:PutObject'
                Resource: !Sub 'arn:aws:s3:::youth-house-notifier-${AWS::AccountId}-${AWS::Region}/*'
        - PolicyName: 'SNSAccessPolicy'
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - 'sns:Publish'
                  - 'sns:Subscribe'
                Resource: '*'

 

잘 되는지 확인하기 위해, 먼저 배포해봅니다.

 

aws cloudformation create-stack \
    --stack-name NotifierStack \
    --template-body file://notifier-template.yaml \
    --capabilities CAPABILITY_NAMED_IAM

- IAM Role 을 생성하기 위해서는 `--capabilities CAPABILITY_NAMED_IAM` 플래그는 필수입니다. 

 

 

CloudFormation 스택이 성공적으로 생성되었습니다.

 

 

 

# SNS Topic 생성 및 구독 설정

위에서 작성한 템플릿에 이어서 써봅니다. 

AWSTemplateFormatVersion: '2010-09-09'
Parameters:
  SubscriptionEmail:
    Type: String
    Description: "The email address to subscribe to the SNS topic"
    
Resources:
  MyS3Bucket:
    Type: 'AWS::S3::Bucket'
    Properties:
      BucketName: !Sub 'youth-house-notifier-${AWS::AccountId}-${AWS::Region}'

  MyIAMRole:
    Type: 'AWS::IAM::Role'
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: 'ec2.amazonaws.com'
            Action: 'sts:AssumeRole'
      Policies:
        - PolicyName: 'S3AccessPolicy'
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - 's3:GetObject'
                  - 's3:PutObject'
                Resource: !Sub 'arn:aws:s3:::youth-house-notifier-${AWS::AccountId}-${AWS::Region}/*'
        - PolicyName: 'SNSAccessPolicy'
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - 'sns:Publish'
                  - 'sns:Subscribe'
                Resource: '*'
  
  MySNSTopic:
    Type: 'AWS::SNS::Topic'
    Properties:
      TopicName: 'youth-house-recruitment-notifier'
  
  MySNSSubscription:
    Type: 'AWS::SNS::Subscription'
    Properties:
      Protocol: 'email'
      Endpoint: !Ref SubscriptionEmail
      TopicArn: !Ref MySNSTopic

 

 

aws cloudformation create-stack \
    --stack-name NotifierStack \
    --template-body file://notifier-template.yaml \
    --capabilities CAPABILITY_NAMED_IAM \
    --parameters ParameterKey=SubscriptionEmail,ParameterValue=thisis.joos@gmail.com

 

정상적으로 되었다면 SNS Topic 이 생성되어 있고, 파라미터로 넘겨준 이메일 주소의 구독이 생성되었을 것입니다.

그리고 해당 메일로 AWS 로부터 확인 메일이 전송되었을 것입니다. 

 

 

 

# Lambda 함수 추가

AWSTemplateFormatVersion: '2010-09-09'
Parameters:
  SubscriptionEmail:
    Type: String
    Description: "The email address to subscribe to the SNS topic"
    
Resources:
  MyS3Bucket:
    Type: 'AWS::S3::Bucket'
    Properties:
      BucketName: !Sub 'youth-house-notifier-${AWS::AccountId}-${AWS::Region}'

  MyIAMRole:
    Type: 'AWS::IAM::Role'
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: 'lambda.amazonaws.com'
            Action: 'sts:AssumeRole'
      Policies:
        - PolicyName: 'S3AccessPolicy'
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - 's3:GetObject'
                  - 's3:PutObject'
                Resource: !Sub 'arn:aws:s3:::youth-house-notifier-${AWS::AccountId}-${AWS::Region}/*'
        - PolicyName: 'SNSAccessPolicy'
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - 'sns:Publish'
                  - 'sns:Subscribe'
                Resource: '*'
  
  MySNSTopic:
    Type: 'AWS::SNS::Topic'
    Properties:
      TopicName: 'youth-house-recruitment-notifier'
  
  MySNSSubscription:
    Type: 'AWS::SNS::Subscription'
    Properties:
      Protocol: 'email'
      Endpoint: !Ref SubscriptionEmail
      TopicArn: !Ref MySNSTopic

  MyLambdaFunction:
    Type: 'AWS::Lambda::Function'
    Properties:
      FunctionName: 'YouthHouseNotifierFunction'
      Runtime: 'nodejs16.x'
      Handler: 'index.handler'
      Role: !GetAtt MyIAMRole.Arn
      Environment:
        Variables:
          SNS_TOPIC_ARN: !Ref MySNSTopic
          S3_BUCKET: !Ref MyS3Bucket
      Layers:
        - 'arn:aws:lambda:ap-northeast-2:764866452798:layer:chrome-aws-lambda:44'
        - 'arn:aws:lambda:ap-northeast-2:637423443861:layer:puppeteer-core:3'
      Code:
        ZipFile: |
          const AWS = require("aws-sdk");
          AWS.config.update({ region: "ap-northeast-2" });

          const chromium = require("@sparticuz/chromium");
          const puppeteer = require("puppeteer-core");

          const s3 = new AWS.S3();
          const sns = new AWS.SNS({ apiVersion: "2010-03-31" });

          exports.handler = async (event, context) => {
            let browser = null;

            const SNS_TOPIC_ARN = process.env.SNS_TOPIC_ARN;
            const S3_BUCKET = process.env.S3_BUCKET;

            if (!SNS_TOPIC_ARN || !S3_BUCKET) {
              console.error("환경 변수가 설정되지 않았습니다.");
              return {
                statusCode: 500,
                body: JSON.stringify({ error: "환경 변수가 설정되지 않았습니다." }),
              };
            }

            try {
              browser = await puppeteer.launch({
                args: [...chromium.args, "--no-sandbox"],
                defaultViewport: chromium.defaultViewport,
                executablePath: await chromium.executablePath(),
                headless: chromium.headless,
                ignoreHTTPSErrors: true,
              });

              const page = await browser.newPage();
              await page.goto("https://soco.seoul.go.kr/youth/bbs/BMSR00015/list.do?menuNo=400008");

              await page.waitForSelector("#boardList > tr:nth-child(1) > td:nth-child(1)");

              const index = await page.$eval("#boardList > tr:nth-child(1) > td:nth-child(1)", (element) => element.textContent.trim());

              const params = { Bucket: S3_BUCKET, Key: "latest-index.txt" };

              let latestIndex;
              try {
                const s3Data = await s3.getObject(params).promise();
                latestIndex = s3Data.Body.toString("utf-8").trim();
              } catch (error) {
                if (error.code === "NoSuchKey") {
                  const putParams = { Bucket: S3_BUCKET, Key: "latest-index.txt", Body: "0" };
                  await s3.putObject(putParams).promise();
                  latestIndex = "0";
                } else {
                  throw error;
                }
              }

              if (latestIndex !== index) {
                const putParams = { Bucket: S3_BUCKET, Key: "latest-index.txt", Body: index };
                await s3.putObject(putParams).promise();

                const message = `서울특별시 청년안심주택의 새로운 모집 공고가 게시되었습니다. 공고 번호: ${index}`;
                const snsParams = { Subject: "[서울특별시 청년안심주택] 새로운 모집 공고", Message: message, TopicArn: SNS_TOPIC_ARN };
                await sns.publish(snsParams).promise();
              }

              return { statusCode: 200, body: JSON.stringify({ message: "작업 완료" }) };
            } catch (error) {
              return { statusCode: 500, body: JSON.stringify({ error: "작업 실패" }) };
            } finally {
              if (browser !== null) {
                let pages = await browser.pages();
                await Promise.all(pages.map((page) => page.close()));
                await browser.close();
              }
            }
          };

 

 

람다 함수까지 잘 생성되었습니다. 

 

 

 

# EventBridge 규칙 생성 

AWSTemplateFormatVersion: '2010-09-09'
Parameters:
  SubscriptionEmail:
    Type: String
    Description: "The email address to subscribe to the SNS topic"
    
Resources:
  MyS3Bucket:
    Type: 'AWS::S3::Bucket'
    Properties:
      BucketName: !Sub 'youth-house-notifier-${AWS::AccountId}-${AWS::Region}'

  MyIAMRole:
    Type: 'AWS::IAM::Role'
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: 'lambda.amazonaws.com'
            Action: 'sts:AssumeRole'
      Policies:
        - PolicyName: 'S3AccessPolicy'
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - 's3:GetObject'
                  - 's3:PutObject'
                Resource: !Sub 'arn:aws:s3:::youth-house-notifier-${AWS::AccountId}-${AWS::Region}/*'
        - PolicyName: 'SNSAccessPolicy'
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - 'sns:Publish'
                  - 'sns:Subscribe'
                Resource: '*'
  
  MySNSTopic:
    Type: 'AWS::SNS::Topic'
    Properties:
      TopicName: 'youth-house-recruitment-notifier'
  
  MySNSSubscription:
    Type: 'AWS::SNS::Subscription'
    Properties:
      Protocol: 'email'
      Endpoint: !Ref SubscriptionEmail
      TopicArn: !Ref MySNSTopic

  MyLambdaFunction:
    Type: 'AWS::Lambda::Function'
    Properties:
      FunctionName: 'YouthHouseNotifierFunction'
      Runtime: 'nodejs16.x'
      Handler: 'index.handler'
      Role: !GetAtt MyIAMRole.Arn
      Environment:
        Variables:
          SNS_TOPIC_ARN: !Ref MySNSTopic
          S3_BUCKET: !Ref MyS3Bucket
      Layers:
        - 'arn:aws:lambda:ap-northeast-2:764866452798:layer:chrome-aws-lambda:44'
        - 'arn:aws:lambda:ap-northeast-2:637423443861:layer:puppeteer-core:3'
      Code:
        ZipFile: |
          const AWS = require("aws-sdk");
          AWS.config.update({ region: "ap-northeast-2" });

          const chromium = require("@sparticuz/chromium");
          const puppeteer = require("puppeteer-core");

          const s3 = new AWS.S3();
          const sns = new AWS.SNS({ apiVersion: "2010-03-31" });

          exports.handler = async (event, context) => {
            let browser = null;

            const SNS_TOPIC_ARN = process.env.SNS_TOPIC_ARN;
            const S3_BUCKET = process.env.S3_BUCKET;

            if (!SNS_TOPIC_ARN || !S3_BUCKET) {
              console.error("환경 변수가 설정되지 않았습니다.");
              return {
                statusCode: 500,
                body: JSON.stringify({ error: "환경 변수가 설정되지 않았습니다." }),
              };
            }

            try {
              browser = await puppeteer.launch({
                args: [...chromium.args, "--no-sandbox"],
                defaultViewport: chromium.defaultViewport,
                executablePath: await chromium.executablePath(),
                headless: chromium.headless,
                ignoreHTTPSErrors: true,
              });

              const page = await browser.newPage();
              await page.goto("https://soco.seoul.go.kr/youth/bbs/BMSR00015/list.do?menuNo=400008");

              await page.waitForSelector("#boardList > tr:nth-child(1) > td:nth-child(1)");

              const index = await page.$eval("#boardList > tr:nth-child(1) > td:nth-child(1)", (element) => element.textContent.trim());

              const params = { Bucket: S3_BUCKET, Key: "latest-index.txt" };

              let latestIndex;
              try {
                const s3Data = await s3.getObject(params).promise();
                latestIndex = s3Data.Body.toString("utf-8").trim();
              } catch (error) {
                if (error.code === "NoSuchKey") {
                  const putParams = { Bucket: S3_BUCKET, Key: "latest-index.txt", Body: "0" };
                  await s3.putObject(putParams).promise();
                  latestIndex = "0";
                } else {
                  throw error;
                }
              }

              if (latestIndex !== index) {
                const putParams = { Bucket: S3_BUCKET, Key: "latest-index.txt", Body: index };
                await s3.putObject(putParams).promise();

                const message = `서울특별시 청년안심주택의 새로운 모집 공고가 게시되었습니다. 공고 번호: ${index}`;
                const snsParams = { Subject: "[서울특별시 청년안심주택] 새로운 모집 공고", Message: message, TopicArn: SNS_TOPIC_ARN };
                await sns.publish(snsParams).promise();
              }

              return { statusCode: 200, body: JSON.stringify({ message: "작업 완료" }) };
            } catch (error) {
              return { statusCode: 500, body: JSON.stringify({ error: "작업 실패" }) };
            } finally {
              if (browser !== null) {
                let pages = await browser.pages();
                await Promise.all(pages.map((page) => page.close()));
                await browser.close();
              }
            }
          };
          
  MyEventBridgeRule:
    Type: 'AWS::Events::Rule'
    Properties:
      ScheduleExpression: 'cron(0 0 * * ? *)'  # UTC 기준 자정 (한국 시간 오전 9시)
      Targets:
        - Arn: !GetAtt MyLambdaFunction.Arn
          Id: 'TargetLambdaFunction'
      State: 'ENABLED'

  LambdaInvokePermission:
    Type: 'AWS::Lambda::Permission'
    Properties:
      Action: 'lambda:InvokeFunction'
      FunctionName: !Ref MyLambdaFunction
      Principal: 'events.amazonaws.com'
      SourceArn: !GetAtt MyEventBridgeRule.Arn

 

 

EventBridge 규칙까지 잘 생성되었다. 

 

 

잘 invoke 되는데,, 람다함수 실행에 실패한다. 

 

람다함수에서는 웹사이트를 크롤링해온다. 

디폴트 값으로 람다함수의 제한시간은 3초이다. 즉, 함수 실행시간이 3초가 초과되면 함수 실행을 완료하지 못한다. 

 

추가로, 람다의 디폴트 메모리 양은 128MB 였는데, 메모리를 많이 요하는 작업이라 1024MB 로 바꿔줘야 한다. 

 

CloudFormation 템플릿에서 Lambda 의 Properties 에 Timeout 을 추가해준다. 

또, MemorySize 도 추가해준다. 

 

그래도 안되길래, 오류 내용을 확인해 보니, s3:ListBucket 정책도 IAM 역할에 추가해주어야 했다. 

그리고 버킷에 대한 권한 역시 추가해주어야 했다.

버킷 내 객체에 대한 권한은 있으나, 버킷 자체에 대한 권한이 없었다. 

이것도 추가.

# 최종

AWSTemplateFormatVersion: '2010-09-09'
Parameters:
  SubscriptionEmail:
    Type: String
    Description: "The email address to subscribe to the SNS topic"
    
Resources:
  YHNotifierS3Bucket:
    Type: 'AWS::S3::Bucket'
    Properties:
      BucketName: !Sub 'youth-house-notifier-${AWS::AccountId}-${AWS::Region}'

  YHNotifierIAMRole:
    Type: 'AWS::IAM::Role'
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: 'lambda.amazonaws.com'
            Action: 'sts:AssumeRole'
      Policies:
        - PolicyName: 'S3AccessPolicy'
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - 's3:GetObject'
                  - 's3:PutObject'
                  - 's3:ListBucket'
                Resource: 
                  - !Sub 'arn:aws:s3:::youth-house-notifier-${AWS::AccountId}-${AWS::Region}'  # 버킷 자체에 대한 권한 
                  - !Sub 'arn:aws:s3:::youth-house-notifier-${AWS::AccountId}-${AWS::Region}/*' # 버킷 내 객체에 대한 권한 
        - PolicyName: 'SNSAccessPolicy'
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - 'sns:Publish'
                  - 'sns:Subscribe'
                Resource: '*'
  
  YHNotifierSNSTopic:
    Type: 'AWS::SNS::Topic'
    Properties:
      TopicName: 'youth-house-recruitment-notifier'
  
  YHNotifierSNSSubscription:
    Type: 'AWS::SNS::Subscription'
    Properties:
      Protocol: 'email'
      Endpoint: !Ref SubscriptionEmail
      TopicArn: !Ref YHNotifierSNSTopic

  YHNotifierLambdaFunction:
    Type: 'AWS::Lambda::Function'
    Properties:
      FunctionName: 'YouthHouseNotifierFunction'
      Runtime: 'nodejs16.x'
      Handler: 'index.handler'
      Role: !GetAtt YHNotifierIAMRole.Arn
      Timeout: 60  # 제한시간 1분 설정
      MemorySize: 1024  # 메모리 1024MB로 설정
      Environment:
        Variables:
          SNS_TOPIC_ARN: !Ref YHNotifierSNSTopic
          S3_BUCKET: !Ref YHNotifierS3Bucket
      Layers:
        - 'arn:aws:lambda:ap-northeast-2:764866452798:layer:chrome-aws-lambda:44'
        - 'arn:aws:lambda:ap-northeast-2:637423443861:layer:puppeteer-core:3'
      Code:
        ZipFile: |
          const AWS = require("aws-sdk");
          AWS.config.update({ region: "ap-northeast-2" });

          const chromium = require("@sparticuz/chromium");
          const puppeteer = require("puppeteer-core");

          const s3 = new AWS.S3();
          const sns = new AWS.SNS({ apiVersion: "2010-03-31" });

          exports.handler = async (event, context) => {
            let browser = null;

            const SNS_TOPIC_ARN = process.env.SNS_TOPIC_ARN;
            const S3_BUCKET = process.env.S3_BUCKET;

            if (!SNS_TOPIC_ARN || !S3_BUCKET) {
              console.error("환경 변수가 설정되지 않았습니다.");
              return {
                statusCode: 500,
                body: JSON.stringify({ error: "환경 변수가 설정되지 않았습니다." }),
              };
            }

            try {
              browser = await puppeteer.launch({
                args: [...chromium.args, "--no-sandbox"],
                defaultViewport: chromium.defaultViewport,
                executablePath: await chromium.executablePath(),
                headless: chromium.headless,
                ignoreHTTPSErrors: true,
              });

              const page = await browser.newPage();
              await page.goto("https://soco.seoul.go.kr/youth/bbs/BMSR00015/list.do?menuNo=400008");

              await page.waitForSelector("#boardList > tr:nth-child(1) > td:nth-child(1)");

              const index = await page.$eval("#boardList > tr:nth-child(1) > td:nth-child(1)", (element) => element.textContent.trim());

              const params = { Bucket: S3_BUCKET, Key: "latest-index.txt" };

              let latestIndex;
              try {
                const s3Data = await s3.getObject(params).promise();
                latestIndex = s3Data.Body.toString("utf-8").trim();
              } catch (error) {
                if (error.code === "NoSuchKey") {
                  const putParams = { Bucket: S3_BUCKET, Key: "latest-index.txt", Body: "0" };
                  await s3.putObject(putParams).promise();
                  latestIndex = "0";
                } else {
                  throw error;
                }
              }

              if (latestIndex !== index) {
                const putParams = { Bucket: S3_BUCKET, Key: "latest-index.txt", Body: index };
                await s3.putObject(putParams).promise();

                const message = `서울특별시 청년안심주택의 새로운 모집 공고가 게시되었습니다. 공고 번호: ${index}`;
                const snsParams = { Subject: "[서울특별시 청년안심주택] 새로운 모집 공고", Message: message, TopicArn: SNS_TOPIC_ARN };
                await sns.publish(snsParams).promise();
              }

              return { statusCode: 200, body: JSON.stringify({ message: "작업 완료" }) };
            } catch (error) {
              return { statusCode: 500, body: JSON.stringify({ error: error.message }) };
            } finally {
              if (browser !== null) {
                let pages = await browser.pages();
                await Promise.all(pages.map((page) => page.close()));
                await browser.close();
              }
            }
          };
          
  YHNotifierEventBridgeRule:
    Type: 'AWS::Events::Rule'
    Properties:
      ScheduleExpression: 'cron(0 0 * * ? *)'  # UTC 기준 자정 (한국 시간 오전 9시)
      Targets:
        - Arn: !GetAtt YHNotifierLambdaFunction.Arn
          Id: 'TargetLambdaFunction'
      State: 'ENABLED'

  LambdaInvokePermission:
    Type: 'AWS::Lambda::Permission'
    Properties:
      Action: 'lambda:InvokeFunction'
      FunctionName: !Ref YHNotifierLambdaFunction
      Principal: 'events.amazonaws.com'
      SourceArn: !GetAtt YHNotifierEventBridgeRule.Arn
aws cloudformation create-stack \
    --stack-name NotifierStack \
    --template-body file://notifier-template.yaml \
    --capabilities CAPABILITY_NAMED_IAM \
    --parameters ParameterKey=SubscriptionEmail,ParameterValue={{MyEMAIL}}

 

'AWS' 카테고리의 다른 글

AWS CDK 를 사용하여 S3 버킷 생성하기 (Python)  (0) 2024.09.21
AWS CloudFormation 와 AWS CDK : 차이점과 장단점, 그리고 추천 방법  (0) 2024.09.21
CloudFormation 을 이용하여 IAM 역할 생성 및 정책 할당 (CLI 사용)  (0) 2024.09.13
AWS CloudFormation vs Terraform ?  (0) 2024.09.13
AWS CloudFormation 의 개념과 구성요소  (0) 2024.09.13
'AWS' Related Articles more