回答
收藏
分享
举报
Nginx转发埋点数据给kafka时,如何做故障恢复
提问于2021-05-20 19:17

浏览 4.1k

Nginx收集埋点数据转发至kafka时,如何保障埋点数据不丢失,有没有类似checkpoint机制
已修改于2023-03-17 02:13



写下您的回答
发表回答
全部回答(1)

按点赞数排序

按时间排序

NGINX无法提供类似机制

赞同

0

回复举报

回答于2022-10-25 15:24



回复为什么不ban猛犸
回复
提问者
WanYi_824
这家伙很懒还未留下介绍~
0
文章
1
问答
0
粉丝
相关问答

新建一个node服务,使用nginx-upload-module提供的upload_pass参数将处理过的请求转发到我们的node服务上面。分两步走:

1. 修改nginx配置

修改nginx配置,将nginx-upload-module存储文件的路径和文件名称塞到请求的body参数中。同时将请求转发到node服务具体配置:

    location /upload {
           upload_pass /res_upload;
            upload_store /usr/local/nginx/html;
            upload_resumable on;
            upload_state_store /tmp/state;
            upload_set_form_field "${upload_field_name}_name" $upload_file_name;
            upload_set_form_field "${upload_field_name}_content_type" $upload_content_type;
            upload_set_form_field "${upload_field_name}_path" $upload_tmp_path;

            upload_aggregate_form_field "${upload_field_name}_md5" $upload_file_md5;
            upload_aggregate_form_field "${upload_field_name}_size" $upload_file_size;
        }
        location /res_upload {
            #default_type text/html;
            #return 200 "success";
            proxy_pass http://localhost:8080;
        }

2. 新建node服务,重命名文件

  1. 查看请求中的form data
  2. 解析请求中的form data为json形式
  3. 根据文件位置,重命名文件
  4. const http = require('http');
    const fs = require('fs');
    
    http.createServer((req, res) => {
      let body = '';
      req.on('data', chunk => {
        body += chunk; // 获取form-data
      });
      req.on('end', () => {
        console.log(body);
        const params = parseForm(body); // json化参数
        rename(params.file_path, params.file_name);
        res.end('success: you can get it from: /' + params.file_name);
      })
    }).listen(8080)
    
    function parseForm(data) {
      const reg = /name="([\w_]+)"\s+(.+)\s/g;
      const params = {};
      let matched;
      while((matched = reg.exec(data))){
        params[matched[1]] = matched[2];
      }
      console.log(params);
      return params;
    }
    
    function rename(source, name){
      const path = require('path');
      const dir = path.dirname(source);
      fs.renameSync(source, path.join(dir, name));
    }
点赞 0
浏览 4.3k
点赞 0
浏览 1.9k