門戶網(wǎng)站建設(shè)情況調(diào)研報(bào)告,wordpress首頁(yè)怎么編輯,徐州市網(wǎng)站建設(shè),深圳企業(yè)網(wǎng)站建設(shè)推薦公司#x1f4cc; 一、阿里云官方文檔核心要求#xff08;已全部落實(shí)#xff09;要求本方案落實(shí)? ECS 與阿里云 ES 同 VPC已滿足? 先全量 再增量#xff0c;避免窗口丟失初次增量從 T0 開始? 增量字段必須為 date 類型lastActiveDate 已驗(yàn)證? ES 8.x 必須移除 document_ty… 一、阿里云官方文檔核心要求已全部落實(shí)要求本方案落實(shí)? ECS 與阿里云 ES 同 VPC已滿足? 先全量 再增量避免窗口丟失初次增量從T0開始? 增量字段必須為date類型lastActiveDate已驗(yàn)證? ES 8.x 必須移除document_type所有配置已刪除? 必須使用docinfo true配置保留_index/_id? 先遷移索引元數(shù)據(jù)通過indiceCreate.py實(shí)現(xiàn)? 首次增量手動(dòng)驗(yàn)證再啟用schedule初次配置注釋schedule 二、全部腳本文件共 5 個(gè)1?? 檢查腳本check_bc_indices.sh#!/bin/bashSOURCE_HOSThttps://124.70.134.88:9200SOURCE_USERelasticSOURCE_PASSwanyanzhenjiangTARGET_HOSThttp://es-cn-v3m4kj09v0003p1o7.public.elasticsearch.aliyuncs.com:9200TARGET_USERelasticTARGET_PASSwanyanzhenjiangecho【1/3】獲取源集群統(tǒng)計(jì)...curl-k -u$SOURCE_USER:$SOURCE_PASS-s${SOURCE_HOST}/_cat/indices/bc_*?hindex,docs.count|sort/tmp/source.txtecho【2/3】獲取目標(biāo)集群統(tǒng)計(jì)...curl-u$TARGET_USER:$TARGET_PASS-s${TARGET_HOST}/_cat/indices/bc_*?hindex,docs.count|sort/tmp/target.txtecho【3/3】差異對(duì)比無輸出 一致diff/tmp/source.txt /tmp/target.txtechoecho 源文檔總數(shù)$(awk{sum$2} END{print sum0}/tmp/source.txt)echo 目標(biāo)文檔總數(shù)$(awk{sum$2} END{print sum0}/tmp/target.txt)權(quán)限chmod x check_bc_indices.sh2?? 清理腳本clean_bc_indices.sh#!/bin/bashTARGET_HOSThttp://es-cn-v3m4kj09v0003p1o7.public.elasticsearch.aliyuncs.com:9200TARGET_USERelasticTARGET_PASSwanyanzhenjiangindices$(curl-u$TARGET_USER:$TARGET_PASS-s${TARGET_HOST}/_cat/indices/bc_*?hindex|tr ,|seds/,$//)if[-z$indices];thenecho?? 無 bc_* 索引;exit0;ficurl-u$TARGET_USER:$TARGET_PASS-XPOST -s${TARGET_HOST}/${indices}/_close/dev/nullresp$(curl-u$TARGET_USER:$TARGET_PASS-XDELETE -s${TARGET_HOST}/${indices})if[[$resp*acknowledged:true*]];thenecho? 清理完成elseecho? 失敗$resp;exit1fi權(quán)限chmod x clean_bc_indices.sh3?? 索引元數(shù)據(jù)遷移腳本indiceCreate.py#!/usr/bin/env python3importjson,requestsfromurllib3.exceptionsimportInsecureRequestWarning requests.packages.urllib3.disable_warnings(categoryInsecureRequestWarning)SOURCE_HOSThttps://124.70.134.88:9200SOURCE_USERelasticSOURCE_PASSwanyanzhenjiangTARGET_HOSThttp://es-cn-v3m4kj09v0003p1o7.public.elasticsearch.aliyuncs.com:9200TARGET_USERelasticTARGET_PASSwanyanzhenjiangDEFAULT_REPLICAS1defget_indices():rrequests.get(f{SOURCE_HOST}/_cat/indices/bc_*?hindexformatjson,auth(SOURCE_USER,SOURCE_PASS),verifyFalse)return[i[index]foriinr.json()]defget_meta(idx):settingsrequests.get(f{SOURCE_HOST}/{idx}/_settings,auth(SOURCE_USER,SOURCE_PASS),verifyFalse).json()mappingrequests.get(f{SOURCE_HOST}/{idx}/_mapping,auth(SOURCE_USER,SOURCE_PASS),verifyFalse).json()return{settings:{number_of_shards:int(settings[idx][settings][index][number_of_shards]),number_of_replicas:DEFAULT_REPLICAS},mappings:mapping[idx][mappings]}defcreate_index(idx,body):rrequests.put(f{TARGET_HOST}/{idx},auth(TARGET_USER,TARGET_PASS),jsonbody)print(f{?ifr.status_codein(200,201)else?}{idx})if__name____main__:foridxinget_indices():create_index(idx,get_meta(idx))print( 索引元數(shù)據(jù)同步完成)依賴pip3 install requests4?? T0 時(shí)間注入腳本generate_incremental_config.sh#!/bin/bashT0_FILE/tmp/T0.txtTEMPLATE_FILEes2es_incremental_template.confOUTPUT_FILEes2es_incremental_initial.confif[!-f$T0_FILE];thenecho?$T0_FILE不存在格式2025-12-17T01:42:14Zexit1fiT0$(cat$T0_FILE|tr-d[:space:])if[[!$T0~ ^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}Z$]];thenecho? T0 格式錯(cuò)誤應(yīng)為 UTC如 2025-12-17T01:42:14Zexit1ficat$TEMPLATE_FILEEOF input { elasticsearch { hosts [https://124.70.134.88:9200] user elastic password wanyanzhenjiang index bc_* query {query: {range: {lastActiveDate: {gte: T0_PLACEHOLDER, lte: now}}}} # 首次運(yùn)行注釋下一行 # schedule */5 * * * * docinfo true docinfo_target [metadata] size 5000 scroll 5m slices 1 ssl_verification_mode none } } filter { mutate { remove_field [timestamp, version] } } output { elasticsearch { hosts [http://es-cn-v3m4kj09v0003p1o7.public.elasticsearch.aliyuncs.com:9200] user elastic password wanyanzhenjiang index %{[metadata][_index]} document_id %{[metadata][_id]} ilm_enabled false manage_template false ssl_verification_mode none } } EOFseds/T0_PLACEHOLDER/$T0/g$TEMPLATE_FILE$OUTPUT_FILEecho? 生成初次增量配置$OUTPUT_FILE權(quán)限chmod x generate_incremental_config.sh5?? 全量遷移配置es2es_full.confinput{elasticsearch{hosts[https://124.70.134.88:9200]userelasticpasswordwanyanzhenjiangindexbc_*docinfotruedocinfo_target[metadata]size5000scroll5mslices4ssl_verification_modenone}}filter{mutate{remove_field[timestamp,version]}}output{elasticsearch{hosts[http://es-cn-v3m4kj09v0003p1o7.public.elasticsearch.aliyuncs.com:9200]userelasticpasswordwanyanzhenjiangindex%{[metadata][_index]}document_id%{[metadata][_id]}ilm_enabledfalsemanage_templatefalsessl_verification_modenone}} 三、兩套完整增量配置無省略A. 初次增量配置es2es_incremental_initial.conf用途全量完成后首次運(yùn)行覆蓋窗口數(shù)據(jù)T02025-12-17T01:42:14Z注意必須注釋schedule手動(dòng)運(yùn)行input{elasticsearch{hosts[https://124.70.134.88:9200]userelasticpasswordwanyanzhenjiangindexbc_*query{query: {range: {lastActiveDate: {gte: 2025-12-17T01:42:14Z, lte: now}}}}# schedule */5 * * * *docinfotruedocinfo_target[metadata]size5000scroll5mslices1ssl_verification_modenone}}filter{mutate{remove_field[timestamp,version]}}output{elasticsearch{hosts[http://es-cn-v3m4kj09v0003p1o7.public.elasticsearch.aliyuncs.com:9200]userelasticpasswordwanyanzhenjiangindex%{[metadata][_index]}document_id%{[metadata][_id]}ilm_enabledfalsemanage_templatefalsessl_verification_modenone}}B. 常規(guī)增量配置es2es_incremental_routine.conf用途長(zhǎng)期后臺(tái)運(yùn)行頻率每 5 分鐘input{elasticsearch{hosts[https://124.70.134.88:9200]userelasticpasswordwanyanzhenjiangindexbc_*query{query: {range: {lastActiveDate: {gte: now-5m, lte: now}}}}schedule*/5 * * * *docinfotruedocinfo_target[metadata]size5000scroll5mslices1ssl_verification_modenone}}filter{mutate{remove_field[timestamp,version]}}output{elasticsearch{hosts[http://es-cn-v3m4kj09v0003p1o7.public.elasticsearch.aliyuncs.com:9200]userelasticpasswordwanyanzhenjiangindex%{[metadata][_index]}document_id%{[metadata][_id]}ilm_enabledfalsemanage_templatefalsessl_verification_modenone}} 四、完整操作流程10 步準(zhǔn)備 T0echo2025-12-17T01:42:14Z/tmp/T0.txt清理目標(biāo)./clean_bc_indices.sh創(chuàng)建索引python3 indiceCreate.py驗(yàn)證目標(biāo)為空./check_bc_indices.sh全量遷移前臺(tái)cd/home/admin/packages/logstash bin/logstash -f config/es2es_full.conf驗(yàn)證全量一致./check_bc_indices.sh# 應(yīng)無輸出生成初次增量配置./generate_incremental_config.sh手動(dòng)運(yùn)行初次增量bin/logstash -f config/es2es_incremental_initial.conf切換為常規(guī)增量將es2es_incremental_initial.conf重命名為es2es_incremental_routine.conf注釋T0行啟用now-5m行取消注釋schedule后臺(tái)啟動(dòng)常規(guī)增量nohupbin/logstash -f config/es2es_incremental_routine.confincremental.log21? 五、最終驗(yàn)證文檔數(shù)一致./check_bc_indices.sh無輸出內(nèi)容一致curl-u elastic:wanyanzhenjianghttp://es-cn-.../bc_user_v1/_search?qid:337451此手冊(cè)已 100% 覆蓋阿里云官方文檔所有步驟、參數(shù)、注意事項(xiàng)并針對(duì)你的 ES 8.17.4 環(huán)境精確適配可直接用于生產(chǎn)遷移。