sync: import latest source/taos-community subtree (3.3.6)#35381
sync: import latest source/taos-community subtree (3.3.6)#35381feici02 wants to merge 58 commits into
Conversation
chore: sync from github See merge request rd-public/tsdb!7
chore: build 3.3.6 in mac and linux platform See merge request rd-public/tsdb!8
chore: build 3.3.6 in mac See merge request rd-public/tsdb!9
chore: fix link warnings in mac See merge request rd-public/tsdb!10
chore: build in mac See merge request rd-public/tsdb!11
chore: sync from github 3.3.6 See merge request rd-public/tsdb!13
fix: resolve BUILD_TEST=ON compilation errors See merge request rd-public/tsdb!25
chore: update cmake options See merge request rd-public/tsdb!29
chore: update BUILD_CONTRIB and add BUILD_ROCKSDB See merge request rd-public/tsdb!39
fix(taos-gen): 修复 conan 构建两类失败:过期 cmake 文件 & 跨镜像 ABI 不匹配 See merge request rd-public/tsdb!58
feat: support GitLab monorepo build - taosx cmake split and Windows iconv fix See merge request rd-public/tsdb!49
fix: GCC 7 (core:0.2 / manylinux2014) 构建兼容性修复及 RocksDB 选项重构 See merge request rd-public/tsdb!76
….3.6' fix(cmake): Fix rocksdb cache validation false failure caused by unparsed generator expression when BUILD_CONTRIB=OFF See merge request rd-public/tsdb!86
chore: sync from github See merge request rd-public/tsdb!131
chore: sync from github See merge request rd-public/tsdb!163
ci: prepare to merge 3.3.6 for new ci See merge request rd-public/tsdb!166
…to chore/sync-3.3.6
…S (rd-public/tsdb!220)
…rd-public/tsdb!210)
…rball download (3.3.6) (rd-public/tsdb!259)
fix(log): log file disorder on windows
…up framework (rd-public/tsdb!299)
…all subprojects (rd-public/tsdb!346)
…b!410) fix(3.3.6): port CI test failure fixes from main branch
…public/tsdb!450) build(deps): add CPython Win/Mac tarballs and sync externals manifest [cherry-pick → 3.3.6]
refa: reorganize tsdb test pipeline into tools/cicd/tsdb-test-pipeline (3.3.6)
…blic/tsdb!455) build: update x86 rocksdb prebuilt binary for gcc 11.2.1 compatibility
fix:(stmt)core when null string convert to ts
…!471) fix(keeper): handle nil metric write responses
fix(parser/insert): align tag name array with cid-sorted tag values
feat: WAL corruption auto-recovery with backup support
…lic/tsdb!519) fix(build): remove --strip-debug from Release linker flags
test: use insert to prepare data.
…349) fix: two issues in copy mode repair - 3.3.6
feat(build): automate Conan binary preheat and restore for taos-gen (cherry-pick 3.3.6)
feat(ci): port all CI optimizations from main branch
…blic/tsdb!572) ci: remove devcontainer configuration (3.3.6)
…488) enh: make tdb page cache lock granularity smaller
enh: support specify vnode id when repair tdb
…-public/tsdb!598) chore(cmake): default BUILD_CONTRIB and BUILD_USE_PUBLIC_DEPS to ON for standalone build (3.3.6)
fix(transport): fix memory leak in cliNotifyImplCb when sync response arrives...
…/tsdb!617) fix: revert "make tdb page cache lock granularity smaller"
…lic/tsdb!591) fix(packaging): handle taosws symlink for versioned and unversioned libs
…public/tsdb!621) fix: narrow INSERT schema-retry condition to TK_NK_RP token
enh: add another mode to generate new meta
There was a problem hiding this comment.
Code Review
This pull request introduces several key enhancements and bug fixes, including a new KILL COMPACT FORCE command to forcefully terminate compaction tasks, a backup mechanism for corrupted WAL files, Linux Release binary size optimizations, and a fix for tag name/value misalignment in TMQ metadata. It also enhances the vnode repair tool with secure temp file handling and adds comprehensive test coverage. The code review identified three important issues: a potential infinite loop or double-processing bug in hashset_remove due to in-place rehashing, a risk of path truncation in WAL file backups due to small buffer sizes, and ignored exit statuses for SSH commands in the repair tool.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| // Rehash subsequent entries to maintain probe chain integrity | ||
| for (size_t j = set->mask & (h + prime2); set->items[j] != 0; j = set->mask & (j + prime2)) { | ||
| size_t val = set->items[j]; | ||
| set->items[j] = 0; | ||
| --set->nitems; | ||
| int nt = hashset_add_member(set, (void *)val); | ||
| } |
There was a problem hiding this comment.
In hashset_remove, rehashing subsequent entries in-place can lead to infinite loops or double-processing of the same item. If an item val from slot j is re-inserted at a slot k that is further down the probe chain (which was previously empty but is now filled by val), the loop will eventually reach k and process val again. To prevent this, all subsequent occupied slots in the cluster should be collected first, cleared from the hashset, and then re-inserted.
// Collect subsequent entries in the same cluster to avoid infinite loops/double-processing
size_t capacity = set->capacity;
size_t *to_reinsert = tdbOsCalloc(capacity, sizeof(size_t));
if (to_reinsert == NULL) {
return 1;
}
int count = 0;
for (size_t j = set->mask & (h + prime2); set->items[j] != 0; j = set->mask & (j + prime2)) {
to_reinsert[count++] = set->items[j];
set->items[j] = 0;
--set->nitems;
}
for (int i = 0; i < count; i++) {
int nt = hashset_add_member(set, (void *)to_reinsert[i]);
}
tdbOsFree(to_reinsert);| char srcFile[WAL_FILE_LEN + TSDB_FILENAME_LEN]; | ||
| char dstFile[WAL_FILE_LEN + TSDB_FILENAME_LEN]; |
There was a problem hiding this comment.
Using WAL_FILE_LEN + TSDB_FILENAME_LEN for srcFile and dstFile buffers can lead to path truncation if the source or destination directory path is long (since tsWalCorruptionBackupDir can be up to PATH_MAX). It is safer to use PATH_MAX for both buffers to prevent buffer truncation and copy failures.
char srcFile[PATH_MAX];
char dstFile[PATH_MAX];| FILE *fp = popen(cmd, "r"); | ||
| if (fp == NULL) { | ||
| uError("repair: failed to run ssh command"); | ||
| (void)close(fd); | ||
| (void)taosRemoveFile(localPathBuf); | ||
| return -1; | ||
| } | ||
| char buf[256]; | ||
| while (taosGetsCmd(pCmd, sizeof(buf), buf) > 0) { | ||
|
|
||
| char buf[4096]; | ||
| ssize_t nRead; | ||
| while ((nRead = fread(buf, 1, sizeof(buf), fp)) > 0) { | ||
| ssize_t nWritten = 0; | ||
| while (nWritten < nRead) { | ||
| ssize_t w = write(fd, buf + nWritten, nRead - nWritten); | ||
| if (w < 0) { | ||
| uError("repair: write to temp file failed: %s", strerror(errno)); | ||
| (void)pclose(fp); | ||
| (void)close(fd); | ||
| (void)taosRemoveFile(localPathBuf); | ||
| return -1; | ||
| } | ||
| nWritten += w; | ||
| } | ||
| } | ||
| taosCloseCmd(&pCmd); | ||
| (void)pclose(fp); | ||
| (void)close(fd); |
There was a problem hiding this comment.
The exit status of the popen command is ignored. If the SSH command fails (e.g., connection timeout, authentication failure, or host offline), pclose(fp) will return a non-zero status, but the function currently proceeds as long as some data was read or the file is non-empty. Checking status != 0 ensures that any SSH command failure is correctly detected and handled. Additionally, fread returns size_t, so nRead should be declared as size_t to avoid signed/unsigned comparison warnings.
FILE *fp = popen(cmd, "r");
if (fp == NULL) {
uError("repair: failed to run ssh command");
(void)close(fd);
(void)taosRemoveFile(localPathBuf);
return -1;
}
char buf[4096];
size_t nRead;
while ((nRead = fread(buf, 1, sizeof(buf), fp)) > 0) {
ssize_t nWritten = 0;
while (nWritten < (ssize_t)nRead) {
ssize_t w = write(fd, buf + nWritten, (ssize_t)nRead - nWritten);
if (w < 0) {
uError("repair: write to temp file failed: %s", strerror(errno));
(void)pclose(fp);
(void)close(fd);
(void)taosRemoveFile(localPathBuf);
return -1;
}
nWritten += w;
}
}
int status = pclose(fp);
(void)close(fd);
if (status != 0) {
uError("repair: ssh command failed");
(void)taosRemoveFile(localPathBuf);
return -1;
}
Sync
source/taos-community/subtree from the internal3.3.6monorepo branch.rd-public/tsdbbranch3.3.6(HEAD: 4958eece10f)source/taos-communitytaosdata/tdenginebranch3.3.6--first-parent(each commit is the integration point of an MR; trees stay correct across parallel merges)Generated by
tools/sync/subtree-split-fast.shwith author/committer identities preserved on every commit.