Skip to content

Commit 3cced85

Browse files
authored
Decrease chunk size in Reddit more comments streams (#1708)
1 parent 9bffcd3 commit 3cced85

File tree

1 file changed

+5
-1
lines changed

1 file changed

+5
-1
lines changed

services/libs/integrations/src/integrations/reddit/processStream.ts

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,11 @@ async function recursiveCommentParser(
3434
)
3535

3636
// Each stream has at most 99 children to expand. If there are more, we make more than one stream.
37-
for (const chunk of partition(comment.children, 99)) {
37+
// Stream identifier has max length of 255 characters, so we need to split the children in chunks of 28.
38+
// 28 * 8 + 22 = 246 < 255
39+
// 8 is a length of chunk_id + ,
40+
// 22 is a length of stream identifier prefix
41+
for (const chunk of partition(comment.children, 28)) {
3842
if (chunk.length > 0) {
3943
await ctx.publishStream<IRedditMoreCommentsStreamData>(
4044
`${RedditStreamType.MORE_COMMENTS}:${metadata.postId}:${chunk.join(',')}`,

0 commit comments

Comments
 (0)