Doing `iceberg_scan` fails in case of AWS Glue

[ferenc-csaky]

Description

When I try to do an Iceberg table query with iceberg_scan that points to an AWS Glue catalog, it raises the below error. The reason I report this here is the same thing works with the native DuckDB client, it fails only through a Java app.

The same thing (the error below) happens on both arm64 and x86 machines.

java.sql.SQLException: INTERNAL Error: Attempted to access index 2 within vector of size 1

Stack Trace:

/root/.duckdb/extensions/v1.4.3/linux_arm64/iceberg.duckdb_extension(+0xaa0dbc) [0xffff72590dbc]
/root/.duckdb/extensions/v1.4.3/linux_arm64/iceberg.duckdb_extension(+0xaa0e88) [0xffff72590e88]
/root/.duckdb/extensions/v1.4.3/linux_arm64/iceberg.duckdb_extension(+0xaa343c) [0xffff7259343c]
/root/.duckdb/extensions/v1.4.3/linux_arm64/iceberg.duckdb_extension(+0x2930dc) [0xffff71d830dc]
/root/.duckdb/extensions/v1.4.3/linux_arm64/iceberg.duckdb_extension(+0xb08f50) [0xffff725f8f50]
/root/.duckdb/extensions/v1.4.3/linux_arm64/iceberg.duckdb_extension(+0xca16b0) [0xffff727916b0]
/root/.duckdb/extensions/v1.4.3/linux_arm64/iceberg.duckdb_extension(+0xbcf290) [0xffff726bf290]
/root/.duckdb/extensions/v1.4.3/linux_arm64/iceberg.duckdb_extension(+0xb08284) [0xffff725f8284]
/root/.duckdb/extensions/v1.4.3/linux_arm64/iceberg.duckdb_extension(+0xb08590) [0xffff725f8590]
/root/.duckdb/extensions/v1.4.3/linux_arm64/iceberg.duckdb_extension(+0x30a804) [0xffff71dfa804]
/root/.duckdb/extensions/v1.4.3/linux_arm64/parquet.duckdb_extension(+0x14b740) [0xffff7104b740]
/tmp/libduckdb_java5599451049912054297.so(+0x917364) [0xffff81d27364]
/tmp/libduckdb_java5599451049912054297.so(+0xf6fbf8) [0xffff8237fbf8]
/tmp/libduckdb_java5599451049912054297.so(+0xf7aa2c) [0xffff8238aa2c]
/tmp/libduckdb_java5599451049912054297.so(+0xf7addc) [0xffff8238addc]
/tmp/libduckdb_java5599451049912054297.so(+0xf71954) [0xffff82381954]
/tmp/libduckdb_java5599451049912054297.so(+0xf7c498) [0xffff8238c498]
/lib/aarch64-linux-gnu/libstdc++.so.6(+0xe1ae0) [0xffff81261ae0]
/lib/aarch64-linux-gnu/libc.so.6(+0x8595c) [0xffffb881595c]
/lib/aarch64-linux-gnu/libc.so.6(+0xeb89c) [0xffffb887b89c]

This error signals an assertion failure within DuckDB. This usually occurs due to unexpected conditions or errors in the program's logic.
For more information, see https://duckdb.org/docs/stable/dev/internal_errors
        at org.duckdb.DuckDBNative.duckdb_jdbc_execute(Native Method)
        at org.duckdb.DuckDBPreparedStatement.execute(DuckDBPreparedStatement.java:194)
        at com.zaxxer.hikari.pool.ProxyPreparedStatement.execute(ProxyPreparedStatement.java:44)
        at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.execute(HikariProxyPreparedStatement.java)
        ...
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:840)

SQL Query

INSTALL iceberg;
INSTALL httpfs;

LOAD iceberg;
LOAD httpfs;
SET unsafe_enable_version_guessing = true;

CREATE SECRET aws_s3_secret (TYPE s3, PROVIDER credential_chain, REGION '...');

SELECT * FROM iceberg_scan('s3://my-aws-glue-bucket/mydatabase.db/mytable', ALLOW_MOVED_PATHS = TRUE);

[staticlibs]

@ferenc-csaky

Hi, thanks for the report! I wonder if you can provide more details about the steps to reproduce this, I cannot get this error from Java so far.

My current steps:

1. create a table in AWS Glue UI choosing Iceberg format
2. adding a few records to this table from Athena UI
3. running the script you provided from DuckDB CLI and with JDBC, this query successfully returns records to me from both CLI and JDBC:

SELECT * FROM iceberg_scan('s3://<bucket>/<table>', ALLOW_MOVED_PATHS = TRUE, version = '?')

Any other details on your setup (specific data column types?, transactions use?) may be helpful!

[ferenc-csaky]

@staticlibs Thanks a lot for responding so quickly here! I am on vacation ATM, but give me a couple days to get back to this with more details. I'll dump out my exact query.

[ferenc-csaky]

Now I had some time to check on this, and I can confirm that simply using DuckDB JDBC the issue is not reproducible. My real query simply includes a dynamic limit and offset params, which I originally thought is orthogonal to the problem, so that's why I did not include it. Like this:

SELECT "col1", "col2" AS "custom_name", "col3"
FROM "iceberg_scan"('s3://mybucket/mydatabase.db/mytable', ALLOW_MOVED_PATHS = TRUE) LIMIT ? OFFSET ?;

And it does not make a difference, cause now I set up a dummy Java project where I use the standard java.sql API to query and it works as expected.

On my real app, the query comes in as a GraphQL query and the JDBC connection is initialized and used through Vert.x. I will get back to this in early January to see if I can make a dummy app to reproduce this, and also check on the Vert.x bits, cause maybe something goes wrong there.

[ferenc-csaky]

@staticlibs I managed to come up with a demo project where I was able to reproduce the issue. I uploaded it to a GitHub repo: https://github.com/ferenc-csaky/duckdb-iceberg-issue

FYI, I populate this Iceberg table with Flink in a streaming manner, and that's why SET unsafe_enable_version_guessing = true is mandatory. I did not tested with another setup, so if it turns out that this also required to reproduce, I can share an S3 bucket with some prepopulated dummy data, just let me know.

Now that I did this whole thing, I managed to also reproduce it in the DudckDB CLI with this:

DuckDB v1.4.0 (Andium) b8a06e4a22
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
D INSTALL iceberg; INSTALL httpfs; LOAD iceberg; LOAD httpfs;
D SET unsafe_enable_version_guessing = true;
D CREATE SECRET aws_s3_secret (TYPE s3, PROVIDER credential_chain, REGION 'eu-north-1');
┌─────────┐
│ Success │
│ boolean │
├─────────┤
│ true    │
└─────────┘
D SELECT * FROM (SELECT AVG("metric") AS "avg_metric", COUNT(*) AS "total_count"
  FROM (SELECT *
    FROM "iceberg_scan"('s3://mybucket/mydatabase.db/metrics', ALLOW_MOVED_PATHS = TRUE)) AS "t"
  WHERE "machineid" = 5 AND "ts" > TIMESTAMPTZ '2024-01-01T00:00:00Z') x LIMIT 5 OFFSET 0;
INTERNAL Error:
Attempted to access index 2 within vector of size 1

Stack Trace:

0        _ZN6duckdb9ExceptionC2ENS_13ExceptionTypeERKNSt3__112basic_stringIcNS2_11char_traitsIcEENS2_9allocatorIcEEEE + 64
1        _ZN6duckdb17InternalExceptionC1ERKNSt3__112basic_stringIcNS1_11char_traitsIcEENS1_9allocatorIcEEEE + 20
2        _ZN6duckdb17InternalExceptionC1IJyyEEERKNSt3__112basic_stringIcNS2_11char_traitsIcEENS2_9allocatorIcEEEEDpT_ + 136
3        _ZN6duckdb6vectorINS_6VectorELb1EEixEm + 136
4        _ZN6duckdb18ExpressionExecutor7ExecuteERKNS_24BoundReferenceExpressionEPNS_15ExpressionStateEPKNS_15SelectionVectorEyRNS_6VectorE + 40
5        _ZN6duckdb18ExpressionExecutor7ExecuteERKNS_10ExpressionEPNS_15ExpressionStateEPKNS_15SelectionVectorEyRNS_6VectorE + 524
6        _ZN6duckdb18ExpressionExecutor6SelectERKNS_25BoundComparisonExpressionEPNS_15ExpressionStateEPKNS_15SelectionVectorEyPS6_S9_ + 152
7        _ZN6duckdb18ExpressionExecutor6SelectERKNS_26BoundConjunctionExpressionEPNS_15ExpressionStateEPKNS_15SelectionVectorEyPS6_S9_ + 1000
8        _ZN6duckdb18ExpressionExecutor6SelectERKNS_26BoundConjunctionExpressionEPNS_15ExpressionStateEPKNS_15SelectionVectorEyPS6_S9_ + 1400
9        _ZN6duckdb22IcebergMultiFileReader20ApplyEqualityDeletesERNS_13ClientContextERNS_9DataChunkERKNS_20IcebergMultiFileListERKNS_20IcebergManifestEntryERKNS_6vectorINS_25MultiFileColumnDefinitionELb1EEE + 3212
10       _ZN6duckdb22IcebergMultiFileReader13FinalizeChunkERNS_13ClientContextERKNS_17MultiFileBindDataERNS_14BaseFileReaderERKNS_19MultiFileReaderDataERNS_9DataChunkESC_RNS_18ExpressionExecutorENS_12optional_ptrINS_26MultiFileReaderGlobalStateELb1EEE + 144
11       duckdb::MultiFileFunction<duckdb::ParquetMultiFileInfo>::MultiFileScan(duckdb::ClientContext&, duckdb::TableFunctionInput&, duckdb::DataChunk&) + 288
12       duckdb::PhysicalTableScan::GetData(duckdb::ExecutionContext&, duckdb::DataChunk&, duckdb::OperatorSourceInput&) const + 80
13       duckdb::PipelineExecutor::FetchFromSource(duckdb::DataChunk&) + 136
14       duckdb::PipelineExecutor::Execute(unsigned long long) + 236
15       duckdb::PipelineTask::ExecuteTask(duckdb::TaskExecutionMode) + 264
16       duckdb::ExecutorTask::Execute(duckdb::TaskExecutionMode) + 192
17       duckdb::TaskScheduler::ExecuteForever(std::__1::atomic<bool>*) + 480
18       void* std::__1::__thread_proxy[abi:ne200100]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, void (*)(duckdb::TaskScheduler*, std::__1::atomic<bool>*), duckdb::TaskScheduler*, std::__1::atomic<bool>*>>(void*) + 56
19       _pthread_start + 136
20       thread_start + 8

This error signals an assertion failure within DuckDB. This usually occurs due to unexpected conditions or errors in the program's logic.
For more information, see https://duckdb.org/docs/stable/dev/internal_errors

So if this should be moved to the core DuckDB project, just let me know. :)

[staticlibs]

@ferenc-csaky

Thanks for this investigation! As this is not directly related to Java, I will raise this internally.

Just the question on the version:

DuckDB v1.4.0 (Andium) b8a06e4a22

There were 3 updates after this one, current stable being 1.4.3. I don't follow Iceberg extension closely so don't know whether any changes in this particular area were added between 1.4.0 and 1.4.3. I would still suggest to re-check with 1.4.3 if possible.

[ferenc-csaky]

@staticlibs Thank you! Good catch, for duckdb_jdbc I already used 1.4.3.0 though, but now I upgraded my native client (DuckDB v1.4.3 (Andium) d1dc88f950) as well, and the issue still present, same stack trace.

[Tmonster]

Thanks for the report! I will look into this 👍

[Tmonster]

Thanks again for the report. I was looking at the reproducer, but it doesn't really help since it requires an already existing table with data. Reproducing the issue then requires trying many glue + flink iceberg table population strategies that may or may not reproduce the issue on my end. The issue is most likely that the iceberg table is in some state that has not yet been tested with DuckDB-Iceberg, but right now I have no way to know how to reproduce the table state.

If you could update the reproduction so that table creation + population are included that would be very helpful. Then debugging becomes much easier.

[ferenc-csaky]

@Tmonster Thanks for checking on this!

Fair point, I guess then the easiest thing would be to run my project in a Docker img. It should be pretty straightforward:

1. Clone git repo with a branch that only contains the necessary files to reproduce the issue:
   git clone --branch duckdb-issue --single-branch https://github.com/DataSQRL/datasqrl-examples
2. cd datasqrl-examples/duckdb-query
3. In package_test_s3_glue.json set the warehouse at L26 to your S3 bucket, and catalog-database at L30 to your existing Glue DB name.
4. Run the project in a Docker img:

docker run -it --rm \
  -v $PWD:/build \
  -e AWS_ACCESS_KEY_ID="..." \
  -e AWS_SECRET_ACCESS_KEY="..." \
  -e AWS_REGION="..." \
  datasqrl/cmd:dev run package_test_s3_glue.json

This pretty much will run a Flink SQL job that will write to the given Glue table. Let it run for cca 30sec to be sure Flink checkpoints, and flushes the data, then stop the img. It most probably will throw an exception about savepoints or checkpints, that can be ignored. Then querying that table should produce the reported error.