Skip to content

Commit c8a270f

Browse files
michalcmichalc
committed
feat: Windows support
It looks like for Windows in the production code: - sqlite3_initialize must be called explicitly to avoid access violation writing 0x0000000000000000 - using SQLITE_TRANSIENT as -1 causes access violation writing 0x00000000FFFFFFFF. This is probably since Python cases -1 to c_int, when it should be a pointer, and so c_void_p. Maybe in a later change could change it to SQLITE_STATIC, but not doing that now to keep this change small. And in tests: - to allow the test SQLite database to be deleted, the cursor and connection objects have to be removed from scope. Otherwise Windows thinks the file object is still open, and refuses to allow the deletion of the database at the end of the test Closes: #46
1 parent 8a88357 commit c8a270f

File tree

3 files changed

+49
-17
lines changed

3 files changed

+49
-17
lines changed

‎.github/workflows/test.yml‎

Lines changed: 42 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -9,21 +9,22 @@ jobs:
99
download-coverage-reporter:
1010
runs-on: ubuntu-latest
1111
steps:
12-
- name: "Download coverage reporter"
12+
- name: "Download coverage reporters"
1313
run: |
1414
mkdir -p ./reporter
15-
curl -L https://codeclimate.com/downloads/test-reporter/test-reporter-latest-linux-amd64 > ./reporter/cc-test-reporter
15+
curl -L https://codeclimate.com/downloads/test-reporter/test-reporter-latest-linux-amd64 > ./reporter/cc-test-reporter-linux
16+
curl -L https://codeclimate.com/downloads/test-reporter/test-reporter-latest-windows-amd64 > ./reporter/cc-test-reporter-windows.exe
1617
- name: "Notify code climate of pending coverage upload"
1718
env:
1819
CC_TEST_REPORTER_ID: ${{ secrets.CC_TEST_REPORTER_ID }}
1920
run: |
20-
chmod +x ./reporter/cc-test-reporter
21-
./reporter/cc-test-reporter before-build
22-
- name: "Save reporter"
21+
chmod +x ./reporter/cc-test-reporter-linux
22+
./reporter/cc-test-reporter-linux before-build
23+
- name: "Save reporters"
2324
uses: actions/upload-artifact@v3
2425
with:
2526
name: reporter
26-
path: ./reporter/cc-test-reporter
27+
path: ./reporter/*
2728

2829
# We want older SQLite amalgamation files, but they are not available to download,
2930
# so must be built from source. And they cannot be build on Windows, even for tests
@@ -69,6 +70,12 @@ jobs:
6970
- {os: "ubuntu-20.04", python: "3.9.0"}
7071
- {os: "ubuntu-20.04", python: "3.10.0"}
7172
- {os: "ubuntu-20.04", python: "3.11.0"}
73+
- {os: "windows-2019", python: "3.6.7"}
74+
- {os: "windows-2019", python: "3.7.1"}
75+
- {os: "windows-2019", python: "3.8.0"}
76+
- {os: "windows-2019", python: "3.9.0"}
77+
- {os: "windows-2019", python: "3.10.0"}
78+
- {os: "windows-2019", python: "3.11.0"}
7279
runs-on: '${{ matrix.os-and-python-version.os }}'
7380
env:
7481
MINIO_ROOT_USER: AKIAIOSFODNN7EXAMPLE
@@ -90,16 +97,36 @@ jobs:
9097
with:
9198
name: sqlite-${{ matrix.sqlite-version }}
9299
path: .
93-
- name: "Compile SQLite from amalgamation"
94-
if: matrix.sqlite-version != 'default'
100+
- name: "Compile SQLite from amalgamation (Windows)"
101+
if: matrix.os-and-python-version.os == 'windows-2019' && matrix.sqlite-version != 'default'
102+
run: |
103+
gcc -shared sqlite3.c -o sqlite3.dll
104+
echo "SQLITE3_VERSION=${{ matrix.sqlite-version }}" >> $env:GITHUB_ENV
105+
echo "LIBSQLITE3_PATH=${PWD}/sqlite3.dll" >> $env:GITHUB_ENV
106+
- name: "Compile SQLite from amalgamation (Ubuntu)"
107+
if: matrix.os-and-python-version.os == 'ubuntu-20.04' && matrix.sqlite-version != 'default'
95108
run: |
96109
gcc -shared -fPIC -o libsqlite3.so.0 sqlite3.c
97110
echo "SQLITE3_VERSION=${{ matrix.sqlite-version }}" >> "$GITHUB_ENV"
98111
echo "LIBSQLITE3_PATH=${PWD}/libsqlite3.so.0" >> "$GITHUB_ENV"
99112
- name: "Install sqlite-s3-query and any dependencies"
100113
run: |
101114
pip install ".[dev]"
102-
- name: "Test"
115+
- name: "Test (Windows)"
116+
if: matrix.os-and-python-version.os == 'windows-2019'
117+
run: |
118+
Invoke-WebRequest -Uri "https://dl.min.io/server/minio/release/windows-amd64/archive/minio.RELEASE.2023-07-21T21-12-44Z" -OutFile "./minio.exe"
119+
mkdir -p ./data
120+
./minio.exe server ./data &
121+
do {
122+
Write-Host "Waiting for MinIO"
123+
sleep 3
124+
} until(Test-NetConnection 127.0.0.1 -Port 9000 | ? { $_.TcpTestSucceeded } )
125+
coverage run -m unittest
126+
coverage xml
127+
./reporter/cc-test-reporter-windows.exe format-coverage --output "./coverage/${{ matrix.os-and-python-version.os }}-${{ matrix.os-and-python-version.python }}-${{ matrix.sqlite-version }}.json"
128+
- name: "Test (Ubuntu)"
129+
if: matrix.os-and-python-version.os == 'ubuntu-20.04'
103130
run: |
104131
wget -O minio https://dl.min.io/server/minio/release/linux-amd64/archive/minio.RELEASE.2023-07-21T21-12-44Z
105132
chmod +x minio
@@ -108,9 +135,9 @@ jobs:
108135
timeout 22 sh -c 'until nc -z $0 $1; do sleep 1; done' 127.0.0.1 9000
109136
coverage run -m unittest
110137
coverage xml
111-
chmod +x ./reporter/cc-test-reporter
112-
COVERAGE_FILE_NAME="./coverage/${{ matrix.os-and-python-version.python }}-${{ matrix.sqlite-version }}.json"
113-
./reporter/cc-test-reporter format-coverage --output "$COVERAGE_FILE_NAME"
138+
chmod +x ./reporter/cc-test-reporter-linux
139+
COVERAGE_FILE_NAME="./coverage/${{ matrix.os-and-python-version.os }}-${{ matrix.os-and-python-version.python }}-${{ matrix.sqlite-version }}.json"
140+
./reporter/cc-test-reporter-linux format-coverage --output "$COVERAGE_FILE_NAME"
114141
- name: "Save code coverage"
115142
uses: actions/upload-artifact@v3
116143
with:
@@ -127,6 +154,6 @@ jobs:
127154
CC_TEST_REPORTER_ID: ${{ secrets.CC_TEST_REPORTER_ID }}
128155
run: |
129156
ls -R
130-
chmod +x ./reporter/cc-test-reporter
131-
./reporter/cc-test-reporter sum-coverage ./coverage/*.json -p 24
132-
./reporter/cc-test-reporter upload-coverage
157+
chmod +x ./reporter/cc-test-reporter-linux
158+
./reporter/cc-test-reporter-linux sum-coverage ./coverage/*.json -p 48
159+
./reporter/cc-test-reporter-linux upload-coverage

‎sqlite_s3_query.py‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,7 @@ def sqlite_s3_query_multi(url, get_credentials=lambda now: (
3535
SQLITE_NOTFOUND = 12
3636
SQLITE_ROW = 100
3737
SQLITE_DONE = 101
38-
SQLITE_TRANSIENT = -1
38+
SQLITE_TRANSIENT = c_void_p(-1)
3939
SQLITE_OPEN_READONLY = 0x00000001
4040
SQLITE_OPEN_NOMUTEX = 0x00008000
4141
SQLITE_IOCAP_IMMUTABLE = 0x00002000
@@ -62,6 +62,8 @@ def sqlite_s3_query_multi(url, get_credentials=lambda now: (
6262
5: lambda pp_stmt, i: None,
6363
}
6464

65+
libsqlite3.sqlite3_initialize()
66+
6567
vfs_name = b's3-' + str(uuid4()).encode()
6668
file_name = b's3-' + str(uuid4()).encode()
6769
body_hash = sha256(b'').hexdigest()

‎test.py‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -690,7 +690,7 @@ def stream(self, method, url, headers, params):
690690
put_object_with_versioning('my-bucket', 'my.db', db)
691691

692692
with server() as server_sock:
693-
with self.assertRaisesRegex(Exception, 'Server disconnected|Connection reset'):
693+
with self.assertRaisesRegex(Exception, 'Server disconnected|Connection reset|WinError 10053|WinError 10054'):
694694
sqlite_s3_query('http://localhost:9000/my-bucket/my.db', get_credentials=lambda now: (
695695
'us-east-1',
696696
'AKIAIOSFODNN7EXAMPLE',
@@ -838,6 +838,9 @@ def get_db(sqls):
838838
cur.execute(sql, params)
839839
cur.execute('COMMIT')
840840

841+
# Really close the file, especially on Windows
842+
del cur, con
843+
841844
def db():
842845
with open(db_path, 'rb') as f:
843846
while True:

0 commit comments

Comments
 (0)