How to write DolphinDB's nanotimestamp column into DuckDB's TIMESTAMP_NS column?

1 week ago 8
ARTICLE AD BOX

I am working on a plugin to transfer data from DolphinDB to DuckDB. I have a test case that tries to append a table containing various DolphinDB types to DuckDB. The table includes a nanotimestamp column (DolphinDB type DT_NANOTIMESTAMP). In DuckDB, I want to store it as TIMESTAMP_NS (nanosecond precision). However, my current approach fails.

Test case (pseudo-code in DolphinDB script):

@testing:case="test_all_types_conversion" conn = duckdb::connect(":memory:") t = table( [true, false] as b, [char(1), char(2)] as c, [short(1), short(2)] as s, [1, 2] as i, [100l, 200l] as l, [1.1f, 2.2f] as f, [1.1, 2.2] as d, ["s1", "s2"] as str, symbol(["sym1","sym2"]) as sym, [2023.01.01, 2023.01.02] as dt, [12:00:00.000, 12:00:01.000] as tm, [2023.01.01 12:00:00.000, 2023.01.01 12:00:01.000] as ts, [nanotimestamp(2023.01.01 12:00:00.000000000), nanotimestamp(2023.01.01 12:00:01.000000000)] as nts ) duckdb::append(conn, t, "all_types", true) res = duckdb::query(conn, "select * from all_types") assert 1, each(eqObj, res.values(), t.values()).min() == true // fails because nts values become 1970-01-01 00:00:00 duckdb::close(conn)

My C++ plugin code for duckdb::append (excerpt):

extern "C" __declspec(dllexport) ConstantSP duckdbAppend(Heap* heap, vector<ConstantSP>& args) { // ... validation and table creation ... duckdb_appender appender; duckdb_appender_create(conn, nullptr, tableName.c_str(), &appender); int rowCount = srcTable->size(); int colCount = srcTable->columns(); for (int r = 0; r < rowCount; ++r) { duckdb_appender_begin_row(appender); for (int c = 0; c < colCount; ++c) { Vector* vec = (Vector*)srcTable->getColumn(c).get(); ConstantSP val = vec->get(r); if (val->isNull()) { duckdb_append_null(appender); continue; } duckdb_state state; DATA_TYPE type = vec->getType(); switch (type) { case DT_BOOL: state = duckdb_append_bool(appender, val->getBool()); break; case DT_CHAR: state = duckdb_append_int8(appender, val->getChar()); break; case DT_SHORT: state = duckdb_append_int16(appender, val->getShort()); break; case DT_INT: state = duckdb_append_int32(appender, val->getInt()); break; case DT_DATE: { duckdb_date d; d.days = val->getInt(); state = duckdb_append_date(appender, d); break; } case DT_LONG: state = duckdb_append_int64(appender, val->getLong()); break; case DT_FLOAT: state = duckdb_append_float(appender, val->getFloat()); break; case DT_DOUBLE: state = duckdb_append_double(appender, val->getDouble()); break; case DT_TIME: { duckdb_time t; t.micros = (int64_t)val->getInt() * 1000; // DolphinDB TIME is ms → DuckDB time needs microseconds state = duckdb_append_time(appender, t); break; } case DT_TIMESTAMP: { duckdb_timestamp ts; ts.micros = (int64_t)val->getLong() * 1000; // DolphinDB TIMESTAMP is ms → DuckDB timestamp needs microseconds state = duckdb_append_timestamp(appender, ts); break; } case DT_NANOTIMESTAMP: { // DolphinDB NANOTIMESTAMP is nanoseconds. // DuckDB C API does not yet have a dedicated nanosecond timestamp type. // Current approach: divide by 1000 to microseconds and append as standard timestamp. duckdb_timestamp ts; ts.micros = val->getLong() / 1000; state = duckdb_append_timestamp(appender, ts); break; } case DT_STRING: case DT_SYMBOL: state = duckdb_append_varchar(appender, val->getString().c_str()); break; default: state = duckdb_append_varchar(appender, val->getString().c_str()); break; } if (state != DuckDBSuccess) { throw RuntimeException("DuckDB Append Error: Failed to append value at row " + to_string(r) + " col " + to_string(c)); } } duckdb_appender_end_row(appender); } duckdb_appender_destroy(&appender); return new Long(rowCount); }

With the above code (converting nanosecond to microsecond and appending as duckdb_timestamp), the test fails because the retrieved values are all 1970-01-01 00:00:00 (zero). The values appear to be lost.

I then tried appending the raw nanosecond value as int64 using duckdb_append_int64, thinking DuckDB might interpret it correctly if the target column is TIMESTAMP_NS. But that results in the following error when running the test:

duckdb::duckdb::append(conn, t, "all_types", 1) => DuckDB Append Error: Failed to append value at row 0 col 12 (Type: 14)

Type 14 corresponds to DT_NANOTIMESTAMP in DolphinDB.

How can I correctly write DolphinDB's nanotimestamp values into a DuckDB TIMESTAMP_NS column using the DuckDB C API? Is there a way to directly append nanosecond timestamps? If not, what is the proper conversion to avoid data loss?

Additional infomation:

DolphinDB version: 3.0.0.4

DuckDB C API version: 1.4.4

OS: Windows 10

Plugin written in C++ using DuckDB C API.

The DuckDB table is created automatically by the plugin using CREATE TABLE IF NOT EXISTS with column types derived from DolphinDB types (using a mapping function ddbTypeToDuckDBSqlType). For DT_NANOTIMESTAMP, I map it to TIMESTAMP_NS.

The DuckDB C API version I'm using does not seem to have a duckdb_append_timestamp_ns function; only duckdb_append_timestamp which expects microseconds.

I verified that the values in DolphinDB are correct: nanotimestamp(2023.01.01 12:00:00.000000000) should represent a nanosecond-precision timestamp.

Read Entire Article