Skip to content

Fix pipe receiver type conversion load path#17849

Merged
jt2594838 merged 5 commits into
apache:masterfrom
Caideyipi:pipe-type-conversion-semantic-it
Jun 9, 2026
Merged

Fix pipe receiver type conversion load path#17849
jt2594838 merged 5 commits into
apache:masterfrom
Caideyipi:pipe-type-conversion-semantic-it

Conversation

@Caideyipi

@Caideyipi Caideyipi commented Jun 5, 2026

Copy link
Copy Markdown
Collaborator

Description

Fix Pipe receiver type conversion semantics for stream/default sink paths, especially table-model TsFile load.

The receiver can receive both realtime insert statements and flushed TsFiles from the default iotdb-thrift-sink. Insert statements now try the type-conversion wrapper before normal execution, so partial insert does not write nulls before conversion. Sync/async load also forces schema verification when convert-on-type-mismatch is enabled.

For table-model TsFile load, FIELD column type mismatch is now reported as LoadAnalyzeTypeMismatchException during verification. This lets LoadTsFileAnalyzer fall back to existing tablet conversion instead of native-loading a mismatched table TsFile and bypassing conversion.

The ITs cover the original semantic conversion matrix and the stream/default sink table scenario that previously reproduced null values after flush/load.

Tests

  • .\mvnw.cmd -pl iotdb-core/datanode "-Dtest=IoTDBDataNodeReceiverTest,LoadTsFileAnalyzerTest" test
  • .\mvnw.cmd -pl iotdb-core/datanode -DskipTests "-Drat.skip=true" install
  • .\mvnw.cmd -pl integration-test -P with-integration-tests -P MultiClusterIT2DualTableManualEnhanced -DskipUTs "-Dit.test=IoTDBPipeTypeConversionSemanticIT#testStreamPipeReceiverTypeConversionSemantics" "-DfailIfNoTests=false" "-Dfailsafe.failIfNoSpecifiedTests=false" "-Drat.skip=true" verify
  • .\mvnw.cmd -pl integration-test -P with-integration-tests -P MultiClusterIT2DualTableManualEnhanced -DskipUTs "-Dit.test=IoTDBPipeTypeConversionSemanticIT#testPipeReceiverTypeConversionSemantics" "-DfailIfNoTests=false" "-Dfailsafe.failIfNoSpecifiedTests=false" "-Drat.skip=true" verify
  • .\mvnw.cmd -pl integration-test -P with-integration-tests -P MultiClusterIT2DualTreeAutoEnhanced -DskipUTs "-Dit.test=IoTDBPipeTypeConversionSemanticIT#testAlignedStreamPipeReceiverTypeConversionSemantics" "-DfailIfNoTests=false" "-Dfailsafe.failIfNoSpecifiedTests=false" "-Drat.skip=true" verify

@Caideyipi Caideyipi left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this expected value is off because the source column is FLOAT, not DOUBLE. The literal 19700103.1 is stored as a 32-bit float before the receiver-side conversion, and it rounds to 19700104.0f. ValueConverter.convertFloatToDate then casts that float to int before parsing the date, so this row should convert to 1970-01-04, not 1970-01-03.

Could we update this expected value, or use a float value that is exactly representable enough for the intended 19700103 case?

@Caideyipi Caideyipi changed the title Add pipe type conversion semantic ITs Fix pipe receiver type conversion load path Jun 5, 2026
@jt2594838 jt2594838 merged commit 90055d5 into apache:master Jun 9, 2026
41 of 43 checks passed
@jt2594838 jt2594838 deleted the pipe-type-conversion-semantic-it branch June 9, 2026 03:21
Caideyipi added a commit to Caideyipi/iotdb that referenced this pull request Jun 9, 2026
Backport receiver-side tree-model changes to dev/1.3.

The 2.x table-model load analyzer changes and table/dual ITs are not applicable to dev/1.3.

(cherry picked from commit 90055d5)
jt2594838 pushed a commit that referenced this pull request Jun 9, 2026
Backport receiver-side tree-model changes to dev/1.3.

The 2.x table-model load analyzer changes and table/dual ITs are not applicable to dev/1.3.

(cherry picked from commit 90055d5)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants