feat(rust): support skipping fields bytes when deserializing in compatible mode#2545
Merged
chaokunyang merged 1 commit intoapache:mainfrom Aug 29, 2025
Merged
feat(rust): support skipping fields bytes when deserializing in compatible mode#2545chaokunyang merged 1 commit intoapache:mainfrom
chaokunyang merged 1 commit intoapache:mainfrom
Conversation
Contributor
Author
|
Oh, maybe this filename isn’t quite appropriate? |
urlyy
commented
Aug 28, 2025
| @@ -113,14 +113,7 @@ fn deserialize_compatible(fields: &[&Field]) -> TokenStream { | |||
| _ => { | |||
| // skip bytes | |||
| println!("no need to deserialize {:?}:{:?}", _field.field_name.as_str(), _field.field_type); | |||
Contributor
Author
There was a problem hiding this comment.
Maybe need to be replaced as println!("skip {:?}:{:?}", _field.field_name.as_str(), _field.field_type);. Or just remove this. I just want to print a warning or log to user.
urlyy
commented
Aug 28, 2025
| Err(anyhow!("Unknown ref flag, value:{ref_flag}"))? | ||
| } | ||
| } else { | ||
| unreachable!() |
Contributor
Author
There was a problem hiding this comment.
Option or Enum or some other will enter here. The implementation and unit tests will be added in other PRs.
Contributor
Author
|
@chaokunyang Please review. |
chaokunyang
approved these changes
Aug 29, 2025
chaokunyang
changed the title
chaokunyang
changed the title
This was referenced Oct 6, 2025
chaokunyang
added a commit
that referenced
this pull request
Oct 13, 2025
…field id (#2758) ## Why? <!-- Describe the purpose of this PR. --> ## What does this PR do? This pr implemented a new fory_read_compatible macro function to use match by assigned field id for better performance. - When read type meta, assign field id based on local field type info. If type info are consistent(excluding nullability), then assign remote field to local field. Otherwise, skip this remote field value when deserializing, use default value instead. - When generating fory_read_compatible code, use match by field id arm, which will compiled to switch. This is much more efficient compared to previous compare field by string - Skip using Option is local field is primitive and not Option. Now the macro generate much more efficient and compact code: ```rust fn fory_read_compatible( fory: &fory_core::fory::Fory, context: &mut fory_core::resolver::context::ReadContext, ) -> Result<Self, fory_core::error::Error> { let remote_type_id = context.reader.read_varuint32(); let meta_index = context.reader.read_varuint32(); let meta = context.get_meta(meta_index as usize); let fields = { let meta = context.get_meta(meta_index as usize); meta.get_field_infos().clone() }; let mut _f7: i16 = 0 as i16; let mut _f5: i8 = 0 as i8; let mut _last: i8 = 0 as i8; let mut _f4: Option<String> = None; let mut _f3: Option<Vec<i8>> = None; let mut _f6: Option<Vec<i16>> = None; let mut _f1: Option<HashMap<i8, Vec<i8>>> = None; let local_type_def = fory .get_type_resolver() .get_type_info(std::any::TypeId::of::<Self>()) .get_type_def(); let high_bytes = &local_type_def[..8]; let local_type_hash = i64::from_le_bytes(high_bytes.try_into().unwrap()); if meta.get_hash() == local_type_hash { <Self as fory_core::serializer::Serializer>::fory_read_data( fory, context, false, ) } else { for _field in fields.iter() { match _field.field_id { 0i16 => { if !&_field.field_type.nullable { _f7 = fory_core::serializer::read_ref_info_data::< i16, >(fory, context, true, true, false)?; } else { if (context.reader.read_bool()) { _f7 = <i16 as fory_core::serializer::ForyDefault>::fory_default(); } else { _f7 = fory_core::serializer::read_ref_info_data::< i16, >(fory, context, true, true, false)?; } } } 1i16 => { if !&_field.field_type.nullable { _f5 = fory_core::serializer::read_ref_info_data::< i8, >(fory, context, true, true, false)?; } else { if (context.reader.read_bool()) { _f5 = <i8 as fory_core::serializer::ForyDefault>::fory_default(); } else { _f5 = fory_core::serializer::read_ref_info_data::< i8, >(fory, context, true, true, false)?; } } } 2i16 => { if !&_field.field_type.nullable { _last = fory_core::serializer::read_ref_info_data::< i8, >(fory, context, true, true, false)?; } else { if (context.reader.read_bool()) { _last = <i8 as fory_core::serializer::ForyDefault>::fory_default(); } else { _last = fory_core::serializer::read_ref_info_data::< i8, >(fory, context, true, true, false)?; } } } 3i16 => { if !&_field.field_type.nullable { _f4 = Some( fory_core::serializer::read_ref_info_data::< String, >(fory, context, true, false, false)?, ); } else { if (context.reader.read_bool()) { _f4 = Some( <String as fory_core::serializer::ForyDefault>::fory_default(), ); } else { _f4 = Some( fory_core::serializer::read_ref_info_data::< String, >(fory, context, true, false, false)?, ); } } } 4i16 => { if !&_field.field_type.nullable { _f3 = Some( fory_core::serializer::read_ref_info_data::< Vec<i8>, >(fory, context, true, false, false)?, ); } else { if (context.reader.read_bool()) { _f3 = Some( <Vec< i8, > as fory_core::serializer::ForyDefault>::fory_default(), ); } else { _f3 = Some( fory_core::serializer::read_ref_info_data::< Vec<i8>, >(fory, context, true, false, false)?, ); } } } 5i16 => { if !&_field.field_type.nullable { _f6 = Some( fory_core::serializer::read_ref_info_data::< Vec<i16>, >(fory, context, true, false, false)?, ); } else { if (context.reader.read_bool()) { _f6 = Some( <Vec< i16, > as fory_core::serializer::ForyDefault>::fory_default(), ); } else { _f6 = Some( fory_core::serializer::read_ref_info_data::< Vec<i16>, >(fory, context, true, false, false)?, ); } } } 6i16 => { if !&_field.field_type.nullable { _f1 = Some( fory_core::serializer::read_ref_info_data::< HashMap<i8, Vec<i8>>, >(fory, context, true, false, false)?, ); } else { if (context.reader.read_bool()) { _f1 = Some( <HashMap< i8, Vec<i8>, > as fory_core::serializer::ForyDefault>::fory_default(), ); } else { _f1 = Some( fory_core::serializer::read_ref_info_data::< HashMap<i8, Vec<i8>>, >(fory, context, true, false, false)?, ); } } } _ => { let field_type = &_field.field_type; let read_ref_flag = fory_core::serializer::skip::get_read_ref_flag( &field_type, ); fory_core::serializer::skip::skip_field_value( fory, context, &field_type, read_ref_flag, ) .unwrap(); } } } Ok(Self { f7: _f7, f5: _f5, last: _last, f4: _f4.unwrap_or_default(), f3: _f3.unwrap_or_default(), f6: _f6.unwrap_or_default(), f1: _f1.unwrap_or_default(), }) } ``` ## Related issues #2492 #2545 Closes #2761 ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/fory/issues/new/choose) describing the need to do so and update the document if necessary. Delete section if not applicable. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. Delete section if not applicable. -->
rajuyadav03
pushed a commit
to rajuyadav03/fory
that referenced
this pull request
Oct 20, 2025
…field id (apache#2758) ## Why? <!-- Describe the purpose of this PR. --> ## What does this PR do? This pr implemented a new fory_read_compatible macro function to use match by assigned field id for better performance. - When read type meta, assign field id based on local field type info. If type info are consistent(excluding nullability), then assign remote field to local field. Otherwise, skip this remote field value when deserializing, use default value instead. - When generating fory_read_compatible code, use match by field id arm, which will compiled to switch. This is much more efficient compared to previous compare field by string - Skip using Option is local field is primitive and not Option. Now the macro generate much more efficient and compact code: ```rust fn fory_read_compatible( fory: &fory_core::fory::Fory, context: &mut fory_core::resolver::context::ReadContext, ) -> Result<Self, fory_core::error::Error> { let remote_type_id = context.reader.read_varuint32(); let meta_index = context.reader.read_varuint32(); let meta = context.get_meta(meta_index as usize); let fields = { let meta = context.get_meta(meta_index as usize); meta.get_field_infos().clone() }; let mut _f7: i16 = 0 as i16; let mut _f5: i8 = 0 as i8; let mut _last: i8 = 0 as i8; let mut _f4: Option<String> = None; let mut _f3: Option<Vec<i8>> = None; let mut _f6: Option<Vec<i16>> = None; let mut _f1: Option<HashMap<i8, Vec<i8>>> = None; let local_type_def = fory .get_type_resolver() .get_type_info(std::any::TypeId::of::<Self>()) .get_type_def(); let high_bytes = &local_type_def[..8]; let local_type_hash = i64::from_le_bytes(high_bytes.try_into().unwrap()); if meta.get_hash() == local_type_hash { <Self as fory_core::serializer::Serializer>::fory_read_data( fory, context, false, ) } else { for _field in fields.iter() { match _field.field_id { 0i16 => { if !&_field.field_type.nullable { _f7 = fory_core::serializer::read_ref_info_data::< i16, >(fory, context, true, true, false)?; } else { if (context.reader.read_bool()) { _f7 = <i16 as fory_core::serializer::ForyDefault>::fory_default(); } else { _f7 = fory_core::serializer::read_ref_info_data::< i16, >(fory, context, true, true, false)?; } } } 1i16 => { if !&_field.field_type.nullable { _f5 = fory_core::serializer::read_ref_info_data::< i8, >(fory, context, true, true, false)?; } else { if (context.reader.read_bool()) { _f5 = <i8 as fory_core::serializer::ForyDefault>::fory_default(); } else { _f5 = fory_core::serializer::read_ref_info_data::< i8, >(fory, context, true, true, false)?; } } } 2i16 => { if !&_field.field_type.nullable { _last = fory_core::serializer::read_ref_info_data::< i8, >(fory, context, true, true, false)?; } else { if (context.reader.read_bool()) { _last = <i8 as fory_core::serializer::ForyDefault>::fory_default(); } else { _last = fory_core::serializer::read_ref_info_data::< i8, >(fory, context, true, true, false)?; } } } 3i16 => { if !&_field.field_type.nullable { _f4 = Some( fory_core::serializer::read_ref_info_data::< String, >(fory, context, true, false, false)?, ); } else { if (context.reader.read_bool()) { _f4 = Some( <String as fory_core::serializer::ForyDefault>::fory_default(), ); } else { _f4 = Some( fory_core::serializer::read_ref_info_data::< String, >(fory, context, true, false, false)?, ); } } } 4i16 => { if !&_field.field_type.nullable { _f3 = Some( fory_core::serializer::read_ref_info_data::< Vec<i8>, >(fory, context, true, false, false)?, ); } else { if (context.reader.read_bool()) { _f3 = Some( <Vec< i8, > as fory_core::serializer::ForyDefault>::fory_default(), ); } else { _f3 = Some( fory_core::serializer::read_ref_info_data::< Vec<i8>, >(fory, context, true, false, false)?, ); } } } 5i16 => { if !&_field.field_type.nullable { _f6 = Some( fory_core::serializer::read_ref_info_data::< Vec<i16>, >(fory, context, true, false, false)?, ); } else { if (context.reader.read_bool()) { _f6 = Some( <Vec< i16, > as fory_core::serializer::ForyDefault>::fory_default(), ); } else { _f6 = Some( fory_core::serializer::read_ref_info_data::< Vec<i16>, >(fory, context, true, false, false)?, ); } } } 6i16 => { if !&_field.field_type.nullable { _f1 = Some( fory_core::serializer::read_ref_info_data::< HashMap<i8, Vec<i8>>, >(fory, context, true, false, false)?, ); } else { if (context.reader.read_bool()) { _f1 = Some( <HashMap< i8, Vec<i8>, > as fory_core::serializer::ForyDefault>::fory_default(), ); } else { _f1 = Some( fory_core::serializer::read_ref_info_data::< HashMap<i8, Vec<i8>>, >(fory, context, true, false, false)?, ); } } } _ => { let field_type = &_field.field_type; let read_ref_flag = fory_core::serializer::skip::get_read_ref_flag( &field_type, ); fory_core::serializer::skip::skip_field_value( fory, context, &field_type, read_ref_flag, ) .unwrap(); } } } Ok(Self { f7: _f7, f5: _f5, last: _last, f4: _f4.unwrap_or_default(), f3: _f3.unwrap_or_default(), f6: _f6.unwrap_or_default(), f1: _f1.unwrap_or_default(), }) } ``` ## Related issues apache#2492 apache#2545 Closes apache#2761 ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/fory/issues/new/choose) describing the need to do so and update the document if necessary. Delete section if not applicable. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. Delete section if not applicable. -->
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Support skipping unneeded field bytes when deserializing in compatible mode.
Related issues
#2531