wsunmoon
  • Joined on Jun 14, 2023
  • Organization
Public Activities Repositories Datasets Models Computing Task Templates Follow Achievement Badge
1
Loading Heatmap…

wsunmoon synced commits to main at wsunmoon/LLaMA-Efficient-Tuning from mirror

  • 63a89710c7 [data] pad position_ids on non-FA2 packing path (fixes rotary crash for Gemma-3/4) (#10737) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

1 week ago

wsunmoon synced commits to main at wsunmoon/data-juicer from mirror

  • 0e40a8659a Release/v1.5.5 (#1037) * Release/v1.5.5 * fix: add sampling_params temperature=0.1 to test_tagging to reduce flaky failures * docs: cover partition checkpoint fix in v1.5.5 news * fix: strip _resume_requested before config re-init to unbreak get_init_configs * update uv.lock
  • 5f958a5839 [Bugfix]: Preserve partition checkpoints across Ray block-layout changes (#1036) * preserve partition checkpoints across Ray block-layout changes * fix issues
  • b397cb295e perf(deduplicator): bound MinHash permutation workspace (#1035) - Reduce token-by-permutation matrices in 8 MiB blocks and fold exact minima. - Preserve byte signatures and empty errors with deterministic differential tests. - Verify 768.0 to 271.9 MiB RSS at 4x with 9 targeted tests passing.
  • a91d3554eb fix(ray): preserve elastic actor pool concurrency (#1032)
  • a070144d5e feat: add RayAnalyzer for distributed data analysis (#1016) * feat: add RayAnalyzer for distributed stats computation - Add RayAnalyzer class that uses Ray for distributed filter stats computation, collects stats to pandas for overall analysis, and skips visualization (ColumnWise/Correlation) which sandbox doesn't use - Add stats_only parameter to RayDataset.process() and _run_single_op() to compute filter stats without actually filtering rows - Add skip_visualization config flag to optionally skip ColumnWise and Correlation analysis in the existing Analyzer - Export RayAnalyzer from data_juicer.core * feat: dispatch RayAnalyzer from dj-analyze CLI when executor_type=ray - analyze_data.py peeks executor_type from args/config to decide which analyzer class to use - RayAnalyzer falls back to local ray.init() when no cluster found * docs: add RayAnalyzer documentation and demo config - Add Distributed Data Analysis section to Distributed.md and Distributed_ZH.md - Add Ray analysis running example to Quick Start sections - Add Ray analyzer note to Data Analysis sections in QuickStart docs - Add demos/process_on_ray/configs/analyze.yaml demo config * feat: update init_configs to allow auto configuration for analyzers * test: add RayAnalyzer unit tests * feat: add RayAnalyzer configuration and update QuickStart documentation * fix: align docs to actual demo path and keep which_entry backward compat - Update Distributed.md/Distributed_ZH.md to reference the actual demos/analyze_simple/ray_analyzer.yaml instead of non-existent demos/process_on_ray/configs/analyze.yaml - Keep deprecated which_entry parameter in init_configs() with DeprecationWarning and auto-conversion to allow_auto
  • Compare 8 commits »

1 week ago

wsunmoon synced commits to gh-pages at wsunmoon/data-juicer from mirror

1 week ago

wsunmoon synced commits to main at wsunmoon/LLaMA-Efficient-Tuning from mirror

  • 887b850813 [assets] fix broken links in README (#10728) Co-authored-by: richboyneedcash <273099414+richboyneedcash@users.noreply.github.com> Co-authored-by: TRAE CLI <noreply@bytedance.com>

1 week ago

wsunmoon synced commits to main at wsunmoon/LLaMA-Efficient-Tuning from mirror

  • 84576b1408 [v1] refactor NPU kernel matching by model type (#10643)

1 week ago

wsunmoon synced commits to pr-a-unified-fs at wsunmoon/data-juicer from mirror

  • e32f1ea9a2 Merge remote-tracking branch 'origin/main' into pr-a-unified-fs
  • 025780274c fix: make scheme detection case-insensitive across the export chain Align the remaining scheme checks in the export chain with the case-insensitive dispatch introduced for exporters, so uppercase schemes (S3://, HDFS://) are handled consistently end to end: - exporter/ray_exporter: encryption-skip check and shard-path construction now use urlparse(...).scheme.lower(); scheme stripping is case-insensitive (split on '://') - default/ray/partitioned executors: S3 credential injection guard uses urlparse(...).scheme.lower() - config: remote-path detection and log-filename path extraction use urlparse (netloc/path) instead of case-sensitive startswith/replace - file_utils.is_remote_path: case-insensitive scheme check
  • 4f0066682c [Bugfix] Process partitioned Ray datasets concurrently (#1022) * Bugfix Process partitioned Ray datasets concurrently * Harden concurrent partition orchestration Co-authored-by: Qirui-jiao <luckybanana404@gmail.com> * Harden partitioned Ray resource planning * Optimize Resource Parallelism and Concurrency Control for Ray Partitioning * Fix Shared State in Concurrent Partition Execution --------- Co-authored-by: fengrui-z <drake.zfr@gmail.com>
  • a92c11f618 perf: Reduce eager Ray dataset actions (#1025) * Reduce eager Ray dataset actions * fix: skip processing for empty datasets with known schema columns() returns [] (not None) for empty datasets with a known schema, e.g. empty parquet/json files, so the previous None check let them fall through to operator processing. Guard on falsy columns_result and add regression coverage; also document the driver-memory cost of get()/ get_column() with large k or k=None.
  • 588a4f7b4e Use public Ray compute strategies (#1024)
  • Compare 6 commits »

1 week ago

wsunmoon synced commits to main at wsunmoon/data-juicer from mirror

  • cc3620d958 perf(filter): stream n-gram frequency counting (#1029) * perf(filter): stream n-gram frequency counting - Count character and word n-grams as generated to remove occurrence lists. - Add adversarial, fusion, memory regression, and benchmark coverage. - Verify 41%/52% RSS cuts at 2.4M occurrences and 21 passing tests. * chore(benchmark): keep n-gram benchmark outside PR - Publish the reproducible harness on a dedicated fork branch. - Keep the PR diff focused and avoid partial CI falling back to all tests.

1 week ago

wsunmoon synced commits to gh-pages at wsunmoon/data-juicer from mirror

  • aed49822c2 deploy: cc3620d95809a312b0f2fcda2f40182120ad3bf7

1 week ago

wsunmoon synced commits to main at wsunmoon/LLaMA-Efficient-Tuning from mirror

1 week ago

wsunmoon synced commits to main at wsunmoon/LLaMA-Efficient-Tuning from mirror

  • 62ae362455 [v1] Support multimodal data training (#10656) Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
  • 3984675dd5 fix(ci): align workflow Python version with requires-python (#10707)
  • Compare 2 commits »

2 weeks ago

wsunmoon synced commits to main at wsunmoon/data-juicer from mirror

  • 4f0066682c [Bugfix] Process partitioned Ray datasets concurrently (#1022) * Bugfix Process partitioned Ray datasets concurrently * Harden concurrent partition orchestration Co-authored-by: Qirui-jiao <luckybanana404@gmail.com> * Harden partitioned Ray resource planning * Optimize Resource Parallelism and Concurrency Control for Ray Partitioning * Fix Shared State in Concurrent Partition Execution --------- Co-authored-by: fengrui-z <drake.zfr@gmail.com>
  • a92c11f618 perf: Reduce eager Ray dataset actions (#1025) * Reduce eager Ray dataset actions * fix: skip processing for empty datasets with known schema columns() returns [] (not None) for empty datasets with a known schema, e.g. empty parquet/json files, so the previous None check let them fall through to operator processing. Guard on falsy columns_result and add regression coverage; also document the driver-memory cost of get()/ get_column() with large k or k=None.
  • 588a4f7b4e Use public Ray compute strategies (#1024)
  • Compare 3 commits »

2 weeks ago

wsunmoon synced commits to gh-pages at wsunmoon/data-juicer from mirror

2 weeks ago

wsunmoon synced commits to main at wsunmoon/LLaMA-Efficient-Tuning from mirror

  • 1b47415a2f [train] Fix hyper parallel tail accumulation loss scaling (#10705) Co-authored-by: wcrzlh <weichaoran@huawei.com>

2 weeks ago

wsunmoon synced commits to main at wsunmoon/data-juicer from mirror

  • d706754991 Pin pandas for supported Ray Data versions (#1023)
  • c1b9aa46fe Enhance and refactor tests for base_op, job utils, and config (#1018) * test: add comprehensive tests for base_op, job utils, and config functions * fix: rewrite Deduplicator/Selector/Aggregator tests to pin differentiated behavior - DeduplicatorTest: run(reduce=True) deduplicates, run(reduce=False) only hashes, compute_hash wrapped with skip_op_error support - SelectorTest: run() operates at dataset level (not .map) - AggregatorRunTest: add test that existing batch_meta is preserved - Remove PipelineInitTest (empty marker class, no behavior to test) - Remove unused Pipeline import * test: refactor and enhance test cases for config, ops, and utils modules
  • eebf92c3d7 feat(filter): support membership operators in general_field_filter (#1000) * feat(filter): support membership operators in general_field_filter * fix(filter): correct NotIn behavior with missing operands and add set-literal test
  • 33c6eebbcf fix(mapper): drop visible split_pattern delimiters in text_chunk_mapper (#999) * fix(mapper): drop visible split_pattern delimiters in text_chunk_mapper * fix(mapper): filter out None chunks from re.split in text_chunk_mapper
  • e1c19f9b36 fix(calibrate_response_mapper): honor output_pattern when set (#1017) parse_output() previously ignored self.output_pattern entirely, always returning the raw stripped text. Now applies the regex when the user provides a custom output_pattern, falling back to the original strip behavior otherwise. Pulp-Item: fine_a11f905aa2b4
  • Compare 6 commits »

2 weeks ago

wsunmoon synced commits to gh-pages at wsunmoon/data-juicer from mirror

2 weeks ago

wsunmoon synced commits to main at wsunmoon/LLaMA-Efficient-Tuning from mirror

  • 9ce6b663e9 [train] support megatron-bridge for PT/SFT training (#10645)

2 weeks ago

wsunmoon synced commits to main at wsunmoon/data-juicer from mirror

  • 29b2feb343 Add elastic data sharding with node-local Ray execution (#1015) * Add elastic data sharding with node-local Ray execution * Update 0723 * Update test. * Fix doc locks and manifests. * Update demos/README

2 weeks ago

wsunmoon synced commits to gh-pages at wsunmoon/data-juicer from mirror

  • d24b36c42f deploy: 29b2feb34398a0dc8c229f02430c185971d7f217

2 weeks ago

wsunmoon synced commits to main at wsunmoon/LLaMA-Efficient-Tuning from mirror

  • 2ebe7be611 [ci] pin ruff version and fix lint errors (#10681) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
  • 3f77101580 [v1] refactor registry plugin structure and params (#10641)
  • 19e9fe3ced [docker] improve NPU image build and distribution (#10664)
  • Compare 3 commits »

3 weeks ago

wsunmoon synced commits to main at wsunmoon/LLaMA-Efficient-Tuning from mirror

  • d0eaa10b0c [docs] update readme (#10678) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
  • a17afe5e1b [docs] update trend badge and promote PenguinHarness in readme (#10677) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
  • Compare 2 commits »

3 weeks ago

wsunmoon
Public Activities Repositories Datasets Models Computing Task Templates Follow Achievement Badge
Loading Heatmap…

wsunmoon synced commits to main at wsunmoon/LLaMA-Efficient-Tuning from mirror

  • 63a89710c7 [data] pad position_ids on non-FA2 packing path (fixes rotary crash for Gemma-3/4) (#10737) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

1 week ago

wsunmoon synced commits to main at wsunmoon/data-juicer from mirror

  • 0e40a8659a Release/v1.5.5 (#1037) * Release/v1.5.5 * fix: add sampling_params temperature=0.1 to test_tagging to reduce flaky failures * docs: cover partition checkpoint fix in v1.5.5 news * fix: strip _resume_requested before config re-init to unbreak get_init_configs * update uv.lock
  • 5f958a5839 [Bugfix]: Preserve partition checkpoints across Ray block-layout changes (#1036) * preserve partition checkpoints across Ray block-layout changes * fix issues
  • b397cb295e perf(deduplicator): bound MinHash permutation workspace (#1035) - Reduce token-by-permutation matrices in 8 MiB blocks and fold exact minima. - Preserve byte signatures and empty errors with deterministic differential tests. - Verify 768.0 to 271.9 MiB RSS at 4x with 9 targeted tests passing.
  • a91d3554eb fix(ray): preserve elastic actor pool concurrency (#1032)
  • a070144d5e feat: add RayAnalyzer for distributed data analysis (#1016) * feat: add RayAnalyzer for distributed stats computation - Add RayAnalyzer class that uses Ray for distributed filter stats computation, collects stats to pandas for overall analysis, and skips visualization (ColumnWise/Correlation) which sandbox doesn't use - Add stats_only parameter to RayDataset.process() and _run_single_op() to compute filter stats without actually filtering rows - Add skip_visualization config flag to optionally skip ColumnWise and Correlation analysis in the existing Analyzer - Export RayAnalyzer from data_juicer.core * feat: dispatch RayAnalyzer from dj-analyze CLI when executor_type=ray - analyze_data.py peeks executor_type from args/config to decide which analyzer class to use - RayAnalyzer falls back to local ray.init() when no cluster found * docs: add RayAnalyzer documentation and demo config - Add Distributed Data Analysis section to Distributed.md and Distributed_ZH.md - Add Ray analysis running example to Quick Start sections - Add Ray analyzer note to Data Analysis sections in QuickStart docs - Add demos/process_on_ray/configs/analyze.yaml demo config * feat: update init_configs to allow auto configuration for analyzers * test: add RayAnalyzer unit tests * feat: add RayAnalyzer configuration and update QuickStart documentation * fix: align docs to actual demo path and keep which_entry backward compat - Update Distributed.md/Distributed_ZH.md to reference the actual demos/analyze_simple/ray_analyzer.yaml instead of non-existent demos/process_on_ray/configs/analyze.yaml - Keep deprecated which_entry parameter in init_configs() with DeprecationWarning and auto-conversion to allow_auto
  • Compare 8 commits »

1 week ago

wsunmoon synced commits to gh-pages at wsunmoon/data-juicer from mirror

1 week ago

wsunmoon synced commits to main at wsunmoon/LLaMA-Efficient-Tuning from mirror

  • 887b850813 [assets] fix broken links in README (#10728) Co-authored-by: richboyneedcash <273099414+richboyneedcash@users.noreply.github.com> Co-authored-by: TRAE CLI <noreply@bytedance.com>

1 week ago

wsunmoon synced commits to main at wsunmoon/LLaMA-Efficient-Tuning from mirror

  • 84576b1408 [v1] refactor NPU kernel matching by model type (#10643)

1 week ago

wsunmoon synced commits to pr-a-unified-fs at wsunmoon/data-juicer from mirror

  • e32f1ea9a2 Merge remote-tracking branch 'origin/main' into pr-a-unified-fs
  • 025780274c fix: make scheme detection case-insensitive across the export chain Align the remaining scheme checks in the export chain with the case-insensitive dispatch introduced for exporters, so uppercase schemes (S3://, HDFS://) are handled consistently end to end: - exporter/ray_exporter: encryption-skip check and shard-path construction now use urlparse(...).scheme.lower(); scheme stripping is case-insensitive (split on '://') - default/ray/partitioned executors: S3 credential injection guard uses urlparse(...).scheme.lower() - config: remote-path detection and log-filename path extraction use urlparse (netloc/path) instead of case-sensitive startswith/replace - file_utils.is_remote_path: case-insensitive scheme check
  • 4f0066682c [Bugfix] Process partitioned Ray datasets concurrently (#1022) * Bugfix Process partitioned Ray datasets concurrently * Harden concurrent partition orchestration Co-authored-by: Qirui-jiao <luckybanana404@gmail.com> * Harden partitioned Ray resource planning * Optimize Resource Parallelism and Concurrency Control for Ray Partitioning * Fix Shared State in Concurrent Partition Execution --------- Co-authored-by: fengrui-z <drake.zfr@gmail.com>
  • a92c11f618 perf: Reduce eager Ray dataset actions (#1025) * Reduce eager Ray dataset actions * fix: skip processing for empty datasets with known schema columns() returns [] (not None) for empty datasets with a known schema, e.g. empty parquet/json files, so the previous None check let them fall through to operator processing. Guard on falsy columns_result and add regression coverage; also document the driver-memory cost of get()/ get_column() with large k or k=None.
  • 588a4f7b4e Use public Ray compute strategies (#1024)
  • Compare 6 commits »

1 week ago

wsunmoon synced commits to main at wsunmoon/data-juicer from mirror

  • cc3620d958 perf(filter): stream n-gram frequency counting (#1029) * perf(filter): stream n-gram frequency counting - Count character and word n-grams as generated to remove occurrence lists. - Add adversarial, fusion, memory regression, and benchmark coverage. - Verify 41%/52% RSS cuts at 2.4M occurrences and 21 passing tests. * chore(benchmark): keep n-gram benchmark outside PR - Publish the reproducible harness on a dedicated fork branch. - Keep the PR diff focused and avoid partial CI falling back to all tests.

1 week ago

wsunmoon synced commits to gh-pages at wsunmoon/data-juicer from mirror

  • aed49822c2 deploy: cc3620d95809a312b0f2fcda2f40182120ad3bf7

1 week ago

wsunmoon synced commits to main at wsunmoon/LLaMA-Efficient-Tuning from mirror

1 week ago

wsunmoon synced commits to main at wsunmoon/LLaMA-Efficient-Tuning from mirror

  • 62ae362455 [v1] Support multimodal data training (#10656) Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
  • 3984675dd5 fix(ci): align workflow Python version with requires-python (#10707)
  • Compare 2 commits »

2 weeks ago

wsunmoon synced commits to main at wsunmoon/data-juicer from mirror

  • 4f0066682c [Bugfix] Process partitioned Ray datasets concurrently (#1022) * Bugfix Process partitioned Ray datasets concurrently * Harden concurrent partition orchestration Co-authored-by: Qirui-jiao <luckybanana404@gmail.com> * Harden partitioned Ray resource planning * Optimize Resource Parallelism and Concurrency Control for Ray Partitioning * Fix Shared State in Concurrent Partition Execution --------- Co-authored-by: fengrui-z <drake.zfr@gmail.com>
  • a92c11f618 perf: Reduce eager Ray dataset actions (#1025) * Reduce eager Ray dataset actions * fix: skip processing for empty datasets with known schema columns() returns [] (not None) for empty datasets with a known schema, e.g. empty parquet/json files, so the previous None check let them fall through to operator processing. Guard on falsy columns_result and add regression coverage; also document the driver-memory cost of get()/ get_column() with large k or k=None.
  • 588a4f7b4e Use public Ray compute strategies (#1024)
  • Compare 3 commits »

2 weeks ago

wsunmoon synced commits to gh-pages at wsunmoon/data-juicer from mirror

2 weeks ago

wsunmoon synced commits to main at wsunmoon/LLaMA-Efficient-Tuning from mirror

  • 1b47415a2f [train] Fix hyper parallel tail accumulation loss scaling (#10705) Co-authored-by: wcrzlh <weichaoran@huawei.com>

2 weeks ago

wsunmoon synced commits to main at wsunmoon/data-juicer from mirror

  • d706754991 Pin pandas for supported Ray Data versions (#1023)
  • c1b9aa46fe Enhance and refactor tests for base_op, job utils, and config (#1018) * test: add comprehensive tests for base_op, job utils, and config functions * fix: rewrite Deduplicator/Selector/Aggregator tests to pin differentiated behavior - DeduplicatorTest: run(reduce=True) deduplicates, run(reduce=False) only hashes, compute_hash wrapped with skip_op_error support - SelectorTest: run() operates at dataset level (not .map) - AggregatorRunTest: add test that existing batch_meta is preserved - Remove PipelineInitTest (empty marker class, no behavior to test) - Remove unused Pipeline import * test: refactor and enhance test cases for config, ops, and utils modules
  • eebf92c3d7 feat(filter): support membership operators in general_field_filter (#1000) * feat(filter): support membership operators in general_field_filter * fix(filter): correct NotIn behavior with missing operands and add set-literal test
  • 33c6eebbcf fix(mapper): drop visible split_pattern delimiters in text_chunk_mapper (#999) * fix(mapper): drop visible split_pattern delimiters in text_chunk_mapper * fix(mapper): filter out None chunks from re.split in text_chunk_mapper
  • e1c19f9b36 fix(calibrate_response_mapper): honor output_pattern when set (#1017) parse_output() previously ignored self.output_pattern entirely, always returning the raw stripped text. Now applies the regex when the user provides a custom output_pattern, falling back to the original strip behavior otherwise. Pulp-Item: fine_a11f905aa2b4
  • Compare 6 commits »

2 weeks ago

wsunmoon synced commits to gh-pages at wsunmoon/data-juicer from mirror

2 weeks ago

wsunmoon synced commits to main at wsunmoon/LLaMA-Efficient-Tuning from mirror

  • 9ce6b663e9 [train] support megatron-bridge for PT/SFT training (#10645)

2 weeks ago

wsunmoon synced commits to main at wsunmoon/data-juicer from mirror

  • 29b2feb343 Add elastic data sharding with node-local Ray execution (#1015) * Add elastic data sharding with node-local Ray execution * Update 0723 * Update test. * Fix doc locks and manifests. * Update demos/README

2 weeks ago

wsunmoon synced commits to gh-pages at wsunmoon/data-juicer from mirror

  • d24b36c42f deploy: 29b2feb34398a0dc8c229f02430c185971d7f217

2 weeks ago

wsunmoon synced commits to main at wsunmoon/LLaMA-Efficient-Tuning from mirror

  • 2ebe7be611 [ci] pin ruff version and fix lint errors (#10681) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
  • 3f77101580 [v1] refactor registry plugin structure and params (#10641)
  • 19e9fe3ced [docker] improve NPU image build and distribution (#10664)
  • Compare 3 commits »

3 weeks ago

wsunmoon synced commits to main at wsunmoon/LLaMA-Efficient-Tuning from mirror

  • d0eaa10b0c [docs] update readme (#10678) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
  • a17afe5e1b [docs] update trend badge and promote PenguinHarness in readme (#10677) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
  • Compare 2 commits »

3 weeks ago

Please read the following content carefully:

Dear OpenI User

Thank you for your continuous support to the Openl Qizhi Community AI Collaboration Platform. In order to protect your usage rights and ensure network security, we updated the Openl Qizhi Community AI Collaboration Platform Usage Agreement in January 2024. The updated agreement specifies that users are prohibited from using intranet penetration tools. After you click "Agree and continue", you can continue to use our services. Thank you for your cooperation and understanding.

For more agreement content, please refer to the《Openl Qizhi Community AI Collaboration Platform Usage Agreement》

Agree and continue
Disagree, exit