Skip to content

Commit

Permalink
Merge pull request #216 from enoch3712/215-doc-refactor---document-lo…
Browse files Browse the repository at this point in the history
…ader-data

adding data to sidebar docs
  • Loading branch information
enoch3712 authored Jan 21, 2025
2 parents 57f9387 + 0d4a218 commit e2cc594
Show file tree
Hide file tree
Showing 3 changed files with 4 additions and 27 deletions.
28 changes: 1 addition & 27 deletions docs/core-concepts/document-loaders/data.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,30 +63,4 @@ The `DataLoaderConfig` class supports the following options:
|--------|------|---------|-------------|
| `content` | Any | None | Initial content to process |
| `cache_ttl` | int | 300 | Cache time-to-live in seconds |
| `supports_vision` | bool | True | Whether vision mode is supported |

## Features

- Handles pre-processed data in standard format
- Supports raw text input
- File and stream processing
- Vision mode support
- Automatic format validation
- Caching support
- Flexible input handling

## Input Types

The loader can handle:
- Pre-formatted list of dictionaries
- Raw text strings
- File paths
- IO streams

## Notes

- Vision mode is configurable
- Validates input format automatically
- Caches results based on content hash
- Supports both file and stream-based loading
- Preserves image data when in vision mode
| `supports_vision` | bool | True | Whether vision mode is supported |
2 changes: 2 additions & 0 deletions docs/core-concepts/document-loaders/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -153,6 +153,8 @@ ExtractThinker provides several specialized Document Loaders:
### Special Purpose
- [Web Loader](web-loader.md): Web page extraction with custom element handling
- [LLM Image](llm-image.md): Vision-enabled LLM processing
- [Data](data.md): Pre-processed data handling with standardized format support


### Coming Soon
- `Adobe PDF Services` <span class="coming-soon">Coming Soon</span>: Adobe's PDF extraction and analysis
Expand Down
1 change: 1 addition & 0 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,7 @@ nav:
- Markitdown: core-concepts/document-loaders/markitdown.md
- Text: core-concepts/document-loaders/txt.md
- Docling: core-concepts/document-loaders/docling.md
- Data: core-concepts/document-loaders/data.md
- Adobe PDF Services: '#'
- ABBYY FineReader: '#'
- PaddleOCR: '#'
Expand Down

0 comments on commit e2cc594

Please sign in to comment.