docs: Add langchain over time (langchain-ai#21434)

Co-authored-by: Erick Friis <[email protected]>

docs/Makefile

@@ -13,7 +13,7 @@ OUTPUT_NEW_DOCS_DIR = $(OUTPUT_NEW_DIR)/docs
 
 PYTHON = .venv/bin/python
 
-PARTNER_DEPS_LIST := $(shell ls -1 ../libs/partners | grep -vE "airbyte|ibm" | xargs -I {} echo "../libs/partners/{}" | tr '\n' ' ')
+PARTNER_DEPS_LIST := $(shell find ../libs/partners -mindepth 1 -maxdepth 1 -type d -exec test -e "{}/pyproject.toml" \; -print | grep -vE "airbyte|ibm" | tr '\n' ' ')
 
 PORT ?= 3001
 

docs/docs/changelog/core.mdx → docs/docs/changes/changelog/core.mdx

@@ -1,27 +1,10 @@
 # langchain-core
 
-## 0.1.7 (Jan 5, 2024)
-
-#### Deleted
-
-No deletions.
+## 0.1.x
 
 #### Deprecated
 
 - `BaseChatModel` methods `__call__`, `call_as_llm`, `predict`, `predict_messages`. Will be removed in 0.2.0. Use `BaseChatModel.invoke` instead.
 - `BaseChatModel` methods `apredict`, `apredict_messages`. Will be removed in 0.2.0. Use `BaseChatModel.ainvoke` instead.
 - `BaseLLM` methods `__call__, `predict`, `predict_messages`. Will be removed in 0.2.0. Use `BaseLLM.invoke` instead.
-- `BaseLLM` methods `apredict`, `apredict_messages`. Will be removed in 0.2.0. Use `BaseLLM.ainvoke` instead.
-
-#### Fixed
-
-- Restrict recursive URL scraping: [#15559](https://github.com/langchain-ai/langchain/pull/15559)
-
-#### Added
-
-No additions.
-
-#### Beta
-
-- Marked `langchain_core.load.load` and `langchain_core.load.loads` as beta.
-- Marked `langchain_core.beta.runnables.context.ContextGet` and `langchain_core.beta.runnables.context.ContextSet` as beta.
+- `BaseLLM` methods `apredict`, `apredict_messages`. Will be removed in 0.2.0. Use `BaseLLM.ainvoke` instead.

docs/docs/changelog/langchain.mdx → docs/docs/changes/changelog/langchain.mdx

@@ -1,16 +1,73 @@
 # langchain
 
+## 0.2.0
+
+### Deleted
+
+As of release 0.2.0, `langchain` is required to be integration-agnostic. This means that code in `langchain`  should not by default instantiate any specific chat models, llms, embedding models, vectorstores etc; instead, the user will be required to specify those explicitly.
+
+The following functions and classes require an explicit LLM to be passed as an argument:
+
+- `langchain.agents.agent_toolkits.vectorstore.toolkit.VectorStoreToolkit`
+- `langchain.agents.agent_toolkits.vectorstore.toolkit.VectorStoreRouterToolkit`
+- `langchain.chains.openai_functions.get_openapi_chain`
+- `langchain.chains.router.MultiRetrievalQAChain.from_retrievers`
+- `langchain.indexes.VectorStoreIndexWrapper.query`
+- `langchain.indexes.VectorStoreIndexWrapper.query_with_sources`
+- `langchain.indexes.VectorStoreIndexWrapper.aquery_with_sources`
+- `langchain.chains.flare.FlareChain`
+
+The following classes now require passing an explicit Embedding model as an argument:
+
+- `langchain.indexes.VectostoreIndexCreator`
+
+The following code has been removed:
+
+- `langchain.natbot.NatBotChain.from_default` removed in favor of the `from_llm` class method.
+
+### Deprecated
+
+We have two main types of deprecations:
+
+1. Code that was moved from `langchain` into another package (e.g, `langchain-community`)
+
+If you try to import it from `langchain`, the import will keep on working, but will raise a deprecation warning. The warning will provide a replacement import statement.
+
+```python
+python -c "from langchain.document_loaders.markdown import UnstructuredMarkdownLoader"
+
+```
+
+```python
+LangChainDeprecationWarning: Importing UnstructuredMarkdownLoader from langchain.document_loaders is deprecated. Please replace deprecated imports:
+
+>> from langchain.document_loaders import UnstructuredMarkdownLoader
+
+with new imports of:
+
+>> from langchain_community.document_loaders import UnstructuredMarkdownLoader
+```
+
+We will continue supporting the imports in `langchain` until release 0.4 as long as the relevant package where the code lives is installed. (e.g., as long as `langchain_community` is installed.)
+
+However, we advise for users to not rely on these imports and instead migrate to the new imports. To help with this process, we’re releasing a migration script via the LangChain CLI. See further instructions in migration guide.
+
+1. Code that has better alternatives available and will eventually be removed, so there’s only a single way to do things. (e.g., `predict_messages` method in ChatModels has been deprecated in favor of `invoke`).
+
+Many of these were marked for removal in 0.2. We have bumped the removal to 0.3.
+
+
 ## 0.1.0 (Jan 5, 2024)
 
-#### Deleted
+### Deleted
 
 No deletions.
 
-#### Deprecated
+### Deprecated
 
 Deprecated classes and methods will be removed in 0.2.0
 
-| Deprecated                | Alternative                       | Reason                                         |
+| Deprecated                      | Alternative                       | Reason                                         |
 |---------------------------------|-----------------------------------|------------------------------------------------|
 | ChatVectorDBChain               | ConversationalRetrievalChain      | More general to all retrievers                 |
 | create_ernie_fn_chain           | create_ernie_fn_runnable          | Use LCEL under the hood                        |

docs/docs/langsmith/walkthrough.ipynb → docs/docs/langsmith/index.ipynb

@@ -7,6 +7,25 @@
     "tags": []
    },
    "source": [
+    "# 🦜🛠️ LangSmith\n",
+    "\n",
+    "[LangSmith](https://smith.langchain.com) helps you trace and evaluate your language model applications and intelligent agents to help you\n",
+    "move from prototype to production.\n",
+    "\n",
+    "Check out the [interactive walkthrough](/docs/langsmith/walkthrough) to get started.\n",
+    "\n",
+    "For more information, please refer to the [LangSmith documentation](https://docs.smith.langchain.com/).\n",
+    "\n",
+    "For tutorials and other end-to-end examples demonstrating ways to integrate LangSmith in your workflow,\n",
+    "check out the [LangSmith Cookbook](https://github.com/langchain-ai/langsmith-cookbook). Some of the guides therein include:\n",
+    "\n",
+    "- Leveraging user feedback in your JS application ([link](https://github.com/langchain-ai/langsmith-cookbook/blob/main/feedback-examples/nextjs/README.md)).\n",
+    "- Building an automated feedback pipeline ([link](https://github.com/langchain-ai/langsmith-cookbook/blob/main/feedback-examples/algorithmic-feedback/algorithmic_feedback.ipynb)).\n",
+    "- How to evaluate and audit your RAG workflows ([link](https://github.com/langchain-ai/langsmith-cookbook/tree/main/testing-examples/qa-correctness)).\n",
+    "- How to fine-tune an LLM on real usage data ([link](https://github.com/langchain-ai/langsmith-cookbook/blob/main/fine-tuning-examples/export-to-openai/fine-tuning-on-chat-runs.ipynb)).\n",
+    "- How to use the [LangChain Hub](https://smith.langchain.com/hub) to version your prompts ([link](https://github.com/langchain-ai/langsmith-cookbook/blob/main/hub-examples/retrieval-qa-chain/retrieval-qa.ipynb))\n",
+    "\n",
+    "\n",
     "# LangSmith Walkthrough\n",
     "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/langchain-ai/langchain/blob/master/docs/docs/langsmith/walkthrough.ipynb)\n",
     "\n",

docs/docs/versions/overview.mdx

@@ -0,0 +1,102 @@
+---
+sidebar_position: 0
+sidebar_label: Overview
+---
+
+# LangChain Over Time
+
+## What’s new in LangChain?
+
+The following features have been added during the development of 0.1.x:
+
+- Better streaming support via the [Event Streaming API](https://python.langchain.com/docs/expression_language/streaming/#using-stream-events)
+- [Standardized tool calling support](https://blog.langchain.dev/tool-calling-with-langchain/)
+- A standardized interface for [structuring output](https://github.com/langchain-ai/langchain/discussions/18154)
+- [@chain decorator](https://python.langchain.com/docs/expression_language/how_to/decorator/) to more easily create **RunnableLambdas**
+- https://python.langchain.com/docs/expression_language/how_to/inspect/
+- In Python, better async support for many core abstractions (thank you [@cbornet](https://github.com/cbornet)!!)
+- Include response metadata in `AIMessage` to make it easy to access raw output from the underlying models
+- Tooling to visualize [your runnables](https://python.langchain.com/docs/expression_language/how_to/inspect/) or [your langgraph app](https://github.com/langchain-ai/langgraph/blob/main/examples/visualization.ipynb)
+- Interoperability of chat message histories across most providers
+- [Over 20+ partner packages in python](https://python.langchain.com/docs/integrations/platforms/) for popular integrations
+
+## What’s coming to LangChain?
+
+- We’ve been working hard on [langgraph](https://python.langchain.com/docs/langgraph/). We will be building more capabilities on top of it and focusing on making it the go-to framework for agent architectures.
+- Vectorstores V2! We’ll be revisiting our vectorstores abstractions to help improve usability and reliability.
+- Better documentation and versioned docs!
+- We’re planning a breaking release (0.3.0) sometime between July-September to [upgrade to full support of Pydantic 2](https://github.com/langchain-ai/langchain/discussions/19339), and will drop support for Pydantic 1 (including objects originating from the `v1` namespace of Pydantic 2).
+
+## What changed?
+
+Due to the rapidly evolving field, LangChain has also evolved rapidly.
+
+This document serves to outline at a high level what has changed and why.
+
+### TLDR
+
+**As of 0.2.0:**
+
+- This release completes the work that we started with release 0.1.0 by removing the dependency of `langchain` on `langchain-community`.
+- `langchain` package no longer requires `langchain-community` . Instead `langchain-community` will now depend on `langchain-core` and `langchain` .
+- User code that still relies on deprecated imports from `langchain` will continue to work as long `langchain_community` is installed. These imports will start raising errors in release 0.4.x.
+
+**As of 0.1.0:**
+
+- `langchain` was split into the following component packages: `langchain-core`, `langchain`, `langchain-community`, `langchain-[partner]` to improve the usability of langchain code in production settings. You can read more about it on our [blog](https://blog.langchain.dev/langchain-v0-1-0/).
+
+### Ecosystem Organization
+
+By the release of 0.1.0, LangChain had grown to a large ecosystem with many integrations and a large community.
+
+To improve the usability of LangChain in production, we split the single `langchain` package into multiple packages. This allowed us to create a good foundation architecture for the LangChain ecosystem and improve the usability of `langchain` in production.
+
+Here is the high level break down of the Eco-system:
+
+- **langchain-core**:  contains core abstractions involving LangChain Runnables, tooling for observability, and base implementations of important abstractions (e.g., Chat Models).
+- **langchain:** contains generic code that is built using interfaces defined in `langchain-core`. This package is for code that generalizes well across different implementations of specific interfaces. For example, `create_tool_calling_agent` works across chat models that support [tool calling capabilities](https://blog.langchain.dev/tool-calling-with-langchain/).
+- **langchain-community**: community maintained 3rd party integrations. Contains integrations based on interfaces defined in **langchain-core**. Maintained by the LangChain community.
+- **Partner Packages (e.g., langchain-[partner])**: Partner packages are packages dedicated to especially popular integrations (e.g., `langchain-openai`, `langchain-anthropic` etc.). The dedicated packages generally benefit from better reliability and support.
+- `langgraph`: Build robust and stateful multi-actor applications with LLMs by modeling steps as edges and nodes in a graph.
+- `langserve`: Deploy LangChain chains as REST APIs.
+
+
+In the 0.1.0 release, `langchain-community` was retained as required a dependency of `langchain`.
+
+This allowed imports of vectorstores, chat models, and other integrations to continue working through `langchain`
+rather than forcing users to update all of their imports to `langchain-community`.
+
+For the 0.2.0 release, we’re removing the dependency of `langchain` on `langchain-community`. This is something we’ve been planning to do since the 0.1 release because we believe this is the right package architecture.
+
+Old imports will continue to work as long as `langchain-community` is installed. These imports will be removed in the 0.4.0 release.
+
+To understand why we think breaking the dependency of `langchain` on `langchain-community` is best we should understand what each package is meant to do.
+
+`langchain` is meant to contain high-level chains and agent architectures. The logic in these should be specified at the level of abstractions like `ChatModel` and `Retriever`, and should not be specific to any one integration. This has two main benefits:
+
+1. `langchain` is fairly lightweight. Here is the full list of required dependencies (after the split)
+
+    ```toml
+    python = ">=3.8.1,<4.0"
+    langchain-core = "^0.2.0"
+    langchain-text-splitters = ">=0.0.1,<0.1"
+    langsmith = "^0.1.17"
+    pydantic = ">=1,<3"
+    SQLAlchemy = ">=1.4,<3"
+    requests = "^2"
+    PyYAML = ">=5.3"
+    numpy = "^1"
+    aiohttp = "^3.8.3"
+    tenacity = "^8.1.0"
+    jsonpatch = "^1.33"
+    ```
+
+2. `langchain` chains/agents are largely integration-agnostic, which makes it easy to experiment with different integrations and future-proofs your code should there be issues with one specific integration.
+
+There is also a third less tangible benefit which is that being integration-agnostic forces us to find only those very generic abstractions and architectures which generalize well across integrations. Given how general the abilities of the foundational tech are, and how quickly the space is moving, having generic architectures is a good way of future-proofing your applications.
+
+`langchain-community` is intended to have all integration-specific components that are not yet being maintained in separate `langchain-{partner}` packages. Today this is still the majority of integrations and a lot of code. This code is primarily contributed by the community, while `langchain` is largely written by core maintainers. All of these integrations use optional dependencies and conditional imports, which prevents dependency bloat and conflicts but means compatible dependency versions are not made explicit. Given the volume of integrations in `langchain-community` and the speed at which integrations change, it’s very hard to follow semver versioning, and we currently don’t.
+
+All of which is to say that there’s no large benefits to `langchain` depending on `langchain-community` and some obvious downsides: the functionality in `langchain` should be integration agnostic anyways, `langchain-community` can’t be properly versioned, and depending on `langchain-community` increases the [vulnerability surface](https://github.com/langchain-ai/langchain/discussions/19083) of `langchain`.
+
+For more context about the reason for the organization please see our blog: https://blog.langchain.dev/langchain-v0-1-0/

docs/docs/packages.mdx → docs/docs/versions/packages.mdx

@@ -1,3 +1,8 @@
+---
+sidebar_position: 3
+sidebar_label: Packages
+---
+
 # 📕 Package Versioning
 
 As of now, LangChain has an ad hoc release process: releases are cut with high frequency by
@@ -23,7 +28,7 @@ Patch version increases will occur for:
 
 ## `langchain`
 
-`langchain` is currently on version `0.1.x`
+`langchain` is currently on version `0.2.x`
 
 Minor version increases will occur for:
 
@@ -34,16 +39,13 @@ Patch version increases will occur for:
 - Bug fixes
 - New features
 - Any changes to private interfaces
-- Any changes to `beta` features
-
-We are working on the `langchain` v0.2 release, which will have some breaking changes to legacy Chains and Agents.
-Additionally, we will remove `langchain-community` as a dependency and stop re-exporting integrations that have been moved to `langchain-community`.
+- Any changes to `beta` features.
 
 ## `langchain-community`
 
-`langchain-community` is currently on version `0.0.x`
+`langchain-community` is currently on version `0.2.x`
 
-All changes will be accompanied by a patch version increase.
+All changes will be accompanied by the same type of version increase as changes in `langchain`.
 
 ## `langchain-experimental`