122 lines
4.8 KiB
Markdown
122 lines
4.8 KiB
Markdown
# LLM QA
|
|
|
|
A proof of concept question-answering system for different types of text data.
|
|
|
|
Currently implemented:
|
|
|
|
- Plain text
|
|
- Markdown
|
|
|
|
## Key Features
|
|
|
|
### Dockerized development environment
|
|
|
|
- Easy, quick and reproducible setup
|
|
|
|
### Automatic pull and serve of declared models
|
|
|
|
- Ollama models are automatically pulled and served by the FastAPI server
|
|
|
|
### Detailed logging
|
|
|
|
- Key potential bottlenecks are timed and logged
|
|
|
|
#### Upsert
|
|
|
|
```console
|
|
2024-02-15 01:10:54,341 - llm_qa.services.upsert - INFO - Split `MARKDOWN` type text into 8 document chunks in 0.01 seconds
|
|
2024-02-15 01:10:54,759 - httpx - INFO - HTTP Request: POST http://text-embeddings-inference/embed "HTTP/1.1 200 OK"
|
|
2024-02-15 01:11:03,121 - httpx - INFO - HTTP Request: POST http://text-embeddings-inference/embed "HTTP/1.1 200 OK"
|
|
2024-02-15 01:11:03,140 - llm_qa.services.upsert - INFO - Upserted 8 document chunks to Qdrant collection `showcase` in 8.80 seconds
|
|
2024-02-15 01:11:03,142 - uvicorn.access - INFO - 127.0.0.1:55868 - "POST /api/v1/upsert-text HTTP/1.1" 200 OK
|
|
```
|
|
|
|
#### Chat
|
|
|
|
```console
|
|
2024-02-15 01:02:03,408 - llm_qa.dependencies - INFO - Ollama auto-pull enabled, checking if model is available
|
|
2024-02-15 01:02:03,441 - httpx - INFO - HTTP Request: POST http://ollama:11434/api/show "HTTP/1.1 200 OK"
|
|
2024-02-15 01:02:03,441 - llm_qa.dependencies - INFO - Ollama model `openchat:7b-v3.5-0106-q4_K_M` already exists
|
|
2024-02-15 01:02:03,645 - httpx - INFO - HTTP Request: POST http://text-embeddings-inference/embed "HTTP/1.1 200 OK"
|
|
2024-02-15 01:02:03,653 - llm_qa.chains.time_logger - INFO - Chain `VectorStoreRetriever` finished in 0.08 seconds
|
|
2024-02-15 01:02:23,192 - httpx - INFO - HTTP Request: POST http://text-embeddings-inference-rerank/rerank "HTTP/1.1 200 OK"
|
|
2024-02-15 01:02:23,194 - llm_qa.chains.time_logger - INFO - Chain `RerankAndTake` finished in 19.54 seconds
|
|
2024-02-15 01:02:29,817 - llm_qa.chains.time_logger - INFO - Chain `ChatOllama` finished in 6.62 seconds
|
|
2024-02-15 01:02:29,817 - llm_qa.services.chat - INFO - Chat chain finished in 26.27 seconds
|
|
2024-02-15 01:02:29,823 - uvicorn.access - INFO - 127.0.0.1:50100 - "POST /api/v1/chat HTTP/1.1" 200 OK
|
|
```
|
|
|
|
### Hierarchical document chunking
|
|
|
|
- Hierarchical text, such as markdown, is split into document chunks by headers
|
|
- All previous parent headers are also included in the chunk, separated by `...`
|
|
- This enriches the context of the chunk and solves the problem of global context being lost when splitting the text
|
|
|
|
Example:
|
|
|
|
```md
|
|
# AWS::SageMaker::ModelQualityJobDefinition MonitoringGroundTruthS3Input<a name="aws-properties-sagemaker-modelqualityjobdefinition-monitoringgroundtruths3input"></a>
|
|
...
|
|
## Syntax<a name="aws-properties-sagemaker-modelqualityjobdefinition-monitoringgroundtruths3input-syntax"></a>
|
|
...
|
|
### YAML<a name="aws-properties-sagemaker-modelqualityjobdefinition-monitoringgroundtruths3input-syntax.yaml"></a>
|
|
``` [S3Uri](#cfn-sagemaker-modelqualityjobdefinition-monitoringgroundtruths3input-s3uri): String ```
|
|
```
|
|
|
|
### Retrieval query rewriting
|
|
|
|
- After the first message, subsequent messages are rewritten to include previous messages context
|
|
- This allows for a more natural conversation flow and retrieval of more relevant chunks
|
|
|
|
Example:
|
|
|
|
```md
|
|
### User: What are all AWS regions where SageMaker is available?
|
|
### AI: SageMaker is available in most AWS regions, except for the following: Asia Pacific (Jakarta), Africa (Cape Town), Middle East (UAE), Asia Pacific (Hyderabad), Asia Pacific (Osaka), Asia Pacific (Melbourne), Europe (Milan), AWS GovCloud (US-East), Europe (Spain), and Europe (Zurich) Region.
|
|
|
|
### User: What about the Bedrock service?
|
|
### Retrieval Query: What is the availability of AWS SageMaker in relation to the Bedrock service?
|
|
```
|
|
|
|
### Reranking
|
|
|
|
- Retrieval of a larger number of document chunks is first performed using a vector store
|
|
- Then, the chunks are reranked using a reranker model
|
|
- This process more precisely selects the most relevant chunks for the user query
|
|
|
|
## Development
|
|
|
|
### Non Nvidia
|
|
|
|
If you don't have an Nvidia GPU, then remove the `nvidia` resource from the `ollama` service in the `compose.yaml` file.
|
|
|
|
### Setup
|
|
|
|
First copy the `.devcontainer/.env.example` file to `.devcontainer/.env` and adjust the settings and models to your needs.
|
|
|
|
Then simply open the project devcontainer in a compatible IDE.
|
|
This will setup all required tools and project dependencies for Python development.
|
|
It will also run Docker containers for all required services.
|
|
|
|
### Configuration
|
|
|
|
Create a `llm-qa/.env` file to override selective default environment variables located in `llm-qa/.env.default`.
|
|
|
|
### Running
|
|
|
|
To run the FastAPI server, run the `llm_qa.web` submodule:
|
|
|
|
```bash
|
|
poetry run python -m llm_qa.web
|
|
```
|
|
|
|
To run the minimal CLI client, run the `llm_qa.client` submodule:
|
|
|
|
```bash
|
|
poetry run python -m llm_qa.client
|
|
```
|
|
|
|
## Deployment
|
|
|
|
Not yet implemented.
|