Development Workflow

Project organization and file layout for the Data-AI-Hub Agent

apps/data-ai-hub-agent

Application deployment and runtime configuration

📁 apps/

📁 data-ai-hub-agent/

📁 databricks/

📁 mypy_cache/

📁 pytest_cache/

📁 ruff_cache/

📁 venv/

📁 dist/

📁 docs/

📁 htmlcov/

📁 mlruns/

📁 src/data_ai_hub_agent/

📁 conf/

⚙️ dev.yml

⚙️ prod.yml

⚙️ staging.yml

📁 infrastructure/

📁 __pycache__/

📁 clients/

📁 endpoints/

📁 event_handlers/

🐍 __init__.py

🐍 api.py

🐍 database.py

🐍 dependencies.py

📁 jobs/

🐍 __init__.py

🐍 data_lifecycle_job.py

📁 services/

📁 __pycache__/

🐍 __init__.py

🐍 langgraph_service.py

🐍 observability_service.py

🐍 production_agent_service.py

🐍 production_agent.py

🐍 slack_thread_service.py

🐍 test_agent.py

🐍 vector_search_service.py

🐍 config.py

🐍 enums.py

🐍 models.py

📁 tests/

📁 .coverage/

⚙️ .gitlab-ci.yml

⚙️ app.yml

⚙️ databricks.yml

📄 poetry.lock

📄 poetry.toml

📄 pyproject.toml

📄 requirements.txt

projects/common/data-ai-hub-agent

Shared libraries and common components

📁 projects/

📁 adas/

📁 commercial/

📁 common/

📁 data-ai-hub-agent/

📁 databricks/

📁 mypy_cache/

📁 pytest_cache/

📁 ruff_cache/

📁 venv/

📁 dist/

📁 docs/

📁 htmlcov/

📁 src/data_ai_hub_agent/

📁 __pycache__/

📁 conf/

⚙️ dev.yml

⚙️ prod.yml

⚙️ staging.yml

⚙️ default.yml

📁 pipelines/

📁 __pycache__/

🐍 __init__.py

🐍 code_doc.py

🐍 confluence.py

🐍 databricks.py

🐍 slack.py

📁 steps/

📁 __pycache__/

📁 code_doc/

📁 confluence/

📁 databricks/

📁 gold/

📁 silver/

📁 slack/

🐍 __init__.py

🐍 configs.py

🐍 enums.py

🐍 fields.py

🐍 main.py

🐍 utils.py

📁 tests/

📁 .coverage/

⚙️ .gitlab-ci.yml

⚙️ databricks.yml

📄 poetry.lock

📄 poetry.toml

📄 pyproject.toml

Key Components

Apps Structure

📁

conf/

Environment-specific configurations (dev, staging, prod)

📁

infrastructure/

API endpoints, database clients, and event handlers

📁

services/

Core agent services including LangGraph and vector search

📁

jobs/

Data lifecycle and batch processing jobs

Projects Structure

📁

conf/

Environment-specific configurations (dev, staging, prod)

📁

pipelines/

Data processing pipeline definitions

📁

steps/

Individual pipeline step implementations

🐍

main.py

Main application entry point

Architecture Overview

The Data-AI-Hub Agent follows a modular architecture with clear separation between application deployment (apps) and shared components (projects/common). This structure enables efficient development, testing, and deployment while maintaining code reusability across different environments.

Modular Design

Clear separation between application logic and shared components

Reusability

Common components can be shared across multiple applications

Scalability

Architecture supports easy scaling and deployment across environments