DTLarchive — Reference Manual v2.2-5

Preface

This manual describes the structure, formats, invariants, and internal interfaces of DTLarchive 2.2-5. It is an architecture and data reference. It deliberately contains no user journey, launch procedure, or step-by-step scenario.

DTLarchive transforms ChatGPT conversation exports into a locally indexed corpus. Its purpose is to make the knowledge contained in these archives searchable, contextual, and reusable by downstream processing, without any remote-service dependency.

Intended audience

This document is intended for developers, maintainers, data owners, and integrators who need to understand engine behavior, validate its outputs, or connect its results to other knowledge-management tools.

Design principles

• Locality: archives, queries, index, and reports remain on the computer.
• Determinism: selection and ranking use explicit lexical rules.
• Traceability: each result retains its source file, identifier, and contexts.
• Incrementality: unchanged sources are not parsed again.
• Bilingual design: the interface and generated outputs are available in French and English.
• No external AI: no language model, embedding, or semantic service is involved.

Architecture

Overview

The architecture separates persistent import, full-text candidate selection, and exact analysis of candidate conversations. This avoids reading and recalculating the complete corpus for every search.

Components

Processing pipeline

File organization

Data and index

ChatGPT source format

The reader accepts an array of conversations or a single conversation object. Each usable conversation contains a mapping dictionary. The active branch is reconstructed backwards from current_node through parent links, then reversed into chronological order.

Only non-empty user and assistant messages are retained. Text comes from content.parts or, as a fallback, content.text. If the active branch produces no messages, a fallback traversal sorts usable messages by date.

Internal models

SQLite schema

Foreign keys are enabled and SQLite uses WAL journaling. Relational indexes cover message order and conversation dates.

Incremental import and deduplication

A source is unchanged when its size and modification timestamp match the stored state. If those metadata differ while the SHA-256 fingerprint remains identical, only metadata are refreshed. A genuinely modified source is parsed inside a transaction.

The ChatGPT identifier is the deduplication key. When absent, a deterministic SHA-256 is computed from path, title, and dates. Existing content is replaced when the incoming (content timestamp, message count) pair is greater than or equal to the stored pair. Provenance links allow one conversation to belong to several exports.

Search engine

Lexical grammar

Duplicates are removed by normalized text, exclusion status, and group number.

Normalization and counting

Normalization applies Unicode NFKD, removes diacritics, lowercases text, replaces typographic apostrophes, and compacts whitespace. Searches are case- and accent-insensitive. Without a wildcard, simple forms accept an s or x plural suffix unless the term already ends in either character.

Indexed candidate selection

For each positive term, FTS5 produces an identifier set constrained by source, role, and date. Terms in one group are intersected; alternative groups are united. The title role is always added to the scope. This phase produces candidates, not final results.

Exact analysis

The title and in-scope messages are combined and normalized. Any excluded-term occurrence rejects the conversation. A positive group is valid only when all its terms occur. Counts, matching roles, and message positions are then calculated against exact text.

Relevance calculation

The score is capped at 100 and follows this deterministic formula:

Final ordering is descending by score and then conversation date. The score is not a probability and does not come from statistical learning.

Context windows

Each matching message generates a window of up to two messages before and two after. Adjacent or overlapping windows are merged. At most six windows are retained per conversation. Context-message text is compacted to 1,200 characters.

Structured outputs

mining_results.json

The root document contains metadata and results. Metadata describe the application, version, logical schema, UTC time, sources, index, dates, query, and scope. Each result is a complete serialization of MiningResult.

HTML reports

DTLarchive-report.html presents metrics, term distribution, the first ranked unique conversation titles, the result table, and context windows. Displayed titles are not a semantic summary. Each conversation-<fingerprint>.html page reproduces the complete branch and places an anchor on the first matching message.

The documents contain their own CSS, use UTF-8, and follow the active interface language. Conversation content remains in its original language.

HTML diagnostic log

The daily logs/DTLarchive_YYYYMMDD.html log receives timestamped information, action, and error entries. It records major phases, summarized parameters, import counters, and exceptions. Log write failures do not interrupt the engine.

Internal reference

Project modules

Function groups

Internationalization

The TRANSLATIONS catalog maps each key to fr and en values. current_language() reads DTLARCHIVE_LANG and falls back to French for unsupported values. t() selects and interpolates a template; plural_key() selects singular and plural variants.

Language coverage includes the console, dialogs, help, errors, reports, and log. It does not translate source conversation content.

Errors, transactions, and integrity

• An unreadable archive may be skipped in tolerant mode or raise an error in strict mode.
• Modified-source import is transactional; a parse failure preserves previously indexed content.
• An incompatible index version raises an explicit error before any search.
• Invalid, reversed, or out-of-corpus periods are rejected before analysis.
• HTML text is escaped before insertion into reports.
• HTML and JSON files use UTF-8; the Windows console is configured for code page 65001.

Appendices

Known limitations

• Search is lexical; it detects neither synonyms, paraphrases, nor semantic proximity.
• FTS candidate selection uses the first significant token of a phrase, after which exact analysis confirms the complete phrase.
• Summary titles are the first five unique titles in ranking order, not topics inferred from the corpus.
• The active branch is preferred; fallback traversal is used only when it yields no usable messages.
• Contexts are limited to six windows and their messages to 1,200 characters.
• The reader reflects the observed ChatGPT export format; source-format changes may require parser adaptation.

Versions and schemas

Glossary