`surveilr` v1.8.11 Release Notes

🎉 New Feature: Automatic Document Processing & Metadata Extraction

🚀 What's New

Surveilr now automatically extracts metadata and converts documents to markdown during ingestion - no configuration required!

Supported File Types

PDF files: Full metadata extraction + markdown conversion
DOCX files: Full metadata extraction + markdown conversion
Images (PNG, JPEG, GIF, etc.): Metadata extraction (dimensions, format, file size)

`surveilr` v1.8.9 Release Notes

🚀 What's New

1. Surveilr ingestion Improvements

Image Ingestion Support - Fixed issues with image format ingestion during file processing
GitHub API Rate Limiting - Enhanced rate limiting handling for GitHub PLM integration

2. Dependencies Update

OIDC/SSO Support - Added OpenID Connect and Single Sign-On support for surveilr web UI
SQLPage Upgrade - Updated to latest SQLPage version

`surveilr` v1.8.8 Release Notes

🚀 What's New

1. Bug-fixes

Ingestion & PLM Issues (#320)

Fixed TLS crypto provider initialization issues
Fixed ingestion PLM issues with github

CSV Transform Issues (#194)

Fixed CSV transform duplicate detection issues

File Carving (#299)

Fixed file carving functionality

`surveilr` v1.8.5 Release Notes

🚀 What's New

1. Surveilr ingestion Improvements

Prevent Duplication of Records in surveilr ingest files --csv-transform-auto Command

2. Changes to Osquery-ms

Platform Consistency: Darwin is the actual kernel name that macOS runs on, making it more technically accurate
OSQuery Compatibility: Aligns with osquery's internal platform detection which uses "darwin"

`surveilr` v1.8.4 Release Notes

🚀 What's New

1. Surveilr ingestion Improvements

Automatically detects and adds missing columns during surveilr ingestion.

2. Installation Config

-The mac archive no longer contains a nested folder — you can now upgrade surveilr by running surveilr upgrade -This fixes installation issues with scripts like install.ps1 and allows surveilr.exe to run immediately after extraction.

`surveilr` v1.8.3 Release Notes

🚀 What's New

1. Admin Merge Improvements

Automatically detects and adds missing columns during database schema merging.

2. Markdown Transformation Enhancements

Introduced transform-md for parsing Markdown files and converting them into structured JSON. Added support for Markdown querying using --md-select with the mdq library.

`surveilr` v1.8.2 Release Notes

🚀 What's New

1. surveilr osquery-ms` Server

Significant enhancements and a complete overhaul of the file carving architecture in osQuery MS server

`surveilr` v1.8.1 Release Notes

🚀 What's New

1. `sureilr osquery-ms` Server

Added distributed queries and file carving to the server.

`surveilr` v1.8.0 Release Notes

🚀 What's New

1. SQLPage

Updated SQLPage to the latest version, v0.34.0, ensuring compatibility and access to the newest features and bug fixes.

2. Introduced `surveilr_notebook_cell_exec`

surveilr_notebook_cell_exec is a function designed to execute queries stored in code_notebook_cells against the RSSD. This is the SQLite function equivalent of the surveilr notebook cat command which only outputs the content of the code_notebook_cell, this function on other hand, executes it. It takes two arguments, the notebook_name and the cell_name and it returns either true or false to denote if the execution was succesful.

Bug Fixes

Fixed the SQL query issue when --persist-raw-logs is passed to the surveilr osquery-ms server.

`surveilr` v1.7.13 Release Notes

This release aims to improve the surveilr osquery-ms server; no new features or bug fixes were added.

`surveilr` v1.7.16 Release Notes

Bug Fixes

Enhanced the CSV transform functionality to correctly include partyID for each ingested CSV table when provided.
Resolved an issue where ingesting multiple CSV files with the same name from different folders resulted in data loss. Now, all files are consolidated into a single table while preserving distinct data sources with the partyID field.

`surveilr` v1.7.13 Release Notes

This release aims to improve the surveilr osquery-ms server; no new features or bug fixes were added.

`surveilr` v1.7.12 Release Notes

🚀 What's New

1. `surveilr osquery-ms` Server

The server has been fully setup, configured with boundaries and the corresponding WebUI, fully configurable with code_notebooks.

`surveilr` v1.7.11 Release Notes

🚀 What's New

1. Upgraded SQLPage

SQLPage has been updated to version 0.33.1, aligning with the latest releases.

Bug Fixes

1. `surveilr admin merge`

Added recent and new tables to the merge structure ensuring all tables in each RSSD are present in the final merged RSSD.

`surveilr` v1.7.10 Release Notes

🚀 What's New

1. Enhancing `surveilr`'s osQuery Management Server

Added support for boundaries to enable grouping of nodes for better viewing

`surveilr` v1.7.9 Release Notes

🚀 What's New

1. Enhancing `surveilr`'s osQuery Management Server

Introduced a new flag --keep-status-logs to indicate whether the server should store status logs received from osQuery in the RSSD.

`surveilr` v1.7.8 Release Notes

This release focuses on enhancing the surveilr osquery-ms UI by adding new tables and optimizing data management. No bugs were fixed or new features introduced. Please review the Web UI for updates.

`surveilr` v1.7.7 Release Notes

This release aims to improve the surveilr osquery-ms server; no new features or bug fixes were added.

`surveilr` v1.7.6 Release Notes

🚀 Bug Fixes

1. `surveilr` Bootstrap SQL

This release fixes the "no such table: device" error introduced in the previous version by propagating any erroors during the SQL initialization of the RSSD with surveilr.

`surveilr` v1.7.5 Release Notes

🆕 New Features

osQuery Management Server Integration:
- surveilr now acts as a management layer for osQuery, enabling secure and efficient monitoring of infrastructure.
- Supports remote configuration, logging, and query execution for osQuery nodes.
Behavior & Notebooks Support:
- Introduced Notebooks, which store predefined queries in the code_notebook_cell table.
- Behaviors allow defining and managing query execution for different node groups.
Secure Node Enrollment:
- Nodes authenticate using an enrollment secret key (SURVEILR_OSQUERY_MS_ENROLL_SECRET).
- Secure communication via TLS certificates (cert.pem, key.pem).
Automated Query Execution:
- Default queries from "osQuery Management Server (Prime)" execute automatically.
- Custom notebooks and queries can be added dynamically via SQL.
Centralized Logging & Config Fetching:
- Osquery logs and configurations are fetched via TLS endpoints (/logger, /config).
- All communication is secured using server-side TLS certificates.
Web UI for Query Results Visualization:
- surveilr web-ui provides an intuitive dashboard to inspect query results across enrolled nodes.
- Simply start with surveilr web-ui -p 3050 --host <server-ip>.

`surveilr` v1.7.1 Release Notes

🚀 What's New

1. Enhancing `surveilr`'s osQuery Management Server

Introduced a new flag--behavioror-b` to specify behavior name to queries to run automatically enrolled nodes.
a new SQLite function called surveilr_osquery_ms_create_behaviour to facilitate the creation of behaviors, making process smooth and easy.

Example

When starting the surveilr osquery-ms server without passing a behavior, a default behavior with the following query configuration is created:

{
  "surveilr-cli": {
    ...
    "osquery_ms": {
      "tls_proc": {
 "query": "select * from processes",
        "interval": 60
      }
    }
  }
}

To use a behavior with the surveilr osQuery management server first create a behavior using the new function:

surveilr shell --cmd "select surveil_osquery_ms_create_behaviour('-behaviour', '{\"tls_proc\": {\"query\": \"select * from processes\", \"interval\": 60}, \"routes\": {\"query\": \"SELECT * FROM routes WHERE destination = ''::1''\", \"interval\": 60}}');"

Then, pass that behavior to the server by:

surveilr osquery-ms --cert ./cert.pem --key ./key.pem --enroll-secret "<secret>" -b "initial-behaviour"

`surveilr` v1.7.0 Release Notes

🚀 What's New

1. `surveilr` OSQuery Management Server

Introducing Osquery Management Server using surveilr, enabling secure and centralized monitoring of your infrastructure. The setup ensures secure node enrollment through TLS authentication and secret keys, allowing only authorized devices to connect. Users can easily configure and manage node behaviors dynamically via surveilr’s behavior tables.

2. OpenDAL Dropbox Integration

The surveilr_udi_dal_dropbox SQLite function, is a powerful new virtual table module that enables seamless interaction with Dropbox files directly within your SQL queries. This module allows users to access and query comprehensive file metadata, including name, path, size, last modified timestamp, content, and more, within specified directories.

`surveilr` v1.6.0 Release Notes

🚀 What's New

1. SQLPage

Updated SQLPage to the latest version, ensuring compatibility and access to the newest features and bug fixes.

`surveilr` v1.5.11 Release Notes

Overview

This release includes updates to dependencies, bug fixes, and performance improvements to enhance stability and functionality.

`surveilr` v1.5.9 Release Notes

🚀 Bug Fixes

1. WebUI Page for About

A dedicated About page has been added in the WebUI to visualize the response of surveilr doctor:
- Dependencies Table:
  - The display of versions and their generation process has been fixed.
- Diagnostic Views:
  - A new section has been added to display the contents and details of all views prefixed with surveilr_doctor*, facilitating the of details and logs for diagnostics.

`surveilr` v1.5.8 Release Notes 🎉

1. WebUI Page for About

Added a dedicated About page in the WebUI visiualizing the response of surveilr doctor:
- Dependencies Table:
  - Displays the versions of sqlpage, rusqlite, and pgwire in a table.
- Extensions List:
  - Lists all synamic and static extensions .
- Capturable executables:
  - Lists all capturable executables that were found in the PATH.
- Env variables
  - Captures all environment variables starting with SURVEILR_ and SQLPAGE_.

`surveilr` v1.5.6 Release Notes 🎉

🚀 What's New

1. Enhanced Diagnostics Command

surveilr doctor Command Improvements:
- Dependencies Check:
  - Verifies versions of critical dependencies: Deno, Rustc, and SQLite.
  - Ensures dependencies meet minimum version requirements for seamless functionality.
- Capturable Executables Detection:
  - Searches for executables in the PATH matching surveilr-doctor*.*.
  - Executes these executables, assuming their output is in JSON format, and integrates their results into the diagnostics report.
- Database Views Analysis:
  - Queries all views starting with the prefix surveilr_doctor_ in the specified RSSD.
  - Displays their contents in tabular format for comprehensive insights.

2. JSON Mode

Added a --json flag to the surveilr doctor command.
- Outputs the entire diagnostics report, including versions, extensions, and database views, in structured JSON format.

3. WebUI Page for Diagnostics

Added a dedicated page in the WebUI for the surveilr doctor diagnostics:
- Versions Table:
  - Displays the versions of Deno, Rustc, and SQLite in a table.
- Extensions List:
  - Lists all detected extensions.
- Database Views Content:
  - Automatically identifies and displays the contents of views starting with surveilr_doctor_ in individual tables.

`surveilr` v1.5.5 Release Notes 🎉

🚀 What's New

Virtual Table: `surveilr_function_docs`

Description
The surveilr_function_docs virtual table offers a structured method to query metadata about surveilr SQLite functions registered in the system.

Columns

name (TEXT): The function's name.
description (TEXT): A concise description of the function's purpose.
parameters (JSON): A JSON object detailing the function's parameters, including:
- name: The name of the parameter.
- data_type: The parameter's expected data type.
- description: An explanation of the parameter's role.
return_type (TEXT): The function's return type.
introduced_in_version (TEXT): The version in which the function was first introduced.

Use Cases

Utilized in the Web UI for generating documentation on the functions.

Virtual Table: `surveilr_udi_dal_fs`

Description
The surveilr_udi_dal_fs virtual table acts as an abstraction layer for interacting with the file system. It enables users to list and examine file metadata in a structured, SQL-friendly manner. This table can list files and their metadata recursively from a specified path.

Columns

name (TEXT): The file's name.
path (TEXT): The complete file path.
last_modified (TEXT): The file's last modified timestamp, when available.
content (BLOB): The content of the file (optional).
size (INTEGER): The size of the file in bytes.
content_type (TEXT): The MIME type of the file or an inferred content type (e.g., based on the extension).
digest (TEXT): The MD5 digest of the file, if available.
arg_path (TEXT, hidden): The base path for querying files, specified in the filter method.

Key Features

Lists files recursively from a specified directory.
Facilitates metadata extraction, such as file size, last modified timestamp, and MDhash).

Virtual Table: `surveilr_udi_dal_s3`

Description
The surveilr_udi_dal_s3 virtual table is an abstraction layer that interacts with the S3 bucket in a given region. It allows listing and inspecting file metadata in a structured, SQL-accessible way.

Columns

name (TEXT): The name of the file.
path (TEXT): The full path to the file.
last_modified (TEXT): The last modified timestamp of the file, if available.
content (BLOB): The file's content (optional).
size (INTEGER): The file size in bytes.
content_type (TEXT): The file's MIME type or inferred content type (e.g., based on the extension).
digest (TEXT): The file's MD5 digest, if available.
arg_path (TEXT, hidden): The base path to query files from, specified in the filter method.

Key Features

Supports metadata extraction (e.g., file size, last modified timestamp, MD5 hash).

Example Queries

Querying Function Documentation

SELECT * FROM surveilr_function_docs WHERE introduced_in_version = '1.0.0';

`surveilr` v1.5.3 Release Notes 🎉

🚀 What's New

1. Open Project Data Extension

surveilr now includes additional data from Open Project PLM ingestion. Details such as a work package's versions and relations are now encapsulated in JSON format in a new elaboration column within the ur_ingest_session_plm_acct_project_issue table. The JSON structure is as follows, with the possibility for extension:

elaboration: {"issue_id": 78829, "relations": [...], "version": {...}}

2. Functions for Extension Verification

Two new functions have been introduced to verify and ensure the presence of certain intended functions and extensions before their use:

The select surveilr_ensure_function('func', 'if not found msg', 'func2', 'if func2 not found msg') function can be used to declaratively specify the required function(s), and will produce an error with guidance on how to obtain the function if it is not found.
The select surveilr_ensure_extension('extn.so', '../bin/extn2.so') function allows for the declarative indication of necessary extensions, and will dynamically load them if they are not already available.

`surveilr` v1.5.2 Release Notes 🎉

🚀 What's New

1. `surveilr` SQLite Extensions

surveilr extensions are now statically linked, resolving all extensions and function usage issues. The following extensions are included by default in surveilr, with additional ones planned for future releases:

`surveilr` v1.4.2 Release Notes 🎉

🚀 What's New

1. Utilizing Custom Extensions with `surveilr`

In the previous release, we introduced the feature of automatically loading extensions from the default sqlpkg location. However, this posed a security risk, and we have since disabled that feature. To use extensions installed by sqlpkg, simply pass --sqlpkg, and the default location will be utilized. If you wish to change the directory from which extensions are loaded, use --sqlpkg /path/to/extensions, or specify the directory with the new SURVEILR_SQLPKG environment variable.

`surveilr` v1.4.2 Release Notes 🎉

🚀 What's New

1. Utilizing Custom Extensions with `surveilr`

Loading extensions is now straightforward with the --sqlite-dyn-extn flag. As long as your extensions are installed via sqlpkg, surveilr will automatically detect the default location of sqlpkg and all installed extensions. Simply install the extension using sqlpkg. To specify a custom path for sqlpkg, use the --sql-pkg-home argument with a directory containing the extensions, regardless of depth, and surveilr will locate them. Additionally, the SURVEILR_SQLITE_DYN_EXTNS environment variable has been introduced to designate an extension path instead of using --sqlite-dyn-extn. Note: Using --sqlite-dyn-extn won't prevent surveilr from loading extensions from sqlpkg's default directory. To disable loading from sqlpkg, use the --no-sqlpkg flag.

Here's a detailed example of using surveilr shell and surveilr orchestrate with dynamic extensions. Using sqlpkg defaults

Download the sqlpkg CLI.
Download the text extension, which offers various text manipulation functions: sqlpkg install nalgeon/sqlean

Run the following command:

surveilr shell --cmd "select text_substring('hello world', 7, 5) AS result" # surveilr loads all extensions from the .sqlpkg default directory

Including an additional extension with sqlpkg Combine --sqlite-dyn-extn with surveilr's ability to load extensions from sqlpkg

Add the path extension to sqlpkg's installed extensions: sqlpkg install asg017/path

Execute:

surveilr shell --cmd "SELECT
      text_substring('hello world', 7, 5) AS substring_result,
      math_sqrt(9) AS sqrt_result,
      path_parts.type,
      path_parts.part 
  FROM 
      (SELECT * FROM path_parts('/usr/bin/sqlite3')) AS path_parts;
  " --sqlite-dyn-extn .extensions/math.so

Specify a Custom Directory to Load Extensions From A --sqlpkg-home flag has been introduced to specify a custom path for extensions. They do not need to be installed by sqlpkg to be used. surveilr will navigate the specified folder and load all compatible extensions for the operating system—.so for Linux, .dll for Windows, and .dylib for macOS. (If you installed with sqlpkg, you don't need to know the file type).

surveilr shell --cmd "SELECT text_substring('hello world', 7, 5) AS substring_result, math_sqrt(9) AS sqrt_result" --sqlpkg-home ./src/resource_serde/src/functions/extensions/

2. Upgraded SQLPage

SQLPage has been updated to version 0.31.0, aligning with the latest releases.

`surveilr` v1.4.1 Release Notes 🎉

🚀 Bug Fixes

1. `surveilr` SQLite Extensions

To temporarily mitigate the issue with surveilr intermittently working due to the dynamic loading of extensions, surveilr no longer supports dynamic loading by default. It is now supported only upon request by using the --sqlite-dyn-extn flag. This flag is a multiple option that specifies the path to an extension to be loaded into surveilr. To obtain the dynamic versions (.dll, .so, or .dylib), you can use sqlpkg to install the necessary extension.

For instance, to utilize the text functions:

Install the extension with sqlpkg: sqlpkg install nalgeon/text

Then execute it:

surveilr shell --cmd "select text_substring('hello world', 7, 5);" --sqlite-dyn-extn ./text.so

`surveilr` v1.3.1 Release Notes 🎉

🚀 Bug Fixes

1. `surveilr` SQLite Extensions

This release fixes the glibc compatibility error that occured with surveilr while registering function extensions.

`surveilr` Release Announcement: Now Fully Compatible Across All Distros 🎉

We are thrilled to announce that surveilr is now fully compatible with all major Linux distributions, resolving the longstanding issue related to OpenSSL compatibility! 🚀

What's New?

Universal Compatibility: surveilr now works seamlessly on Ubuntu, Debian, Kali Linux, and other Linux distributions, across various versions and architectures. Whether you're using Ubuntu 18.04, Debian 10, or the latest Kali Linux, surveilr is ready to perform without any hiccups.
Resolved OpenSSL Bug: We’ve fixed the recurring OpenSSL-related issue that caused headaches for users on older and varied systems. With this update, you no longer need to worry about OpenSSL version mismatches or missing libraries.

`surveilr` v2.2.0 Release Notes

🚀 What's New

JSONL File Ingestion Support

New Format Support: Added comprehensive JSONL (JSON Lines) file format ingestion capabilities
Streaming Data Processing: Efficiently handles large streaming JSON datasets line-by-line
Automatic Schema Detection: Intelligently detects and processes JSONL file structures

How JSONL Ingestion Works

Unlike regular JSON files that contain a single JSON object or array, JSONL files contain one valid JSON object per line:

{"id": 1, "name": "Alice", "timestamp": "2025-09-01T10:00:00Z"} {"id": 2, "name": "Bob", "timestamp": "2025-09-01T10:01:00Z"} {"id": 3, "name": "Charlie", "timestamp": "2025-09-01T10:02:00Z"}

Ingestion Process:

Line-by-Line Reading: File is read sequentially, one line at a time
JSON Validation: Each line is validated as proper JSON
Individual Processing: Each JSON object is processed as a separate resource
Schema Evolution: Supports varying schemas across lines in the same file
Line-Specific URIs: Each line gets a unique URI with line number reference for precise tracking

URI Structure for JSONL:

Each JSON line creates a unique uniform_resource entry with line-specific URI: /path/to/events.jsonl#L1 # First JSON object /path/to/events.jsonl#L2 # Second JSON object /path/to/events.jsonl#L3 # Third JSON object

Example:

File: /data/user-events.jsonl Line 1: {"user": "alice", "action": "login", "timestamp": "2025-09-01T10:00:00Z"} Line 2: {"user": "bob", "action": "logout", "timestamp": "2025-09-01T10:05:00Z"}

Results in uniform_resource entries with URIs:

/data/user-events.jsonl#L1 /data/user-events.jsonl#L2

This unique URI scheme allows precise tracking of which specific line in the JSONL file each resource originated from, enabling accurate data lineage and debugging capabilities.

`surveilr` v1.2.0 Release Notes 🎉

What's New?

This update introduces two major additions that streamline file system integration and ingestion session management.

New Features

1. `surveilr_ingest_session_id` Scalar Function

The surveilr_ingest_session_id function is now available, offering robust management of ingestion sessions. This function ensures efficient session handling by:

Reusing existing session IDs for devices with active sessions.
Creating new ingestion sessions when none exist.
Associating sessions with metadata for improved traceability.

2. `surveilr_udi_dal_fs` Virtual Table Function

The surveilr_udi_dal_fs virtual table function provides seamless access to file system resources directly within your SQL queries. With this feature, you can:

Query file metadata, such as names, paths, sizes, and timestamps.
Retrieve file content and calculate digests for integrity checks.
Traverse directories recursively to handle large and nested file systems effortlessly.

`surveilr` v2.1.0 Release Notes

🚀 What's New

1. AI-Powered Natural Language to SQL (ask-ai)

New Command: surveilr ask-ai sql converts natural language queries into SQL and executes them against your RSSD
Smart Context Integration: Automatically retrieves relevant AI context from Surveilr notebook cells for domain-specific knowledge
Multiple Output Formats: Support for table, JSON, CSV, and markdown output formats
Flexible LLM Support: Works with OpenAI-compatible endpoints including local models (Ollama, etc.)

Basic natural language queries

surveilr ask-ai sql "show me all files ingested in the last week" surveilr ask-ai sql "what devices have been scanned?" surveilr ask-ai sql "find all JSON files larger than 1MB"

Different output formats

surveilr ask-ai sql "show device information" --output json surveilr ask-ai sql "list recent sessions" --output csv surveilr ask-ai sql "security audit summary" --output markdown

Development and debugging

surveilr ask-ai sql "show database tables" --show-query surveilr ask-ai sql "count all records" --sql-only

🔧 Configuration

🤖 AI Engine Compatibility

The new ask-ai feature supports a wide range of AI engines through OpenAI-compatible APIs:

Supported AI Engines

OpenAI: GPT-4, GPT-3.5-turbo, GPT-4-turbo
Local AI Servers: Ollama, LM Studio, text-generation-webui
Cloud Providers: Azure OpenAI, Anthropic Claude (via compatible proxies)
Open Source Models: Any model served via vLLM, LocalAI, or other OpenAI-compatible servers

Technical Implementation

Surveilr uses the ureq HTTP client to communicate with any OpenAI-compatible API endpoint. For a comprehensive list of supported providers and configuration examples, see the OpenAI-compatible providers documentation.

Quick Configuration Examples

# OpenAI (default)
export SURVEILR_LLM_API_KEY="sk-your-key"

# Ollama (local)
export SURVEILR_LLM_ENDPOINT="http://localhost:11434/v1/chat/completions"

# LM Studio (local)
export SURVEILR_LLM_ENDPOINT="http://localhost:1234/v1/chat/completions"

# Azure OpenAI
export SURVEILR_LLM_ENDPOINT="https://your-resource.openai.azure.com/openai/deployments/your-deployment/chat/completi
ons?api-version=2023-05-15"

`surveilr` v1.1.0 Release Notes 🎉

🚀 New Features

1. Integrated Documentation in Web UI

This release introduces a comprehensive update to the RSSD Web UI, allowing users to access and view all surveilr-related SQLite functions, release notes, and internal documentation directly within the interface. This feature enhances user experience by providing integrated, easily navigable documentation without the need to leave the web environment, ensuring that all necessary information is readily available for efficient reference and usage.

2. `uniform_resource` Graph Infrastructure

The foundational framework for tracking uniform_resource content using graph representations has been laid out in this release. This infrastructure allows users to visualize uniform_resource data as connected graphs in addition to the traditional relational database structure. To facilitate this, three dedicated views—imap_graph, plm_graph, and filesystem_graph—have been created. These views provide a structured way to observe and interact with data from different ingestion sources:

imap_graph: Represents the graphical relationships for content ingested through IMAP processes, allowing for a visual mapping of email and folder structures.
plm_graph: Visualizes content from PLM (Product Lifecycle Management) ingestion, showcasing project and issue-based connections.
filesystem_graph: Illustrates file ingestion paths and hierarchies, enabling users to track and manage file-based data more intuitively.

This release marks an important step towards enhancing data tracking capabilities, providing a dual approach of relational and graphical views for better data insights and management.

Surveilr v3.0.0 - Drizzle ORM Foundation

Summary

Migrated internal schema generation from SQLa to Drizzle ORM - a lightweight, type-safe TypeScript ORM. This establishes the foundation for optional type-safe database queries while maintaining our SQL-first philosophy.

What Changed

Schema Generation (Internal)

Replaced: SQLa-based lifecycle.sql.ts → Drizzle-generated bootstrap SQL
New: Type-safe schema definitions in lib/std/drizzle/models.ts and views.ts
Result: Same RSSD structure with enhanced TypeScript support

Developer Experience

Added: Optional type-safe query helpers for complex scenarios
Maintained: SQL views remain the preferred approach for business logic

File Organization

lib/std/drizzle/
├── models.ts              # RSSD table schemas
├── views.ts              # SQL view definitions
├── bootstrap.sql.ts       # Schema generator
└── drizzle-lifecycle.ts   # Migration cells

Migration Impact

End Users: No changes to surveilr CLI commands
Developers: Optional access to type-safe queries when needed
Databases: Same RSSD structure, generated via Drizzle instead of SQLa

Technical Validation

✅ Identical 65+ table structure generated
✅ All foreign key constraints preserved
✅ File ingestion, transforms, multitenancy functional
✅ Web UI and code notebook systems working
✅ Complete test suite passing

`surveilr` v1.0.0 Release Notes 🎉

We’re thrilled to announce the release of surveilr v1.0, a significant milestone in our journey to deliver powerful tools for continuous security, quality and compliance evidence workflows. This release introduces a streamlined migration system and a seamless, user-friendly experience for accessing the surveilr Web UI.

🚀 New Features

1. Database Migration System

This release introduces a comprehensive database migration feature that allows smooth and controlled updates to the RSSD structure. Our migration system includes:

Structured Notebooks and Cells: A structured system organizes SQL migration scripts into modular code notebooks, making migration scripts easy to track, audit, and execute as needed.
Idempotent vs. Non-Idempotent Handling: Ensures each migration runs in an optimal and secure manner by tracking cell execution history, allowing for re-execution where safe.
Automated State Tracking: All state changes are logged for complete auditing, showing timestamps, transition details, and the results of each migration step.
Transactional Execution: All migrations run within a single transaction block for seamless rollbacks and data consistency.
Dynamic Migration Scripts: Cells marked for migration are dynamically added to the migration script, reducing manual effort and risk of errors.

This system ensures safe, controlled migration of database changes, enhancing reliability and traceability for every update.

2. Enhanced Default Command and Web UI Launch

The surveilr executable now starts the Web UI as the default command when no specific CLI commands are passed. This feature aims to enhance accessibility and ease of use for new users and teams. Here’s what happens by default:

Automatic Web UI Startup: By default, running surveilr without additional commands launches the surveilr Web UI.
Auto-Browser Launch: Opens the default browser on the workstation, pointing to the Web UI’s URL and port, providing a user-friendly experience right from the first run.

`surveilr` v2.0.0 Release Notes

🚀 What's New

1. Enhanced Markdown Transformation Workflow

Improved mdq Integration: Fixed mdq selector syntax and added comprehensive selector support
Content Preservation: Markdown transforms no longer null out original content by default
Better URI Tracking: Transform results now preserve source file paths (e.g., document.pdf/md-select:headers)

2. Dependencies Update

Upgraded to SQLPage 0.36.1.

Release Notes for surveilr Versions

v1.8.11

surveilr v1.8.11 Release Notes

🎉 New Feature: Automatic Document Processing & Metadata Extraction

🚀 What's New

v1.8.9

surveilr v1.8.9 Release Notes

🚀 What's New

1. Surveilr ingestion Improvements

2. Dependencies Update

v1.8.8

surveilr v1.8.8 Release Notes

🚀 What's New

1. Bug-fixes

Ingestion & PLM Issues (#320)

CSV Transform Issues (#194)

File Carving (#299)

v1.8.5

surveilr v1.8.5 Release Notes

🚀 What's New

1. Surveilr ingestion Improvements

2. Changes to Osquery-ms

v1.8.4

surveilr v1.8.4 Release Notes

🚀 What's New

1. Surveilr ingestion Improvements

2. Installation Config

v1.8.3

surveilr v1.8.3 Release Notes

🚀 What's New

1. Admin Merge Improvements

2. Markdown Transformation Enhancements

v1.8.2

surveilr v1.8.2 Release Notes

🚀 What's New

1. surveilr osquery-ms` Server

v1.8.1

surveilr v1.8.1 Release Notes

🚀 What's New

1. sureilr osquery-ms Server

v1.8.0

surveilr v1.8.0 Release Notes

🚀 What's New

1. SQLPage

2. Introduced surveilr_notebook_cell_exec

Bug Fixes

v1.7.25

surveilr v1.7.13 Release Notes

v1.7.16

surveilr v1.7.16 Release Notes

Bug Fixes

v1.7.13

surveilr v1.7.13 Release Notes

v1.7.12

surveilr v1.7.12 Release Notes

🚀 What's New

1. surveilr osquery-ms Server

v1.7.11

surveilr v1.7.11 Release Notes

🚀 What's New

1. Upgraded SQLPage

Bug Fixes

1. surveilr admin merge

v1.7.10

surveilr v1.7.10 Release Notes

🚀 What's New

1. Enhancing surveilr's osQuery Management Server

v1.7.9

surveilr v1.7.9 Release Notes

🚀 What's New

1. Enhancing surveilr's osQuery Management Server

v1.7.8

surveilr v1.7.8 Release Notes

v1.7.7

surveilr v1.7.7 Release Notes

v1.7.6

surveilr v1.7.6 Release Notes

🚀 Bug Fixes

1. surveilr Bootstrap SQL

v1.7.5

`surveilr` v1.8.11 Release Notes

`surveilr` v1.8.9 Release Notes

`surveilr` v1.8.8 Release Notes

`surveilr` v1.8.5 Release Notes

`surveilr` v1.8.4 Release Notes

`surveilr` v1.8.3 Release Notes

`surveilr` v1.8.2 Release Notes

`surveilr` v1.8.1 Release Notes

1. `sureilr osquery-ms` Server

`surveilr` v1.8.0 Release Notes

2. Introduced `surveilr_notebook_cell_exec`

`surveilr` v1.7.13 Release Notes

`surveilr` v1.7.16 Release Notes

`surveilr` v1.7.13 Release Notes

`surveilr` v1.7.12 Release Notes

1. `surveilr osquery-ms` Server

`surveilr` v1.7.11 Release Notes

1. `surveilr admin merge`

`surveilr` v1.7.10 Release Notes

1. Enhancing `surveilr`'s osQuery Management Server

`surveilr` v1.7.9 Release Notes

1. Enhancing `surveilr`'s osQuery Management Server

`surveilr` v1.7.8 Release Notes

`surveilr` v1.7.7 Release Notes

`surveilr` v1.7.6 Release Notes

1. `surveilr` Bootstrap SQL

`surveilr` v1.7.5 Release Notes

`surveilr` v1.7.1 Release Notes

1. Enhancing `surveilr`'s osQuery Management Server

`surveilr` v1.7.0 Release Notes

1. `surveilr` OSQuery Management Server

`surveilr` v1.6.0 Release Notes

`surveilr` v1.5.11 Release Notes

`surveilr` v1.5.9 Release Notes

`surveilr` v1.5.8 Release Notes 🎉

`surveilr` v1.5.6 Release Notes 🎉

`surveilr` v1.5.5 Release Notes 🎉

Virtual Table: `surveilr_function_docs`

Virtual Table: `surveilr_udi_dal_fs`

Virtual Table: `surveilr_udi_dal_s3`

`surveilr` v1.5.3 Release Notes 🎉

`surveilr` v1.5.2 Release Notes 🎉

1. `surveilr` SQLite Extensions

`surveilr` v1.4.2 Release Notes 🎉

1. Utilizing Custom Extensions with `surveilr`

`surveilr` v1.4.2 Release Notes 🎉

1. Utilizing Custom Extensions with `surveilr`

`surveilr` v1.4.1 Release Notes 🎉

1. `surveilr` SQLite Extensions

`surveilr` v1.3.1 Release Notes 🎉

1. `surveilr` SQLite Extensions

`surveilr` Release Announcement: Now Fully Compatible Across All Distros 🎉