<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[yet another BLOB]]></title><description><![CDATA[yet another BLOB]]></description><link>https://waqassiddiqi.com</link><generator>RSS for Node</generator><lastBuildDate>Mon, 16 Nov 2026 22:57:20 GMT</lastBuildDate><atom:link href="https://waqassiddiqi.com/rss.xml" rel="self" type="application/rss+xml"/><language><![CDATA[en]]></language><ttl>60</ttl><item><title><![CDATA[Step towards Self-Serve Analytics: Natural Language Queries on Databases]]></title><description><![CDATA[This blog post is connected to my earlier post here in which I documented the UI part - This post here talks about what happens under-the-hood when user…]]></description><link>https://waqassiddiqi.com/step-towards-self-serve-analytics-natural-language-queries-on-databases</link><guid isPermaLink="true">https://waqassiddiqi.com/step-towards-self-serve-analytics-natural-language-queries-on-databases</guid><category><![CDATA[Azure OpenAI]]></category><category><![CDATA[LlamaIndex]]></category><category><![CDATA[data-engineering]]></category><dc:creator><![CDATA[Waqas Hussain Siddiqui]]></dc:creator><pubDate>Sat, 06 Sep 2025 16:56:01 GMT</pubDate><content:encoded><![CDATA[<p>This blog post is connected to my earlier post here <a target="_blank" href="https://waqassiddiqi.com/from-text-to-charts-generative-ui-using-assistant-ui">https://waqassiddiqi.com/from-text-to-charts-generative-ui-using-assistant-ui</a> in which I documented the UI part - This post here talks about what happens under-the-hood when user asks questions like “How many orders placed yesterday?”</p>
<h2 id="heading-what-is-llamaindex">What is LlamaIndex?</h2>
<p>LlamaIndex is an open-source framework that connects <strong>large language models (LLMs)</strong> with private data. It takes unstructured or structured data (databases, documents, APIs, etc.), builds indices, and allows natural-language queries on top of them.</p>
<p>At its core, LlamaIndex does three things:</p>
<ol>
<li><p><strong>Data ingestion:</strong> Pulls data from multiple sources: databases, data lakes, APIs, or documents.</p>
</li>
<li><p><strong>Indexing:</strong> Creates efficient structures so that LLMs can understand and query the data.</p>
</li>
<li><p><strong>Querying:</strong> Let uses plain English to ask questions, with the LLM turning them into the right underlying queries.</p>
</li>
</ol>
<h3 id="heading-building-blocks-of-our-implementation">Building Blocks of Our Implementation</h3>
<ul>
<li><p><strong>SQLDatabase</strong>: A thin wrapper over SQLAlchemy that exposes our database to LlamaIndex. It examines <strong>tables, columns, table relations</strong> and can sample rows</p>
</li>
<li><p><strong>Nodes &amp; Schemas</strong>: LlamaIndex represents table schemas as structured <strong>Nodes</strong> (e.g. <code>SQLTableSchema</code>). These become part of the model’s context, so the model <em>knows</em> what exists.</p>
</li>
<li><p><strong>ObjectIndex</strong>: An index over schema objects (tables/views). Useful to send questions to the right subset of tables.</p>
</li>
<li><p><strong>NLSQLTableQueryEngine</strong>: It create prompt which includes (a) the user’s question (b) relevant schema (c) examples, and asks the LLM to <strong>suggest SQL</strong>.</p>
</li>
<li><p><strong>Settings / Providers</strong>: We configure the global LLM &amp; embedding models. For our use case, we are using <strong>Azure OpenAI</strong> deployment.</p>
</li>
</ul>
<h3 id="heading-the-request-lifecycle-what-happens-on-ask-sql">The Request Lifecycle (What happens on <code>/ask-sql</code>)</h3>
<ol>
<li><p><strong>User asks:</strong> “Top 5 customers by total invoice amount for 2024.”</p>
</li>
<li><p><strong>Schema context:</strong> <code>SQLDatabase</code> exposes table/column metadata. Pre-index these with <code>ObjectIndex</code> for cleaner prompts.</p>
</li>
</ol>
<pre><code class="lang-python"><span class="hljs-comment"># app/sql_engine.py (excerpt)</span>
<span class="hljs-keyword">from</span> sqlalchemy <span class="hljs-keyword">import</span> create_engine
<span class="hljs-keyword">from</span> llama_index.core <span class="hljs-keyword">import</span> SQLDatabase
<span class="hljs-keyword">from</span> llama_index.core.objects <span class="hljs-keyword">import</span> SQLTableNodeMapping, ObjectIndex
<span class="hljs-keyword">from</span> llama_index.core.indices.struct_store <span class="hljs-keyword">import</span> SQLTableSchema
<span class="hljs-keyword">from</span> llama_index.core <span class="hljs-keyword">import</span> VectorStoreIndex

engine = create_engine(settings.sqlalchemy_url, pool_pre_ping=<span class="hljs-literal">True</span>)
sqldb = SQLDatabase(engine)

node_mapping = SQLTableNodeMapping(sqldb)
schema_nodes = [SQLTableSchema(table_name=t) <span class="hljs-keyword">for</span> t <span class="hljs-keyword">in</span> sqldb.get_usable_table_names()]
obj_index = ObjectIndex.from_objects(schema_nodes, node_mapping, VectorStoreIndex)
</code></pre>
<ol start="3">
<li><strong>Prompt build:</strong> <code>NLSQLTableQueryEngine</code> assembles a system prompt with rules (only SELECT, use explicit JOINs, respect LIMIT), the <strong>filtered schema</strong>, and examples.</li>
</ol>
<pre><code class="lang-python"><span class="hljs-comment"># app/sql_engine.py (excerpt)</span>
<span class="hljs-keyword">from</span> llama_index.core.query_engine <span class="hljs-keyword">import</span> NLSQLTableQueryEngine

table_retriever = obj_index.as_retriever(similarity_top_k=<span class="hljs-number">3</span>)

query_engine = NLSQLTableQueryEngine(
    sql_database=sqldb,
    table_retriever=table_retriever,
    synthesize_response=<span class="hljs-literal">True</span>,
)

resp = query_engine.query(<span class="hljs-string">"Top 5 customers by total invoice amount in 2024"</span>)
candidate_sql = getattr(resp, <span class="hljs-string">"metadata"</span>, {}).get(<span class="hljs-string">"sql_query"</span>) <span class="hljs-keyword">or</span> str(resp)
</code></pre>
<ol start="4">
<li><p><strong>LLM proposes SQL:</strong> Azure OpenAI returns a candidate query (e.g., a SELECT with joins/filters).</p>
</li>
<li><p><strong>Guardrails:</strong> We <strong>validate</strong> the SQL (deny DDL/DML, multiple statements, comments), <strong>whitelist</strong> tables, and <strong>enforce LIMIT</strong>.</p>
</li>
</ol>
<pre><code class="lang-python"><span class="hljs-comment"># app/guardrails.py usage (excerpt)</span>
<span class="hljs-keyword">from</span> sqlalchemy <span class="hljs-keyword">import</span> text
<span class="hljs-keyword">from</span> .guardrails <span class="hljs-keyword">import</span> validate_sql, whitelist_tables, enforce_limit

validate_sql(candidate_sql)
whitelist_tables(candidate_sql)
safe_sql = enforce_limit(candidate_sql, settings.max_rows)
</code></pre>
<ol start="6">
<li><strong>Execution:</strong> If validation passes, we run it via SQLAlchemy to get results.</li>
</ol>
<pre><code class="lang-python">df = pd.read_sql(text(safe_sql), engine)
</code></pre>
<h3 id="heading-prompting-why-it-matters">Prompting (Why it matters)</h3>
<p>Good prompts lower hallucinations and push the model toward minimal, correct SQL.</p>
<p>We encode:</p>
<ul>
<li><p><strong>Rules</strong>: <em>No</em> <code>SELECT *</code><em>, explicit JOINs, always include LIMIT, no data modification.</em></p>
</li>
<li><p><strong>Scope</strong>: Only the whitelisted tables/columns (keeps search space small and safer).</p>
</li>
<li><p><strong>Few‑shot examples</strong>: Show the model how we want dates, GROUP BYs, and aliases formatted.</p>
</li>
<li><p><strong>Fallback behavior</strong>: If the question is ambiguous, ask a <strong>brief clarifying question</strong> (we allow the engine to respond with a follow‑up, or you can implement a retry policy).</p>
</li>
</ul>
<h3 id="heading-schema-exposure-amp-scoping">Schema Exposure &amp; Scoping</h3>
<p>Large schemas overwhelm the prompt and increase error rates. To Keep the context <strong>small and relevant</strong>:</p>
<ul>
<li><p>Create <strong>read‑only reporting views</strong> that reflect analytics needs (clean joins, intuitive names).</p>
</li>
<li><p>Maintain a <strong>synonym dictionary</strong> (e.g., “AU” → <code>country='Australia'</code>, “last quarter” → <code>[date range]</code>).</p>
</li>
<li><p>For multi‑tenant apps, scope by <strong>RLS</strong> (Row-Level Security) or inject tenant predicates into generated SQL.</p>
</li>
</ul>
<h3 id="heading-guardrails-what-we-block-amp-why">Guardrails (What we block &amp; why)</h3>
<ul>
<li><p><strong>Forbidden tokens</strong>: Reject anything that looks like DDL/DML or comment injection: <code>DROP</code>, <code>UPDATE</code>, <code>--</code>, <code>/* ... */</code>, etc.</p>
</li>
<li><p><strong>Whitelist tables</strong>: Only allow known‑safe tables/views.</p>
</li>
<li><p><strong>Single‑statement only</strong>: Block semicolons and multi‑statements.</p>
</li>
<li><p><strong>LIMIT</strong>: Always enforce an upper bound on rows.</p>
</li>
<li><p><strong>AST validation (optional)</strong>: For stricter control, parse the SQL (e.g., via <code>sqlglot</code>) and inspect the abstract syntax tree.</p>
</li>
</ul>
<p>As I heard a guy speaking on this topic:</p>
<blockquote>
<p>Treat the model like an untrusted user—you must validate and constrain everything it produces.</p>
</blockquote>
<h3 id="heading-common-pitfalls-and-how-we-are-going-to-mititage">Common Pitfalls (and how we are going to mititage)</h3>
<ol>
<li><p><strong>Ambiguous business terms</strong><br /> <em>“Sales”</em> could mean gross, net, or invoiced.</p>
<p> <strong>Mitigation</strong>: a glossary/synonym map and/or clarify‑question policy.</p>
</li>
<li><p><strong>Join path confusion</strong><br /> With multiple possible join routes, the model guesses.</p>
<p> <strong>Mitigation</strong>: expose <strong>reporting views</strong> with the joins baked in; hide raw tables.</p>
</li>
<li><p><strong>Date math &amp; time zones</strong><br /> “Last month” depends on locale/time zone.</p>
<p> <strong>Mitigation</strong>: convert relative periods to exact ranges server‑side (e.g., Australia/Perth), and teach the model the pattern.</p>
</li>
<li><p><strong>NULL semantics &amp; aggregates</strong><br /> SUM/COUNT with NULLs can be tricky.</p>
<p> <strong>Mitigation</strong>: steer model via few‑shots to use <code>COALESCE</code> and correct grouping.</p>
</li>
<li><p><strong>Performance outliers</strong><br /> A generated query may be logically correct but slow.</p>
<p> <strong>Mitigation</strong>: LIMITs, timeouts, and a vault of <strong>approved views</strong> with proper indexes.</p>
</li>
<li><p><strong>Prompt bloat: cost/latency</strong><br /> Too much schema in context increases tokens.</p>
<p> <strong>Mitigation</strong>: scope tables, cache schema summaries, and keep few‑shots tight.</p>
</li>
<li><p><strong>Security drift</strong><br /> New tables accidentally become queryable.</p>
<p> <strong>Mitigation</strong>: explicit <strong>allowlist</strong>.</p>
</li>
</ol>
<h3 id="heading-known-shortcomings-what-llamaindex-wont-solve-for-you">Known Shortcomings / What LlamaIndex won’t solve for you</h3>
<ul>
<li><p><strong>Ground truth vs. business logic</strong>: LlamaIndex can <em>suggest SQL</em>, but it doesn’t know the business semantics; we must add them (views/glossary).</p>
</li>
<li><p><strong>Optimiser hints / fine‑grained planning</strong>: The LLM cannot guarantee optimal queries; we still need indexes and materialized views.</p>
</li>
<li><p><strong>100% correctness</strong>: No framework can fully prevent occasional hallucinations or mis‑joins.</p>
</li>
<li><p><strong>Complex multi‑step analytics</strong>: Long, multi‑CTE analytics may exceed token budgets. Consider prebuilt views or “tooling” that lets the app expose safe stored procedures.</p>
</li>
<li><p><strong>Stateful chats</strong>: Carrying lots of conversation context increases tokens and error surface. Be deliberate with memory.</p>
</li>
</ul>
<h3 id="heading-performance-amp-cost-tuning-recipe">Performance &amp; Cost Tuning Recipe</h3>
<ul>
<li><p><strong>Keep schema small</strong> (views, not raw tables).</p>
</li>
<li><p><strong>Short prompts</strong> with a handful of quality few‑shots.</p>
</li>
<li><p><strong>Cache</strong> schema summaries and reuse the engine.</p>
</li>
<li><p><strong>LIMIT early</strong>, and use <strong>Postgres statement timeouts</strong>.</p>
</li>
<li><p>Prefer a <strong>smaller model</strong> for straightforward queries; upgrade only when needed.</p>
</li>
<li><p>Introduce <strong>result caching</strong> or materialised views for popular questions.</p>
</li>
</ul>
<h3 id="heading-future-improvements">Future Improvements</h3>
<ul>
<li><p><strong>AST validator</strong>: Integrate <code>sqlglot</code> to parse and verify only safe SELECT shapes (no cross joins, bounded GROUP BY, approved functions).</p>
</li>
<li><p><strong>Query templates</strong>: Allow approved parameterized templates for high‑risk analytics while still supporting free‑text for low‑risk ones.</p>
</li>
<li><p><strong>Observability to Azure</strong>: Push traces/metrics to <strong>Azure Monitor / App Insights</strong>; add OpenTelemetry spans for each stage.</p>
</li>
<li><p><strong>Synonym &amp; unit dictionary</strong>: Map business terms (e.g., <strong>“AU”</strong>, <strong>“LTM”</strong>) and units (e.g., <em>k</em>, <em>m</em>) to canonical forms pre‑prompt.</p>
</li>
<li><p><strong>Partial structured outputs</strong>: Ask the model for YAML‑typed intents before emitting SQL, it can be easier to validate.</p>
</li>
</ul>
<p><em>Link to example github repository:</em> <a target="_blank" href="https://github.com/waqassiddiqi/llamaindex-postgres-nlp-starter">https://github.com/waqassiddiqi/llamaindex-postgres-nlp-starter</a></p>
]]></content:encoded></item><item><title><![CDATA[From LLM to RAG: How Azure OpenAI Powers Smarter Maritime ERP at mBluewave]]></title><description><![CDATA[Large Language Models (LLMs) like GPT are changing how we interact with systems. They can generate human-like answers, write summaries, and even reason…]]></description><link>https://waqassiddiqi.com/from-llm-to-rag-how-azure-openai-powers-smarter-maritime-erp-at-mbluewave</link><guid isPermaLink="true">https://waqassiddiqi.com/from-llm-to-rag-how-azure-openai-powers-smarter-maritime-erp-at-mbluewave</guid><category><![CDATA[RAG ]]></category><category><![CDATA[generative ai]]></category><category><![CDATA[openai]]></category><category><![CDATA[azureml]]></category><category><![CDATA[Azure OpenAI]]></category><category><![CDATA[AI]]></category><category><![CDATA[Artificial Intelligence]]></category><dc:creator><![CDATA[Waqas Hussain Siddiqui]]></dc:creator><pubDate>Sat, 30 Aug 2025 18:29:12 GMT</pubDate><content:encoded><![CDATA[<h3 id="heading-introduction">Introduction</h3>
<p>Large Language Models (LLMs) like GPT are changing how we interact with systems. They can generate human-like answers, write summaries, and even reason across documents. But there is a catch: LLMs don’t know your company’s data, asking any question specific to that data will result in convincing but <strong>wrong</strong> answer. A serious risk in industries like banking, finance and maritime, where compliance and safety are non-negotiable.</p>
<p>That is where <strong>Retrieval-Augmented Generation (RAG)</strong> comes in: you ground the LLM with your own data. At <strong>mBluewave</strong>, our AI-first Maritime ERP platform, we built a RAG-powered assistant using <strong>Azure OpenAI + Azure AI Search</strong> to make vessel manuals and procedures accessible using simple human-like interaction.</p>
<h3 id="heading-why-rag">Why RAG?</h3>
<p>Ask GPT: <em>“How do I reset the fire suppression system?”</em> It might give you a plausible but incorrect procedure. With RAG, the answer is pulled directly from the vessel’s own safety manual, ensuring compliance and accuracy.</p>
<p><strong>Architecture at a glance:</strong></p>
<ul>
<li><p>Manuals and safety documents in <strong>Azure Blob Storage</strong></p>
</li>
<li><p>Chunked, embedded, and indexed with <strong>Azure AI Search</strong></p>
</li>
<li><p>Retrieval with metadata filters (by vessel, department, doc type)</p>
</li>
<li><p><strong>Azure OpenAI</strong> generates grounded answers with citations</p>
</li>
</ul>
<p><img src="https://cdn.hashnode.com/res/hashnode/image/upload/v1756627528124/3624654c-d25c-480a-a9b7-be3399b3be8f.png" alt class="image--center mx-auto" /></p>
<h3 id="heading-mbluewave-rag-exahttpsgithubcomgeekabyte-auazure-search-openai-demoutmsourcechatgptcommple"><a target="_blank" href="https://github.com/geekabyte-au/azure-search-openai-demo?utm_source=chatgpt.com">mBluewave RAG Exa</a>mple</h3>
<p><strong>User query:</strong></p>
<blockquote>
<p>“What are the steps to reset the fire suppression system on Vessel A?”</p>
</blockquote>
<p><strong>Workflow:</strong></p>
<ol>
<li><p>Search service retrieves the most relevant passages from Vessel A’s manuals.</p>
</li>
<li><p>Context is injected into the GPT prompt.</p>
</li>
<li><p>Model answers <em>only from sources</em>, returning step-by-step instructions.</p>
</li>
<li><p>Citations show exactly which manual and page were used.</p>
</li>
</ol>
<p><strong>Result:</strong> A precise, compliant answer tailored to that vessel — not a hallucination.</p>
<p><img src="https://cdn.hashnode.com/res/hashnode/image/upload/v1756622836512/72533056-3813-46db-8bdc-fe3b5668cfa0.png" alt="Enterprise GTP using Azure Open AI and Azure AI Search" class="image--center mx-auto" /></p>
<p>(Not an actual example but similar to ours, copied from an open-source GitHub repository)</p>
<h3 id="heading-sample-code-c-aspnethttpaspnet-minimal-api">Sample Code (C# / <a target="_blank" href="http://ASP.NET">ASP.NET</a> Minimal API)</h3>
<pre><code class="lang-csharp"><span class="hljs-comment">// Ask endpoint: retrieve, ground, and answer</span>
app.MapPost(<span class="hljs-string">"/ask"</span>, <span class="hljs-keyword">async</span> (AskRequest req, SearchClient search, OpenAIClient openai) =&gt;
{
    <span class="hljs-comment">// 1) Vector search with vessel filter</span>
    <span class="hljs-keyword">var</span> options = <span class="hljs-keyword">new</span> SearchOptions { Size = <span class="hljs-number">5</span> };
    options.Filter = <span class="hljs-string">$"vessel eq '<span class="hljs-subst">{req.Vessel}</span>'"</span>;
    options.Select.Add(<span class="hljs-string">"chunkText"</span>);
    options.Select.Add(<span class="hljs-string">"source"</span>);

    <span class="hljs-keyword">var</span> results = <span class="hljs-keyword">await</span> search.SearchAsync&lt;SearchDocument&gt;(<span class="hljs-string">""</span>, options);
    <span class="hljs-keyword">var</span> top = results.Value.GetResults().Select(r =&gt; (<span class="hljs-keyword">string</span>)r.Document[<span class="hljs-string">"chunkText"</span>]).ToList();

    <span class="hljs-comment">// 2) Build grounded prompt</span>
    <span class="hljs-keyword">var</span> sys = <span class="hljs-string">"Answer ONLY from provided sources. If unknown, say so."</span>;
    <span class="hljs-keyword">var</span> context = <span class="hljs-keyword">string</span>.Join(<span class="hljs-string">"\n\n"</span>, top);
    <span class="hljs-keyword">var</span> user = <span class="hljs-string">$"<span class="hljs-subst">{req.Question}</span>\n\nSources:\n<span class="hljs-subst">{context}</span>"</span>;

    <span class="hljs-keyword">var</span> chat = <span class="hljs-keyword">new</span> ChatCompletionsOptions
    {
        DeploymentName = <span class="hljs-string">"gpt-4.1-mini"</span>,
        Temperature = <span class="hljs-number">0</span>,
        Messages =
        {
            <span class="hljs-keyword">new</span> ChatRequestSystemMessage(sys),
            <span class="hljs-keyword">new</span> ChatRequestUserMessage(user)
        }
    };

    <span class="hljs-keyword">var</span> completion = <span class="hljs-keyword">await</span> openai.GetChatCompletionsAsync(chat);
    <span class="hljs-keyword">return</span> Results.Ok(completion.Value.Choices[<span class="hljs-number">0</span>].Message.Content[<span class="hljs-number">0</span>].Text);
});
</code></pre>
<h3 id="heading-handling-wrong-answers">Handling Wrong Answers</h3>
<p>No AI system is perfect. Besides putting a disclaimer that <em>AI can make mistakes,</em> the way we are handling these scenarios are:</p>
<ul>
<li><p><strong>Guardrails in design</strong> if the answer isn’t in the docs, the AI says <em>“I don’t know, please check the manual”</em>.</p>
</li>
<li><p><strong>Citations</strong> every answer points back to the source, building user trust.</p>
</li>
<li><p><strong>Feedback loop</strong> crew can flag answers, helping us refine and retrain.</p>
</li>
</ul>
<h3 id="heading-keeping-costs-in-check">Keeping Costs in Check</h3>
<p>It is not cheap to keep this running but cost are manageable if right practices are followed:</p>
<ul>
<li><p><strong>Use smaller models where possible</strong> (e.g., GPT-4.1-mini instead of full GPT-4.1 for everyday queries).</p>
</li>
<li><p><strong>Embed once, reuse often</strong> embeddings are only re-generated when documents change.</p>
</li>
<li><p><strong>Chunk smartly</strong> splitting docs into ~700–900 token sections balances accuracy and token usage.</p>
</li>
<li><p><strong>Cache common queries</strong> frequent lookups (e.g., “When is next inspection due?”) can be cached for near-zero cost.</p>
</li>
<li><p><strong>Monitor usage</strong> track tokens, queries per vessel, and adjust.</p>
</li>
</ul>
<hr />
<h3 id="heading-future-directions">Future Directions</h3>
<ul>
<li><p><strong>Multi-modal RAG</strong> combining text with diagrams, checklists, and images.</p>
</li>
<li><p><strong>Offline/on-board sync</strong> ensuring crews have access even without internet.</p>
</li>
</ul>
]]></content:encoded></item><item><title><![CDATA[Why Most Data Teams Don’t Need Real-Time Streaming]]></title><description><![CDATA[“Real-time” is one of those phrases that shows up almost every time some one is talking about data platforms. It sounds impressive and essential. But the…]]></description><link>https://waqassiddiqi.com/why-most-data-teams-dont-need-real-time-streaming</link><guid isPermaLink="true">https://waqassiddiqi.com/why-most-data-teams-dont-need-real-time-streaming</guid><category><![CDATA[BatchProcessing]]></category><category><![CDATA[data-engineering]]></category><category><![CDATA[streaming]]></category><category><![CDATA[Batch Processing]]></category><category><![CDATA[Data Architecture]]></category><dc:creator><![CDATA[Waqas Hussain Siddiqui]]></dc:creator><pubDate>Fri, 29 Aug 2025 17:05:17 GMT</pubDate><content:encoded><![CDATA[<h2 id="heading-the-streaming-hype">The Streaming Hype</h2>
<p>“Real-time” is one of those phrases that shows up almost every time some one is talking about data platforms. It sounds impressive and essential. But the reality? Most teams do not actually need true real-time streaming.</p>
<h2 id="heading-when-streaming-is-justified">When Streaming Is Justified</h2>
<p>There are use cases where streaming is genuinely valuable:</p>
<ul>
<li><p><strong>Fraud detection</strong>: catching suspicious transactions as they happen.</p>
</li>
<li><p><strong>IoT / telemetry</strong>: where devices continuously emit signals.</p>
</li>
<li><p><strong>Real-time personalization</strong>: serving recommendations while the user is still on the page.</p>
</li>
</ul>
<h2 id="heading-when-streaming-is-overkill">When Streaming Is Overkill</h2>
<ul>
<li><p>Daily or hourly reporting pipelines built on Kafka “because it is real-time.”</p>
</li>
<li><p>Teams burning engineering hours debugging offsets, lag, and exactly-once semantics for data that could have been processed every 15 minutes in a batch job.</p>
</li>
<li><p>Costs ballooning from always-on clusters and high-throughput message brokers — for data that did not need millisecond delivery.</p>
</li>
</ul>
<h2 id="heading-the-batch-alternative">The Batch Alternative</h2>
<p>For the majority of analytics and ML workflows, <strong>batch or micro-batch</strong> pipelines are simpler, cheaper, and more reliable:</p>
<ul>
<li><p><strong>ETL jobs</strong>: running hourly or daily is enough for most cases.</p>
</li>
<li><p><strong>Model training</strong>: retraining weekly or daily does not benefit from real-time data.</p>
</li>
<li><p><strong>Dashboards</strong> – refreshing every 15 minutes feels real-time to most stakeholders.</p>
</li>
</ul>
<p>Modern platforms (like Databricks with Delta Live Tables, Azure Synapse pipelines, Airflow, etc.) make batch pipelines robust and scalable without the operational overhead of streaming.</p>
<h2 id="heading-the-hidden-costs-of-streaming">The Hidden Costs of Streaming</h2>
<p>Building streaming systems is not free and easy:</p>
<ul>
<li><p><strong>Operational overhead</strong>: monitoring lag, offsets, consumer groups.</p>
</li>
<li><p><strong>Complexity</strong>: exactly-once delivery, late-arriving events, schema evolution.</p>
</li>
<li><p><strong>Cost</strong>: brokers, always-on compute, storage of raw + processed streams.</p>
</li>
</ul>
<p>Unless there’s a clear business case for &lt;1 minute latency, the overhead usually outweighs the benefit.</p>
<h2 id="heading-final-thoughts">Final Thoughts</h2>
<p>Streaming is really powerful when done right and used in correct context. But for most data teams, the value comes from reliable, maintainable pipelines that deliver data when the business actually needs it.</p>
]]></content:encoded></item><item><title><![CDATA[The Hidden Costs of Data Pipelines (and How to Control Them)]]></title><description><![CDATA[Data pipelines are the backbone of modern platforms, but they come at a cost. Costs creep in through compute, storage, and networking — and unless they are…]]></description><link>https://waqassiddiqi.com/the-hidden-costs-of-data-pipelines-and-how-to-control-them</link><guid isPermaLink="true">https://waqassiddiqi.com/the-hidden-costs-of-data-pipelines-and-how-to-control-them</guid><category><![CDATA[Azure]]></category><category><![CDATA[cloud cost optimization  ]]></category><category><![CDATA[finops]]></category><category><![CDATA[dataengineering]]></category><category><![CDATA[Databricks]]></category><dc:creator><![CDATA[Waqas Hussain Siddiqui]]></dc:creator><pubDate>Fri, 29 Aug 2025 16:43:41 GMT</pubDate><content:encoded><![CDATA[<p>Data pipelines are the backbone of modern platforms, but they come at a cost. Costs creep in through compute, storage, and networking — and unless they are being monitored, they show up not in design docs, but in the cloud bill.</p>
<p>For small and busy teams, this can hit hard. In our case, it once meant an <strong>unexpected A$8,000 spike in a single month</strong>, all from a default storage configuration we hadn’t even realised was enabled.</p>
<p>In this post, let’s look at some of the most common hidden costs in data pipelines, along with practical steps we’ve taken to control them.</p>
<h2 id="heading-1-compute-overprovisioning-vs-smart-reservations">1. Compute Overprovisioning vs Smart Reservations</h2>
<p><strong>The hidden cost:</strong> Always-on clusters, oversized VMs, or idle containers running 24/7 even though data only arrives once or twice an hour.</p>
<p><strong>What I have seen:</strong> ML training VMs, inference endpoints, and containerized services left running on-demand around the clock. The workloads were steady and predictable but the on-demand pricing was quietly draining the budget.</p>
<p><strong>Control:</strong></p>
<ul>
<li><p>For unpredictable workloads: use <strong>autoscaling</strong> (Databricks jobs, Kubernetes, Container Apps).</p>
</li>
<li><p>For predictable workloads: switch to <strong>Reserved Instances</strong> or <strong>Savings Plans</strong>.</p>
</li>
</ul>
<p>We moved our live inference endpoints, model training VMs, Container Apps, Container Instances, and eligible databases to reservations and savings plans.</p>
<p>This immediately lowered our baseline compute costs without changing a single line of pipeline code.</p>
<h2 id="heading-2-networking-amp-storage-replication-the-grs-surprise">2. Networking &amp; Storage Replication: The GRS Surprise</h2>
<p>One of the biggest hidden costs in cloud data platforms is data replication and cross-region movement.</p>
<p>Learned this the hard way: the Azure Blob Storage attached to our Databricks workspace was set to <strong>Geo-Redundant Storage (GRS)</strong> by default. That meant every piece of data written was being replicated across regions, even though geo-redundancy was not needed.</p>
<p>The result? An <strong>extra A$8,000</strong> in a single billing cycle, made up of unnecessary storage replication and cross-region egress charges.</p>
<p><strong>Control:</strong></p>
<ul>
<li><p>Be explicit with storage redundancy:</p>
<ul>
<li><p><strong>LRS (Locally Redundant)</strong> for most workloads.</p>
</li>
<li><p><strong>ZRS (Zone Redundant)</strong> if you need availability across zones.</p>
</li>
<li><p>Reserve <strong>GRS</strong> for critical data that <em>truly</em> needs cross-region failover.</p>
</li>
</ul>
</li>
</ul>
<h2 id="heading-3-pipeline-failures-amp-retries">3. Pipeline Failures &amp; Retries</h2>
<p><strong>The hidden cost:</strong> Pipelines failing silently and retrying endlessly. Each retry consumes compute and storage, multiplying costs.</p>
<p>Daily ETL jobs failing due to schema drift, retrying 10+ times before alerting anyone wasting hours of compute.</p>
<p><strong>Control:</strong></p>
<ul>
<li><p>Add <strong>schema validation and contracts</strong> at ingestion.</p>
</li>
<li><p>Configure <strong>sensible retry policies</strong> with proper alerts.</p>
</li>
<li><p>Monitor pipeline SLAs, if a job fails send an alerts - for us, it is Microsoft Teams channel where notifications are being sent</p>
</li>
</ul>
<h2 id="heading-4-orphaned-resources">4. Orphaned Resources</h2>
<p><strong>The hidden cost:</strong> Forgotten dev clusters, staging environments, or test containers left running long after they are needed.</p>
<p><strong>Control:</strong></p>
<ul>
<li><p>Automate teardown of non-prod resources after inactivity.</p>
</li>
<li><p>Apply tags and budgets to all environments.</p>
</li>
<li><p>Until automated clean-up scheduled jobs are not setup, manually review prediocally</p>
</li>
</ul>
<hr />
<h2 id="heading-bringing-in-finops-practices">Bringing in FinOps Practices</h2>
<p>Managing costs is not just about cutting spend, it is about visibility and accountability.</p>
<ul>
<li><p>Use tags to attribute costs by project, team, and environment - we are still not at the top of this game.</p>
</li>
<li><p>Set up budget alerts in Azure Cost Management (or AWS Budgets, GCP Billing) - we have one setup for each development and production enviroment.</p>
</li>
<li><p>Review spend regularly with the team, cost awareness is part of engineering.</p>
</li>
</ul>
<hr />
<h2 id="heading-final-thoughts">Final Thoughts</h2>
<p>The hidden costs of data pipelines are real: oversized compute, cross-region replication, retries, and orphaned resources. I have seen them all.</p>
<p>The fix is not to cut corners, it is to spend smarter:</p>
<ul>
<li><p>Reserve what is predictable.</p>
</li>
<li><p>Autoscale burstable workload.</p>
</li>
<li><p>Audit defaults before they burn you.</p>
</li>
</ul>
]]></content:encoded></item><item><title><![CDATA[Shifting Security Left in Data Platforms: Docker Image Scanning with Trivy]]></title><description><![CDATA[For busy data team like ours, the priority has always been to keep the platform reliable and fast while delivering new features. But reliability is not…]]></description><link>https://waqassiddiqi.com/shifting-security-left-in-data-platforms-docker-image-scanning-with-trivy</link><guid isPermaLink="true">https://waqassiddiqi.com/shifting-security-left-in-data-platforms-docker-image-scanning-with-trivy</guid><category><![CDATA[DevSecOps]]></category><category><![CDATA[dataengineering]]></category><category><![CDATA[Docker]]></category><category><![CDATA[cloud security]]></category><category><![CDATA[githubaction]]></category><category><![CDATA[trivy]]></category><dc:creator><![CDATA[Waqas Hussain Siddiqui]]></dc:creator><pubDate>Fri, 29 Aug 2025 15:53:45 GMT</pubDate><content:encoded><![CDATA[<p>For busy data team like ours, the priority has always been to keep the platform reliable and fast while delivering new features. But reliability is not just about performance, it is about security as well.</p>
<p>Parts of our data pipeline and live inference / scoring services runs inside containerised services on Azure, handling sensitive data. Without any guardrails, a vulnerable base image or outdated dependency could slip into production and put the entire platform at risk. For a team that needs to move quickly, manual checks weren’t realistic.</p>
<p>That is where we introduced automated container vulnerability scanning in the CI/CD pipeline. We used Trivy, an open-source scanner, together with GitHub Actions. It became a simple but powerful way to make sure that every Docker image we build and deploy is scanned before it ever reaches production.</p>
<hr />
<h2 id="heading-how-it-works">How It Works</h2>
<p>The workflow is straightforward:</p>
<ol>
<li><p><strong>Build</strong> – Use Docker buildx in GitHub Actions to build the image.</p>
</li>
<li><p><strong>Scan with Trivy</strong> – Run Trivy against the freshly built image. Check for critical/high vulnerabilities. Fail the build if any are found.</p>
</li>
<li><p><strong>Deploy</strong> – Only if the scan passes.</p>
</li>
</ol>
<p>Here’s the Trivy scan step in the pipeline:</p>
<pre><code class="lang-bash">- name: Run Trivy vulnerability scanner
  uses: aquasecurity/trivy-action@master
  with:
    image-ref: <span class="hljs-variable">${{ env.CONTAINER_REGISTRY_LOGIN_SERVER }</span>}/<span class="hljs-variable">${{ env.PROJECT_NAME_FOR_DOCKER }</span>}:<span class="hljs-variable">${{ github.sha }</span>}
    format: <span class="hljs-string">'table'</span>
    exit-code: <span class="hljs-string">'1'</span>
    ignore-unfixed: <span class="hljs-literal">true</span>
    vuln-type: <span class="hljs-string">'os,library'</span>
    severity: <span class="hljs-string">'CRITICAL,HIGH'</span>
  env:
    TRIVY_USERNAME: <span class="hljs-variable">${{ secrets.CONTAINER_REGISTRY_USERNAME }</span>}
    TRIVY_PASSWORD: <span class="hljs-variable">${{ secrets.CONTAINER_REGISTRY_PASSWORD }</span>}
</code></pre>
<p>Where snippet above:</p>
<ul>
<li><p>Fails the build if any <em>Critical/High</em> vulnerabilities are found.</p>
</li>
<li><p>Ignores unfixed issues to reduce noise.</p>
</li>
<li><p>Scans both OS and app libraries.</p>
</li>
</ul>
<p>This ensures that security scanning is built into the delivery pipeline, not something you rely on to remember manually.</p>
<hr />
<h2 id="heading-benefits-of-this-approach">Benefits of This Approach</h2>
<ul>
<li><p><strong>Shift security left</strong> - issues are caught during the build, not after deployment.</p>
</li>
<li><p><strong>Developer visibility</strong> - vulnerabilities appear directly in PRs, making remediation faster.</p>
</li>
<li><p><strong>Automated enforcement</strong> - no reliance on manual checks or periodic scans.</p>
</li>
<li><p><strong>Lightweight and fast</strong> - Trivy is quick enough to run on every build without slowing down pipelines.</p>
</li>
</ul>
<h2 id="heading-how-this-helps-a-data-platform">How This Helps a Data Platform</h2>
<p>Here’s where the real impact shows up:</p>
<ul>
<li><p><strong>Security at the source</strong> - Vulnerabilities are caught before an image goes live.</p>
</li>
<li><p><strong>Reliable pipelines</strong> - Fewer runtime issues from broken OS libraries or insecure dependencies. That means ingestion and ML jobs run consistently.</p>
</li>
<li><p><strong>Shift-left culture</strong> - Developers see issues directly in their pull requests, making security everyone’s responsibility.</p>
</li>
<li><p><strong>Scalable practice</strong> - As the number of microservices and pipelines grows, automated scanning scales far better than manual checks.</p>
</li>
</ul>
<hr />
<h2 id="heading-better-solutions-at-scale">Better Solutions at Scale?</h2>
<p>Larger organisations may want more than Trivy-in-CI/CD:</p>
<ul>
<li><p><strong>Centralized registry scanning</strong> Azure Container Registry and AWS ECR can automatically scan images after push. This complements CI/CD scans by catching vulnerabilities in existing images over time.</p>
</li>
<li><p><strong>Continuous monitoring</strong> vulnerabilities evolve. An image built last month may be flagged today. A dedicated security tool (e.g., Aqua, Prisma, Anchore) provides continuous scanning and alerting.</p>
</li>
<li><p><strong>Policy integration</strong> tying scans into a broader DevSecOps workflow ensures vulnerabilities are prioritized, tracked, and remediated, not just blocked.</p>
</li>
</ul>
<hr />
<h2 id="heading-final-thoughts">Final Thoughts</h2>
<p>For a small and busy team, this practice struck the right balance: low effort, high impact. By adding Trivy scanning into GitHub Actions, we made security part of our delivery pipeline and reduced the risk of shipping vulnerable containers into production.</p>
]]></content:encoded></item><item><title><![CDATA[Journey Through the MLOps Maturity Model]]></title><description><![CDATA[When I first joined Cash Converters, the data science setup was pretty raw. There was no proper repository for training data or models. Just a few Python…]]></description><link>https://waqassiddiqi.com/journey-through-the-mlops-maturity-model</link><guid isPermaLink="true">https://waqassiddiqi.com/journey-through-the-mlops-maturity-model</guid><category><![CDATA[mlops]]></category><category><![CDATA[Machine Learning]]></category><category><![CDATA[Data Science]]></category><category><![CDATA[Azure Machine Learning]]></category><category><![CDATA[azureml]]></category><category><![CDATA[Azure]]></category><category><![CDATA[mlflow]]></category><category><![CDATA[dataops]]></category><category><![CDATA[AI]]></category><dc:creator><![CDATA[Waqas Hussain Siddiqui]]></dc:creator><pubDate>Thu, 28 Aug 2025 14:43:22 GMT</pubDate><content:encoded><![CDATA[<p>When I first joined Cash Converters, the data science setup was pretty raw. There was no proper repository for training data or models. Just a few Python files and few pickled model. That was it.</p>
<p>Reproducibility was non-existent. There was no way to figure out which version of data had been used to train a model. If someone asked anyone from a Data Science &amp; AI teams to rerun an experiment, they couldn’t. Traceability was also missing—we had a model in production, good, how it went there, no clear answer.</p>
<p>And deployment? It was as fragile as it gets. A Python script running inside an ASP.NET scoring service container, loading the pickle file on every inference request. Every prediction started with reloading the model. Slow, brittle, and error-prone - restarting Azure Container Instance was not running on periodically was normal.</p>
<p>When I first started working on automating this entire process, I didn’t know about the <strong>MLOps Maturity Model from Microsoft</strong> (<a target="_blank" href="https://learn.microsoft.com/en-us/azure/architecture/ai-ml/guide/mlops-maturity-model?utm_source=waqassiddiqi.com">link</a>). What I did know—coming from a software engineering background—was that we couldn’t keep going with manual scripts, no reproducibility, and fragile deployments.</p>
<p>For me, the natural step was to bring in CI/CD practices being used for years in software development. Automate whatever could be automated. Track everything. Make deployments predictable.</p>
<p>Quite late in our journey of automating ML, we came across maturity model, and it perfectly described the path we had already taken. We had unknowingly moved through the levels—starting at <strong>Level 0 (No MLOps)</strong> and gradually building toward higher levels of maturity.</p>
<h2 id="heading-level-0-level-1">Level 0 → Level 1</h2>
<p>The first step was to at least bring in some DevOps practices. Since our code was already in GitHub, we started with <strong>GitHub Actions</strong>. That gave us automated builds and tests for the service, but the ML side was still manual. This was basically <strong>Level 1 (DevOps but no MLOps)</strong>.</p>
<h2 id="heading-level-2-automated-training">Level 2: Automated Training</h2>
<p>Things started to change when we brought in <strong>Azure ML SDK</strong> and <strong>MLflow</strong>.</p>
<ul>
<li><p>With Azure ML pipelines, training became reproducible.</p>
</li>
<li><p>With MLflow, every run was logged—parameters, metrics, artifacts. Suddenly, Data Scientist could compare experiments and track what was happening.</p>
</li>
<li><p>Data and models were versioned properly, finally giving us traceability.</p>
</li>
</ul>
<p>This was the first big win. Moving from “I have no idea what’s running” to “I know exactly which model was trained, when, and with which data.”</p>
<h2 id="heading-level-3-automated-deployment">Level 3: Automated Deployment</h2>
<p>Next came deployment. No more copying pickle files into a container.</p>
<p>Now, when a pipeline finishes, the trained model is registered in Azure ML. From there, <strong>GitHub Actions</strong> takes over—deploying the model to staging or production environments. Approvals are built in, so we can control when things go live.</p>
<p>This was a game-changer. Deployments went from fragile manual steps to a reliable process that just works.</p>
<h2 id="heading-level-4-full-mlops">Level 4: Full MLOps</h2>
<p>We’re not fully there yet, but we’re moving in the right direction. Monitoring and retraining are being wired up. Azure ML SDK provides functionality to implement data drift &amp; data quality monitoring and prediction drift in production.</p>
<hr />
<h2 id="heading-why-we-chose-azure-ml-mlflow-and-why-we-didnt-stick-with-databricks-registry">Why We Chose Azure ML + MLflow (and Why We didn’t stick with Databricks Registry)</h2>
<p>When we were evaluating tools, we initially tried <strong>MLflow + Databricks Model Registry</strong>. It gave us basic versioning, but it was tightly coupled with the Databricks ecosystem and always had issues setting up tracking URI and logging experiments running locally.</p>
<p><strong>Azure ML</strong> offered a more <strong>centralized model registry</strong> that fit naturally with our wider Azure ecosystem (we were already using Azure heavily in other part of the business). It also provided out-of-the-box support for deployment endpoints and (evolving) but excellent Python SDK.</p>
<p><strong>MLflow stayed in the picture</strong> for several reasons:</p>
<ul>
<li><p>It was becoming the <em>de facto</em> standard in the community.</p>
</li>
<li><p>Extensive documentation.</p>
</li>
<li><p>Excellent support of saving / loading custom models.</p>
</li>
<li><p>Lightweight and easy for data team to adopt.</p>
</li>
</ul>
<p>As of today, Azure ML has become our <strong>production-grade platform</strong>, while MLflow remained our <strong>experiment tracking layer</strong>. This combination gave us the best of both worlds.</p>
<hr />
<h2 id="heading-what-i-learned-along-the-way">What I Learned Along the Way</h2>
<p>Looking back, a few things stand out:</p>
<ul>
<li><p><strong>Start small.</strong> Even automating one pipeline reduces headaches.</p>
</li>
<li><p><strong>Traceability matters.</strong> The moment we had lineage, conversations changed. We could stop guessing and start deciding.</p>
</li>
<li><p><strong>CI/CD isn’t just for app code.</strong> Treating ML like software made everything more stable.</p>
</li>
</ul>
]]></content:encoded></item><item><title><![CDATA[From Text to Charts: Generative UI using assistant-ui]]></title><description><![CDATA[Build Generative UI with Assistant UI to turn natural language queries into charts using MCP, makeAssistantTool, and makeAssistantToolUI.]]></description><link>https://waqassiddiqi.com/from-text-to-charts-generative-ui-using-assistant-ui</link><guid isPermaLink="true">https://waqassiddiqi.com/from-text-to-charts-generative-ui-using-assistant-ui</guid><category><![CDATA[assistant-ui]]></category><category><![CDATA[generative-ui]]></category><category><![CDATA[llm-tools]]></category><category><![CDATA[data visualization]]></category><dc:creator><![CDATA[Waqas Hussain Siddiqui]]></dc:creator><pubDate>Mon, 07 Jul 2025 14:35:20 GMT</pubDate><content:encoded><![CDATA[<h2 id="heading-what-is-generative-ui">What is Generative UI?</h2>
<p><strong>Generative UI</strong> refers to user interfaces that are <em>not fixed ahead of time</em> but are instead generated dynamically — often by an AI model — in response to user queries, application state, or real-time data.</p>
<h2 id="heading-why-generative-ui-at-all">Why Generative UI at all?</h2>
<p>I was working on an LLM project to enable users to query a database using natural language<br />(<em>more on this in a separate post on MCP: Model Context Protocol</em>).</p>
<p>At first, displaying results as <strong>Markdown tables</strong> worked well.</p>
<p>But I realized if the assistant was going to be used by <strong>non-technical users</strong>, it needed to go beyond rows and columns. Tables show data, but <strong>charts make insights instantly clear</strong> — especially for questions like:</p>
<ul>
<li><p>“Which loan type has the most applications?”</p>
</li>
<li><p>“What’s the breakdown of applications by status?”</p>
</li>
</ul>
<p>This is where <strong>Generative UI</strong> comes into play — that is dynamically generating UI components based on intent, not hardcoded logic.</p>
<h2 id="heading-putting-generative-ui-into-practice">Putting Generative UI into Practice</h2>
<p>At its core, <strong>Generative UI</strong> is about shifting from <em>static</em> interfaces to <em>adaptive, AI-driven experiences</em>. Rather than designing every possible UI screen ahead of time, we let the system <strong>generate the interface dynamically</strong> based on user intent, data context, and goals.</p>
<p>As described in <a target="_blank" href="https://medium.com/design-bootcamp/generative-ui-the-future-of-dynamic-user-experiences-880b1781fcf4">this excellent overview on Medium</a>, building a generative UI requires three fundamental capabilities:</p>
<h3 id="heading-1-understanding-intent">1. Understanding Intent</h3>
<p>The system must deeply understand what the user wants — not just in keywords, but semantically. For example:<br />“Show me the number of applications by type” should be interpreted as a request for a <strong>grouped count</strong>, not just a text summary.</p>
<h3 id="heading-2-selecting-the-right-interaction-pattern">2. Selecting the Right Interaction Pattern</h3>
<p>Once the intent is known, the UI needs to choose the <strong>most effective way to present it</strong>:</p>
<ul>
<li><p>Table</p>
</li>
<li><p>Chart</p>
</li>
<li><p>Form</p>
</li>
<li><p>Summary</p>
</li>
</ul>
<p>This requires mapping intent to a suitable component.</p>
<h3 id="heading-3-generating-and-rendering-ui-elements-dynamically">3. Generating and Rendering UI Elements Dynamically</h3>
<p>Finally, the system must be able to <strong>construct and display</strong> the chosen interface, in context — ideally within a fluid conversational or task-driven flow.</p>
<h3 id="heading-how-assistant-ui-fulfills-these-requirements">How assistant-ui Fulfills These Requirements</h3>
<p><strong>assistant-ui</strong> makes it practical to implement all three layers of Generative UI, without building everything from scratch:</p>
<ul>
<li><p>✅ <strong>Intent understanding</strong> is handled via language models — enhanced by tool descriptions and structured schemas.</p>
</li>
<li><p>✅ <strong>Interaction mapping</strong> is enabled through <strong>tool definitions</strong> using <code>makeAssistantTool</code>, where each tool maps to a specific capability (e.g., draw a chart).</p>
</li>
<li><p>✅ <strong>UI generation</strong> happens with <code>makeAssistantToolUI</code>, which binds the tool’s output to a fully interactive visual component — rendered directly in the chat interface.</p>
</li>
</ul>
<p>Execution is powered by <code>streamText</code>, which allows the assistant to:</p>
<ol>
<li><p>Parse the user’s question</p>
</li>
<li><p>Call the appropriate backend tool (e.g., via MCP)</p>
</li>
<li><p>Invoke a rendering tool (e.g., <code>drawBarchart</code>)</p>
</li>
<li><p>Stream the result back into the assistant window as a chart or component</p>
</li>
</ol>
<h3 id="heading-example-let-the-assistant-render-visual-charts">Example: Let the Assistant Render Visual Charts</h3>
<p>Here’s what we’ll build:</p>
<ul>
<li><p>A <code>drawBarchart</code> tool to visualize <code>{ key, value }</code> grouped data</p>
</li>
<li><p>A corresponding chart renderer using <code>makeAssistantToolUI</code></p>
</li>
<li><p>The same for <code>drawPiechart</code></p>
</li>
<li><p>Assistant instructions that guide the AI to use these tools automatically</p>
</li>
</ul>
<h3 id="heading-step-1-define-chart-ui-with-makeassistanttoolui">Step 1: Define Chart UI with <code>makeAssistantToolUI</code></h3>
<p>The UI layer connects tool results to React visual components. Here’s how I built it:</p>
<p><strong>Bar Chart Tool UI</strong></p>
<pre><code class="lang-typescript"><span class="hljs-keyword">export</span> <span class="hljs-keyword">const</span> GetBarchartToolUI = makeAssistantToolUI&lt;{}, 
    { key: <span class="hljs-built_in">string</span>; value: <span class="hljs-built_in">number</span> }[]&gt;({
  toolName: <span class="hljs-string">"drawBarchart"</span>,
  render: <span class="hljs-function">(<span class="hljs-params">{ result, status }</span>) =&gt;</span> {
    <span class="hljs-keyword">if</span> (status.type === <span class="hljs-string">"running"</span>) {
      <span class="hljs-keyword">return</span> &lt;div&gt;Loading summary data...&lt;/div&gt;;
    }
    <span class="hljs-keyword">if</span> (!<span class="hljs-built_in">Array</span>.isArray(result) || result.length === <span class="hljs-number">0</span>) {
      <span class="hljs-keyword">return</span> &lt;div&gt;No data found.&lt;/div&gt;;
    }

    <span class="hljs-keyword">return</span> (
      &lt;div className=<span class="hljs-string">"w-full h-72"</span>&gt;
        &lt;ResponsiveContainer&gt;
          &lt;BarChart data={result}&gt;
            &lt;CartesianGrid strokeDasharray=<span class="hljs-string">"3 3"</span> /&gt;
            &lt;XAxis dataKey=<span class="hljs-string">"key"</span> /&gt;
            &lt;YAxis /&gt;
            &lt;Tooltip /&gt;
            &lt;Bar dataKey=<span class="hljs-string">"value"</span> fill=<span class="hljs-string">"#4f46e5"</span> /&gt;
          &lt;/BarChart&gt;
        &lt;/ResponsiveContainer&gt;
      &lt;/div&gt;
    );
  },
});
</code></pre>
<h3 id="heading-step-2-define-the-tool-logic">Step 2: Define the Tool Logic</h3>
<p>Using <code>makeAssistantTool</code>, I exposed a tool the assistant can call when it needs to draw a chart.</p>
<pre><code class="lang-typescript"><span class="hljs-keyword">const</span> drawBarchart = tool({
  description: <span class="hljs-string">"Draw a bar chart from tabular data"</span>,
  parameters: z.object({
    data: z.array(
      z.object({
        key: z.string(),
        value: z.number(),
      })
    ),
  }),
  execute: <span class="hljs-keyword">async</span> ({ data }) =&gt; {
    <span class="hljs-keyword">return</span> data;
  },
});
</code></pre>
<p>You can create a similar <code>drawPiechart</code> tool the same way.</p>
<hr />
<h3 id="heading-step-3-register-tools-in-the-chat-runtime">Step 3: Register Tools in the Chat Runtime</h3>
<p>This is where it all comes together: wiring the tool into the AI assistant runtime.</p>
<pre><code class="lang-typescript"><span class="hljs-keyword">export</span> <span class="hljs-keyword">async</span> <span class="hljs-function"><span class="hljs-keyword">function</span> <span class="hljs-title">POST</span>(<span class="hljs-params">req: Request</span>) </span>{
  <span class="hljs-keyword">const</span> session = <span class="hljs-keyword">await</span> auth();
  <span class="hljs-keyword">const</span> jwt = <span class="hljs-keyword">await</span> session.getToken();

  <span class="hljs-keyword">const</span> mcpClient = <span class="hljs-keyword">await</span> createMCPClient({...});

  <span class="hljs-keyword">const</span> mcpTools = <span class="hljs-keyword">await</span> mcpClient.tools();
  <span class="hljs-keyword">const</span> { messages, system, tools } = <span class="hljs-keyword">await</span> req.json();

  <span class="hljs-keyword">const</span> result = streamText({
    model: azure(<span class="hljs-string">"gpt-4o"</span>),
    messages,
    toolCallStreaming: <span class="hljs-literal">true</span>,
    system,
    tools: {
      ...frontendTools(tools),
      ...mcpTools,
      drawBarchart,
    },
    onError: <span class="hljs-built_in">console</span>.log,
  });

  <span class="hljs-keyword">return</span> result.toDataStreamResponse();
}
</code></pre>
<hr />
<h2 id="heading-step-4-instruct-the-assistant-on-chart-use">🧠 Step 4: Instruct the Assistant on Chart Use</h2>
<p>assistant-ui gives you the ability to guide the model’s behavior via <code>useAssistantInstructions</code>.</p>
<pre><code class="lang-typescript">useAssistantInstructions(<span class="hljs-string">`
  You are an intelligent assistant that helps users query a financial services database.
  You ONLY answer questions that can be answered using the provided database schema.

  If the user asks to visualize data (e.g., applications by type), first use the MCP tool 
  to group the results. Then call the "drawBarchart" tool and pass in the grouped data 
  using the { key, value } format.

  Avoid general financial advice or any data not available in the MCP schema.
`</span>);
</code></pre>
<h3 id="heading-example-in-action">Example in Action</h3>
<p><img src="https://cdn.hashnode.com/res/hashnode/image/upload/v1751898609786/3f1b5b35-efac-4ba7-aa62-9389d05bc5e8.png" alt class="image--center mx-auto" /></p>
<p><strong>User:</strong><br />"Can you show me the number of loan applications by status?"</p>
<p><strong>Assistant does:</strong></p>
<ol>
<li><p>Queries database using MCP to get the grouped result.</p>
</li>
<li><p>Receives:</p>
<pre><code class="lang-typescript"> [{ <span class="hljs-string">"key"</span>: <span class="hljs-string">"Approved"</span>, <span class="hljs-string">"value"</span>: <span class="hljs-number">140</span> }, { <span class="hljs-string">"key"</span>: <span class="hljs-string">"Declined"</span>, <span class="hljs-string">"value"</span>: <span class="hljs-number">75</span> }]
</code></pre>
</li>
<li><p>Calls <code>drawBarchart</code>.</p>
</li>
<li><p>The UI renders a live, interactive bar chart in the chat.</p>
</li>
</ol>
<p>That’s Generative UI at work — dynamically transforming natural language into charts, based entirely on real-time context and data.</p>
<p><strong>References</strong></p>
<ul>
<li><p><a target="_blank" href="https://www.assistant-ui.com/docs">Assistant UI – Official Documentation</a><br />  Core docs on <code>makeAssistantTool</code>, <code>makeAssistantToolUI</code>, and building copilots.</p>
</li>
<li><p><a target="_blank" href="https://www.assistant-ui.com/docs/copilots/make-assistant-tool">makeAssistantTool Guide</a><br />  Defines how to expose structured tools for the assistant to call.</p>
</li>
<li><p><a target="_blank" href="https://www.assistant-ui.com/docs/copilots/make-assistant-tool-ui">makeAssistantToolUI Guide</a><br />  Shows how to render tool results as dynamic UI elements.</p>
</li>
<li><p><a target="_blank" href="https://www.assistant-ui.com/docs/guides/ToolUI">Tool UI Integration Guide</a><br />  High-level overview of how Assistant UI enables generative interfaces.</p>
</li>
<li><p><a target="_blank" href="https://medium.com/design-bootcamp/generative-ui-the-future-of-dynamic-user-experiences-880b1781fcf4">Generative UI: The Future of Dynamic User Experiences</a><br />  A great conceptual introduction to generative UI patterns and their design implications.</p>
</li>
</ul>
]]></content:encoded></item><item><title><![CDATA[Back to the future: MLflow]]></title><description><![CDATA[MLflow is an open-source platform for managing the machine learning lifecycle. It provides a set of tools and APIs tracking experiments, deploying models…]]></description><link>https://waqassiddiqi.com/back-to-the-future-mlflow</link><guid isPermaLink="true">https://waqassiddiqi.com/back-to-the-future-mlflow</guid><category><![CDATA[mlflow]]></category><category><![CDATA[mlops]]></category><category><![CDATA[Azure Machine Learning]]></category><category><![CDATA[Data Science]]></category><dc:creator><![CDATA[Waqas Hussain Siddiqui]]></dc:creator><pubDate>Sun, 19 Nov 2023 03:11:02 GMT</pubDate><content:encoded><![CDATA[<h3 id="heading-what-is-mlflow"><strong>What is MLflow?</strong></h3>
<p>MLflow is an open-source platform for managing the machine learning lifecycle. It provides a set of tools and APIs tracking experiments, deploying models, and managing model versions. MLflow is provider agnostic and can be used with Azure, AWS or GCP.</p>
<h3 id="heading-why-mlflow">Why MLflow?</h3>
<p>When I joined the Data Science "team", we had a (<em>broken</em>) data pipeline to get training data (<em>more on this some other day)</em> and a model deployed as "Python script" (yes, as a script, which gets executed as command line, loading pickled model for every scoring request) in Azure Container Instance.</p>
<p>Besides Github repository with training and scoring code, and pickled model in shared drive, there was no information available on how the model was trained or what is required to retrain a model. To make matter worse, it took someone to spend days in recreating the environment without any success and when it came to redeploy the container to update API contract, I had to manually create a new docker image using the one running and manually overwrite DLL files.</p>
<p>It became quite clear to me that without proper MLOps in place, things will get even worse. This is where MLflow came to into play, it offered:</p>
<ul>
<li><p><strong>Experiment tracking:</strong> MLflow provided comprehensive experiment tracking capabilities, offering Azure Machine Learning workspaces as a centralized repository (albeit with some additional work) for experiment details, parameters, and metrics.</p>
</li>
<li><p><strong>Model Packaging and Versioning:</strong> Managing models in a consistent format is important for sharing and deploying across different environments. MLflow simplified this process, allowed us to package our models in efficient, consistent and reproducible manner.</p>
</li>
<li><p><strong>Collaborative Development:</strong> With MLflow, collaborative development became seamless, easing the burden of environment recreation. By leveraging MLflow, we were able to easily recreated the exact environment used for training, saving significant time and effort.</p>
</li>
<li><p><strong>Efficient Model Deployment:</strong> MLflow with it's integration with Azure Machine Learning workspace made deployment to Azure Container Instance simpler and consistent. No more manual Docker image creation, copying pickle files around or overwriting DLL files.</p>
</li>
<li><p><strong>Facilitated Retraining:</strong> MLflow's support for packaging code and dependencies made it easier to reproduce machine learning environments.</p>
</li>
</ul>
<h3 id="heading-why-did-we-move-away-from-mlflow"><strong>Why did we move away from MLflow?</strong></h3>
<p>Since everything from our operational sources to reporting was already on Azure, it was only natural for us to choose Azure Machine Learning workspaces to manage our machine learning experiments.</p>
<p>This is where Azure ML SDK v1 proved easier to setup and use then MLflow:</p>
<ul>
<li><p>Azure ML SDK v1 provided everything we needed from experiment logging to model deployment and scoring.</p>
</li>
<li><p>Running experiments locally with MLflow tracking them to Azure Machine Learning workspace turned out of be quite problematic and required fair bit of bootstrap code.</p>
</li>
<li><p>SDK gave us multiple options for deploying models without much effort. We could deploy models to Azure Container Instances, Azure Kubernetes Service, or even locally.</p>
</li>
</ul>
<h3 id="heading-why-are-we-back-to-mlflow"><strong>Why are we back to MLflow?</strong></h3>
<p>MLflow is the only and recommended way to log metrics, parameters and files in Azure ML SDK v2. As Azure Machine Learning workspaces are MLflow-compatible, now it is as easy as code below to start MLflow tracking:</p>
<pre><code class="lang-python">mlflow.set_experiment(<span class="hljs-string">"model-v1"</span>)
mlflow_run = mlflow.start_run()
</code></pre>
<p>Besides easy of use and no dependency on Azure Machine Learning in training routines:</p>
<ul>
<li><p>Using MLflow makes training routines cloud-agnostic. This means that we can train our models on any cloud platform or even locally.</p>
</li>
<li><p>Provides consistent set of tools regardless of where experiments are running.</p>
</li>
<li><p>We can use Azure Machine Learning workspaces as our tracking server, even if it runs on AWS or, as a matter of fact, anywhere - as long as MLflow is configured to point to the workspace where the tracking should happen.</p>
</li>
</ul>
<p>In addition to these benefits, MLflow is opensource, mature and feature-rich platform for end-to-end MLOps with over 13 million monthly downloads. It has a large and active community, and it is constantly being updated with new features.</p>
]]></content:encoded></item><item><title><![CDATA[Reading subset of data from Azure Machine Learning "folder" dataset]]></title><description><![CDATA[Training a model is an interactive and iterative process which requires fair bit of tuning and quite a lot of debugging. Ability to write, run and debug…]]></description><link>https://waqassiddiqi.com/reading-subset-of-data-from-azure-machine-learning-folder-dataset</link><guid isPermaLink="true">https://waqassiddiqi.com/reading-subset-of-data-from-azure-machine-learning-folder-dataset</guid><category><![CDATA[Azure Machine Learning]]></category><dc:creator><![CDATA[Waqas Hussain Siddiqui]]></dc:creator><pubDate>Fri, 17 Nov 2023 15:41:49 GMT</pubDate><content:encoded><![CDATA[<p>Training a model is an interactive and iterative process which requires fair bit of tuning and quite a lot of debugging. Ability to write, run and debug locally makes this process significantly faster.</p>
<p>However, data that we use to train our models has grown big enough that it is no longer feasible or possible to use entire dataset during development.</p>
<p>Azure ML SDK v1 offered <code>TabularDataset</code> and <code>FileDataset</code> types which had a function called <code>take(count)</code>, which we were using to get a subset of our data. Azure ML SDK v2 replaced them with new data type: <code>uri_folder</code>.</p>
<p><code>uri_folder</code> refers to a folder containing data, either non-tabular (e.g. folder containing large number of images) or tabular with multiple files (e.g. folder containing partitioned parquet files or csv output from a Spark job). We have two way to read data (or subset of data) from <code>uri_folder</code> dataset:</p>
<h2 id="heading-using-mltable">Using mltable</h2>
<p><code>mltable</code> is feature rich Python library to work with tabular dataset. It comes with built-in functionality to get the subset large dataset using <code>take(count)</code> method.</p>
<p>Once we have <code>mltable</code> installed using following command:</p>
<pre><code class="lang-bash">pip install -U mltable azureml-dataprep[pandas]
</code></pre>
<p>It is pretty straight forward to read the Azure ML dataset:</p>
<pre><code class="lang-python"><span class="hljs-keyword">from</span> azure.ai.ml <span class="hljs-keyword">import</span> MLClient
<span class="hljs-keyword">from</span> azure.identity <span class="hljs-keyword">import</span> DefaultAzureCredential
<span class="hljs-keyword">from</span> azure.ai.ml.entities <span class="hljs-keyword">import</span> Data

<span class="hljs-keyword">import</span> mltable

<span class="hljs-comment"># get credentials</span>
credential = DefaultAzureCredential()

<span class="hljs-comment"># initialise Azure ML SDK v2 client</span>
ml_client = MLClient.from_config(path=<span class="hljs-string">"."</span>, credential=creds)

<span class="hljs-comment"># get reference to Azure ML dataset</span>
apps_dataset = ml_client.data.get(name=<span class="hljs-string">"applications"</span>, version=<span class="hljs-number">1</span>)

<span class="hljs-comment"># initialise mltable</span>
<span class="hljs-comment"># apps_dataset.path refers to a folder containing multiple parquet files</span>
tbl = mltable.from_parquet_files([{
    <span class="hljs-string">"pattern"</span>: apps_dataset.path
}])

<span class="hljs-comment"># take subset of data as pandas dataframe</span>
df = tbl.take(<span class="hljs-number">1000</span>)
</code></pre>
<div data-node-type="callout">
<div data-node-type="callout-emoji">💡</div>
<div data-node-type="callout-text"><code>mltable</code> can read any tabular dataset even if it is saved as Azure ML dataset of type <code>uri_folder</code> . Most common example are parquet / csv outputs generated by Apache Spark job, where Spark create a folder and writes one or many files inside it. The best part: code shown above can be used to read <code>uri_folder</code> data type containing tabular data without <strong>any </strong>change.</div>
</div>

<h3 id="heading-using-azuremachinelearningfilesystem"><strong>Using AzureMachineLearningFileSystem</strong></h3>
<p>For some reason if we don't want to use <code>mltable</code> or we have non-tabular data, there is no direct way but to use <code>AzureMachineLearningFileSystem</code> to read subset of data.</p>
<p>Once installed and configured, it provides filesystem like commands (such as <code>ls</code> , <code>open</code> and many other) to interact with Azure ML dataset.</p>
<p>First thing first, installing the library:</p>
<pre><code class="lang-bash"> pip install -U azureml-fsspec
</code></pre>
<p>Then, all we need is to initialise the filesystem using Azure ML dataset:</p>
<pre><code class="lang-python"><span class="hljs-keyword">from</span> azure.ai.ml <span class="hljs-keyword">import</span> MLClient
<span class="hljs-keyword">from</span> azure.identity <span class="hljs-keyword">import</span> DefaultAzureCredential
<span class="hljs-keyword">from</span> azure.ai.ml.entities <span class="hljs-keyword">import</span> Data
<span class="hljs-keyword">from</span> azureml.fsspec <span class="hljs-keyword">import</span> AzureMachineLearningFileSystem

<span class="hljs-comment"># get credentials</span>
credential = DefaultAzureCredential()

<span class="hljs-comment"># initialise Azure ML SDK v2 client</span>
ml_client = MLClient.from_config(path=<span class="hljs-string">"."</span>, credential=creds)

<span class="hljs-comment"># get reference to Azure ML dataset</span>
apps_dataset = ml_client.data.get(name=<span class="hljs-string">"applications"</span>, version=<span class="hljs-number">1</span>)

<span class="hljs-comment"># initialise file system</span>
fs = AzureMachineLearningFileSystem(apps_dataset.path)

temp_dir = tempfile.mkdtemp()
df_list = []
count = <span class="hljs-number">0</span>
limit = <span class="hljs-number">1000</span>

<span class="hljs-comment"># iterate through individual files in folder</span>
<span class="hljs-comment"># read then as pandas dataframe</span>
<span class="hljs-comment"># concate to single dataframe at the end</span>
<span class="hljs-keyword">for</span> path <span class="hljs-keyword">in</span> fs.glob(<span class="hljs-string">f'<span class="hljs-subst">{apps_dataset.path}</span>/*.parquet'</span>):
    _, tail = os.path.split(path)
    dest_path = os.path.join(temp_dir, tail)
    fs.download(rpath=path, lpath=dest_path)

    df_part = pd.read_parquet(dest_path)
    df_list.append(df_part)
    count = count + len(df_part)

    <span class="hljs-keyword">if</span> count &gt;= limit:
        <span class="hljs-keyword">break</span>

df = pd.concat(df_list)
</code></pre>
]]></content:encoded></item></channel></rss>