Updates & Releases

Summary

This feature allows users to download a file directly from Storytell. When a user clicks on a file, they will be able to see the raw content of the file. Instead of displaying this content in a pop-up modal with a markdown version, the implementation now focuses on providing a direct download from a cloud storage URL. Here’s the documentation on how to do this

Technical details

The implementation of this feature involves creating endpoints for both markdown rendering and signed URLs. When the user clicks on a file, the application will generate a signed URL that grants temporary access to the raw content stored in the cloud. This is achieved by integrating the cloud storage API to facilitate smooth and secure file downloading.

1. Endpoint Creation: We set up two endpoints:

   • One for handling markdown rendering.
   • Another for providing signed URLs for direct content access.

2. Security Protocol: The signed URL ensures that the files are accessible only for a limited time, providing an additional layer of security and protecting against unauthorized access.

3. User Experience: The user simply clicks on the file link, which triggers the application to provide the download link, ensuring a seamless experience without unnecessary pop-ups.

Explain it to me like I’m five

Imagine you have a really cool drawing that you want to share with your friend. Instead of showing it to them on your phone or computer where they can’t touch it, you just let them take it home. When you click on the drawing, it doesn’t go on the screen; instead, it gets sent directly to them, so they can keep it forever. That’s exactly what we’ve done here—now when someone clicks on a file, they get to download it right away!

Specific items shipped:

• Direct File Download:

  • Users can click on a file link to download the actual file instead of viewing it in a pop-up. This streamlines the process and reduces user distractions.

• Signed URL Generation:

  • The system generates a temporary link that allows users to access their files securely. This URL ensures that only authorized users can view or download the files, maintaining data security.

• Endpoint Setup:

  • We’ve put in place specific endpoints to manage file access. One allows users to view markdown content, and the other gives them a quick link to download files directly.

The problem we solved with this feature

Before this feature was implemented, users had to view files in a modal pop-up, which could be an inconvenient experience. Users expressed a need for a simpler way to access and download files directly without the hassle of navigating through additional views. By enabling direct downloads, we improved user experience, allowing for quicker access to important documents and enhancing overall functionality.

Who we built this for

This feature was built primarily for content creators and end-users who frequently need to download files for offline use or sharing. Specifically, it addresses:

• Content Creators: Who need to provide raw data or documents to others easily.
• End-Users: Who prefer quick access to downloadable content rather than having to view it in a pop-up and then navigate away to download it.

Users we especially built file downloads for

• Brandon from Siemens

Summary

We have successfully replicated the Web home page design onto the Storytell.ai web app. This feature includes significant enhancements, such as integrating a prompt bar and an animated graphic that replaces the product screenshot.

Technical Details

The replication process involved several technical steps to ensure the new design elements were effectively integrated into the Storytell.ai web app. The key technical components include:

• Prompt Bar Integration: The prompt bar is a critical feature allowing users to interact directly with the product from the homepage. It was implemented to be persistent, ensuring it remains accessible even when outside the user’s viewport. This required carefully considering the app’s layout and user interface design to maintain functionality across different screen sizes and devices.

• Animation Replacement: The product screenshot in the “How Storytell turns data into insights” section was replaced with an animation. This animation dynamically demonstrates the product’s capabilities, providing a more engaging and informative user experience. The technical challenge was to ensure smooth animation performance without affecting the app’s load time.

• Responsive Design Adjustments: The new homepage design required adjustments to ensure responsiveness across various devices. This involved using CSS media queries and JavaScript to adapt the layout and functionality based on the user’s device and screen size.

Explain it to Me Like I’m Five

Imagine you have a cool new toy and want to show it to your friends. Instead of just telling them about it, you create a fun and colorful picture that shows exactly how the toy works. That’s what we did with our website! We made it look super cool and added a special button that immediately lets people play with our toy (the app). It’s like having a magic door that takes you straight to the fun part!

Specific Items Shipped

• Prompt Bar Integration:
  We added a special bar at the top of the page that lets users start using our app right away. This bar stays in place even if you scroll down, so it’s always easy to find.

• Animation Replacement:
  Instead of a boring picture, we now have a moving animation that shows how our app works. This makes it easier for people to understand what we do and why it’s awesome.

• Responsive Design Adjustments:
  We ensured our website looks great on any device, whether a phone, tablet, or computer. This means everyone can enjoy the new design no matter how they visit our site.

The Problem We Solved with This Feature

The primary problem we addressed with this feature was the need to enhance user engagement and streamline the transition from the homepage to the product experience. By integrating the prompt bar, we provide users with immediate access to the app, reducing friction and improving the likelihood of user interaction. The animation replacement offers a more dynamic and informative way to showcase the product’s capabilities, making it easier for users to understand and appreciate the value of Storytell.ai.

Who We Built This For

This feature was primarily built for potential users visiting the Storytell.ai web app for the first time. The goal was to provide them with an engaging and intuitive experience that quickly demonstrates the app’s value and encourages them to start using it. By making the homepage more interactive and informative, we aim to attract and retain users who are looking for a powerful tool to turn data into insights.

Summary

Our latest feature enables the conversion of XLS files into multiple CSV files, each corresponding to a tab in the original XLS file. This feature is designed to streamline data processing and enhance compatibility with various data analysis tools that prefer CSV format.

Technical Details

The XLS → CSV conversion feature processes each tab within an XLS file independently, converting it into a separate CSV file. The conversion process involves several key steps:

1. File Parsing: The XLS file is parsed to identify individual tabs. Each tab is treated as a separate dataset.
2. Data Extraction: Data from each tab is extracted, ensuring that both structured and semi-structured data are handled appropriately.
3. Classification and Validation: The data is classified to determine its structure. This involves checking for consistent column counts and identifying header rows.
4. Conversion: The extracted data is converted into CSV format. Special attention is given to handling non-tabular data and ensuring that data integrity is maintained.
5. Error Handling: The system includes mechanisms to handle errors such as varied headers and unstructured data, with fallback strategies to ensure conversion success.

Explain It to Me Like I’m Five

Imagine you have a big book with lots of pages, and each page has a different story. Our new feature takes that book and turns each page into its own little book. So, if you have a book with five stories, you end up with five smaller books, each with one story. This makes it easier to read and share just the story you want.

Specific Items Shipped

• Multi-Tab Conversion: Converts each tab in an XLS file into a separate CSV file. This allows for easier data management and analysis.
• Structured Data Handling: Ensures that data with consistent columns is accurately converted. This improves reliability for structured datasets.
• Semi-Structured Data Support: Implements fallback strategies for semi-structured data, enhancing flexibility in data conversion.
• Error Mitigation: Includes error handling for varied headers and unstructured data, reducing conversion failures.
• Validation and Testing: Comprehensive testing with various datasets to ensure robustness and accuracy.

The Problem We Solved with This Feature

We developed this feature to address the challenge of working with XLS files that contain multiple tabs. Many data analysis tools and workflows require data in CSV format, which is more universally compatible and easier to process. By converting XLS files into CSVs, we simplify data handling and improve accessibility for users who need to analyze or share data efficiently.

Who We Built This For

This feature is particularly beneficial for data analysts, researchers, and business professionals who frequently work with large datasets in Excel format. It is designed to support use cases such as:

• Data Analysis: Facilitating the import of data into analysis tools that prefer CSV format.
• Data Sharing: Simplifying the process of sharing specific datasets without the need to share entire Excel files.
• Data Management: Enhancing the organization and management of data by separating it into individual, manageable files.

Summary

The recently shipped feature enables users to perform various operations on collections, specifically Create, Read, Update, and Delete (CRUD). This functionality gives users the flexibility to manage their collections more effectively. They can create new collections, rename existing ones, update them with new data, and delete unwanted collections. Additionally, we’ve implemented a new API endpoint to facilitate the deletion of collections directly.

Explain it to me like I’m five

Imagine you have a toy box. This feature is like a special way to organize your toys. You can add new toys to the box, change the name of your toy box, take out toys when you don’t want them anymore, or even throw the whole toy box away. Also, there’s a button you can press to make sure you can throw away the toy box whenever you decide!

Specific Items Shipped

• Create Collection: Users can now create new collections.

  • This allows users to group related items easily, making organization easier.

• Rename Collection: Existing collections can be renamed.

  • Users can change the name of a collection to better reflect its contents or purpose.

• Move Collection: Moving collections helps users to organize their content more effectively.

  • This feature provides flexibility for users to rearrange their collections, ensuring that their organization systems remain current and intuitive.

• Delete Collection: Added API endpoint for deleting collections.

  • Users can remove collections they no longer need, ensuring the interface remains clutter-free and relevant.

The Problem We Solved with This Feature

We built this feature to address the challenges users faced in managing their collections. Before this, users struggled with limited organization capabilities, leading to confusion as collections grew. By providing a comprehensive set of CRUD functionalities, we empower users to keep their collections tidy and easily accessible, which ultimately enhances their overall experience.

Who We Built This For

This feature is designed for users who frequently manage collections, such as project managers, researchers, and hobbyists. Specifically, it addresses those looking to streamline their organization of items, whether for projects, studies, or personal interests. By enhancing their ability to create, update, and delete collections effortlessly, we improve their workflow and productivity.

Summary

The newly shipped feature addresses the challenge of processing non-tabular data within CSV files. Unlike traditional CSV files that contain well-structured tabular data with consistent columns and headers, many files we receive lack these characteristics. This feature enables the system to handle such “report” type CSV files by leveraging a Language Learning Model (LLM) to generate data “chunks” effectively, ensuring no data points are lost in the process.

Technical Details

The feature utilizes an LLM to parse and process CSV files that do not conform to standard tabular formats. The LLM is prompted to generate data chunks from the CSV content, even in the absence of headers or consistent columns. This involves modifying the prompt to ensure the LLM does not summarize or drop data points, which was a challenge in earlier implementations. Additionally, a “source” identifier is included in each chunk to enhance searchability and traceability of the data.

Explain it to Me Like I’m Five

Imagine you have a messy box of toys instead of a neat row of blocks. Our new feature is like a special robot that helps organize these toys into groups, even if they don’t fit perfectly into rows. It makes sure every toy is in the right group and doesn’t lose any toys while doing so. Plus, it puts a little tag on each group so we can find them easily later.

Specific Items Shipped

• LLM-Generated Data Chunks: The feature uses an LLM to create data chunks from non-tabular CSV files, ensuring all data points are captured.
• Prompt Modifications: Adjustments were made to the LLM prompts to prevent data summarization and ensure complete data retention.
• Source Inclusion in Chunks: Each data chunk now includes a “source” tag to improve searchability and data traceability.
• Regression Testing: Comprehensive testing was conducted using various datasets to validate the feature’s performance and reliability.

The Problem We Solved with This Feature

The primary problem addressed by this feature is the inability to process non-tabular CSV files effectively. Traditional methods failed due to the lack of headers and inconsistent columns, leading to incomplete data processing. By implementing this feature, we ensure that all data points are captured and organized, even in non-standard formats, which is crucial for accurate data analysis and decision-making.

Who We Built This For

This feature was primarily built for data analysts and engineers who frequently work with CSV files that do not adhere to standard tabular formats. It is particularly useful for some organizations, where such files are common, and accurate data processing is essential for generating insights and reports.

Summary

The newly shipped feature classifies CSV content as either “structured” or “unstructured.” This classification allows for the application of different processing strategies, ensuring that CSV files are handled appropriately based on their content type. Structured CSVs typically have a clear header row followed by data rows, while unstructured CSVs may not follow this format and require different handling.

Technical Details

The feature employs a classification algorithm to analyze the content of CSV files. It inspects the first few rows to determine if they follow a structured format, characterized by a consistent header and data row pattern. If the pattern is not detected, the file is classified as unstructured. This classification is crucial for selecting the appropriate processing prompt, which optimizes data handling and ensures accuracy in data interpretation. The implementation is designed to be efficient, minimizing processing time and resource usage.

Explain it to Me Like I’m Five

Imagine you have two kinds of boxes. One box has toys neatly organized with labels, and the other box has toys all jumbled up. This feature is like a helper that looks at each box and tells us if the toys are organized or not. If they are organized, we use one way to play with them, and if they are jumbled, we use a different way. This helps us make sure we always know how to handle the toys correctly.

Specific Items Shipped

• Classification Algorithm: We built a smart system that looks at CSV files and decides if they are organized (structured) or not (unstructured).
• Efficient Processing: The feature is designed to quickly and accurately classify files without slowing down the system.
• Local Validation: We tested the feature with real data to make sure it works perfectly and doesn’t cause any problems.

The Problem We Solved with This Feature

In the real world, CSV files can vary significantly in format. Some are neatly structured, while others are not. This inconsistency can lead to processing errors and inefficiencies. By classifying CSV content, we ensure that each file is processed using the most suitable method, improving accuracy and efficiency in data handling.

Who We Built This For

This feature is designed for data analysts and engineers who frequently work with CSV files. It addresses the need for accurate data processing in environments where CSV files may not always follow a standard structure. By automating the classification process, we reduce the manual effort required to handle diverse data formats, allowing users to focus on more critical tasks.

Summary

We have enhanced our LLM (Large Language Model) router by adding a new feature that categorizes “comparative” queries. This improvement allows the router to better understand and process queries that involve comparisons, enhancing the model selection process. The router now returns a structured response to the prompt builder, which includes a list of categories identified by the LLM for the given prompt. This categorization helps in selecting the most appropriate model for processing the query.

Technical details

The LLM router has been updated to include a categorization mechanism specifically for “comparative” queries. When a query is received, the router analyzes the content to determine if it involves any form of comparison. If a comparison is detected, the query is categorized accordingly. The router then returns a structured response to the prompt builder, which includes a PromptCategories list. This list contains the categories identified by the LLM, which are used to guide the model selection process. If the router cannot define a specific category, it assigns a generic category to ensure that the query is still processed effectively.

Explain it to me like I’m five

Imagine you have a big box of crayons, and you want to pick the right color to draw a picture. Our new feature is like a helper that looks at your picture idea and tells you which crayon to use. If you’re drawing something that needs comparing, like “Is the sky bluer than the ocean?” our helper will know exactly which crayon to pick to make your picture look just right.

Specific items shipped

• Comparative Query Categorization: We have implemented a system that identifies and categorizes queries involving comparisons. This helps in selecting the most suitable model for processing such queries.
• Structured Response to Prompt Builder: The router now returns a structured response that includes a PromptCategories list. This list helps in guiding the model selection process by providing identified categories for the prompt.
• Generic Category Assignment: In cases where a specific category cannot be determined, a generic category is assigned to ensure the query is processed effectively.

The problem we solved with this feature

This feature addresses the challenge of accurately processing queries that involve comparisons. Previously, the LLM router might not have effectively identified and categorized comparative queries, leading to suboptimal model selection. By introducing this categorization, we ensure that comparative queries are processed more accurately, improving the overall performance and reliability of our system.

Who we built this for

This feature was primarily developed for users who frequently engage in queries that involve comparisons. It is particularly useful for applications where understanding and processing comparative language is crucial, such as data analysis, research, and decision-making tools. By enhancing the model selection process for these queries, we aim to provide a more accurate and efficient user experience.

Summary

The Web scraper MVP is an initial release of our web scraping tool designed to extract data from websites efficiently. This feature leverages the Firecrawl API to crawl and scrape web pages, currently supporting the downloading of YouTube videos and their transcripts. The scraper is capable of handling multiple URLs, storing each page’s HTML as an asset within its respective Collection. While the current implementation focuses on downloading transcripts to prevent blocking the control plane, future updates will introduce job architecture for more extensive scraping capabilities.

Technical Details

The Web scraper MVP utilizes several Python libraries, including the youtube-transcript-api, to download YouTube videos and their transcripts. In its first generation, the scraper integrates the Firecrawl API to perform crawling and scraping tasks. Here’s a deeper look into its technical components:

• Firecrawl Integration: Firecrawl is chosen for its ability to crawl entire websites rather than just single pages. It offers configuration options like maximum pages to crawl and crawl depth, allowing for controlled data extraction.

• Endpoint Management: Currently, the scraper operates under a single endpoint, which handles the downloading of transcripts to ensure that the control plane remains responsive. Future iterations will implement asynchronous processing to manage larger scraping tasks without hindering performance.

• Asset Storage: Each scraped page’s HTML is stored as an asset in the Collection from which it was added. This organized storage facilitates easy retrieval and management of scraped data.

• Scalability Considerations: To support scaling, the system plans to switch to Scrapfly for better credit cost efficiency and implement a robust job architecture using our existing queueing and job processing frameworks. Rate limiting and credit management are addressed to ensure sustainable usage of scraping resources.

• Challenges and Future Work: Upcoming enhancements include allowing users to specify multiple URLs for scraping, integrating anti-bot bypass strategies, and implementing proxy rotations to prevent IP blocking. These advancements aim to make the scraper more versatile and resilient against common web scraping obstacles.

Explain it to me like I’m five

Imagine you have a magic robot that can visit websites and copy down all the important information for you. The Web scraper MVP is like that robot. Right now, it can grab videos and their captions from YouTube and save the information neatly. It’s built to handle one website at a time, but soon it will be able to visit many websites at once without getting stuck. This helps us collect data quickly and keep everything organized.

Specific Items Shipped

• YouTube Transcript Downloading
  Enabled the scraper to download transcripts from YouTube videos using Python libraries.
  This feature allows users to automatically fetch and store the transcript of any YouTube video, facilitating content analysis and accessibility.

• Firecrawl API Integration
  Implemented Firecrawl for efficient website crawling and scraping.
  By leveraging Firecrawl, the scraper can navigate entire websites, respecting limits on the number of pages and crawl depth to manage resource usage effectively.

• Single Endpoint Operation
  Configured the scraper to run under one endpoint to maintain control plane performance.
  This design choice ensures that fetching transcripts does not interfere with other system operations, maintaining overall stability.

• Asset Storage System
  Developed a system to store each scraped page’s HTML as an asset within its Collection.
  This organized storage method allows for easy access and management of the scraped data, categorizing it based on the source Collection.

• Scalability Framework
  Set the foundation for future scalability by planning job architecture and rate limiting.
  Preparing for growth, this framework will support more extensive scraping tasks and manage usage effectively as the number of users and scraped URLs increases.

The Problem We Solved with This Feature

Before the Web scraper MVP, extracting data from websites, especially large ones, was time-consuming and resource-intensive. Manually downloading videos and transcripts from platforms like YouTube was inefficient and prone to errors. Our scraper automates this process, enabling rapid and accurate data collection. This matters because it saves significant time and resources for users who rely on up-to-date and comprehensive web data for analysis, research, or content creation.

Who We Built This For

We developed the Web scraper MVP for engineers and researchers who need to gather large amounts of web data efficiently. Use cases include:

• Content Aggregators: Automatically collecting and organizing video transcripts for content libraries.
• Data Analysts: Gathering web data for trend analysis and market research.
• Educational Institutions: Compiling resources and transcripts for academic projects and studies.

Storytell is now HIPAA compliant, ensuring that we adhere to the highest standards for protecting the privacy and security of your health information. This certification underscores our commitment to maintaining the confidentiality of sensitive data and meeting regulatory requirements.

For more information about our security practices and policies, please visit our Trust Center.

Summary

We have implemented a feature that stops sending transformed prompts to the Language Learning Model (LLM) when creating suggestions. Instead, the system now uses the raw prompts provided by the user. This change simplifies the suggestion process and enhances performance by reducing the complexity and processing time involved in generating suggestions.

Technical details

Previously, when generating suggestions, the system injected additional system prompts into the user’s input, transforming the original prompt before sending it to the LLM. This transformation included predefined instructions and context that were not part of the user’s original request. While this approach provided a standardized context for the LLM, it introduced unnecessary complexity and increased the processing time, leading to slower suggestion generation.

With the new feature, the system bypasses the transformation process and sends the user’s raw prompt directly to the LLM. This change reduces the payload size, minimizes processing overhead, and accelerates the response time for generating suggestions. By eliminating the additional system prompts, the LLM focuses solely on the user’s input, which streamlines the suggestion process and improves overall efficiency.

Explain it to me like I’m five

Imagine you’re asking a friend for ideas on a drawing. Before, someone was adding extra instructions to your request, which made it take longer for your friend to come up with suggestions. Now, we let you ask your friend directly without any extra steps, so you get faster and simpler ideas for your drawing.

Specific items shipped

• Direct Use of Raw Prompts: The system now sends the exact prompts provided by users to the LLM without any additional transformations or injected system prompts. This ensures that the suggestions are based solely on the user’s original input.

• Enhanced Performance: By removing the transformation layer, the time taken to process and generate suggestions has been significantly reduced, resulting in quicker response times and a more efficient user experience.

• Simplified Suggestion Pipeline: The elimination of system prompt injection streamlines the suggestion generation process, making it easier to maintain and reducing potential points of failure.

• Improved Accuracy of Suggestions: With the LLM receiving only the raw user input, the relevance and accuracy of the suggestions have improved, as there is less noise and fewer instructions that could potentially misguide the model.

The problem we solved with this feature

We identified that the process of transforming user prompts by injecting additional system prompts before sending them to the LLM was causing unnecessary complexity and delays in generating suggestions. This transformation not only slowed down the response time but also introduced potential inaccuracies in the suggestions by adding extraneous information. By switching to using raw prompts, we simplified the suggestion generation process, making it faster and more reliable. This improvement matters because it enhances the user experience by providing quicker and more relevant suggestions, which is crucial for maintaining user engagement and satisfaction.

Who we built this for

We built this feature for our end-users who rely on quick and accurate suggestions to enhance their productivity and creativity. Specifically, users who interact with our platform for content creation, brainstorming ideas, or seeking assistance in writing will benefit from faster and more relevant suggestions. By optimizing the suggestion generation process, we cater to professionals, writers, and creative individuals who need efficient tools to support their workflow and enhance their output quality.

Summary

The primary objective of this update is to generate responses that are both faster and more user-centric. This feature enhances the generation of responses, making interactions more user-friendly and efficient. Its aim is to ensure that responses feel natural while also processing them more swiftly.

Technical Details

For engineers, understanding the mechanics of this feature is crucial. Here’s a deeper dive:

• System Context Settings: Adjustments have been made to the context settings to promote a more natural conversation flow, encouraging direct interaction with the user.
• Prompt Structure: The prompts have been redesigned to leverage reference materials, reducing the reliance on external training data unless necessary. This helps the system remain aligned with the specific context at hand.
• Diagnostics Integration: New diagnostic tools are implemented to track the accuracy and speed of prompt-response cycles, enabling ongoing improvements and ensuring reliability.

Explain It to Me Like I’m Five

Imagine writing a message to a friend. This feature helps the system respond in a way that’s easy to understand and quick, like getting a faster, friendlier text back from a friend. It’s designed to make conversations smoother and answers come quicker without any confusion.

Specific Items Shipped

• More Friendly Responses: Improved Interaction - The new structure of prompts facilitates more conversational and natural interactions, smoothing out the dialogue between users and the system.
• Faster Responses: Enhanced Speed - With a more efficient processing system, prompts are turned into answers faster than before, reducing wait times.
• Diagnostics in Cloud Environment: Increased Reliability - By incorporating diagnostic tools, any discrepancies or issues can be rapidly identified and corrected, boosting the system’s overall dependability.

The Problem We Solved with This Feature

Previously, the system’s responses often seemed robotic and sometimes misaligned with what users were looking for. This feature addresses that by making responses more human-like and natural, enhancing the overall user experience and ensuring that interactions with the platform meet users’ expectations.

Who We Built This For

This improvement benefits all Storytell users who interact through queries. By refining response quality and reducing wait times, users from different sectors, particularly those requiring prompt and clear interactions for their tasks, will experience a more efficient workflow.

Summary

This feature enhances the routing logic by automatically excluding routes when the search results predominantly originate from CSV files. By implementing this, the system ensures that users receive the most relevant and diverse search results, improving overall user experience.

Technical Details

The feature integrates a filtering mechanism within the existing search algorithm. It analyzes the source of search results and identifies the proportion of results derived from CSV files. If the CSV file results exceed a predefined threshold, the routing process bypasses these results, directing the search to alternative sources. This is achieved by modifying the routing API to include a validation step that checks the origin of each result and applies the exclusion criteria accordingly. Additionally, caching strategies are employed to minimize performance overhead during this verification process.

Explain it to Me Like I’m Five

Imagine you’re looking for your favorite toy in a big toy store. Sometimes, most of the toys look the same and come from the same packaging (like CSV files). This feature helps the store staff make sure that when you search for a toy, you get a variety of options from different places, not just the ones that look alike. This way, you have a better chance of finding exactly what you want.

Specific Items Shipped

• Routing Filter Module: Introduced a new module that scans search results to determine the origin of each result.

  This module evaluates whether the majority of search outcomes are from CSV files. If the condition is met, it triggers the exclusion process to remove these results from the routing path.

• Threshold Configuration Settings: Added configurable settings to define the percentage threshold for CSV file dominance in search results.

  Administrators can adjust the threshold based on varying needs, allowing flexibility in how strictly the routing exclusion is applied.

• Performance Optimization Techniques: Implemented caching mechanisms to store previously evaluated search results.

  These optimizations ensure that the additional filtering does not significantly impact the search performance, maintaining a seamless user experience.

• Enhanced Logging and Monitoring: Developed comprehensive logging features to track the filtering process and its impact on search routing.

  This facilitates easier troubleshooting and performance assessment, ensuring the feature operates as intended.

The Problem We Solved with this Feature

Previously, when users performed searches, a significant portion of the results came from CSV files, leading to repetitive and less diverse information being displayed. This repetition could frustrate users and reduce the perceived quality of the search functionality. By implementing the “Remove from Routing if the Search Results Contain Mostly Results from CSV Files” feature, we ensure that search results are more varied and relevant, enhancing user satisfaction and engagement.

Who We Built This For

This feature is designed for users who rely on Storytell to access a wide range of data sources. Specifically:

• Data Analysts: Who require diverse datasets for comprehensive analysis without being confined to CSV file limitations.
• Researchers: Seeking varied sources to support robust and multifaceted research outcomes.
• General Users: Looking for diverse information without redundancy, ensuring a more enriching search experience.

Summary

The Supportability(Search) feature enhances our search system by providing comprehensive visibility into tenant-specific assets and their embeddings. This improvement allows our team to monitor and understand customer issues more effectively by displaying detailed information about the assets and embeddings associated with each tenant.

Technical Details

This feature integrates directly with our existing search and ingest services, enabling a seamless flow of information. A new user interface was designed, which offers real-time metrics reflecting the ingested assets and their corresponding embeddings. This is achieved through the backend’s connection to our databases.

The feature employs optimized API endpoints that minimize performance overhead while ensuring that data retrieval is both efficient and accurate. It allows for queries that provide:

• The total count of ingested assets
• The number of embeddings generated for those assets
• A breakdown of embeddings organized by asset for each tenant

The front end is developed using React to maintain a responsive and user-friendly experience. For visualizing data, D3.js is utilized, allowing the creation of dynamic and interactive charts, which present complex data in an easily digestible format. All these technical advancements work harmoniously to enhance the search system’s supportability.

Explain It to Me Like I’m Five

Imagine you have a big toy box filled with lots of toys, and your parents want to help you play with them. Sometimes it’s hard to know what toys you have and how they’re organized. The Supportability(Search) feature is like putting labels on each toy and organizing them into groups so you can easily see how many toys there are, what kinds they are, and how they are arranged. This way, you can find your favorite toys quickly and enjoy playing without any confusion!

Specific Items Shipped

• Enhanced Dashboard Interface: Introduced a new dashboard within the search service that displays real-time metrics on ingested assets and embeddings.
  This dashboard provides a clear overview of the number of assets ingested and the embeddings generated for each tenant. Users can navigate to detailed views for specific organizations to see more granular data.

• Organizational Detail Views: Added detailed views for each organization, showing the number of assets and embeddings per asset.
  These detail views allow engineers to drill down into specific tenants to better understand their data landscape. This granularity aids in troubleshooting and optimizing search performance for individual organizations.

• Optimized API Endpoints: Developed new API endpoints to fetch and serve asset and embedding data efficiently.
  These optimized endpoints ensure that data retrieval is fast and reliable, minimizing latency and providing up-to-date information for the dashboard and detail views.

• Minimalistic User Interface: Maintained a clean and simple UI design for ease of use and quick navigation between services.
  The minimalistic approach ensures that users can access the search and ingest services without unnecessary complexity, focusing on the functionality that matters most.

The Problem We Solved with This Feature

Before implementing Supportability(Search), our team lacked the necessary visibility into the search system’s inner workings, which made diagnosing and addressing customer issues related to search functionalities a challenge. This gap hindered our ability to understand which assets and embeddings were present for each tenant, leading to slower response times in troubleshooting and optimizing the search experience.

Who We Built This For

Primarily for our engineering and support teams. These users require detailed visibility into the search system to monitor performance, diagnose issues, and optimize the search experience for our customers. Additionally, product managers benefit from these insights to make informed decisions about feature enhancements and resource allocation based on how different tenants utilize the search capabilities.