Data Connections
Designing a No-Code Solution to Simplify Data Warehouse Connections
TLDR;
The process of connecting to data within the product was cumbersome, requiring extensive programming expertise, resulting in user frustration and inefficiencies. Data scientists, our primary users, struggled to complete a core task essential for using our analytics platform.
I redesigned the data connection experience by introducing a streamlined, secure workflow that minimized setup time, improved usability, and ensured seamless collaboration across teams.
“It took you how long?”
“Six hours. And I needed to ask for help twice.”
-Plotly Co-Founder
As we all stared at our co-founder dumbfounded he began walking us through his experience of using our product. Hours of recordings documenting the process of getting access to data, connecting to it, and then using our libraries to answer a question he had about some analytics.
The process was tedious and code heavy, and it was clear: our users were struggling
Most of our users store their data in data warehouses: centralized repositories optimized for querying and reporting structured data. Unfortunately, the initial setup process to access data from these warehouses required programming expertise that many lacked. We realized our focus had been on helping users develop and deploy data dashboards, but we had neglected a critical part of the user journey: connecting to the data itself.
How Datawarehousing works
Pain points with our current infrastructure:
Pain Point 1
Repeated setup for each connection instance
Every time users needed to connect a new data source or reuse an existing one, they were required to go through the entire setup process from scratch. This included locating credentials stored in separate systems, re-entering the same information repeatedly, and debugging minor errors in configurations like typos or mismatched parameters.
Paint Point 2
Risky credential management practices.
Many users resorted to hardcoding credentials directly into code, creating platform wide environment variables or sharing them via unsecured methods like email. Without centralized management, credentials were frequently misplaced or out-of-date, resulting in reliance on administrators or broken applications.
Paint Point 3
High barriers of entry for non-programmers.
Initial configurations often required scripting or using command-line tools, which alienated less technical users. Errors during the process lacked clear feedback, leaving many stuck and reliant on IT teams for support. Additionally, Plotly’s documentation catered primarily to developers, offering insufficient visual guides or examples for other user groups.
With many enterprise products locked behind paywalls, I relied on available what was accessible publicly to play around with and supplementary resources like YouTube tutorials to analyze four competitors and identify patterns in their user experiences.
The findings revealed a consistent user journey across all products, with following industry trends:
Form Based Connections
No-Code connection to projects
Access Control
Our competitors all offered similar solutions, creating internal pressure to follow their lead and design a comparable system. While I recognized the strengths of these established approaches, I couldn’t help but ask: Is this truly the best we can do?
Was there an opportunity to innovate and push the boundaries of what users could expect from the industry?
To challenge the status quo and inspire stakeholders, I developed a series of low-fidelity designs. These concepts weren’t focused on the granular details but aimed to explore alternative workflows and re-imagine how users could connect to and interact with their data.
Solution 1: Stick to the status Quo
This approach mirrored the traditional credential-based connection setups used by our competitors. Users would manually enter their credentials into a straightforward form to establish a connection with their data source.
Solution 2: Higher Security, Fewer Steps
In this option, users would connect to data sources using their personal credentials through OAuth or similar authentication protocols. This method would automatically allow users to view only the data they have access to without needing additional permissions logic on our end of the system.
Option 3: Integrations with data warehouses
This solution envisioned a collaboration between Plotly and data warehouse providers. Users would continue to configure their data sources directly within their warehouse environment, and would then have the option to push the data to Plotly through prebuilt integrations.
Ultimately, we decided to proceed with Option 1 due to time and resource constraints. This allowed us to deliver a functional solution quickly while laying the groundwork for future enhancements. We also incorporated plans to introduce OAuth support in subsequent iterations, moving closer to the streamlined and secure experience envisioned in Option 2.
In addition to designing the form-based flow for connecting to data, I also designed a way for make improve the experience for accessing the data.
The two additional features were:
Data Browser: To search though and view the connected data without code
Custom SQL Cells: Integrate SQL queries directly into notebooks without requiring a python wrapper