so this is not per user, it's per pipeline manager.
what happens to your view if someone else is moving pipelines from another browser?

I'm a bit confused why we went with folders when users were asking for tags and in the issue everyone seemed to like tags?

@gz

We're already strictly structured, we dont have to switch to the dark side of using unstructured data on top of a relational database.

The model I am proposing is not "unstructured", it just doesn't bind relational structure to the shape structure for a family of pipeline information fields with similar semantics - they are not used by the backend, not relevant to backend or pipeline deployment lifecycle, only used by the clients - "description", "tags", "path". JSON operations on a small dataset (for the number of pipelines) are fine in Postgres.

Are you saying you would prefer a e.g. "tags" column in PipelineDescriptor, with the same semantics as the "metadata" field I added? (can be changed without triggering pipeline version updates, when the pipeline is running)?

... a well structured field that is reserved for use by webconsole so we dont have arbitrary clients overwriting/using it differently.

Two points I want to make here:

• database structure =/= API structure; if we wanted to harden the API contract we would add validation at endpoint level, not modify the relational schema
• I don't expect rogue clients to not respect the structure or overwriting the field, losing web-console's information. I guess I misused the term "free-form"; I propose "metadata" as a JSON field which contains various fields clients can set for e.g. organizing the pipelines - like "tags" or "path". Clients would respect this convention and not overwrite the field with e.g. plaintext, but individual clients can add their unique metadata fields they want to without affecting metadata set by other clients, and without having to add support for it in pipeline-manager.
  With the current, simpler model of pipeline-manager not validating "metadata" field it is easier for clients to implement related features, and, again, we don't expect rogue clients that just overwrite other clients' data

The customer did not ask for tags specifically, they listed tags and folders as ways to solve their UX problem. I need to discuss tags with Anna because the feature is not trivial - good location to show and edit tags, and e.g. version can be displayed as another tag, which could improve the design and UX

Whether tags or folders, the feature is likely to be used by tooling around Feldera some of our customers have built, not just web-console

what happens to your view if someone else is moving pipelines from another browser?

They see the change immediately - within the polling period of web-console, which is 2 seconds now

What I would like to do is merge this PR with folders as a simpler solution, and then open a separate PR for user-defined tags and search-by-tag. Tags and folders are orthogonal, both are optional, both are useful.
The folder feature is not invasive, it is only "visible" through the small drag icon at the left of the pipeline list item

Regarding tags vs. nested directory structure: it seems to be most simple to just have tags, as it allows a pipeline to belong to multiple groups, and it is a concept users are already familiar with (e.g., GitHub). Directory structure is also a big departure from the current list.

For design of UI, this seems most straightforward (mock-up):

• In search bar, you can hint Search pipelines... e.g., "example", "tag:example"
• There could be a dropdown filter by tags as well, that shows all the tags and once selected shows all pipelines with that tag

NACK from me on folders too. Folders are used in environments where there is a natural filesystem hierarchy for projects and a mix of content types (e.g. workspace, catalogs, schemas and separate files for sql like in dbt and other warehouse cloud products). It's not a good fit here.

Tags are the most appropriate here. There's many other things in the backlog we should get to first, but let's go through UI design here as well before implementing anything.

I'll remove the folders and make this PR pipeline-manager only. Tags will be waiting for the UI design

I agree that "description" API field should not be deprecated in the PR. More so, "description" and "tags" fields should be top-level in the SelectedPipelineInfo API struct and others. I have updated the PR to reflect that.

I argue against having separate "description" and "tags" columns in the Pipeline table.
I believe there is a stronger technical case to have the single client_metadata JSON column that stores the "description" and "tags":

• For the current requested feature set, the backend does not need to know about the concept of tags. This is purely a client-driven feature; the only thing pipeline-manager cares about is the semantics of the family of the client-driven metadata fields: they are independent from pipeline versioning and deployment runtime status, so need special handling when PATCH-ing the pipeline. There is no feature in the near future that would require pipeline-manager knowing about any of these fields - past the strong API typing.
• Simon cited worse performance of querying the JSON field compared to the SQL ARRAY TEXT. In our case, the performance impact is not material - firstly, all handling of the tags and other metadata is client-side, and there is no use case to process it in pipeline-manager, secondly - pipeline-manager would need to store tens of thousands of pipelines for impact of querying over JSON vs TEXT ARRAY to be noticeable. PostgreSQL 11 we currently use has plenty of operations for querying over JSON. For example, GIN index can be declared to speed up set queries on an array inside JSONB column.
• If needed, writing a SQL query over JSON(B) field is only a marginal complication, and if we really add more features that require SQL querying over metadata we can easily migrate the schema to dedicated SQL columns because the structure is well known by convention and API contract.

The benefits of a single client_metadata column are clear: the pipeline-manager remains simpler, and clients can iterate on the functionality with trivial changes to the backend without the need for DB migrations.

Does it pass CI?

Does it pass CI?

I haven't run CI on it yet

I'll see if some tests can be folded, but the new functionality is not trivial - skipping version bumps for certain fields - so the complexity is justified

A new commit fixes HTTPS test failure caused by the new pipeline test.

The new test test_pipeline_metadata_edits_at_deployment_stages is the first test in test_pipeline_crud.py that actually starts a pipeline and then pauses, resumes, and stops it - every other test in that file only creates and edits pipelines without ever running one. This matters because of how the pipeline manager handles secure connections. When you run the manager in HTTPS mode, it needs a security certificate to prove its identity. Normally there are two distinct kinds of certificate: an "authority" certificate, whose only job is to vouch for other certificates, and an ordinary "server" certificate, which a server presents to identify itself. The manager hands each pipeline it launches a server certificate, and then connects to that pipeline to control it (e.g. to pause it). The secure-connection library we use is strict: it refuses to accept an authority-type certificate when what it expects is a server's own identity, and reports this as CaUsedAsEndEntity.

The problem is in one of our CI jobs, "Make sure manager runs with HTTPS," which runs exactly this test file with the manager in HTTPS mode. That job generates a throwaway test certificate on the fly, and it was marking that certificate as an authority (basicConstraints = critical,CA:TRUE). But this job doesn't give the manager a separate certificate to hand out to pipelines, so the manager reuses its own certificate as each pipeline's server identity. As long as no test ever started a pipeline, this was harmless - the manager never had to connect to a running pipeline, so the certificate's wrong "kind" never mattered. The new test changed that: it starts a pipeline (which works, because the runner uses a more lenient connection path to first reach it), and then asks the manager to pause it. That pause goes through the strict connection path, which sees the authority certificate being used as the pipeline's server identity and rejects it - producing the 503 Service Unavailable every time.

The fix is a one-line change in the CI job: generate the throwaway certificate as an ordinary server certificate (drop the CA:TRUE marking) instead of an authority certificate. This brings it in line with how our other HTTPS test suite already works. That suite (the "OSS platform tests" job) runs the manager with the testing feature flag enabled, which makes the manager auto-generate its own test certificate at startup - and that auto-generated certificate is created as a plain server certificate with no authority marking at all (see ensure_testing_https_cert_key in crates/pipeline-manager/src/config.rs, which sets the host names but never sets basicConstraints). Because that suite's certificate is an ordinary server certificate, it has always been able to start, pause, resume, and stop pipelines over HTTPS without trouble - which is exactly the behavior the new test needs. After this change, both HTTPS jobs generate the same kind of certificate, so the new test passes. No product code changes are required: the tags/metadata feature itself was working correctly the whole time; only the test certificate's configuration was wrong.